Securing AI systems: New challenges and research priorities

At the request of the National Science Foundation, the National Academies of Sciences, Engineering, and Medicine will hold a convening to identify research priorities for securing machine learning and AI-enabled systems. The meeting will examine frameworks and concepts for measuring AI security, assess how existing cybersecurity tools and practices can be adapted, identify novel and unique AI security challenges and explore emerging risks in high-impact applications such as scientific research, drug discovery, and financial services. Bringing together experts in cybersecurity and AI, the meeting will frame research priorities and inform future research programs and strategies for academia, industry, and government aimed at advancing the security and resilience of AI systems. 1. Defining AI Security This session will clarify what “AI security” encompasses, distinguishing it from traditional cybersecurity and AI safety. It will identify concepts, stakeholders, system boundaries, and a set of shared vocabulary to guide research and policy discussions. 2. Threat Models for AI Security This session will examine the range of adversaries, present and future, new attack surfaces and failure modes, and trust boundaries. It will consider how threat modeling must evolve to account for AI’s constantly emerging capabilities. 3. Adapting Classical Cybersecurity to AI Security This session will explore how established cybersecurity principles—such as prevention, audit, defense-in-depth, identity, least privilege, and secure development lifecycles—can be translated to AI systems. It will identify where existing approaches remain effective and where they require modification. 4. Security of Agentic AI This session will focus on AI systems that act autonomously, use tools, and interact dynamically with other systems. It will examine orchestration, identity, access control, containment, integration, resiliency challenges specific to agentic and multi-component environments. 5. Measurement and Evaluation Frameworks and Infrastructure This session will address how to assess and benchmark AI security. It will consider metrics, red-teaming methodologies, concepts of leaderboards, and evaluation infrastructure to support rigorous research and accountability. 6. Guidance and Forward-Looking Framework This session will synthesize insights from the workshop to identify research priorities and practical guidance for industry and government. It will highlight priority areas, collaboration gaps, and strategic investments needed to strengthen AI security over the coming decade. ______________________________________________________________________________________ PROVIDE INPUT We welcome your input to enrich the meeting discussions. Please share your input here:https://survey.alchemer.com/s3/8777659/Securing-AI-Systems