Schmidt Sciences: Scaling AI Safety for a Multi-Agent World

(Funded jointly by Schmidt Sciences, Google DeepMind, Advanced Research & Invention Agency (ARIA), the Cooperative AI Foundation, and Google.org)

How do we ensure safety in a world with millions of interacting agents, built and deployed by many different actors?

This funding call aims to … catalyse the foundational scientific research needed to understand, evaluate, and control risks emerging from large-scale ecosystems of interacting AI agents, deployed by multiple actors.

The call has been inspired by three recent papers. First, Google DeepMind’s “Distributional AGI Safety” outlines the safety implications of highly capable AI systems emerging not as single monolithic agents, but through coordinated networks of specialised sub-AGI systems with differential access to tools, data, memory, and resources. Second, ARIA’s “Scaling Trust” programme thesis argues that, in a world of increasingly capable networked agents acting across digital and physical environments, coordination infrastructure that lets agents enter into ‘contracts’ securely, programmatically, at scale, and without intermediaries can preserve pluralism and unlock new forms of coordination. Finally, the Cooperative AI Foundation’s “Multi-Agent Risks from Advanced AI” report argues that interacting populations of AI agents introduce qualitatively new failure modes beyond single-agent systems, including collusion, conflict, destabilising dynamics, emergent agency, and novel multi-agent security vulnerabilities. These perspectives in turn build on earlier work by Minsky (1986), Huberman (1988), Wooldridge & Jennings (1995), Manheim (2018), Drexler (2019), Critch & Krueger (2020), Clifton (2020), Dafoe et al. (2020), Conitzer & Oesterheld (2023), Chan et al. (2025), Kolt (2025), Hadfield & Koh (2025), and Tomašev et al. (2025), among many others.

We organise this call into four sections, corresponding to the following research clusters:

1. Sandboxes and Testbeds address the first major bottleneck: without realistic, reproducible multi-agent environments, progress on the remaining sections is hard to evaluate or compare.
2. The Science of Agent Networks focuses on the safety-relevant properties of interacting agent populations: how collective capabilities emerge and scale, how networks of agents fail or become volatile, and how dangerous population-level properties can be detected.
3. Strengthening Agent Infrastructure concerns the evaluation and stress-testing of the technical primitives – identity, verifiability, reputation, communication, commitment – on which trustworthy multi-agent interactions will depend.
4. Multi-Agent Oversight and Control covers the detection, attribution, security, and intervention methods needed to keep deployed agent populations safe at scale.

We expect work in the latter clusters to build on work in the former clusters. Proposals may therefore target one cluster or span several, but we will prioritise those that target depth rather than breadth.

* Funding: Tier 1: Up to $300,000 over 2 years;  Tier 2: up to $1,000,000 over 2 years

* Informational Webinars:  Tuesday, June 30 12 pm ET. Register here;  Thursday, July 23 10 am ET. Register here

https://schmidtsciences.smapply.io/prog/scaling_ai_safety_for_a_multi_agent_world

FAQ: https://docs.google.com/document/d/1LLRZvrKZD9rtKAWjWXPBVZpA5PTpWZatLBkPFT1IuzU