Funding Call on Scaling AI Safety for a Multi-Agent World

Overview

AI agents are increasingly being deployed in multi-agent settings. While most present-day cases involve teams of agents orchestrated by a single actor (or ‘principal’), we are beginning to see the emergence of more complex ecosystems of agents deployed by different actors across shared digital infrastructure. These multi-principal, multi-agent interactions create new opportunities for cooperation and shared benefit (Dafoe et al., 2021), but also new risks, which means focusing only on the safety and alignment of individual models is insufficient (Hammond et al., 2025).

More research is therefore urgently needed to understand safety and risk through a system-level, multi-agent lens – developing methods to analyse emergent collective dynamics, building infrastructure for trustworthy interaction between agents, and creating scalable approaches for monitoring and control of increasingly complex networks of AI systems. While some of these problems will be addressed by market forces, we expect others to fall through the gaps. This funding call aims to fill those gaps, catalysing the foundational scientific research needed to understand, evaluate, and control risks emerging from large-scale ecosystems of interacting AI agents, deployed by multiple actors.

The call has been inspired by three recent papers. First, Google DeepMind’s “Distributional AGI Safety” outlines the safety implications of highly capable AI systems emerging not as single monolithic agents, but through coordinated networks of specialised sub-AGI systems with differential access to tools, data, memory, and resources. Second, ARIA’s “Scaling Trust” programme thesis argues that, in a world of increasingly capable networked agents acting across digital and physical environments, coordination infrastructure that lets agents enter into ‘contracts’ securely, programmatically, at scale, and without intermediaries can preserve pluralism and unlock new forms of coordination. Finally, the Cooperative AI Foundation’s “Multi-Agent Risks from Advanced AI” report argues that interacting populations of AI agents introduce qualitatively new failure modes beyond single-agent systems, including collusion, conflict, destabilising dynamics, emergent agency, and novel multi-agent security vulnerabilities. These perspectives in turn build on earlier work by Minsky (1986), Huberman (1988), Wooldridge & Jennings (1995), Manheim (2018), Drexler (2019), Critch & Krueger (2020), Clifton (2020), Dafoe et al. (2020), Conitzer & Oesterheld (2023), Chan et al. (2025), Kolt (2025), Hadfield & Koh (2025), and Tomašev et al. (2025), among many others.

Research Agenda

We organise this call into four sections, corresponding to the following research clusters:

1. Sandboxes and Testbeds address the first major bottleneck: without realistic, reproducible multi-agent environments, progress on the remaining sections is hard to evaluate or compare.
2. The Science of Agent Networks focuses on the safety-relevant properties of interacting agent populations: how collective capabilities emerge and scale, how networks of agents fail or become volatile, and how dangerous population-level properties can be detected.
3. Strengthening Agent Infrastructure concerns the evaluation and stress-testing of the technical primitives – identity, verifiability, reputation, communication, commitment – on which trustworthy multi-agent interactions will depend.
4. Multi-Agent Oversight and Control covers the detection, attribution, security, and intervention methods needed to keep deployed agent populations safe at scale.

We expect work in the latter clusters to build on work in the former clusters. Proposals may therefore target one cluster or span several, but we will prioritise those that target depth rather than breadth. In particular, we stress the importance of realistic sandboxes and testbeds (Section 1) for enabling scientific progress across the broader agenda. Where appropriate, we also encourage collaborations between teams addressing Section 1 with those addressing other sections, and welcome suggestions or requests for sandboxes and testbeds from those submitting proposals under other sections.

Eligibility and Funding Tiers

We invite applicants to apply to either or both funding tiers: Tier 1 (Up to $300,000) or Tier 2 ($300,000-$1,000,000). Project durations can range from one to two years. Tier 1 aims to support exploratory research projects, pilot studies, or focused technical investigations, whereas Tier 2 targets more ambitious or collaborative projects.

We invite individual researchers, research teams, research institutions, and multi-institution collaborations across universities, national laboratories, institutes, and nonprofit research organisations. We are open globally and encourage collaborations across geographic boundaries. For any projects funded by Schmidt Sciences, indirect costs must be at or below 10% to comply with our policy. Projects funded under this RFP must comply with all applicable law and may not include lobbying, efforts to influence legislation or political activity.

Selection Criteria

Proposals will be evaluated holistically. Key considerations include:

1. Research Agenda Fit. Does the proposal clearly engage with the intention behind the scientific questions and objectives in the research agenda?
2. Scientific Quality and Rigour. Is the proposed work technically sound, well-motivated, and capable of producing generalizable insight?
3. Potential Impact. If successful, would the project materially advance scientific understanding relevant to this call, or meaningfully improve our ability to understand, evaluate, or control risks posed by multi-principal, multi-agent AI systems?
4. Philanthropic fit. Is there a clear market, coordination, or incentive failure that means commercial interests are unlikely to solve this problem?
5. Feasibility and Scope. Is there sufficient evidence to suggest that the proposal’s milestones and deliverables are ambitious yet well-defined and feasible enough to be achievable within the stated time duration?
6. Team Expertise. Is the team well-suited to execute the proposed work, with relevant technical expertise, sufficient capacity, and a time commitment commensurate with the project’s ambition?
7. Cost Appropriateness. Is the proposed budget reasonable and well-justified given the project’s goals and planned activities?
8. Additional funder-specific considerations. Applications will be considered by the parties jointly supporting this funding call. If your proposal is selected for funding, the specific party or parties funding your project may provide additional limitations or guidance applicable to your award, which would be documented, if agreed upon, in your final award documentation.

Application and more details: https://schmidtsciences.smapply.io/prog/scaling_ai_safety_for_a_multi_agent_world/