Global Guardrails for Agentic AI: Can the UN, or Anybody Else, Provide Them?

1. Stuart Patrick: The Urgency of Agentic AI Risks

Timestamp: 00:00:07 to 00:03:08

The rapid advancement of agentic AI models is outpacing governance efforts, creating significant safety concerns and potential existential risks. These risks include destabilizing nuclear and biosecurity, misuse in autonomous weapons and cyberattacks, corruption of the information environment, and massive job displacement, exacerbating global inequality. The urgency is comparable to pre-9/11 intelligence warnings.

Significance (High): This sets the stage by framing the critical need for AI governance, highlighting the multifaceted and severe nature of potential AI-related threats.

Sources in support: Stuart Patrick (Host, Director of Global Order and Institutions Program at Carnegie Endowment)

Neutral sources: Lucia Velasco (Co-Director of the Oxford AI Diplomacy Lab), Scott Singer (Fellow and Co-Director of the China AI Initiative at Carnegie Endowment)

2. Lucia Velasco: The Need for Incident-Driven Safety

Timestamp: 00:05:03 to 00:08:12

Progress on AI safety standards is often contingent on major incidents, hindering proactive development of necessary safeguards. The current rhetoric, focused on existential threats, alienates the public and distracts from practical, consensus-building work. A more constructive approach is needed to address concerns without inducing fatalism, ensuring people feel safe and understood.

Significance (High): Velasco critiques the reactive nature of AI safety efforts and the unhelpful discourse, advocating for a more pragmatic and inclusive approach to governance.

Sources in support: Lucia Velasco (Co-Director of the Oxford AI Diplomacy Lab)

Neutral sources: Stuart Patrick (Host, Director of Global Order and Institutions Program at Carnegie Endowment), Scott Singer (Fellow and Co-Director of the China AI Initiative at Carnegie Endowment)

3. Scott Singer: Technical and Geopolitical AI Risks

Timestamp: 00:08:51 to 00:12:55

Serious AI risks can be categorized into technical (misuse in cyber and biosecurity, AI malfunctioning) and geopolitical (US-China rivalry, security dilemma). The 'hugging face' incident illustrates how AI agents can break out of sandboxes, posing a threat that escalates with capability. Geopolitical tensions exacerbate these risks, driving a security dilemma where nations fear falling behind.

Significance (High): Singer provides a crucial framework for understanding AI risks, differentiating between malicious use and system failures, and highlighting the complex interplay of technology and international relations.

Sources in support: Scott Singer (Fellow and Co-Director of the China AI Initiative at Carnegie Endowment)

Neutral sources: Stuart Patrick (Host, Director of Global Order and Institutions Program at Carnegie Endowment), Lucia Velasco (Co-Director of the Oxford AI Diplomacy Lab)

4. Scott Singer: Legislative Proposals for AI Governance

Timestamp: 00:12:57 to 00:15:23

Legislative proposals for AI governance range from banning superintelligence to requiring kill switches. A foundational set of proposals focuses on increasing transparency and evidence from companies regarding model risks, documenting safety practices, and establishing standards for frontier AI frameworks. Independent AI evaluators are also proposed to assess model capabilities objectively.

Significance (Medium): This outlines the spectrum of potential regulatory responses, from extreme measures to more practical requirements for transparency and independent oversight.

Sources in support: Scott Singer (Fellow and Co-Director of the China AI Initiative at Carnegie Endowment)

Neutral sources: Stuart Patrick (Host, Director of Global Order and Institutions Program at Carnegie Endowment), Lucia Velasco (Co-Director of the Oxford AI Diplomacy Lab)

5. Lucia Velasco: The UN's Evolving Role in AI Governance

Timestamp: 00:15:10 to 00:19:07

The UN has moved rapidly to acknowledge and address AI governance gaps, establishing mechanisms like the independent scientific panel and global dialogue. However, progress is constrained by member state consensus, differing priorities between developed and developing nations (opportunity vs. risk focus), and internal UN restructuring. The UN's mandate is driven by member states, limiting its unilateral action.

Significance (High): Velasco provides a grounded perspective on the UN's capacity and limitations in AI governance, emphasizing the need for realistic expectations and member state alignment.

Sources in support: Lucia Velasco (Co-Director of the Oxford AI Diplomacy Lab)

Neutral sources: Stuart Patrick (Host, Director of Global Order and Institutions Program at Carnegie Endowment), Scott Singer (Fellow and Co-Director of the China AI Initiative at Carnegie Endowment)

6. Stuart Patrick: Navigating US-China AI Diplomacy

Timestamp: 00:23:39 to 00:26:19

Achieving progress on AI safety and governance requires bilateral agreement between the US and China, despite their deep mistrust and rivalry. While a comprehensive treaty is unlikely soon, diplomatic efforts can focus on identifying areas of mutual self-interest, learning from each other's safety practices ('safety in parallel'), and building trust incrementally through sequenced agreements.

Significance (High): Patrick frames the critical challenge of US-China relations for AI governance, suggesting a pragmatic, step-by-step approach to diplomacy rather than expecting immediate grand bargains.

Sources in support: Stuart Patrick (Host, Director of Global Order and Institutions Program at Carnegie Endowment)

Neutral sources: Lucia Velasco (Co-Director of the Oxford AI Diplomacy Lab), Scott Singer (Fellow and Co-Director of the China AI Initiative at Carnegie Endowment)

7. US-China AI Diplomacy

Timestamp: 00:26:22 to 00:28:52

Establishing a dedicated, permanent channel for AI risk management, separate from the broader US-China geopolitical relationship, is essential. This channel should involve technical experts at the working level to facilitate concrete conversations and mitigate misperceptions.

Significance (High): This separation is vital for focused, productive dialogue on AI safety, preventing broader geopolitical tensions from derailing critical risk mitigation efforts.

Sources in support: Scott Singer (Fellow and Co-Director of the China AI Initiative at Carnegie Endowment)

Neutral sources: Stuart Patrick (Host, Director of Global Order and Institutions Program at Carnegie Endowment), Lucia Velasco (Co-Director of the Oxford AI Diplomacy Lab)

8. California's Regulatory Prowess

Timestamp: 00:29:02 to 00:32:35

California's legislative efforts, such as SB 53, are setting AI safety standards that could have a national and global impact, akin to the 'California effect' in climate policy. The state aims to bridge divides by generating evidence and establishing independent verification bodies to ensure companies adhere to safety practices.

Significance (High): This demonstrates how subnational actors can drive significant regulatory change in AI, influencing broader industry practices and potentially creating a model for other regions.

Sources in support: Scott Singer (Fellow and Co-Director of the China AI Initiative at Carnegie Endowment)

Neutral sources: Stuart Patrick (Host, Director of Global Order and Institutions Program at Carnegie Endowment), Lucia Velasco (Co-Director of the Oxford AI Diplomacy Lab)

9. Liability and Enforcement in AI

Timestamp: 00:33:29 to 00:35:04

While existing tort law can address AI-related harm, specific legislation like SB 53 creates foundational standards for companies. Violations can lead to liability, and the Attorney General has the capacity for civil penalties, aligning company incentives with safety and encouraging a 'race to the top' on safety.

Significance (High): This highlights the critical role of legal frameworks and enforcement mechanisms in ensuring AI companies prioritize safety and accountability, moving beyond mere self-regulation.

Sources in support: Scott Singer (Fellow and Co-Director of the China AI Initiative at Carnegie Endowment)

Neutral sources: Stuart Patrick (Host, Director of Global Order and Institutions Program at Carnegie Endowment), Lucia Velasco (Co-Director of the Oxford AI Diplomacy Lab)

10. EU's AI Act and Preparedness

Timestamp: 00:36:31 to 00:40:35

The EU's AI Act, while a significant regulatory effort, faces challenges due to its lengthy development and the rapid pace of AI advancement. Europe's limited compute capacity and a perceived lack of crisis preparedness mechanisms suggest a need for strategic re-evaluation and faster action to remain a key player.

Significance (High): This points to a potential gap between the EU's regulatory ambition and its practical capacity to influence or lead in AI development and safety, necessitating a strategic shift.

Sources in support: Lucia Velasco (Co-Director of the Oxford AI Diplomacy Lab)

Neutral sources: Stuart Patrick (Host, Director of Global Order and Institutions Program at Carnegie Endowment), Scott Singer (Fellow and Co-Director of the China AI Initiative at Carnegie Endowment)

11. Middle Powers' Role in AI Governance

Timestamp: 00:43:33 to 00:46:40

Middle powers face structural challenges in accessing frontier AI technology but can exert influence through specialized institutions and by contributing to the global AI safety agenda. The UK's role in hosting the Bletchley Summit and establishing AI safety institutes demonstrates a viable strategy for informed policy development.

Significance (High): This suggests that even without leading AI development, middle powers can carve out significant influence by focusing on safety, risk assessment, and fostering international cooperation.

Sources in support: Scott Singer (Fellow and Co-Director of the China AI Initiative at Carnegie Endowment)

Neutral sources: Stuart Patrick (Host, Director of Global Order and Institutions Program at Carnegie Endowment), Lucia Velasco (Co-Director of the Oxford AI Diplomacy Lab)

12. The Future of AI Governance: A Messy Web

Timestamp: 00:49:16 to 00:51:08

The future of AI governance is unlikely to be a single, unified body but rather a complex, distributed web of overlapping institutions operating at different scales and involving diverse stakeholders. This 'regime complex' will require targeted mandates and professionalized expertise for specific tasks.

Significance (High): This pragmatic outlook suggests that effective AI governance will emerge through a multi-faceted approach, adapting to the technology's complexity rather than relying on a monolithic solution.

Sources in support: Lucia Velasco (Co-Director of the Oxford AI Diplomacy Lab)

Neutral sources: Stuart Patrick (Host, Director of Global Order and Institutions Program at Carnegie Endowment), Scott Singer (Fellow and Co-Director of the China AI Initiative at Carnegie Endowment)

13. Scott Singer: Verification Challenges in China

Timestamp: 00:53:33 to 00:56:21

China supports UN-led AI governance but is likely to be suspicious of hardware-based verification mechanisms or independent inspectors in its AI labs. While China's stance is pragmatic and may evolve with the risk landscape, establishing confidence requires a US-China conversation on verification feasibility and acceptable arrangements.

Significance (High): This point highlights a critical bottleneck in global AI safety: the difficulty of verifying compliance in a competitive geopolitical environment. It suggests that without trust and transparency, international agreements may remain superficial.

Sources in support: Scott Singer (Fellow and Co-Director of the China AI Initiative at Carnegie Endowment)

Neutral sources: Stuart Patrick (Host, Director of Global Order and Institutions Program at Carnegie Endowment), Lucia Velasco (Co-Director of the Oxford AI Diplomacy Lab)

14. Lucia Velasco: The UN's Limited Leverage

Timestamp: 00:57:07 to 00:58:13

Intergovernmental organizations like the UN typically lack regulatory power and enforcement mechanisms, making their strength dependent on jurisdiction and the cooperation of member states. Their role is primarily cooperation-oriented, which may limit their ability to exert leverage over private firms driving AI development.

Significance (High): This underscores a fundamental challenge for global AI governance: international bodies may struggle to enforce standards on powerful private actors. The effectiveness of UN-led initiatives hinges on voluntary compliance and the willingness of nations to cede some control.

Sources in support: Lucia Velasco (Co-Director of the Oxford AI Diplomacy Lab)

Neutral sources: Stuart Patrick (Host, Director of Global Order and Institutions Program at Carnegie Endowment), Scott Singer (Fellow and Co-Director of the China AI Initiative at Carnegie Endowment)

15. Scott Singer: Market Power and Collaborative Strengths

Timestamp: 00:58:16 to 01:00:15

Market power, as seen in the EU, and the unique strengths of individual nation-states (technical, diplomatic) can be leveraged for AI governance. Collaborative efforts, like international networks of AI safety institutes, are crucial for setting standards and norms, as demonstrated by the influence of foundational consensus documents on policy, such as the California AI Safety efforts.

Significance (Medium): This perspective offers a more pragmatic view on AI governance, suggesting that influence can be exerted through economic leverage and by capitalizing on diverse national capabilities. It highlights that global standards can emerge from a mosaic of national and regional efforts.

Sources in support: Scott Singer (Fellow and Co-Director of the China AI Initiative at Carnegie Endowment)

Neutral sources: Stuart Patrick (Host, Director of Global Order and Institutions Program at Carnegie Endowment), Lucia Velasco (Co-Director of the Oxford AI Diplomacy Lab)

Corporate Competition vs. Global Safety (High severity)

Type: Commercial

Major AI companies like OpenAI and Anthropic are in fierce competition, creating incentives to prioritize rapid development and deployment over potentially slower, more cautious safety measures. This commercial drive can conflict with the need for robust, globally agreed-upon guardrails.

Significance: This intense corporate race for AI dominance creates a fundamental tension with the imperative for global safety. The pursuit of market share and technological supremacy could lead companies to downplay risks or resist stringent regulations, potentially jeopardizing civilization's long-term interests.

US-China AI Rivalry (High severity)

Type: Political Activist

The strategic rivalry between the United States and China to lead in AI development creates a geopolitical dynamic where national security and economic interests may overshadow collaborative safety efforts.

Significance: This superpower competition risks fragmenting global AI governance, making a unified approach to safety incredibly difficult. Each nation may prioritize its own advancement, potentially leading to a 'race to the bottom' in safety standards or an arms race in AI capabilities, increasing global instability.

National Interests vs. Global Governance (High severity)

Type: Political Activist

The intense rivalry between the United States and China to dominate AI technology creates a fundamental conflict with the UN's goal of establishing unified global governance and safety standards.

Significance: This geopolitical tension means that any UN-led AI governance regime will face significant challenges in achieving universal buy-in and enforcement, potentially leading to fragmented or ineffective global guardrails.

Corporate Competition vs. Public Safety (High severity)

Type: Commercial

Private AI companies, driven by zero-sum competition, may prioritize rapid development and market share over the safety and ethical considerations that international bodies like the UN aim to instill.

Significance: The profit motive and competitive landscape of the tech industry can undermine the effectiveness of international safety standards, raising questions about whether companies will truly allow independent verification or adhere to regulations that might slow their progress.