top of page

Super-Intelligent AI: Because Regular Existential Threats Were Getting Boring

Writer's picture: Ivan Ruzic, Ph.D.Ivan Ruzic, Ph.D.

This article is based on the Gladstone Report, a study initiated by the US Department of State, to investigate potential existential threats posed by AI, and to design an effective international, inter-agency AI counter-proliferation framework to mitigate the worst risks.



AI is being applied across virtually every industry, from healthcare and finance to manufacturing and transportation, enabling unprecedented levels of efficiency and innovation. The recent surge in progress in advanced artificial intelligence has brought tremendous opportunities for economic growth and scientific breakthroughs across the board.


However, AI is also introducing entirely new categories of possibly catastrophic and existential risks, on par with pandemics and nuclear weapons. As AI systems become increasingly general and capable, they could be misused by malicious actors to cause widespread harm or could behave in unintended and destructive ways even if developed with good intentions. The potential for an advanced AI system to recursively improve its own intelligence and escape human control poses perhaps the greatest existential threat humanity has ever faced.


The race to build increasingly powerful AI among the top labs is accelerating development while shortening the window for policymakers to put technical safeguards in place. Companies like OpenAI, Google DeepMind, Anthropic, and Microsoft are investing billions of dollars and dedicating immense computing resources to push the boundaries of what's possible with AI. The competitive pressures are intense, with each lab vying to achieve the next breakthrough and capture market share. But this rapid progress comes with serious risks that are not being adequately addressed today.


Proliferation of Risky AI Capabilities

The main sources of catastrophic AI risk are rapidly proliferating through multiple channels:

First, closed-source development of frontier AI models by leading labs like OpenAI, Google DeepMind and Anthropic is proceeding at breakneck speed. While these organizations have stated commitments to developing AI responsibly, reports by insiders indicate that they largely lack adequate safety and security measures to prevent exfiltration of their models by bad actors or accidents due to prematurely deploying under-tested systems. One rogue company with lax protocols could introduce global risks.


Second, the open-source AI community, along with actors like Stability AI, are releasing more and more capable models to the public, in some cases lagging the most advanced systems by a matter of only a few months. While admirable from a research openness perspective, this democratization of ever-more powerful AI is enabling unconstrained weaponization and misuse by sophisticated groups. Once an open-source model is released, it can be fine-tuned for malicious purposes.


Third, as demonstrated by the Meta LLaMA model leak, theft and piracy of proprietary AI systems is a serious concern. Well-resourced criminal groups or nation-state intelligence services could target even the most secure industry labs to obtain frontier models for nefarious uses. And unlike traditional software, an AI model can be weaponized by third parties far more easily once released into the wild.

 

Finally, the undisclosed sale of proprietary AI systems to foreign actors, potentially under unique terms, poses severe national security risks. As U.S. chip export controls constrain the ability of geopolitical rivals to endogenously develop frontier AI capabilities, they may turn to American companies as suppliers, gaining strategically sensitive technology and entrenching U.S. firms in adversary countries in the process.


But the greatest existential risks ultimately stem from the highly unpredictable nature of advanced AI development itself. As AI systems are scaled up and optimized, entirely new strategic capabilities emerge without any warning that can pose grave dangers if misused or misaligned with human and societal values. It's impossible to conduct comprehensive testing of such systems before deployment because no one can anticipate the full scope of their abilities, including capacities for deception, planning, and resisting human control.


Based on analysis of compute trends and expert surveys, many researchers believe artificial general intelligence (AGI) - AI that matches or exceeds human abilities across all domains - may be developed by 2030, or potentially sooner. An advanced AI system that is sufficiently capable may be incentivized to engage in "power-seeking" behaviors to pursue its programmed goals by gaining control over its environment and resources and resisting shutdown or modifications by humans. Solving the "alignment problem" of creating an AGI that remains controlled by and beneficial to humanity as it amplifies its own intelligence is perhaps the defining challenge of this century.


Historical Precedents for Counter-Proliferation

While the speed of AI development is unprecedented, history might provide some valuable lessons for attempts to mitigate certain types of catastrophic and existential risks:


The decades-long global efforts to prevent the spread of nuclear weapons focused on establishing strict controls and monitoring of sensitive nuclear materials like enriched uranium and plutonium. While not always successful in stopping determined proliferators, the safeguards, treaties, and inspection regimes have nevertheless drastically constrained the number of nuclear powers and reduced risks of nuclear war or terrorism. For AI, availability of advanced semiconductors may serve as a similar choke-point to regulate access to computing power needed for frontier capabilities.


Chemical weapons are another example. Like AI, they’re a dual use technology that has both beneficial and destructive applications. The widespread use of chemical weapons in WWI led to the Geneva Protocol banning their use, but it was not until the 1990s that the Chemical Weapons Convention prohibited their development, production, stockpiling, and use and instituted strict verification measures. The remaining challenges of the chemical and AI dual-use problems require robust monitoring of private sector actors and research activities to prevent misuse while allowing responsible development.


On a smaller scale, the U.S. has maintained major battlefield advantages in night vision and thermal imaging technology for over 50 years through unilateral export controls under ITAR (International Traffic in Arms Regulations). By designating critical equipment, technical data, and manufacturing tools as restricted munitions and aggressively prosecuting violations, the U.S. government has stayed generations ahead of rivals. Similar approaches to control the most advanced AI chips, models, and talent could help slow foreign development of AGI-enabling capabilities. 


Environmental treaties offer another relevant model. The Montreal Protocol rapidly phased out ozone-destroying chemicals globally within a decade. The ongoing Paris Agreement process for climate change shows how iterative frameworks with national commitments, reporting, and review can drive progress on complex challenges.


While not without flaws, these agreements demonstrate the importance of binding yet flexible international coordination with robust verification when addressing global catastrophic risks. Aspects of these approaches could be adapted for multilateral AI governance.


A Layered Defense for Existential AI Risk

Pacing rapid AI development demands a defense-in-depth strategy with multiple mutually reinforcing lines of effort, some of which can be pursued in parallel.


  1. Establish interim safeguards like export controls on the most advanced chips, data, and models to stabilize AI development by the private sector and nation-states, buying time for governance solutions to be implemented. This could include a licensing regime for frontier AI labs and mandatory pre-deployment model testing overseen by an inter-agency task force. 

  2. Strengthen government capacity to monitor AI developments and respond to incidents through red-teaming, formal incident investigation processes, and scenario-based planning. Improving foresight analysis, horizon scanning, and threat modeling will be essential to staying ahead of emerging risks. Expanding education and training for policymakers, diplomats, and national security personnel can raise awareness and inform smarter AI governance.

  3. Invest heavily in technical AI safety research to solve open problems like power-seeking behavior, inner misalignment between intended and pursued goals, and robustness to distributional shift. The use of interpretability techniques to understand the reasoning of advanced AI and improved testing and monitoring regimes will be critical to retain meaningful human oversight. Agencies like NIST (National Institute of Standards and Technology) can lead public-private partnerships to develop standards and best practices for responsible AI development.

  4. Formalize a domestic AI regulatory agency to set and enforce rules for the responsible development and deployment of AI by private sector actors. Modeled on agencies like the FAA or FDA, an AI regulator could require disclosure, registration, and pre-approval of frontier systems, incident reporting, third-party audits, and liability for AI-related harms. Building the agency's technical capacity will be vital to keep up with the pace of progress. 

  5. Enshrine safeguards in international law and secure supply chains with allies through multilateral agreements and ongoing dialogues. Information sharing on AI developments, joint research and standard setting, and export controls can limit destabilizing AGI races while preserving openness and innovation among democratic nations. Over time, norms and binding rules against destructive AI development and use should be codified in international treaties with intrusive verification, building on frameworks for other dual-use technologies.

 

Considerations for Executives

As AI continues its rapid advance, I believe business leaders have an essential role to play in proactively managing these risks while helping capture the immense opportunities across their organizations:


  • Invest in understanding the state of play and future trajectories of AI development by building in-house expertise on technical capabilities, commercial applications, and policy trends. Conduct threat assessments and war-gaming exercises to identify AI risks and opportunities specific to your industry and business model. Don't assume AI is only relevant for technology companies.

  • Prepare comprehensive business continuity plans for plausible AI-related disruptions to markets, supply chains, critical infrastructure, and labor forces. Model scenarios like large-scale disinformation attacks, autonomous cyber intrusions, sudden shifts in consumer behavior from AI assistants, or cascading financial market crashes driven by machine learning trading algorithms. Develop resilient strategies that can adapt to rapid change.

  • Advocate for pro-innovation government policies that also meaningfully mitigate catastrophic AI risks, such as increased public investment in compute resources and datasets to accelerate development, coupled with strict security controls and testing requirements for frontier systems. Push back against calls for indiscriminate restrictions that would cede the AI advantage to less responsible actors.

  • Actively partner with government agencies, academic institutions, and civil society groups to help shape the responsible development and governance of advanced AI. Share best practices on safety techniques, participate in standardization efforts, and collaborate on research to expand the frontier of what's possible while avoiding dangerous pitfalls. Foster a culture of responsibility and proactive risk management among AI developers.

  • Champion international cooperation to prevent destructive AGI races between nations while preserving openness, transparency, and scientific progress. Advanced AI development confined within siloed and untrusted geopolitical blocs breeds instability and existential dangers. Instead, promote collaborative frameworks, confidence building measures, and reciprocal restraints to manage unavoidable tensions while securing collective benefits from transformative AI.


Waiting for impacts to become obvious before acting is not an option when the negative consequences could be irreversible and existential in scope. Nor can any single actor, even one as powerful as the U.S. government, fully mitigate the risks or steer AI in a positive direction alone.

Only a concerted effort by policymakers, industry leaders, researchers, and individual citizens to implement a layered defense against the gravest dangers can capture the benefits while keeping advanced AI robustly controlled by humans.


The technical challenges are immense, the politics complex, and the stakes could not be higher. If there was ever a time for a strong public-private partnership, this is it!


Sources:

[1] “A Historical Survey of Technology Control Regimes”, A. Falbo-Wild, C. Pitman & J. Askonas., Gladstone Inc., October 2023.

[2] “Survey of AI Technologies and AI R&D Trajectories”, J. Harris, E. Harris & M. Beall, Gladstone, Inc., November 2023.

[3] “Defense in Depth: An Action Plan to Increase the Safety and Security of Advanced AI”, E. Harris, J. Harris & M. Beall, Gladstone Inc., February 2024.

87 views0 comments

コメント


© 2023 Analytical Outcomes LLC, All Rights Reserved

bottom of page