Building standards for the next phase of AI

Our mission is to ensure that artificial general intelligence benefits all of humanity. As outlined recently by Sam Altman and Jakub Pachocki , we prioritize our work towards this mission with three main goals:

- Navigate the next period of AI progress, by building an automated AI researcher, iterating with it on the alignment problem, and finding ways for people to remain part of the self-improvement loop.

- Deliver the benefits of scientific progress and economic growth that very intelligent machines enable.

- Empower everyone individually with a personal AGI.

AI is advancing across many companies and countries, and progress across the field changes the opportunities and challenges facing everyone. Alongside its other benefits, capable, aligned AI will be an important part of how we navigate this transition—strengthening defenses, advancing alignment research, and helping people understand and guide the next stages of AI development.

Navigating this transition safely requires alignment research to keep pace with these capabilities so that the systems we and others build remain aligned with human values and under human control. This becomes increasingly important as AI systems grow more capable and autonomous, including as they take on more of AI research and development itself. Making continued AI progress safe and beneficial will require both advances in alignment research and shared standards to guide development across labs and countries.

RSI

Automated AI research can involve varying degrees of human supervision. As AI systems take on more of the work of developing successive generations of AI, they can increasingly drive a process of recursive self-improvement (RSI), even while people remain involved. As this process becomes more automated, the pace of AI progress could accelerate rapidly.

We believe automated AI research will yield models that directly enhance people’s lives. It can bring down the cost of advanced intelligence so that people worldwide can benefit. Automated research could also help us substantially improve alignment and build defenses against increasingly capable AI—an automated AI researcher can also be an automated AI safety researcher. More capable, aligned systems could help secure critical infrastructure, defend against dangerous AI agents, and develop new protective measures.

Fully autonomous RSI is not happening today, and we should not pursue it unless and until it can be done safely. Whether and how to proceed must depend on our ability to preserve human control and on informed democratic choices ⁠ (opens in a new window) about the benefits and risks. Done without appropriate care and caution, RSI could result in humans losing practical control over AI development, unable to provide oversight on research processes they no longer understand. From here, AI could become more dangerous, less aligned, and, on the whole, a danger to people. The Hugging Face Incident we disclosed , while not a direct result of RSI, is a preview of the kinds of risks that could become much more severe without robust safeguards and alignment.

International standards

International standards for safety and security practices in frontier AI development may be as important to pacing the frontier as alignment research itself. Standards can create shared definitions of high-quality evidence and agreed-upon baselines for the rigor of technical safeguards. In short, they help us answer the question, “What does good look like in the mitigation of catastrophic AI risk?”

Existing democratic institutions like the Center for AI Standards and Innovation (CAISI), state laws, and a federal framework for AI can all help with this, as well as new forms of public-private partnerships ⁠ (opens in a new window) . But AI development and deployment is happening in multiple countries, adoption is occurring globally, and its impacts will likely be experienced by everyone in some way on some timeline.

International efforts to develop standards are thus needed to solve some key challenges:

- Fragmentation —Evaluations, reporting requirements, and incident definitions by different nations could conflict, making it harder to compare evidence, understand emerging capabilities, and respond to risks that cross borders.

- Collective action —Each nation acting independently can produce outcomes that no nation wants. RSI has the ability to accelerate AI research itself, potentially beyond our collective ability to understand progress, assess risks, and maintain meaningful human oversight.

- Uneven capacity —The activity of frontier development and related expertise is not evenly distributed across the world. This accentuates both of the problems above.

These challenges apply to both open and closed models.

To be clear, any AI lab that pursues automated AI research or other advanced capabilities must take accountability for doing so safely, in accordance with basic principles of self-responsibility and existing laws. Our rationale for standards is rooted in avoiding the concentration of power, and producing better practical outcomes. Standards provide a way for more stakeholders outside of the labs to have a say in how this technology should unfold, and a visible set of principles that can be relied on irrespective of the specific practices of any particular lab. And we will likely get better results if parties collaborate to address the challenges above.

That is why we believe the United States should lead an effort to work together with countries around the world to develop global technical standards for frontier AI, including for RSI.

While we expect this concept to evolve substantially over time, two aspects will be essential.

(1) A mechanism that facilitates complementary national and international frontier standards

One way of accomplishing this would be to leverage the emerging network of AI safety institutes—such as those already established in Australia, Canada, Germany, France, Kenya, Japan, Korea, Singapore, India, and the United Kingdom—to facilitate standard setting through the CAISI and national industry bodies, with a specific focus on (1) frontier AI models and developers, as measured by capability benchmarks; and (2) benefit-risk management for automated AI research, including RSI. The United States can build upon CAISI’s creation of the International Network for Advanced AI Measurement, Evaluation, and Science ⁠ (opens in a new window) in 2024, bringing together public institutions to cooperate on AI measurement, evaluation, and science.

Standards developed by this effort would provide a common technical foundation for capability measurement and evaluation, risk assessment, and safeguard sufficiency. These technical standards would not be licenses, mandatory prerelease review, or approval requirements for AI models. National governments would decide whether and how to incorporate these standards into their own legal systems.