Anthropic warns that it’s very close to RSI capability, aka self-building AI

While much attention has focused on the updated financials Anthropic revealed in its revised S-1/A on Monday (revenue is soaring, but so are expenses), the AI lab dropped another little bombshell in the fine print.

The filing seems to warn that the company is on the cusp of creating a model capable of recursive self-improvement (RSI).

RSI is the point at which AI grows so good at training itself that it spawns a dramatic acceleration of AI capabilities. To the AI doomers, RSI represents the cataclysm from which there is no return, in which AI builds itself into something powerful enough that it slips beyond human control and may be indifferent to the damage it causes while achieving its goals.

To AI enthusiasts, RSI, or some version of AI reliably using itself to improve, represents the technology’s promise, where it becomes a tool of unimaginable productivity, scientific discovery, and more.

Anthropic seems to believe that RSI is both a promise and a peril. It says in the S-1 that it uses AI in its R&D, but it cautions: “Using AI to help build more capable AI could also pose a considerable safety risk, particularly if it sharply accelerates the pace of progress. This is one of the areas covered by our Responsible Scaling Policy, which sets out our plans to monitor and mitigate the risks as we approach what has been termed ‘recursive self-improvement.’”

When the term RSI became popular earlier this year, many believed it was a buzzword or, at best, a goal too hard to achieve, define, or prove.

But on Monday, Anthropic added new statements to its Risk Factors section, saying 1) it has already used AI to develop new models whose leaps in capability it didn’t recognize at the time, and 2) it may soon meet its own definition of RSI (bold emphasis ours, below):

In addition, our increasing use of AI systems to accelerate our own research and development may cause the capabilities of our models to advance more rapidly than our ability to develop, validate, and implement corresponding safety, alignment, and security measures. As a result, we have in the past not identified, and may in the future not identify, a significant acceleration in our rate of development until after it has occurred.

For example, our internal estimates and our expectations regarding future models, rather than our current models,indicate that we may imminently meet our Responsible Scaling Policy’s definition of “dramatically accelerated progress attributable to AI.” We have not designated any of our current models as exceeding this threshold, and we continue to assess the underlying data.

However, Anthropic also said in the filing that none of its models have crossed this RSI threshold yet and that it is promising to keep humans in the R&D process for the foreseeable future. “We do not anticipate being able to safely use Claude to fully autonomously design and build successive generations of frontier models without meaningful human supervision or understanding, if and when that becomes technically possible. Accordingly, we are considering updates to our Responsible Scaling Policy to keep humans in control of frontier model development at Anthropic.”

Skeptics might point out that an S-1 functions as both disclosure and marketing, and that claims of being close to a model so powerful that it can build itself could fall into the latter category.

Nevertheless, there’s no question that Claude’s capabilities, especially in terms of coding, math, and cybersecurity, have grown leaps and bounds this year. If Anthropic thinks the possibility of RSI is close, maybe we should believe it and act — perhaps govern — accordingly.

When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.

Julie Bort is the Startups/Venture Desk editor for TechCrunch.

You can contact or verify outreach from Julie by emailing [email protected] or via @Julie188 on X.

View Bio