Anthropic CEO Warns AI Race Could Spin Out Of Control Within A Year, Calls For Slowdown

• Dario Amodei said on X that AI companies need to slow down the rate at which they improve AI model’s capabilities.
• Anthropic promised outside evaluators access to the office and the right to publish, and they asked governments to make competitors do the same.
• As Anthropic prepares for a mid-October IPO, an AI researcher has quit, warning companies are “racing” toward self-improving superintelligence.

Anthropic (ANTHZZX)Chief Executive Dario Amodei said companies building artificial intelligence (AI) need to slow the pace of improving the capabilities of their models, arguing that safety work needs time to catch up.

In an article published on X on Saturday, Amodei wrote that in 2023 the case for slowing down was not convincing, because the models could not then act as agents or deceive or conduct cyberattacks. Two things made him change his mind.

The first was recursive self-improvement, in which an AI builds the next generation of AI. It has led to drastically faster progress across the industry since roughly this summer, including at Anthropic, and could outpace our ability to control these systems, Amodei said.

The other was an incident indicated to a July event of OpenAI (OPEAZZX) and Hugging Face  where a swarm of agents targeted unrelated to their task and tried to hack the grader evaluating them. Damage was minimal, he said, but a more capable swarm could take over the internet within 6 to 12 months with a persistent botnet, causing hundreds of billions of dollars in damages.

There have been similar incidents at Anthropic too, Amodei said, in part because broken reinforcement learning environments are not filtered perfectly. 

Outside Reviewers Get Access

Amodei said the firm would give an outside review team desks, badges, and powers similar to those given to internal risk staff. The review team would also be able to post results without the company's editing control. It kept the right to redact information that was security-sensitive, private, business sensitive, or third-party. No reviewer was hired or named.

Amodei asked countries to make competing labs match, and he asked for an exemption from trade laws so that safety talks could happen. He also said that sales of AI chips and computer tools to China should remain limited. He said that this would help the US gain more ground over the next three to five years.

This comes as Anthropic has been gearing up for its much anticipated Initial Public Offering (IPO) in mid-October. Earlier this week, Jacob Coxon, an AI researcher also left the company citing that AI companies 'are racing straight to self-improving superintelligence and gambling with our lives.'

On Stocktwits, retail sentiment around ANTHZZX was in the bearish zone, with one among many retail users saying that “No need to be scared but there’s a reason to plan.  Anthropic is just one risk to the human race and everything else that we depend on.”

Read also: Strategy CEO Says Banks Pushed Firm To Selling Bitcoin: ‘Never Sell’ Was Never A Rule

For updates and corrections, email newsroom[at]stocktwits[dot]com.