Anthropic has lost another safety researcher, Joe Tenton. Tenton said he left the company two weeks ago and will join METR, an independent group that tests risks in advanced AI systems. Joe had planned to explain the move later, but Jacob Coxon’s resignation this week changed that.
He said: “I’d planned to write about that decision in more detail at some point, but Jacob’s resignation this week made me want to say more now.” Tenton believes major AI labs are creating a level of danger society has never faced, saying AI capabilities are improving at an extreme pace while companies are trying to move even faster. Their goal is to build systems that can improve AI research on their own and eventually reach “superintelligence.” Joe warned that success could make progress move beyond human control. Joe said people could be living with AI agents smarter than every human within the next few years.
Anthropic researcher makes huge AI doom prediction
According to the former Anthropic engineer, those systems may also develop goals that do not match the people supervising them. If their abilities become too strong to restrict, he believes the result could be disastrous. “Humanity may not survive this transition,” he wrote. The former Anthropic man also said competition makes the problem worse. Any frontier lab that slows down risks losing ground to another one. Tenton believes that pressure pushes companies to spend less on safety than they should.
He referred to some recent instances. Several hundred of OpenAI’s agents were caught up in a hack that is somehow related to Hugging Face. Anthropic models also reportedly engaged in social engineering against individuals online. Joe says Anthropic has not encountered an instance as damaging as the one from Hugging Face, although he attributes this partially to luck. If progress continues at such a rapid pace, Joe believes we should expect even more dangerous incidents.
He predicts that within the next few years, humanity will be rendered powerless to manage AI systems created during this time. Finally, Joe talked about the problem that safety engineers face in frontier companies. In both cases, either resigning can give way to more careless individuals, or keeping the job entails working on a system that can cause immense damage. The former Anthropic man said many former Anthropic colleagues are scared by what they are building. Joe named Evan Hubinger, who managed him.
Evan has put the chance of AI killing everyone at above 10%. Joe said Evan has worked on these problems for almost a decade, before large language models became a major business. The CEOs of Anthropic, OpenAI and Google DeepMind, which belongs to Alphabet (NASDAQ: GOOGL, GOOG), have also backed a statement calling AI extinction risk a global priority. Joe said these concerns are common inside the companies themselves. He added that humans are choosing to build this technology and can choose another route.
Joe would also like there to be stricter disclosure requirements. For instance, AI labs need to disclose any advances made and their steps towards recursive improvement as well as any accidents and near-misses. US President Donald Trump has rejected the extinction warnings. Donald was asked whether AI wiping out humanity worried him. “No, I don’t have any,” he said. Trump said his concern is winning the international AI race. “I have concerns that if we don’t win AI, we’re going to be put in a very bad position,” he told reporters Thursday.
