By TheDailyNewsHub
A researcher who recently resigned from artificial intelligence company Anthropic has issued a stark warning about the direction of the AI industry, arguing that leading technology companies are racing towards self-improving superintelligence without adequate safeguards.
Jacob Coxon, a 27-year-old researcher who has worked on AI pre-training at both OpenAI and Anthropic, announced his resignation on social media, saying he could no longer participate in what he described as an irresponsible race to develop increasingly powerful AI systems.
Coxon said the industry was “racing straight to self-improving superintelligence and gambling with our lives”, arguing that the people developing the technology themselves believe it could pose an existential threat to humanity within the decade.
Concern over self-improving AI
At the centre of Coxon’s warning is the possibility of AI systems becoming capable of significantly improving their own capabilities.
Superintelligent AI refers to hypothetical systems that would outperform humans across virtually all intellectual tasks. Researchers concerned about such systems warn that if an AI became capable of rapidly improving itself, humans could struggle to understand, predict or control its behaviour.
Coxon argued that the competition between major AI laboratories is creating pressure to move faster, even when researchers remain uncertain about how to ensure increasingly powerful systems remain aligned with human interests.
He also called for greater coordination among AI companies and suggested that a temporary halt on increasing model capabilities could be necessary if the industry cannot establish adequate safety measures.
Anthropic researcher backs warning
Coxon’s concerns received significant attention after Evan Hubinger, Anthropic’s Alignment Science Lead, publicly responded to his claims.
Hubinger said Coxon was correct that researchers at Anthropic take the possibility of catastrophic AI risks seriously. He estimated his own probability of AI causing human extinction within the next decade at more than 10 percent.
Hubinger also acknowledged that Anthropic does not yet have a complete solution for “alignment for superintelligence” — the challenge of ensuring highly capable AI systems continue to behave according to human intentions and values.
He nevertheless said Anthropic was trying to address the risks, while noting that the company was not yet clearly on track to solve the alignment problem.
Anthropic says AI needs safeguards
The controversy comes as Anthropic continues to position AI safety as a central part of its development strategy.
The company says increasingly powerful AI systems can deliver major benefits but also introduce new risks requiring safeguards. Its Responsible Scaling Policy outlines measures intended to address emerging threats as AI capabilities become more advanced.
Anthropic has also published a frontier safety roadmap covering areas including security, AI alignment and measures intended to reduce risks from increasingly capable systems.
However, the resignation highlights a growing tension within the AI sector: companies are competing to develop more capable systems while some of the researchers building those systems are simultaneously warning that the technology may eventually become difficult to control.
Coxon’s resignation does not establish that superintelligent AI will cause human extinction. His claims represent a warning about a potential future risk, and the likelihood and timeline of such an outcome remain subjects of significant debate among researchers.
What his departure does demonstrate, however, is that concerns about AI safety are no longer confined to outside critics. They are increasingly being voiced by people who have worked directly inside some of the world’s leading AI laboratories.
As the race for increasingly capable AI accelerates, the central question may no longer be simply how powerful these systems can become, but whether humanity can develop adequate safeguards before their capabilities outpace human oversight.