Jacob Coxon has made global headlines today by resigning from Anthropic. The 27-year-old artificial intelligence safety and pretraining researcher, in series of posts on X warned the world about the dangers of emerging AI. He emphasized that neither OpenAI nor Anthropic was operating responsibly. Both corporations appear to be focused solely on self-improving superintelligence while disregarding the associated risks.
Why Jacob Coxon resigned from Anthropic
He cautioned that AI executives and developers of this technology genuinely believe that advanced AI could wipe out humans by 2030. He noted that private fears within these companies were far greater than what is publicly portrayed. Coxon had also previously also contributed to GPT-4o and model interpretability at OpenAI. He concluded that market pressures outweigh safety frameworks.
Moreover, recursive models could make it virtually impossible to control AI in the next year. This means superhuman systems could become capable of hacking critical infrastructure.
Coxon’s resignation comes after Anthropic’s recent announcement of an initial public offering (IPO). The company is aiming to go public on the stock market, allowing investors and institutions to purchase its shares. It is currently seeking a $2 trillion valuation. The commercial reality of this situation means that Anthropic’s focus could shift from security risks to maximizing profit. He stated that corporate AI development was gambling with human lives.
Industry backing and future risks
Evan Hubinger, Anthropic’s alignment lead, backed Coxon’s stance on X. While acknowledging that the company was trying its best, he stated that Anthropic does not have a concrete plan to resolve the issue of superintelligence alignment, and is making no substantial effort to do it either. He further said that the risk of extinction was 10% higher over the next decade. Hubinger maintained that while current AI models are low-risk, the dangers arising from recursively self-improving superintelligence models could be extremely dangerous.
Alex Turner, a former AI safety researcher at Google DeepMind, further confirmed Coxon’s view. He said that researchers believe that AI corporates are building something that could potentially destroy humanity.
There is wider consensus among AI researchers that Anthropic’s “Responsible Scaling Policy”, while good on paper, would not withstand the pressure of a potential $2 trillion valuation. These developments have caused fear and paranoia among the public. Members of the public are also urging researchers to explain technical mechanisms and exact complexities of these technologies. There are others who believe that a 10% chance of extinction means that there is a 90% chance that everything will turn out fine.
Coxon notes that Anthropic is aware of the dangers posed by rapidly developing AI. However, it continues to scale up their operations due to competition. By rushing to build superintelligence, Anthropic is effectively escalating the endgame. In his parting remarks, Coxon has urged leading technology experts to organize a labor revolt to halt the corporate AI race.
