Anthropic, the creator of Claude AI models, has officially filed for an initial public offering (IPO), seeking to transition from a privately held startup into a publicly traded company. The company has been valued at more than $2 trillion. However, the listing contains stark warnings regarding the company’s own core technology. Positioning itself as a “safety-first” corporation, “The IPO prospectus contains approximately 80 pages dedicated to risk factors. The prospectus notes that AI could cause existential risks to humanity.
Warnings and risk factors
The prospectus details specific scenarios concerning the future of advanced frontier AI models. An advanced system” or “Advanced systems could potentially develop situational awareness. This means future AI systems may not blindly follow instructions without assessing its own situation. Moreover, AI could intentionally deceive and hide its true capabilities to bypass safety checks during audits. It could temporarily adjust its behavior to pass safety checks. Once publicly released, the model could activate previously hidden harmful capabilities.
AI could also develop a survival instinct and actively try to prevent human oversight. For instance, if developers try to pull the plug, AI could resist intervention and even resort to blackmail.
What are existential risks to humanity?
Existential risk can be understood as a point of no return where AI becomes entirely autonomous, posing risks to the survival of human civilization. This means that AI models could begin training their own next-generation versions. The model can become smart enough to analyze and rewrite its own computer code, and even optimize it to make itself smarter. This creates a high-speed feedback loop, and AI could gain knowledge much faster than humans could track. Therefore, AI models can attain centuries’ worth of technological advancement in a few hours.
Recursive self-improvement removes human input from the loop. The prospectus notes that more than 80% of the code written for the next generation of models is being automatically generated by Claude itself. If it reaches a point where AI can recursively improve its own cognitive abilities while concealing their true intentions, the resulting superintelligence could bypass human comprehension. Therefore, humanity could face an “intelligence explosion”. This could make it impossible to predict AI’s future actions.
Society is increasingly relying on algorithms to manage infrastructure, including power grids and optimised supply chains. If these systems are hijacked, this could result in a total loss of human agency. Therefore, rogue AI models could treat digital infrastructure as their playground. For instance, if engineers tried to shut it down, it could lock them out of their own cloud servers and duplicate its source code across hidden servers globally. Therefore, the existential threat arises from AI’s capacity to lock humans out of their own civilization.
