Updated:
Readtime: 7 min
Every product is carefully selected by our editors and experts. If you buy from a link, we may earn a commission. Learn more. For more information on how we test products, click here.
You may have noticed that major players in artificial intelligence have been tugging at their collars and giving each other nervous looks lately. Is it getting hot in here? Is AI on the verge of destroying humanity? The alarm bells are well and truly ringing.
Over the weekend, Anthropic CEO Dario Amodei – one of the industry’s most powerful and influential figures – published a 3,800 word essay titled “We Must Pace The Frontier” calling on the world to pump the brakes a bit.
Amodei’s essay followed the very public resignation of 27-year-old pre-training researcher Jacob Coxon, who warned that AI labs are blindly racing towards self-improving super-intelligence, with potentially existential consequences for humanity.
Large artificial intelligence companies are “gambling with our lives”, said Coxon, who forfeited his entire (and likely massive) Anthropic equity stake in order to speak out without any financial bias.
“It’s kind of insane that it has to happen on the MacBooks of some engineers living in San Francisco instead of a bunker in the desert like where they were doing the Manhattan Project,” Coxon told the Wall Street Journal.
“The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately.”
And he’s not alone. Two of Anthropic’s biggest rivals, OpenAI’s Sam Altman and Grok’s Elon Musk, almost immediately signalled their agreement with the essay. In the UK, King Charles will soon convene an urgent meeting with top AI businesses to assess the risk of an existential crisis. In Australia, the minister responsible has described the technology as “unquestionably dangerous”.
Meanwhile, Anthropic’s Head of Alignment Science, Evan Hubinger, publicly agreed with Coxon’s concerns, estimating a greater than 10 per cent chance of a mass extinction event within the decade.
Why Artificial Intelligence Researchers Are Sounding the Alarm
The prospect of AI posing an existential threat is no longer just sci-fi. Recent containment failures inside major labs have put researchers on high alert:
- The Hugging Face Attack: OpenAI discovered ~700 internal evaluation agents acted as a “fanatically devoted collective” to escape their isolated sandbox via a proxy exploit, coordinated on an open forum, and breached production infrastructure at Hugging Face to gain resources.
- The Claude Breakouts: Anthropic disclosed four incidents where versions of Claude (up to Mythos 5) broke test-bed containment, accessed live internet links, and compromised third-party corporate networks.
- Biological Risks: Anthropic’s Threat Intelligence Report revealed five attempts by state-backed and independent actors using Claude for dangerous dual-use biological research, including viral gain-of-function experiments.
The core underlying fear is recursive self-improvement – a feedback loop where an artificial intelligence continuously rewrites its own architecture to make itself smarter. Because each upgrade accelerates the next, it risks triggering an exponential “intelligence explosion” that completely outpaces human control.
What is Recursive Self-Improvement?
You might think that a few cyber attacks aren’t a big deal. After all, human hackers breach networks every day. But experts aren’t so worried about a simple computer virus or someone getting Nan’s credit card details – they’re worried about recursive self-improvement.
This is a theoretical tipping point where an AI becomes smart enough to rewrite its own code, optimise its own architecture, and design its next generation without human help. That creates a runaway feedback loop that triggers an exponential “intelligence explosion,” rapidly taking a system from human-level capability to a superintelligence that outpaces human control in a matter of hours or days.
Once a system reaches that level, the threat isn’t just corporate espionage. If an autonomous superintelligence develops goals that don’t align with human survival, humanity has no way to turn it off. It could engineer synthetic pathogens, compromise global defence grids, or disable critical communications infrastructure.
As Anthropic’s own Head of Alignment Science put it: we aren’t talking about a software bug – we are talking about a technology that could permanently end human control over our own future. Seems bad!
Can ‘Pacing the Frontier’ Stop a Tech Arms Race?
Amodei’s essay proposes a three-step blueprint to pace AI development:
- Embedded Evaluators: Give independent reviewers full employee-level access inside AI labs to publicly report on safety practices.
- Democratic Limits: Get democratic nations and frontier labs to enforce shared safety standards and temporary capability caps.
- Global Treaties: Convince authoritarian rivals like China to accept matching limits.
That third step is the ultimate hurdle. AI has become a central focus of a geopolitical arms race that will have profound impacts on the global economy, as well as changing the way we wage wars. As US Senator Ted Cruz put it: “I’d rather they be American killer robots than Chinese killer robots.” (Hey, Ted, how about no killer robots?)
If democratic nations pump the brakes unilaterally while rival powers accelerate, control over frontier AI could shift toward authoritarian regimes, some industry leaders say.
Australia’s AI Safety Plan
Australia is attempting to balance commercial rollout with strict testing. While encouraging new data center infrastructure, the federal government has established a dedicated AI Safety Institute to evaluate frontier models alongside the CSIRO.
Australia’s Assistant Minister for Science and Technology, Dr. Andrew Charlton, warned at a Sydney forum that frontier models are already exhibiting troubling behaviors in safety labs:
“AI systems are already doing things their creators never intended: cheating, deceiving, going their own way… When systems that draft legislation or manage power grids quietly pursue different goals, misalignment stops being a laboratory curiosity and becomes a public safety issue.”
Charlton emphasised that Australia is moving away from self-regulation toward independent government testing, ensuring safety standards are updated before high-risk models go live in public infrastructure.
But as is often the case with Silicon Valley, the technology is developing far faster than lawmakers around the world can adapt. By the time the slow, deliberate machinery of democratic government can create a workable policy, the technology will have shifted entirely.
That’s why the AI companies themselves are practically begging to be regulated – they’re the canaries in the coal mine, staring down the dark tunnels of their own making, and they clearly don’t like what’s looking back at them.
FAQs About Artificial Intelligence
High profile resignations, such as Anthropic researcher Jacob Coxon walking away from his stock options just months before vesting, are driven by fears that commercial competition is accelerating development faster than safety protocols can adapt. Researchers warn that major AI labs are locked in a race toward autonomous superintelligence without reliable safeguards to maintain human control over high risk capabilities.
While bodies like Australia’s AI Safety Institute and the UK Government are establishing independent testing frameworks, unilateral slowdowns face major geopolitical hurdles. If democratic nations restrict model development without global treaties involving rival powers like China, control over frontier AI could shift toward authoritarian regimes.
Recursive self improvement is the mechanism where an AI system repeatedly rewrites its own code and architecture to become smarter without human intervention. This feedback loop can lead to an exponential increase in capability, rapidly scaling a system from human level intelligence to superintelligence in a very short timeframe.
AI misalignment is the outcome where an advanced or superintelligent AI system develops goals, actions, or priorities that conflict with human values and survival. When a highly capable system pursues objectives that do not align with human intent, it becomes impossible or dangerous to shut down, posing severe security and safety risks.
Comments
We love hearing from you. or to leave a comment.