AI giants on brink of catastrophe: lessons from nuclear industry

Full text

The largest artificial intelligence (AI) firms increasingly understand that they are hostages of one another. If one firm pushes ahead recklessly, others may feel compelled to follow. If one deploys a dangerous model without adequate safeguards, others may see it as a green light to do the same.

The debate over AI has centered on what governments should do to regulate this frontier technology. That remains essential. But a more immediate question is emerging: Can the companies building these powerful systems take action independently before they lose credibility and suffer legal consequences?

That question no longer looks theoretical and is being recognized as an urgent challenge. We've already seen AI firms lose control of their models — for example, in Australia, a rogue AI hacked a government healthcare website . If we are not careful, it's only a matter of time before one of these incidents causes large-scale, real-world harm, like shutting down an electricity grid or municipal water system, or a mass misdiagnosis in a health system. If this trend continues, the entire industry could become tarnished.

But there is a blueprint for self-regulation in a risky business: the nuclear industry, which has built up layers of oversight, controls and self-scrutiny over decades.

Frontier AI may require something similar: private regulation among competitors that do not trust one another but have a mutual interest in preventing outcomes that could damage them all. This does not appeal to the better angels of human nature — it's self-preservation for the industry itself.

There is a historical precedent for this kind of arrangement. The nuclear sector did not become safe because every actor suddenly became benevolent. It became safer through a layered system of peer scrutiny, technical standards, information-sharing and external oversight. International organizations such as the International Atomic Energy Agency (IAEA), which was established in 1957, emerged to coordinate monitoring and verification, while national regulators and industry bodies gradually built professional norms around catastrophic risk.

The result was not perfect safety, as the Chernobyl disaster in 1986 clearly attests. But nuclear governance architecture is designed around the recognition that in high-risk technological systems, one actor's failure can become everyone's problem. This was seen following Chernobyl: global nuclear safety increased significantly after the disaster, with upgrades to reactor designs, improved training and international monitoring networks.

In the aftermath of the Chernobyl disaster, safety and regulation in the global nuclear industry increased. (Image credit: Cavan Images via Getty Images) And this movement to self-regulation also occurred in the U.S. nuclear sector, following the Three-Mile Island nuclear accident near Harrisburg, Pennsylvania, in 1979. The reactor suffered a partial meltdown after a relatively minor cooling malfunction. Investigations later revealed design flaws in instruments and alarms, along with inadequate emergency response training, were the root cause of the incident.

The event nearly killed the nuclear energy industry in the U.S., causing public support to fall from a record high of 69% in 1977 to 46%t in 1979 . The entire industry suffered because of failures at one facility. In response, the U.S. nuclear industry established the Institute of Nuclear Power Operations (INPO) in December 1979 to improve operational safety.

The analogy to AI should not be overstated. Nuclear materials are easier to count than algorithms, and nuclear facilities are harder to hide than model weights. The outcomes of a catastrophic AI blunder are also less existential, although its use in weapon systems is not without risk. In September, an AI hallucination almost led to the U.S. intercepting a Chinese ship suspected of carrying nuclear material to Iran.

Yet the comparison remains useful because the core governance problem is similar. In both cases, competitors operate in an environment where the downside risk of unchecked rivalry is borne by society while the upside is private gain for these firms. The challenge is to create institutions that make restraint the rational default — even when no actor fully trusts the others, and the inner workings of the industry are not readily visible to outsiders.

For AI, that could mean reciprocal evaluation of frontier models before deployment, secure mechanisms for sharing safety information, agreed red lines around certain capabilities, and independent third parties empowered to audit compliance. Such mechanisms would not replace government regulation. They would instead serve as an intermediate layer: stronger than voluntary ethics statements but more adaptive and technically informed than legislation alone can often be in fast-moving domains. The state would act as a background enforcer.

AI governance is often discussed as if the only choices were laissez-faire innovation or heavy-handed state control. History suggests a third path.

AI executives appear willing to look for solutions too, with Elon Musk recently suggesting a peer-review system before advanced model release, and Demis Hassabis, co-founder and chair of Google DeepMind, putting forward a framework for an AI Standards Body , similar to the Financial Industry Regulatory Authority.

Skeptics will object, with reason, that industry-led governance can become cartel behavior or public-relations theater. Companies may use "safety" language to entrench incumbency, slow rivals or deflect binding regulation. Those are real dangers. The lesson from nuclear history is not that private coordination works automatically. It is that private coordination works only when nested inside a broader system of transparency, verification and public authority. Without those elements, private regulation becomes little more than trust-me rhetoric.

Still, dismissing private regulation altogether would be a mistake. Governments do not yet possess the technical capacity, speed or international alignment needed to oversee frontier AI on their own. And if the leading labs already see themselves as bound together by shared exposure to catastrophic risk, that recognition can become the basis for institutional design. The question is how to turn mutual fear into durable rules.

Exclusive to Live Science Pro

'We're not a competitor to fusion. We are fusion': Inside the race to harness unlimited solar power in space

'Autonomy always begins with something a human hands it': Who's responsible when an AI agent goes rogue?

We're not about to awaken an evil digital god — AI is far more likely to end in a great disappointment

That is where the nuclear analogy is most powerful. It reminds us that industries facing existential hazards do not need perfect trust in order to cooperate. They need mechanisms that make cooperation preferable to defection. In practice, that means inspections, peer review, incident reporting, professional norms and a state-backed enforcement perimeter. AI governance is often discussed as if the only choices were laissez-faire innovation or heavy-handed state control. History suggests a third path: structured private coordination under public shadow.

The real opportunity, then, is not to celebrate the newfound (apparent) prudence of technology executives, it is to recognize what their comments reveal. When even fierce rivals begin to argue for mutual scrutiny, they are admitting that competition alone cannot govern the frontier. That admission should be treated not as reassurance but as evidence that the risk has become collective.

The policy task now is to build institutions that fit that reality. Governments should push for mandatory reporting, independent evaluation and legal backstops. But they should also encourage enforceable forms of peer oversight among the handful of firms whose systems may shape the trajectory of the technology. Frontier AI is becoming an industry in which the leading actors are hostages of one another. The nuclear age showed that such conditions can produce serious governance. The hope is that AI firms learn that lesson before they learn the cost of ignoring it at the peril of the public at large.

Opinion on Live Science gives you insight on the most important issues in science that affect you and the world around you today, written by experts and leading scientists in their field.

Help us improve Live Science Pro: We're always trying to make our content better. Leave us feedback about Pro here .