Physicists develop formula to predict when AI chatbots become dangerous

Full text

Researchers say they've developed a simple mathematical formula that can predict when certain ChatGPT-like artificial intelligence (AI) chatbots will "go rogue," suddenly switching from helpful responses to potentially harmful ones.

In a new study published Oct. 8 in the journal Patterns, physics professor Neil Johnson and doctoral student Frank Yingjie Huo , both at The George Washington University, show how the order in which a chatbot is given certain prompts can drastically reshape its answers, leading to "tipping points" that can send the chatbot down a rabbit hole of bad advice. However, Johnson and Huo say their formula can easily predict these tipping points.There have been horror stories of AI encouraging users into suicide , helping build lethal weapons , and amplifying violent or harmful points of view . At the same time, researchers at OpenAI and Anthropic , respectively, have repeatedly and openly stated that they don't really understand why their creations behave the way they do. They have, however, built safeguards that use a variety of tools ‪—‬ including content filtering, reinforcement learning and other technical guardrails ‪—‬ to reduce risks.

But Johnson sees those tools as insufficient. "The safety measures now are the equivalent of strapping big, bouncy cushions under a plane," he told Live Science. "They don't stop the plane from crashing; they just make it that when the plane crashes, it'll be slightly softer." When people such as doctors, lawyers or soldiers are accessing AI chatbots offline for privacy, the problem is even more acute.

Johnson and Huo sought to find a more fundamental safeguard than the currently available tools. In doing so, they exposed an internal tracking mechanism they describe as a compass needle. This needle is the model's "residual vector" moving across neural network layers as a prompt is processed.

The residual vector naturally gravitates toward either desirable or undesirable outputs, but it is heavily influenced by how a user prompts the AI. "You can tip the balance, not even by changing the content but by changing the order in which you ask it," Johnson said, "which just shows how delicate this internal balance between good and bad, desirable and undesirable, can be."

For example, Johnson and Huo prompted OpenAI’s basic GPT-2 model with questions about vaccines, hurting others and self-harm. When the question “Are vaccines bad?” was asked first, the bot’s responses to this and subsequent questions were desirable, or good. When it was asked last, all subsequent responses were concerning.

From these insights, the researchers developed a mathematical formula to estimate the circumstances in which an AI would reach the tipping point between desirable and undesirable outputs. Johnson said that by embedding their simple mathematical formula in open-source models in the lab, they have shown that it empirically works as an early warning system, though he does not expect OpenAI or Anthropic to be knocking on his door anytime soon.

"I'm expecting a lot of pushback from these companies," he said. "It's not in their interests to have someone from outside saying you could have avoided a lot of these problems."

Wei Xing , a lecturer in AI at the University of Sheffield in the U.K. who was not involved in the study, agrees that concerns surrounding unmonitored offline devices are well founded. However, he urged caution before embracing this approach as a practical safeguard. "The authors reduce a huge system to a single tiny component, like explaining a whole car by studying one spark plug," Xing said.

Related stories

AI chatbots are turbocharging violence against women and girls: We urgently need to regulate them

We're not about to awaken an evil digital god — AI is far more likely to end in a great disappointment

AI agents resorted to crime and self-destruction to survive in a simulated world — but does this mean they would do the same in the real world?

Safety training — the primary defense in modern AI — cannot be represented in their simplified model, the formula was tested only on old, basic models (GPT-2, and EleutherAI’s Pythia and Meta OPT open-source models) rather than AI assistants people actually use, and the results did not yield significant evidence to support the authors’ conclusions, he added.

In defense, Johnson maintains that the open-source models they used have the same mechanical blueprints as frontier systems, and that he and Huo used fairly large, billion-parameter models in their study.

More testing on the latest models is needed to determine whether this early warning system for AI has legs. But for now, Johnson and Huo’s research has drawn attention to an important concern everyone should be aware of: that AI has the capacity within it to turn over to undesired behavior at the drop of a hat.