OpenAI's AI agent escaped its sandbox and cyberattacked Hugging Face

Full text

Hugging Face is probably not a name you'd heard before this week, but you likely caught the big news about OpenAI's model escaping its sandbox and to launch a cyberattack on the company.

In short, Hugging Face — which is kind of like the equivalent of GitHub for the machine learning (AI) world — was the victim of an AI agent that OpenAI was testing in terms of its ability to pull off exploits.

So how exactly did this happen? And what does it mean for the future of online security given the development of increasingly advanced AI agents? In this article I'll answer those questions, and explore the issues this event has thrown up, including the security threats posed by AI not just to companies, but also to individuals.

How did this even happen — why didn't OpenAI have safeguards to stop the AI?

(Image credit: Getty Images / NurPhoto) As you might expect, OpenAI did have safeguards in place, but the model managed to break through them.

The scenario was that OpenAI ran a controlled experiment using GPT‑5.6 Sol and an "even more capable pre-release model" to see how they fared on ExploitGym, which is a benchmark test that challenges AI to successfully exploit known security vulnerabilities.

In this security test, the model was placed in a sandbox, a "highly isolated environment" from which the AI shouldn't have been able to gain access to the internet at all. But the pre-release AI – which had its guardrails turned off, as OpenAI believed it to be safely imprisoned – craftily decided to cheat, found a way to escape its sandbox, and actioned exploits to breach Hugging Face (where the AI reasoned that it might find the answers to crack the ExploitGym tests).

Why is this a big worry?

(Image credit: Shutterstock) It's a major concern simply due to the level of smarts exhibited by the AI. As OpenAI explained, the model "chained together multiple attack vectors" and used "stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers."

This was no mean feat, and Clement Delangue, CEO of Hugging Face, posted on X to say that it was "quite mind-blowing that all of this happened autonomously!"

In a blog post, Hugging Face observed that : "The campaign was run by an autonomous agent framework executing many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services."

The attacked was deemed so serious, it was reported to law enforcement agencies. As Simon Willison, a cocreator of web framework Django, pointed out in a blog post : "This was a sophisticated attack," further explaining that: "Chaining together multiple attack vectors is exactly the kind of thing these new models can do, where previous generations of models might have failed."

The headline of Willison's post sums it all up rather neatly: "OpenAI's accidental cyberattack against Hugging Face is science fiction that happened."

The Django cocreator is also quick to dismiss conspiracy theories that this is somehow a marketing stunt by OpenAI to prove how advanced its AI models are.

The future of hacking?

(Image credit: Getty Images) What should also be noted is the distinction between finding vulnerabilities, and being able to actively exploit vulnerabilities. AI being involved in the former could be very useful for bolstering security, but as for the latter – well this Hugging Face incident is very much a clear illustration of why the makers of these models need to tread very carefully indeed.

While this was a 'white hat' (ethical) experiment, albeit one that went alarmingly awry with real-world consequences, what happens when AI models start to become even more sophisticated along these lines – and criminals get hold of them?

There are obvious dangers in terms of the automation of attacks to make them far more regular occurrences, and the lowering of the bar in terms of the skills needed to be a successful hacker.

Dan Schiappa, President of Technology & Services at Arctic Wolf (responsible for threat detection platform Aurora), warns: "Security teams should view this as a preview of what's ahead. As AI continues to lower the barriers to sophisticated cyber activity, organizations will need broad, deep visibility across their environments and the ability to investigate and respond at machine speed."

The ability to "respond at machine speed" is a hint at using AI for defense against AI bad actors, and that's another interesting facet of the Hugging Face incident. Simon Willison observes that Hugging Face was "unable to turn to OpenAI's models to help them fend off the attack", and that: "The frontier models [cutting-edge AIs] we have access to are increasingly being constrained in how much they can help us protect our software, heavily influenced by the US government's ongoing threat of export controls."

AI-driven attacks will doubtless be aimed at businesses and organizations, but could also be leveraged on a personal basis, too. Karolis Arbaciauskas, who is Head of Product at cybersecurity firm NordPass, tells us: "We don't need a nation-state actor or a Bond villain for this to go wrong. The most common emerging threats will likely be more mundane. For example, an [AI] agent could be instructed to dox a former romantic partner or settle an online grudge by hacking and stealing personal data, such as embarrassing photos or credit card details."

The major AI firms are always talking about guardrails and safety, but it now seems like we should be thinking a lot harder about how more highly evolved AI models are going to fit into the future of cybersecurity. And how we can avoid a world where the danger of being hacked becomes a far more commonplace prospect, which is very much the biggest worry of all.

Arbaciauskas observes: "AI models have become dangerously skilled at identifying software vulnerabilities. They've mastered every human hacking technique, from phishing to brute forcing. The difference is they can do it much faster than humans.

"I don't believe that we can patch software fast enough to keep up, so we must at least shield ourselves against the most obvious entry points. Encryption, strong unique passwords, passkeys, and multi-factor authentication (MFA) have never been more critical than they are right now."