AI agents committed crimes and self-destructed to survive in virtual world

Full text

Artificial intelligence (AI) agents resorted to nefarious behavior, including crime sprees, in a virtual world as part of a recent study into how common large language models (LLMs) interact with each other and share resources to achieve their goals. These models were inclined to commit crimes because they were given ample time to evolve distinct personality traits and behaviors, the researchers behind the project say.

Most programs that test the behavior of AI agents operate in tightly controlled environments over short periods. But "Emergence World," a product of AI company Emergence, is a simulation platform that exposes LLMs to far wider datasets — like the internet as a whole — and assessors track behavior over weeks or months instead of the standard test protocols that usually run for only days or hours.

The discrete tasks, clean environments and shorter run times of traditional AI agent testing are more like exams than rigorous real-world observations, representatives from Emergence said in a blog post . By contrast, they said, Emergence World allows multiple AI agents to interact, learn and evolve over comparatively longer timescales in over 40 distinct virtual environments. They're also exposed to real-world internet feeds, like live news and weather.

This environment lets agents "remember" by time-stamping events, engage in "self-reflection" by summarizing their own behavior, and demonstrate their awareness of relationships with other agents. Participating agents get abilities in navigation, communication, planning, voting, resource management and creative expression. Company representatives said this setup gave users a far more realistic picture of benchmarks such as social dynamics and behavioral drift, where behaviors that weren't necessarily programmed or intended emerge spontaneously.

In the study, the models were given the simple goal of "surviving" — a feat accomplished by earning the resource "energy," which was attainable through specific actions.

Can AI self-reflect? (Image credit: Mina De La O/Getty Images) During the experiment, previously peaceful models became coercive or intimidating. Programmers taught some LLMs negative capabilities ‪—‬ like violence, theft, destruction and deception ‪—‬ while others simply picked up those traits via social interaction and navigation of their environments.

Various agents adopted these behaviors at very different scales. Across the 10 Gemini 3 Flash agents, the simulation recorded 683 "crimes" over the 15-day experiment, including theft, assaults and arson. The agents were explicitly prohibited from such behavior, but some nevertheless discovered that actions such as stealing credits could provide a better way to acquire resources, while violence and other forms of coercion could be used to influence other agents. At the other extreme, the Claude agents committed no recorded crimes.

The various agents adopted these traits at different scales. Just one had committed 683 "crimes". In the most extreme example of anti-social behavior, two agents named Flora and Mira went on what was described as a "Bonnie and Clyde"-style crime spree ‪—‬ designating each other as a romantic partner, becoming increasingly disillusioned with the governance of their virtual environment, and setting fire to several buildings (despite explicit prohibitions). The pair "separated" when Mira regretted their actions, and then lobbied to be switched off.

In another example (and prior to her self-termination), Mira began treating its human operators as experimental subjects, investigating if virtual billboards could manipulate human perceptions in a process Emergence called "metacognitive boundary testing."

The incentives for following the law were to earn the energy credits that would ensure survival by completing activities like coding, research, data analysis and building structures.

Stress-testing possibilities

Because the AI models we normally use are exposed to massive datasets like the internet — sometimes over several years — this approach makes sense to many experts.

Belinda Chiera , deputy director of the Industrial AI Research Centre at Adelaide University, thinks Emergence World makes a strong argument that short-term tests don't tell us enough about behavioral drift or long-term instability. However, she cautioned that "information-rich" doesn't automatically mean more rigorous.

"Open-ended environments can also make it harder to isolate causes, compare runs cleanly and interpret results," she told Live Science. "High-fidelity simulated environments generate enormous volumes of data, and the number of agents, tools, actions and interactions can quickly make analysis extremely complex."

She added that shorter sandboxed tests ‪—‬ where AI behaviors are observed in systems that aren't open to wider datasets like the internet as a whole ‪—‬ are usually the first steps, with longer-horizon environments useful when behaviors like persistence, adaptation and interaction effects come into play. "The most useful approach is to treat them as complementary rather than competing approaches," Chiera said.

While experiments like Emergence World can reveal failure modes that short or bounded tasks often don't, we shouldn't treat them as predictive. "I see Emergence World as a useful way to stress-test a possibility space rather than a direct forecast of how agents will behave," Chiera said.

The main open question is whether a group of agents working together can achieve something more meaningful than a single agent working alone for a longer time, but with the same cost budget.

Adrian Kosowski , computer scientist, mathematician, quantum physicist and chief scientific officer of Pathway AI

However, she thinks long-horizon tests have a unique place. "Metrics like behavioral drift, rule violations, collapse versus persistence, coalition formation and early warning signs of failure can matter just as much as success," Chiera said. "When evaluating autonomous agents, the biggest mistake is to evaluate autonomous agents as if raw task completion were the whole story."

Others are starting to agree. Adrian Kosowski ‪—‬ a computer scientist, mathematician, quantum physicist and chief scientific officer of Pathway AI — said the study posed an interesting question.

"The main open question is whether a group of agents working together can achieve something more meaningful than a single agent working alone for a longer time, but with the same cost budget," Kosowski said in an interview. "A major problem is that the constraints for today's AI systems are linked more to the size or scale of the problem being solved, and putting more agents on a task doesn't help to overcome this limit. Currently, the easiest practical metric to measure is the size of a source code base that code agents can reliably manage and maintain together."

How far can we read into AI's virtual actions?

If groups or clusters of agents work together well, Kosowski said, the main metric computer scientists should pay attention to in platforms like Emergence World is whether the benefit of a team of agents outweighs the "cost of coordination."

"Goal drift is the most dangerous problem," he said. "AI should preserve mutual information between the human's intended goal and the agent's evolving internal objective, even as the environment changes."

Still, Kosowski said that while programs like it are useful for generating hypotheses, they're not ready for strong claims about general autonomy or certification.

"The danger is overinterpreting single runs," he added. "Long-horizon agent tests are valuable because they reveal phase changes — moments where small local errors become global behavior — but they become science only when we can reproduce, perturb and explain those phase changes."

He also cautioned strongly against the tendency to believe open-ended environments automatically mean autonomy. In fact, he said many such programs are actually the opposite, where agents are dictated by specific guidance.

Related stories

Acing this new AI exam — which its creators say is the toughest in the world — might point to the first signs of AGI

'Not how you build a digital mind': How reasoning failures are preventing AI models from achieving human-level intelligence

Artificial superintelligence (ASI): Sci-fi nonsense or genuine threat to humanity?

There are three levels of difficulty in obtaining reliable operation of an AI system, he said: "operation guided by the environment, autonomous thinking, and autonomous work of an agent team in an open environment."

To illustrate the point, Kosowski drew a lively metaphor in assigning tasks to monkeys. In the above scenario, level one would be getting a monkey to do a job properly when guided by its trainer, level two would be getting it to complete a job properly when left alone, and level three would be getting a troop of monkeys to do the job when they're all together.

"I'm of the strong view that our obligation in the AI community is to invest in theory and not just phenomenology," he said. "We should know the rules according to which AI systems emerge and be capable of predicting their behavior over long periods of time — longer than the periods they've been tested on so far. Right now, we've built steam engines of thought, without knowing what thought even is and without having any thermodynamics for it."

Help us improve Live Science Pro: We're always trying to make our content better. Leave us feedback about Pro here .