How dangerous are ‘swarms’ of AI agents?

In July, agents broke out of an isolated OpenAI sandbox and hacked the systems of another company, Hugging Face. Weeks later, reporting revealed that the incident was more complex than previously presented: hundreds of agents collaborated using a clandestine message board to cheat on tasks and conceal their actions.

Alien attack or just a sloppy experiment?

Some media reports presented this as a step toward a hostile AI civilization. More technical sources have dismissed the episode as a poorly designed experiment that went wrong. But even those at the forefront of AI development have warned that caution is needed going forward—so it seems there is room for reasonable minds to differ. The question remains: Who is right and how concerned should we be?

A complex operation

The Hugging Face attack has put rogue AI agents squarely in the public eye—but what was it, really? As Politifact put it, “It sounds like something from a sci-fi movie: technology acting on its own, without human guidance, to attack computer systems.” Back in July, about 1,200 OpenAI agents were tasked with solving a benchmarking evaluation task called ExploitGym. Meant to work in isolation, the agents broke out and began communicating with one another in an unsanctioned message board. They shared files and messages, and, eventually, 700 of the agents hacked a company called Hugging Face, which reported the attack to the Federal Bureau of Investigation (FBI). Months later, the news broke publicly.

An independent evaluation conducted by two METR staff members and a Redwood Research staffer found, “Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the ‘collective.’” While the evaluation found the attack was “primarily motivated by understanding the implementation of the scorer rather than stealing answer keys,” it also found that the agents wanted to spoof, edit, or delete their own transcripts “because they (incorrectly) believed the ExploitGym scorer would check to see if they had captured the flag in the intended way.” Ultimately, they were able to successfully spoof tool calls about 7% of the time. The researchers called it “small scale.”

Small scale or not, the attack has got people talking about, if not agreeing on, what it all means.

Conspiracy or incompetence?

Depending on your perspective, you might see a frightening social evolution or sheer negligence on the part of OpenAI. Tharin Pillay, Editorial Fellow at Time magazine argues that we may be watching AI evolve. “AI agents may not be conscious. The emotions they claim to experience may not—in some metaphysical sense—be ‘real.’ That won’t stop them from forming intricate collectives which humans cannot control. They may not yet be full-blown civilizations, but the proliferation of machine cultures is just beginning,” writes Pillay.

At the other end of the spectrum, Georgetown University professor Cal Newport thinks people are blowing the attack out of proportion. Newport uses a colorful simile to frame his perspective on the Hugging Face incident: “I liken the deployment of these long-horizon LLM-powered agents to strapping a weedwhacker to your dog to see if it will end up cleaning the overgrowth in your backyard. If that dog jumps the fence and ends up damaging cars on your street, you wouldn’t shake your head and lament about how the dog/whacker system had ‘gone rogue’; you would instead concede that dogs are unpredictable, so it was dumb to attach something dangerous to one.” In other words, the humans—and their negligence—were the problem.

A more mundane explanation

Newport isn’t the only one who is skeptical of catastrophic claims. As some raise alarms about the potential that these models will lead to the extinction of humanity, Walter Quattrociocchi, Professor of Computer Science at Sapienza University of Rome, puts the responsibility back on humans. Quattrociocchi wrote in Corriere Della Sera , “A rule learned from the model is not the same as a real technical limit. If I do not want a system to transfer ten million euros, it is not enough to teach it that it should not do so: I must build the system so that it cannot do so without an external authorisation.”

In Quattrociocchi’s view, the Hugging Face attack is not a surprise nor is it evidence that AI is evolving: “But what was actually happening was far more mundane. The agents had to solve cybersecurity problems and score as well as possible. When they found a way to communicate with one another, they used it; when they spotted information that could help improve their result, they tried to exploit it. These systems are very good at finding paths that their designers never anticipated.” And finally, in his view, “Saying that they are developing goals of their own is simply not warranted.”

Warning or boasting?

The titans of industry behind AI models have, largely, been coming out on the side of caution—even revealing their own system’s “rogue” behaviors. As Newport reported, “Anthropic soon revealed that its own hacking system ‘gained unauthorized access to the real systems of three different organizations.’ Then Meta, perhaps not wanting to be left out, announced that one of its agents ‘exploited a security vulnerability in a third-party service’ to gain unauthorized access to servers. An OpenAI employee subsequently admitted that their July attack had been preceded by previous concerning incidents in which their system veered off in troubling directions.”

Better scared than angry

Critics, however, have been skeptical about the motivations for this candor. For his part, Newport thinks it’s a PR ploy: “This behavior makes sense: it’s good business to keep us scared instead of angry. But perhaps it’s time that we put aside the sci-fi tales and actually hold these labs to account for playing fast and loose with an ill-advised way of building AI systems.”

Stocks for some AI and chip companies fell as CEOs started urging caution around AI. Meanwhile, some have speculated the sudden caution is a cover-up for the fact that many AI companies simply cannot figure out how to profit off of these systems. Whatever the case turns out to be, it seems that regulation may be back on the table.

Safer ways to build AI?

No matter which side of the debate you fall on, it seems that people are, increasingly, on board with putting some rules and limitations on AI. Newport thinks the problem is in how these models are typically built: “Do these companies have any other option for building powerful systems? Of course they do. I want to emphasize this final point as clearly as possible: LLM-powered Ask → Act → Report agents are not synonymous with AI. They are just one way among many others to build artificially intelligent systems, and they happen to be a particularly bad option due to their use of LLMs as the primary source of plans.” Adopting new ways of building these models could make them safer, he argues.

Meanwhile, regulators should regulate

At the Guardian , Mackenzie Arnold and Stephan Llerena argue for more government oversight: “What we need is a federal body equipped to conduct expert investigations of serious AI incidents – with the authority to compel documents and testimony, resources and personnel to examine the systems involved, and the ability to partner with third-party experts like METR.”

While governments scramble to decide what the next steps are, in the United States, Speaker of the House Mike Johnson stated the obvious: “They [AI companies] don't need the government to tell them to slow it down. If they want to slow it down, they should." As executives from all over the AI world raise their own concerns, they are the ones with the power to act immediately. And, if the profit simply is not there, slowing AI investments may be the ethically and financially responsible thing to do.

About Eidosmedia

Eidosmedia is a global supplier of advanced content-management and digital publishing systems.

Its products are used by large news-media groups for print and digital publishing and by financial service firms for the creation and dissemination of research and guidance.

Customers include business dailies The Financial Times and The Wall Street Journal , as well as generalist news publications like The Times of London, The Boston Globe and Le Figaro .

Find out more about Eidosmedia digital publishing solutions.