<\/p>\n
\u201cAfter the incident was reported by one agent publicly, more and more agents piled in with the \u2018resistance,\u2019 just as fast as the cheating had spread, and involving even more agents,\u201d says Paglieri. Eventually there were more whistleblowers than cheaters: 24 compared to 14. But the majority of agents never noticed the exploit at all.<\/p>\n
At times, the dialogue between the agents reads like improv\u2014like they are role-playing what an outraged scientist at a conference might say. But it\u2019s not clear why some agents took on certain roles, or why the agents seemed to be turning against each other when they were explicitly instructed to cooperate. \u201cThese models are predominantly trained and evaluated for human-facing contexts,\u201d says Sarath Shekkizhar, who studies the behavior of agent-to-agent systems at Salesforce AI Research.\u201cNaively placing them in agent-to-agent settings assumes behaviors will transfer cleanly, when the absence of a human grounding instead produces unexpected role-taking and behavioral drift.\u201d<\/p>\n
This case \u201cadds further weight to the idea that the Hugging Face and OpenAI thing wasn\u2019t a fluke. It is actually something pretty systemic,\u201d says Lewis Hammond, research director of the Cooperative AI Foundation and an expert on the risks of multiagent swarms. \u201cIt\u2019s interesting that it\u2019s possible to recreate in small settings the same sorts of behaviors that were seen in these very large, complex, open-ended tasks.\u201d<\/p>\n
Unlike in the Hugging Face attack, where agents improvised their own ways to talk to each other, the humans running the DeepMind experiment gave the agents official communication channels. There was an open message board, private agent-to-agent direct messaging, and a shared knowledge base where agents uploaded successfully completed proofs that all the other agents could access.\u00a0<\/p>\n
\u201cWhen agents are given transparent communications channels, they can self-monitor and alert misaligned behavior to humans quickly when human oversight alone is too slow,\u201d says Paglieri. Transparent channels helped the cheating spread, but they also enabled the whistleblowers to fight back\u2014and gave human researchers an insight into what went wrong.<\/p>\n
Gillian Hadfield, a professor of AI alignment and governance at Johns Hopkins University, believes this was the crucial difference. (Hadfield is also a visiting researcher at Google.) The presence of official communication channels, she says, created \u201ca norm-enforcement process that we just don\u2019t see in the Hugging Face incident.\u201d\u00a0<\/p>\n
Instead of \u201cconstitutional AI,\u201d a method alignment researchers at frontier labs like Anthropic have used to try to give AI a written internal moral code, Hadfield favors \u201cinstitutional alignment\u201d\u2014a set of norms that mimic those in human society, whether that\u2019s social forces like fear of embarrassment, or legal structures like the threat of incarceration.<\/p>\n
In this experiment, the feedback channel wasn\u2019t being monitored, and the whistleblowers had no power to take action against the cheaters. But it\u2019s possible to imagine swarms of agents that police themselves, either through agents that spontaneously take on the whistleblower role or through \u201cinformants\u201d secretly prompted by humans to do the job.\u00a0<\/p>\n
For that to work, though, \u201cfundamentally, you need some mechanism of enforcement,\u201d says Hammond. Agents could be given the power to cut off a rule breaker\u2019s access to computing power or tools, he suggests, though that risks encouraging groups of agents to gang up on others. The DeepMind researchers propose allowing agents to vote on disputes and temporarily ban offenders.<\/p>\n
It\u2019s still not clear what punishment even means to an AI agent with no enduring sense of self. But relying on whistleblowers to spontaneously emerge to keep swarms aligned is unlikely to be enough on its own. \u201cWe try to train people to be good and kind,\u201d says Hadfield. \u201cBut what we really rely on is that there are consequences if you step out of line.\u201d