How “reverse game theory” could keep AI agents in check

A second approach, as envisioned by Elad Hazan, a professor of computer science at Princeton, is to design mechanisms that take inspiration from the ways human society keeps people “aligned”: social norms, the justice system, the monetary incentives of free-market capitalism. “We want to align superintelligent AI,” he says. “Why don’t we first talk about humans? How do we align humans?”

Hazan’s suggested mechanism involves having AI models take turns as either “solvers” that tackle problems or “auditors” that check the others’ work. “One agent acts like law enforcement; the other agent acts like the citizen,” he says. 

During training, the models are rewarded depending on outcomes: Solvers that get caught cheating are punished, but so are auditors that fail to catch the cheaters. In situations where there’s a disagreement between auditors and solvers, the case can be sent to “trial”—perhaps a human expert will decide who is right. 

There is, Hazan admits, a risk that agents will collude with each other—as auditors have colluded with the businesses they’re meant to be auditing, or corrupt police officers have colluded with criminals. But that problem can be solved by rewarding the auditors well enough, so that being caught isn’t worth the risk (a lot easier in the world of AI training than the real one, where money for police salaries is tight), and by auditing the auditors using techniques akin to employing mystery shoppers. 

Hazan points to Nobel Prize–winning work by the economist Gary Becker, who introduced the idea of the “rational offender” in the 1960s. Criminals don’t break the law because they’re inherently bad people, he said. Instead, they do it when it’s the most rational choice given the incentives on offer. Crime makes sense only when the proceeds outweigh the potential punishment. 

Alignment shouldn’t be thought of as a property of a model, he argues, but as a descriptor of its behavior at a particular moment. “You are, like, a nice person,” he says. “Are you aligned? I don’t know. Probably you are aligned most of the time, but there could be a situation where you’re not.”

One potential issue with applying mechanism design to AI: It relies on the assumption that AI agents behave rationally. Mechanism design has a mixed record with humans, who often fail to behave the way economic models say they should. 

It’s ironic: Economists have spent decades trying to reduce human behavior to mathematics, and now, in large language models, we’ve built mathematical systems that run on human language and could be just as inscrutable. AI models exhibit behaviors that seem more human than machine, and they can be swayed by subtle factors that are difficult to understand and explain. “Is it even reasonable to assume that they will respond to incentives?” asks Helen Qu, a research fellow at the Flatiron Institute, a nonprofit dedicated to advancing scientific discovery through computational methods, who has written about mechanism design and AI safety.