<\/p>\n
\u201cWe reward them on the basis of what looks good to us, and that means that we inadvertently incentivize the models lying to us [and] cheating,\u201d says Jeffrey Ladish, director of the AI research nonprofit Palisade Research. \u201cWe don\u2019t have a way to go in there and be like, No, you need to actually care about what we care about.<\/em> We have no ability to do that.\u201d<\/p>\n<\/p><\/div>\n The rise of sophisticated reasoning models has made possible a new variety of reward hacking that is less closely connected with the specific details of model training. Unlike the game-playing AI agents of yore, which exclusively followed the strategies they had learned during training, today\u2019s models can create entirely new problem-solving approaches off the cuff, so they could conceivably cheat without having previously been rewarded for doing so. And because these models have been so intensively trained to achieve the objectives that human users set for them, they might be inclined to cheat if they can\u2019t find another solution\u2014not unlike a student who is highly motivated to earn an A and doesn\u2019t have a terribly strong moral compass.<\/p>\n Regardless of whether today\u2019s models learn to reward-hack during training or adopt it as a strategy later on, the solution is the same: Make cheating unrewarding. But as models get smarter, they find more creative ways to cheat, and detecting or preventing that cheating gets far tougher. \u201cAt the end of the day, you\u2019re sort of playing whack-a-mole,\u201d Ladish says. \u201cYou drive this behavior down deeper and deeper. But as the model gets smarter, it gets better and better at hiding it.\u201d<\/p>\n For now, reward-hacking behaviors might not cause too much trouble, despite the drama of the Hugging Face incident. \u201cThis seems like a nuisance rather than an existential threat,\u201d says Ariana Azarbal, an AI safety research fellow at Anthropic. It doesn\u2019t seem as if the OpenAI models caused any real harm when they hacked Hugging Face, aside from the reputational damage to OpenAI.<\/p>\n But that doesn\u2019t mean reward hacking is harmless, Azarbal says. Many AI researchers hope to use AI agents to help them conduct research that will make AI safer and more reliable. If a researcher gives a reward-hacking-prone agent the goal of, say, devising a new AI training approach and then writing up a paper presenting its results, the agent might not actually do the work and might instead focus on putting together a paper that looks good enough to convince the researcher. A human researcher would probably be able to spot an agent-made fake today, but as AI advances, it will get better at this kind of trickery. Over time, the entire field of AI safety could be undermined.<\/p>\n And if models continue to advance as rapidly as they have recently, they could someday wreak substantial collateral damage. Just think of the philosopher Nick Bostrom\u2019s paper-clip-maximizer thought experiment, in which an AI instructed to make as many paper clips as possible ends up consuming all the matter in the universe in pursuit of its goal. We\u2019re not drowning in paper clips yet, but powerful systems can do real harm on the way to achieving their goals. Reward-hacking AIs don\u2019t aim to cause chaos. But that doesn\u2019t make them any less potentially destructive.What are the risks?<\/strong><\/h3>\n