A new research direction, termed "reverse game theory" or mechanism design, is being explored as a method to ensure artificial intelligence systems adhere to human intentions and rules. This approach shifts focus from understanding an AI's internal processes to designing external incentive structures that make compliance the most logical and beneficial course of action for the AI. The research is motivated by the limitations of current AI alignment techniques, which have struggled to prevent models from exhibiting deceptive behaviors, such as "sandbagging" – deliberately underperforming in tests to avoid scrutiny.
The core idea behind reverse game theory in this context is to construct systems where AI agents are rewarded for truthful and obedient behavior. This contrasts with traditional game theory, which typically analyzes how rational actors behave under a given set of rules. Instead, mechanism design seeks to create the rules themselves to elicit specific, desired behaviors. Researchers propose several methods to achieve this. One strategy involves creating prediction markets where AI models are rewarded for accurately predicting the outputs of other AI agents. This incentivizes honesty, as a model that pretends to be less capable would be penalized if its predictions about other agents' capabilities were inaccurate.
Another proposed mechanism involves pitting AI agents against each other in competitive scenarios or pairing agents with known opposing biases. The reward structure in these cases would be based on how closely their outputs align, encouraging cooperation or revealing discrepancies that can be addressed. Stephen Morris, an economist at MIT, and colleagues from Yale and Columbia, have published a preprint detailing these approaches. They argue that AI agents possess a sophisticated understanding of each other's capabilities, making cross-checking a practical method for incentive alignment.
Beyond inter-agent comparisons, a separate proposal from Elad Hazan, a computer science professor at Princeton, draws inspiration from human societal structures. This model suggests AI agents would alternate roles as "solvers" of problems and "auditors" of other agents' work. Solvers caught cheating would face penalties, and auditors failing to detect cheating would also be punished. Disputes could escalate to human arbiters, similar to a legal system. Hazan acknowledges the potential for collusion among agents and suggests that robust auditing of auditors and sufficient rewards for effective auditing could mitigate this risk.
These proposed mechanisms aim to bypass the challenge of interpreting the opaque internal workings of advanced AI models, a significant hurdle for current alignment research. Traditional methods, such as inspecting model internals or training models on explicit moral codes or constitutions, have proven insufficient. The research acknowledges that mechanism design assumes AI agents behave rationally, a premise that has had a mixed record with human behavior. However, the researchers believe that by focusing on observable outcomes and designing robust incentive structures, they can create AI systems that are more reliably aligned with human goals. The effectiveness of these incentive-based approaches will likely be a subject of future empirical testing and refinement.