Rational Inattention Reinforcement Learning for Bounded Rationality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional reinforcement learning frameworks fail to accurately simulate human-like behavior in multi-agent systems due to their reliance on rationality assumptions, which do not account for human irrationality and cognitive biases, leading to unrealistic outcomes in complex scenarios like principal-agent problems.
Innovation Solution
The rational inattention reinforcement learning (RIRL) framework models bounded rationality by incorporating cognitive costs into the reward function using mutual information, allowing agents to behave irrationally and account for the costliness of mental effort, thereby mimicking human-like behavior in simulations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional reinforcement learning frameworks are used to simulate multi-agent systems, then the simulation can be implemented with standard RL algorithms, but the agents exhibit unrealistic rational behavior that does not match human decision-making
Solution Approach 1:
The patent changes the fundamental parameter of agent optimization from pure utility maximization to utility maximization minus cognitive cost. By introducing the cognitive cost parameter λ and the mutual information term I(o_t; a_t), the framework transforms RL agents from perfectly rational to boundedly rational, enabling them to exhibit human-like irrational behavior while still optimizing their objectives.
Solution Approach 2:
The patent introduces mutual information I(o_t; a_t) as an intermediary term between the observation o_t and action a_t. This intermediary captures the cognitive cost of processing information and translating it into action, serving as a mediator that bridges the gap between rational RL optimization and human bounded rationality without requiring direct specification of human behavioral rules.
2Productivity
If agents are designed to maximize utility perfectly, then optimization is achieved, but cognitive costs of information processing are ignored
Solution Approach 1:
The patent converts the previously harmful assumption of unlimited rationality into a beneficial model of bounded rationality. By recognizing that perfect optimization is costly and often unachieving, the framework transforms the limitation of cognitive resources into a useful constraint that makes agents more realistic and adaptable to human decision-making patterns.
Solution Approach 2:
The patent modifies the objective function parameter from pure utility maximization to (Utility - λ × Cognitive Cost). This parameter change allows agents to trade off between optimization efficiency and cognitive effort, enabling them to achieve satisfactory but not perfect optimization while accounting for the energy and resources required for information processing.
3Reliability
If human irrationality is fully modeled, then realistic human behavior is captured, but the complexity of specifying behavioral rules increases
Solution Approach 1:
The patent extracts the complex specification of human behavioral rules from the simulation design. Instead of requiring designers to manually specify how agents should exhibit irrationality, the framework extracts and automates this behavior through the automatic optimization of the modified objective function that includes cognitive cost terms, simplifying the design process while maintaining accuracy.
Solution Approach 2:
The framework enables agents to self-regulate their cognitive effort by automatically optimizing their own behavior according to the modified objective function. Agents independently balance utility maximization with cognitive cost minimization based on their own observations and actions, eliminating the need for external specification of human-like irrationality patterns.
Data Source
AI summary
A rational inattention reinforcement learning (RIRL) framework determines actions of actors based on observations while modeling human irrationality or rational inattention. The RIRL framework decomposes observations into a set of observations, and passes the set through multiple information channels modeled as encoders having different information costs. Discriminators of the encoders measure a cost of mutual information (MI) associated with the observations. A stochastic action module of the RIRL framework receives encodings of the encoders and a history of encoded information from a previous iteration, and generates a distribution of actions. The stochastic action module includes a discriminator for measuring a cost of MI associated with the stochastic action module. The RIRL framework computes a reward based on the cost of MI of stochastic encoders, the cost of MI of the stochastic action module, and the distribution of actions. From the reward, the actions of the actors are determined.