Action Selection Policy Generation for Imperfect Information Games
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Solving Imperfect Information Games (IIGs) is computationally expensive and time-consuming, especially for large games with many possible states and actions, as existing methods like original CFR require significant memory and do not handle large games efficiently.
Innovation Solution
The implementation of Monte Carlo Counterfactual Regret Minimization (MCCFR) with vector-form sampling policies, variance reduction techniques, and hybrid variants that include skipping mechanisms and discounting updates to accelerate convergence and reduce variance, allowing for more efficient action selection policy generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If original CFR method is used to solve Imperfect Information Games, then solution accuracy is improved, but computational time and memory usage increase significantly
Solution Approach 1:
The patent segments the game tree into multiple components and uses Monte Carlo sampling to selectively evaluate only certain portions of the tree rather than exhaustively traversing all nodes. This segmentation allows the algorithm to focus computational resources on critical decision points while approximating less critical areas, thereby reducing overall computational time while maintaining solution accuracy.
Solution Approach 2:
The patent applies partial action by using Monte Carlo sampling to evaluate only a subset of possible game states and actions rather than all possible combinations. The counterfactual regret minimization is updated based on sampled trajectories rather than complete game tree analysis, achieving acceptable solution accuracy with significantly reduced computational effort.
2Measurement precision
If original CFR method is used to solve large games, then solution accuracy is improved, but memory requirements increase significantly
Solution Approach 1:
The patent extracts and stores only the essential regret values and sampling information needed for counterfactual regret minimization, rather than maintaining complete game tree structures in memory. By extracting only the necessary computational elements and using Monte Carlo sampling to generate experiences on-the-fly, the algorithm reduces memory requirements while preserving solution accuracy.
3Loss of time
If Monte Carlo sampling is used to reduce computational load, then computational time is improved, but convergence speed decreases
Solution Approach 1:
The patent implements feedback mechanisms where the sampled experiences are used to update counterfactual regret values, which in turn influence future sampling decisions. The algorithm continuously refines its policy based on accumulated sampling feedback, accelerating convergence by learning from past sampling results and focusing subsequent sampling on more informative trajectories.
Solution Approach 2:
The patent uses periodic updates of the action selection policy based on accumulated sampling data. Rather than updating continuously after every single sample, the algorithm periodically aggregates sampling results and performs policy updates, which stabilizes convergence while maintaining computational efficiency. This periodic action balances the trade-off between sampling frequency and convergence speed.
Data Source
AI summary
Disclosed herein are methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating an action selection policy for completing a task in an environment. The method includes identifying multiple possible actions in a state, wherein the state corresponds to a vector of information sets; identifying a vector of current action selection policies in the state, wherein each current action selection policy in the vector of current action selection policies corresponds to an information set in the vector of information sets; computing a sampling policy based on the vector of current action selection policies in the state; sampling an action among the multiple possible actions in the state according to a sampling probability of the action specified in the sampling policy; and updating each current action selection policy of the execution device in the state based on the action.