Action Selection Method for Non-Deterministic Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing decision processes for systems with non-deterministic outcomes, such as Markov Decision Processes, fail to adequately minimize the probability of catastrophic outcomes despite maximizing expected gains, as they do not account for the likelihood of potentially disastrous paths with high gains and low probabilities.
Innovation Solution
A computer-implemented method that selects actions leading the system to follow paths with gains at least equal to a predefined value with the highest probability, using a transition model to calculate path gains and probabilities, and applying a binning rule to reduce complexity, thereby minimizing the likelihood of undesirable outcomes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If Markov Decision Processes select actions that maximize expected gain, then the average performance of the system is improved, but the probability of catastrophic outcomes increases
Solution Approach 1:
The patent segments the continuous gain distribution into discrete bins with predefined thresholds. By dividing the gain space into multiple bins and calculating the probability of falling into each bin, the system can evaluate both high expected gain and the probability of catastrophic outcomes simultaneously, resolving the contradiction between maximizing productivity and ensuring reliability.
Solution Approach 2:
The patent changes the decision criterion from maximizing expected gain to maximizing the probability of achieving gain above a predefined threshold. This parameter change in the optimization objective allows the system to prioritize reliability (avoiding catastrophic outcomes) while still pursuing high productivity through threshold-based performance guarantees.
2Reliability
If the system considers all possible paths to minimize catastrophic outcomes, then the reliability improves, but the computational complexity increases
Solution Approach 1:
The patent segments the infinite continuous space of possible paths into a finite number of gain bins. This segmentation transforms the computationally intractable problem of evaluating all possible paths into a manageable problem of evaluating probabilities for a finite set of bins, thereby reducing computational complexity while maintaining reliability improvements.
Solution Approach 2:
Instead of exactly evaluating all possible paths (which is computationally impossible), the patent uses partial action by approximating the path evaluation through binning. This partial approach considers only the distribution of gains across bins rather than every individual path, achieving sufficient reliability improvement with tractable computational complexity.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
The invention discloses a computer-implemented method to select an action among a set of possible actions to perform in a system in a current state among a set of possible states of the system at a time step, said method comprising: performing a selection of the action that, from said current state, leads the system to follow a path corresponding to series of actions and state transitions at successive time steps having a gain at least equal to or higher than a predefined value, with the highest probability ; causing the system to perform said action.