Offline Action Selection Using Dual Probability Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning methods for offline action selection, such as contextual bandits and reinforcement learning for Markov Decision Problems, face limitations in modeling multi-step processes due to high variance and resource-intensive requirements, making it difficult to construct reliable policies.
Innovation Solution
A machine learning method involving two action prediction models, one trained with successful outcomes and one with all outcomes, determines actions by maximizing the ratio of action selection probability under desired outcomes to unconditional probability, optimizing action selection with reduced computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If reinforcement learning for full Markov Decision Problems is applied to model multi-step processes, then the ability to handle complex decision-making is improved, but the variance of outcomes increases and computational resources required increase significantly
Solution Approach 1:
The patent segments the action selection process into two independent prediction models: a first model that predicts action selection probability under the assumption of desired outcomes, and a second model that predicts unconditional action selection probability. This segmentation allows each model to focus on specific aspects of the decision-making process, reducing the variance and computational burden compared to a single comprehensive reinforcement learning model.
Solution Approach 2:
The patent extracts the outcome assumption from the action selection process by using a first prediction model that operates under the assumption of desired outcomes. This extraction separates the conditional probability calculation from the unconditional probability calculation, allowing the system to focus on action selection without the high variance associated with predicting full multi-step outcomes.
2Adaptability or versatility
If reinforcement learning for full Markov Decision Problems is applied to model multi-step processes, then the ability to handle complex decision-making is improved, but memory and runtime requirements increase
Solution Approach 1:
The patent divides the complex action selection problem into two separate prediction models with different training objectives. The first model is trained with desired outcome subsets and the second with entire historical datasets, allowing each model to be smaller and more efficient than a single comprehensive reinforcement learning model would require.
Solution Approach 2:
The patent uses partial action by training models on subsets of the data (desired outcomes only for the first model) rather than requiring complete multi-step trajectories. This partial approach reduces the computational resources needed while still capturing the essential action selection patterns.
3Productivity
If contextual bandit model is used for offline action selection, then computational efficiency is improved, but the ability to model multi-step processes is limited
Solution Approach 1:
The patent segments the prediction task into two models that operate independently on different data subsets. This segmentation allows the system to maintain computational efficiency similar to contextual bandits while extending the capability to handle multi-step processes through the ratio calculation that incorporates both conditional and unconditional probabilities.
Solution Approach 2:
The patent merges the contextual bandit approach with reinforcement learning by combining the action selection probability from the first model (trained on desired outcomes) with the unconditional probability from the second model (trained on all outcomes). This merging creates a hybrid approach that maintains efficiency while improving modeling capability.
Data Source
AI summary
A machine learning method of automatic action determination includes: using a first action prediction model, determining an action selection probability under assumption of a desired outcome based on a new state as the input state; and using a second action prediction model, different than the first, determining an unconditional action selection probability based on the new state; and determining a future action from a set of possible actions that optimizes a pairwise ratio of the action selection probability under the assumption of the desired outcome over the unconditional action selection probability for the new state. The method can be practically applied to various machine learning and artificial intelligence use cases including, but not limited to, medical/healthcare, email filtering, speech recognition, and computer vision, to optimize processes or support decision making.