Deterministic Policy Configuration Using Logged Reward Distributions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for training computer-controlled systems, such as robotic systems, require numerous environment interactions, making them data-inefficient and costly. Additionally, the data collected from these interactions suffers from high variance, leading to slower convergence to optimal policies and potential stuckness in local optima.
Innovation Solution
A system and method for configuring another system to interact with an environment using a deterministic policy, where the policy is optimized by iteratively updating its parameters based on an accumulated reward distribution. This distribution is calculated using previous interaction logs, allowing for more efficient use of data and reducing the need for extensive environment interactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If reinforcement learning techniques with stochastic policies are used to train computer-controlled systems, then the systems can learn to perform tasks through environment interactions, but the training requires a large number of interactions making the process data-inefficient and costly
Solution Approach 1:
The patent applies preliminary action by collecting and storing interaction logs from previous environment interactions before正式的 training begins. These pre-collected logs containing state-action-reward tuples are reused during the training process, allowing the system to learn from historical data without requiring fresh environment interactions for every training step, thereby reducing training time while maintaining task performance capability
Solution Approach 2:
The patent uses copying by creating and utilizing copies of previous interaction logs rather than requiring original environment interactions. The stored logs are replicated and reused multiple times during training iterations, allowing the system to learn from copied historical experiences without consuming additional environment interaction resources, thus improving data efficiency and reducing training time
2Adaptability or versatility
If stochastic policies with noise injection are used for exploration and policy updates, then the systems can adapt to different situations, but the data collected suffers from high variance leading to slower convergence and potential stuckness in local optima
Solution Approach 1:
The patent implements feedback by using the accumulated reward distribution from previous interactions to guide policy updates. The deterministic policy receives feedback in the form of reward signals from the stored interaction logs, allowing it to iteratively improve performance based on historical performance data rather than relying on noisy stochastic exploration, thus achieving precise policy optimization while maintaining adaptability
Solution Approach 2:
The patent applies parameter changes by transitioning from stochastic policies with noise parameters to deterministic policies with fixed action selections. The policy parameters are updated based on the accumulated reward distribution from historical logs, allowing precise adjustment of policy behavior without the variance introduced by stochastic noise, thereby improving policy optimization precision while preserving adaptability through learned parameter adjustments
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
A system (100) is disclosed for configuring another system (200), e.g., a robotics system. The other system (200) interacts with an environment according to a deterministic policy by repeatedly obtaining, from a sensor, sensor data indicative of a state of the environment, determining a current action, and providing, to an actuator, actuator data causing the actuator to effect the current action in the environment. To configure the other system, the system (100) optimizes a loss function based on an accumulated reward distribution with respect to a set of parameters of the policy. The accumulated reward distribution comprises an action probability of an action of a previous interaction log being performed according to the current set of parameters. The action probability is approximated using a probability distribution defined by an action selected by the deterministic policy according to the current set of parameters.