Offline Control Learning With Feedback-Corrected State Transitions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing learning methods for control targets using offline data face accuracy issues when insufficient data is available for certain states, leading to suboptimal control performance.
Innovation Solution
A learning device that utilizes a model acquisition means to train a policy using feedback information, a display device to evaluate accuracy, and a recording medium to improve model accuracy by incorporating user feedback and correcting pseudo trajectories.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If offline learning is performed using data obtained in advance, then the control learning can be completed without real-time data collection, but the accuracy of learning in states with insufficient data will be low
Solution Approach 1:
The patent implements a feedback mechanism where the learned policy is executed in the actual environment, and the resulting state transitions are fed back as new training data. This allows the system to iteratively improve learning accuracy in states where initial data was insufficient, resolving the contradiction between offline learning efficiency and learning accuracy in data-scarce states.
Solution Approach 2:
The patent performs preliminary offline learning with available data to obtain an initial policy, then uses this policy to generate additional training data through actual execution. This preliminary action enables the system to start with limited data and progressively improve, rather than requiring complete data upfront.
2Measurement precision
If more data is collected to improve learning accuracy in all states, then the learning accuracy improves, but the time and resources required for data collection increase
Solution Approach 1:
The system performs self-service by using its own learned policy to generate training data through actual execution in the environment. This eliminates the need for external data collection efforts, allowing the system to improve its own learning accuracy without additional time investment in data gathering.
Solution Approach 2:
The patent enables continuous improvement of learning accuracy by maintaining a loop where the policy is continuously executed and the results are continuously fed back for retraining. This continuous useful action allows incremental accuracy improvements without interrupting the operational workflow for data collection.
3Reliability
If a complex model is used to handle states with limited data, then the learning accuracy in those states improves, but the device complexity increases
Solution Approach 1:
The patent employs a dynamic approach where the policy is continuously updated through iterative retraining with feedback data. Rather than using a statically complex model, the system maintains a relatively simple model structure that evolves and adapts over time through continuous learning from actual execution data, achieving high accuracy without excessive complexity.
Data Source
AI summary
A learning device includes: a model acquisition means that, through training using data that links a state of an environment where an agent performs an action, an action that is executable in the state, and a next state in a case where the action is performed in the state, acquires a model takes a state and an action as input and a next state as output; a feedback information acquisition means that, based on the acquired model, acquires feedback information that is information that is used for training the model or for training a new model that takes a state and an action as input and a next state as output; and a policy management means that trains a policy indicating an action of the agent according to a state by using the model acquired through training using the feedback information.