Value-Function Planner for Real-Time Action Decision Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies require excessive calculation time to decide on appropriate actions for control targets, such as robots, due to the need for deep trajectory searches and numerous pattern generations, which is not feasible within the limited time frames available for tasks like gait control.
Innovation Solution
A planner device and method that utilize a pre-trained value function to quickly decide on actions by calculating the value related to the sum of rewards based on future states, allowing for efficient action decision-making with minimal calculation, proportional to the number of action candidates rather than the search depth and pattern number.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the search depth P and number of patterns Q are increased to improve action decision accuracy, then the decision accuracy is improved, but the calculation time becomes excessively long and cannot be completed within the limited control period
Solution Approach 1:
The patent pre-calculates and stores value functions for multiple future time points before actual control execution. These value functions represent pre-evaluated trajectory qualities at different depths, allowing the system to skip real-time deep trajectory searches by simply querying pre-computed values, thus resolving the contradiction between search depth and calculation time
Solution Approach 2:
The patent prepares multiple candidate trajectory patterns and their corresponding value functions in advance, creating a buffer of pre-processed information. This cushioning allows the control system to make decisions by comparing pre-evaluated options rather than performing exhaustive searches during the limited control period, effectively cushioning against the time constraint
2Reliability
If the trajectory search depth and pattern number are increased to generate more appropriate actions, then the quality of decided actions is improved, but the calculation amount proportional to P×Q becomes too large for real-time control
Solution Approach 1:
The system performs trajectory evaluations and value function computations in advance, storing results for multiple future time points. During actual control, it only needs to retrieve and compare pre-computed value functions rather than performing full trajectory searches, thus maintaining high action appropriateness while dramatically improving calculation efficiency
Solution Approach 2:
The patent creates copies of trajectory patterns and their associated value functions for multiple future time points. Instead of重新 computing trajectories at each control step, the system uses these pre-generated copies, reducing the calculation amount from proportional to P×Q to a much smaller value while maintaining the reliability of action decisions
Data Source
AI summary
A state acquisition means acquires a state of a control target at a first time. An action decision means decides on an action at a second time that is a control timing subsequent to the first time such that a value calculated when the state has been input to a pre-trained value function is largest. The value function is trained such that a value related to a sum of rewards based on states of the control target at control timings between the second time and a third time subsequent to the second time is calculated when a process of deciding on an action between the second time and the third time from the state of the control target at the first time and the action at the second time has been iterated.