Automated Shaped Reward Function Generation for Reinforcement Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Defining suitable reward functions for reinforcement machine learning systems is challenging, especially when dealing with complex tasks and sparse state spaces, as it requires balancing competing objectives and providing adequate feedback for training.
Innovation Solution
A method that automatically translates a programmer-defined training specification into a shaped reward function, using pre-defined reinforcement machine learning criterion statements, allowing the system to score and adjust the model configuration based on sub-goals without explicit programmer input.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a shaped reward function is manually defined to guide reinforcement learning training, then training efficiency is improved, but device complexity and development time increase
Solution Approach 1:
The system automatically generates shaped reward functions by having the reinforcement learning system itself evaluate candidate reward functions and select the most effective one, eliminating the need for manual definition by developers
Solution Approach 2:
A computer-implemented method acts as an intermediary between the reinforcement learning system and the reward function definition, automatically translating high-level training specifications into shaped reward functions
2Reliability
If a shaped reward function is defined to provide rich feedback, then training effectiveness is improved, but the difficulty of defining and measuring performance increases
Solution Approach 1:
The system implements automated feedback loops where the reinforcement learning system evaluates candidate reward functions and provides performance feedback, enabling automatic selection and refinement of shaped reward functions
Solution Approach 2:
The system automatically adjusts reward function parameters and shaping factors based on observed training performance, transforming the static manual definition process into a dynamic automated optimization process
Data Source
AI summary
A method of training a reinforcement machine learning computer system. The method comprises providing a machine-learning computer programming language including a pre-defined plurality of reinforcement machine learning criterion statements, and receiving a training specification authored in the machine-learning computer programming language. The training specification defines a plurality of training sub-goals with a corresponding plurality of the reinforcement machine learning criterion statements supported by the machine-learning computer programming language. The method further comprises computer translating the plurality of training sub-goals from the training specification into a shaped reward function configured to score a reinforcement machine learning model configuration with regard to the plurality of training sub-goals. The method further comprises running a training experiment with the reinforcement machine learning model configuration, scoring the reinforcement machine learning model in the training experiment with the shaped reward function, and adjusting the reinforcement machine learning model configuration based on the shaped reward function.