Policy Model Training With Time-Window Reward Alignment

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing reinforcement learning technologies face challenges in accurately determining reward values due to the lack of consideration for temporal ordering in the agent execution and expert demonstration trajectories, leading to low accuracy in policy model training.

Innovation Solution

The method involves dividing observation and expert data sequences into time windows to determine cost and transport matrices, excluding interference from data outside the window, thereby improving the accuracy of reward values used in reinforcement learning.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If the entire observation data sequence and expert data sequence are used to determine the cost matrix and transport matrix, then more data is available for calculation, but interference from data outside the relevant time window increases, reducing reward value accuracy

Engineering Contradiction:

Improvereward value accuracyVSAvoidinterference from irrelevant data

Core Design Contradiction:

Measurement precisionVSObject-affected harmful factors

Solution Approach 1:

The patent segments the observation data sequence and expert data sequence into multiple time windows. For each observation data point, only the corresponding time window is used to calculate the cost matrix and transport matrix, rather than using the entire data sequence. This segmentation isolates relevant temporal context and excludes interference from data outside the current time window, thereby improving reward value accuracy.

Inventive Principle:

Principle #1Segmentation

2Measurement precision

If a smaller time window is used, then interference from irrelevant data is reduced, but temporal context information is lost, potentially reducing learning effectiveness

Engineering Contradiction:

Improvereward value accuracyVSAvoidtemporal context information

Core Design Contradiction:

Measurement precisionVSLoss of information

Solution Approach 1:

The patent applies local quality by making the time window size adaptive rather than fixed. The time window size is adjusted based on the specific task requirements and data characteristics. For tasks requiring broader context, larger windows are used; for tasks requiring precise temporal matching, smaller windows are used. This allows the system to optimize between including sufficient temporal context and excluding irrelevant interference data.

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS20260070214A1Method and electronic device for training policy model based on reinforcement learning

Publication Date: 2026.03.12 HORIZON ROBOTICS INC

AI summary

The embodiment of the present discloses a method for training a policy model based on reinforcement learning and apparatus, and an electronic device. The method includes: determining an observation data sequence generated by an agent during a time period of executing a target task, a preset time window, and an expert data sequence corresponding to the target task; determining a cost matrix and a transport matrix based on the preset time window, the observation data sequence, and the expert data sequence; determining a reward value for each observation data in the observation data sequence based on the cost matrix and the transport matrix; and training the policy model based on the observation data and the reward value of the observation data.