Causal Transformer Trajectory Processing for Stable Reinforcement Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional reinforcement learning methods face challenges in large online systems due to moving target problems, instability in systems with larger state features, reliance on strict Markov decision processes, and lack of quantitative metrics for policy convergence, leading to inefficiencies and instability in decision-making.
Innovation Solution
A reinforcement learning system utilizing long-term user trajectory data for sequential trajectory modeling with causal transformers, which addresses these challenges by incorporating historical sequences of states, actions, and rewards, reducing reliance on bootstrapping and Markov decision processes, and providing more stable and adaptable decision-making.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional reinforcement learning methods are used in large online systems, then decision-making can be implemented, but the system suffers from moving target problems and instability due to changing user dynamics over time
Solution Approach 1:
The patent applies dynamics by transitioning from static bootstrapping methods to dynamic trajectory processing. The system processes sequences of states, actions, and rewards over time, allowing the reinforcement learning model to adapt to changing user dynamics while maintaining stability through systematic temporal processing of experience data.
Solution Approach 2:
The patent implements preliminary action by collecting and storing extensive trajectory data (sequences of states, actions, and rewards) before training the reinforcement learning model. This pre-collection of comprehensive temporal data enables the model to learn from historical patterns, reducing the moving target problem when making future decisions.
2Productivity
If bootstrapping is used in reinforcement learning, then learning can proceed with limited data, but the system relies on strict Markov decision processes and produces unstable results with larger state features
Solution Approach 1:
The patent applies segmentation by breaking down the reinforcement learning experience into discrete trajectory segments consisting of sequences of states, actions, and rewards. This segmentation allows systematic processing of temporal dependencies while maintaining computational efficiency, avoiding the instability of bootstrapping with large state spaces.
Solution Approach 2:
The patent introduces trajectory data as an intermediary between the environment and the reinforcement learning model. Instead of direct bootstrapping from current estimates, the system uses stored trajectory sequences as intermediate training data, providing more stable and reliable learning with larger state features.
3Productivity
If long-term user trajectory data is processed, then more accurate and efficient resource allocation is achieved, but the system complexity increases
Solution Approach 1:
The patent segments long-term trajectory data into manageable sequences of states, actions, and rewards. This segmentation enables efficient processing and training of reinforcement learning models while maintaining the benefits of long-term temporal information, reducing system complexity compared to processing entire long-term trajectories at once.
Data Source
AI summary
Methods, systems, and apparatuses include preprocessing data for input to a trained transformer system. User trajectory data for a user of an online system is received, the user trajectory data includes tuples for times at which the user interacted with the online system. The tuples are sorted based on the times to create a tuple sequence. Tuple subsequences are extracted from the tuple sequence based on a subsequence length. Cumulative rewards are calculated for the tuple subsequence based on rewards for the tuple subsequence. Input data is generated including cumulative rewards, action options, and the tuple subsequence. A best action option is determined by applying the trained transformer system to the input data. The best action option is implemented for the user of the online system.