Bootstrapping Reinforcement Learning for Sequential Content Recommendations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional reinforcement learning-based sequential recommendation systems require significant exploration and costly data collection to determine an optimized recommendation policy, making them impractical for recommending tutorial content in software applications due to high user churn and resource constraints.
Innovation Solution
The system bootstraps a reinforcement learning agent using passive data and a small amount of active data, iteratively updating the recommendation policy and generating additional active data, leveraging passive data to quickly learn an optimal policy, and employs a clustering approach to balance model expressiveness and data sparsity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional reinforcement learning is used to train a sequential recommendation system, then an optimized recommendation policy can be learned, but the exploration process is costly and requires significant data collection
Solution Approach 1:
The patent applies preliminary action by using passive data collected before the recommendation system is deployed to bootstrap the reinforcement learning agent's training. This allows the system to learn an initial recommendation policy without requiring extensive active data collection during deployment, thereby reducing the costly exploration phase while still achieving an optimized policy.
2Reliability
If conventional reinforcement learning exploration is implemented, then an optimized recommendation policy is determined, but user churn increases due to the time and resources required
Solution Approach 1:
The patent uses passive data collected in advance (before deployment) to perform preliminary training of the reinforcement learning agent. This preliminary action reduces the training time required during the actual deployment phase, preventing user churn that would otherwise occur while waiting for the system to learn through extensive exploration.
3Productivity
If passive data is used to bootstrap the reinforcement learning agent, then training efficiency is improved, but the system must handle data sparsity issues
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting the clustering approach and model complexity based on the amount of active data available. As more active data becomes available during deployment, the system can refine its parameters and clustering to better handle data sparsity while maintaining the efficiency gains from using passive data for initial training.
Data Source
AI summary
Systems and methods provide a recommendation system for recommending sequential content. The training of a reinforcement learning (RL) agent is bootstrapped from passive data. The RL agent of the sequential recommendations system is trained using the passive data over a number of epochs involving interactions between the sequential recommendation system and user devices. At each epoch, available active data from previous epochs is obtained, and transition probabilities are generated from the passive data and at least one parameter derived from the currently available active data. Recommended content is selected based on a current state and the generated transition probabilities, and the active data is updated from the current epoch based on the recommended content and a resulting new state. A clustering approach can also be employed when deriving parameters from active data to balance model expressiveness and data sparsity.