Session-Based Recommendation Policy Training via Distributional Reinforcement Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing session-based recommendation systems rely on costly online interactions or potentially biased user-behavior models, struggling with overestimation bias and stochastic user environments, especially in batch learning scenarios where true user behavior is difficult to model.
Innovation Solution
A system and method for training session-based recommendation policies using historical data, partitioned into four chronological sets, employing supervised learning to pre-train item and action embeddings, modeling session embeddings, and updating recommendation models to handle stochasticity and overestimation bias through batch-constrained distributional reinforcement learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If model-free reinforcement learning approaches are used for session-based recommendation, then the system can maximize long-term cumulative reward, but it requires costly online interactions with real users
Solution Approach 1:
The patent applies preliminary action by pre-training the recommendation policy using batch reinforcement learning on historical offline data before deployment. This allows the system to learn from past user interactions without requiring extensive real-time online interactions, thus reducing time loss while maintaining the ability to maximize long-term cumulative reward through pre-learned policies that capture user preferences and behaviors
2Loss of time
If batch reinforcement learning is used to train recommendation policies, then online interactions are reduced, but overestimation bias and stochastic user environments make robust reward estimation challenging
Solution Approach 1:
The patent applies parameter changes by transforming the reinforcement learning problem into a supervised learning problem with modified parameters. It uses quantile regression to estimate the distribution of cumulative rewards, changing the parameter estimation approach from direct RL value functions to quantile-based distributional estimates. This reduces overestimation bias by modeling the full reward distribution rather than relying on potentially biased RL value estimates from offline data
Solution Approach 2:
The patent introduces an intermediary supervised learning model that bridges the gap between offline batch data and reliable reward estimation. This intermediary model is trained on historical data with known outcomes and uses quantile regression to provide robust estimates of reward distributions, mediating between the limitations of batch data and the requirements for accurate reward estimation in reinforcement learning
3Ease of operation
If deep Q-learning approaches are used in off-policy learning setting, then learning can proceed without on-policy interactions, but overestimation bias occurs when learning the value function
Solution Approach 1:
The patent converts the harmful overestimation bias into a beneficial estimation approach by using quantile regression. Instead of treating overestimation as an error to be eliminated, it models the full distribution of rewards including the overestimated values, and uses quantile estimates to provide robust, bias-resistant value function estimates. This transforms the problematic overestimation into useful information about reward variability and uncertainty
Data Source
AI summary
Session-based Recommendation (SR) is the task of recommending the next item based on previously recorded user interactions. However, most existing approaches for SR either rely on costly online interactions with real users (model-free approaches) or rely on potentially biased rule-based or data-driven user-behavior models (model-based approaches) for learning. This disclosure relates to a system and method for selecting session-based recommendation policies using historical recommendations and user feedback. Herein, the learning of recommendation policies given offline or batch data from old recommendation policies based on a Distributional Reinforcement Learning (DRL) based recommender system in the offline or batch-constrained setting without requiring access to a user-behavior model or real-interactions with the users.