Reinforcement Learning Item Push List Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional customer service systems are labor-intensive and inefficient in predicting user questions, as existing methods focus on immediate rewards without considering the impact of previous recommended items on subsequent recommendations, leading to a need for more effective solutions for determining an item push list.
Innovation Solution
A reinforcement learning model is used to determine an item push list for a user, where each item list includes static and dynamic features, with the model outputting a weight vector to calculate scores for candidate items, and updated lists are determined using beam search algorithms to optimize long-term accumulated rewards.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional classification methods are used to predict user questions, then immediate click rewards are maximized, but the impact of previous recommended items on following recommended items is ignored
Solution Approach 1:
The patent applies dynamic programming to transform the static classification problem into a dynamic sequential decision-making process. The policy function is updated iteratively to account for the sequential impact of recommended items, where each recommendation decision considers both immediate click rewards and future recommendation effectiveness. This resolves the contradiction by making the system adaptive to the sequential nature of item recommendations while maintaining high click throughput.
Solution Approach 2:
The patent implements a feedback mechanism where the policy function is continuously updated based on observed user interactions and the actual impact of recommended items on subsequent recommendations. This feedback loop allows the system to learn from past recommendations and adjust future recommendations to maximize long-term accumulated rewards rather than just immediate clicks, thereby addressing the information loss about item sequence impact.
2Reliability
If reinforcement learning models are used to optimize long-term accumulated rewards, then user satisfaction increases, but computational complexity and training time increase
Solution Approach 1:
The patent employs pre-training and fine-tuning strategies where the reinforcement learning model is first pre-trained on large-scale historical data to learn general user behavior patterns, then fine-tuned on more recent data to adapt to changing user preferences. This preliminary action reduces the computational burden during deployment while maintaining high user satisfaction, as the model has already learned fundamental patterns before being applied to specific recommendation scenarios.
Solution Approach 2:
The patent segments the recommendation process into distinct phases: policy function initialization, iterative updating based on user feedback, and final deployment. Each phase handles specific computational tasks, allowing for optimized resource allocation and reducing overall system complexity. The segmentation enables parallel processing of different recommendation streams and facilitates incremental model updates without retraining the entire system, thereby managing computational complexity while maintaining reliability.
3Manufacturing precision
If beam search algorithms are used to determine updated item lists, then recommendation quality improves, but processing time increases
Solution Approach 1:
The patent applies beam search with a limited beam width parameter, exploring only the top K most promising recommendation paths at each step rather than exhaustively searching all possible sequences. This partial action approach achieves sufficient recommendation accuracy for practical applications while dramatically reducing processing time compared to complete beam search. The beam width is tuned to balance between recommendation quality and computational efficiency, providing a practical solution that doesn't require exhaustive search.
Data Source
AI summary
This disclosure is related to determining an item push list for a user based on a reinforcement learning model. In one aspect, a method includes obtaining M first item lists that have been predetermined for a first user. Each first item list includes i−1 items. For each first item list, an ith state feature vector is obtained. The ith state feature vector includes a static feature and a dynamic feature. The ith state feature vector is provided as input to the reinforcement machine learning model. The reinforcement model outputs a weight vector including weights of sorting features. A sorting feature vector of each item in a candidate item set corresponding to the first item list is obtained. The sorting feature vector includes feature values of sorting features. M updated item lists are determined for the first item lists based on a score for each item in M candidate item sets.