Search Query Suggestions Using Reinforcement Learning Rewards
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing search-query suggestion systems often provide repetitive, unnatural, or irrelevant suggestions, which can be offensive and fail to enhance user engagement and relevance.
Innovation Solution
A deep reinforcement learning model is employed to generate high-quality search-query suggestions by leveraging contextual semi-supervised models and user feedback, optimizing for naturalness, relatedness, and user engagement through a reward-based reinforcement learning approach.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional query suggestion systems are used, then query suggestions can be provided quickly, but the suggestions become repetitive, unnatural, or irrelevant
Solution Approach 1:
The patent changes the fundamental parameters of the query suggestion system by transitioning from traditional statistical or rule-based methods to deep reinforcement learning. This involves changing the learning paradigm, the objective function (reward signal composition), and the model architecture to generate more natural and relevant queries while maintaining efficiency through optimized training and inference processes
Solution Approach 2:
The patent replaces traditional mechanical or statistical query suggestion mechanisms with an intelligent agent based on deep reinforcement learning. The system substitutes rule-based generation with a learned policy that can adaptively generate queries based on user feedback and contextual understanding, achieving higher quality suggestions
2Adaptability or versatility
If query suggestions are generated without filtering, then all possible suggestions can be provided, but offensive or erroneous content appears
Solution Approach 1:
The patent implements feedback mechanisms at multiple levels: (1) User feedback through implicit signals (clicks, dwell time) and explicit ratings is incorporated into the reward signal to guide the reinforcement learning agent toward generating safe and helpful queries. (2) Automated safety filters and moderation systems provide continuous feedback to block or modify potentially harmful suggestions before they reach users
Solution Approach 2:
The patent introduces intermediary components between the query generation process and user exposure, including safety filters, content moderation systems, and reward functions that penalize harmful content. These intermediaries act as mediators that maintain the versatility of the suggestion system while blocking offensive or erroneous content
3Manufacturing precision
If reinforcement learning is applied to improve query suggestions, then suggestion quality improves, but system complexity increases
Solution Approach 1:
The patent segments the complex reinforcement learning system into distinct functional modules: (1) The deep reinforcement learning agent for query generation, (2) Reward signal composition modules that integrate multiple objectives, (3) Safety filter and moderation systems, (4) Training data preparation and validation pipelines. This segmentation makes the complex system more manageable, maintainable, and deployable
Data Source
AI summary
Methods, systems, and computer programs are presented for suggesting related search queries. One method includes an operation for obtaining a supervised model by training a machine-learning (ML) program with training data that includes search queries entered by users of an online service. Further, the method includes operations for initializing a generator model with the supervised model, and for improving the generator model using reinforcement learning. The reinforcement learning is being based on a reward based on naturalness, relatedness, and a user having a positive session on the online service. Further, the result of the improvement of the generator model is a roll-out model, which is utilized to generate query suggestions for a user of the online service based on a search query provided by the user.