Graph-Based Negative Training Example Generation for ML Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for generating training sets for machine learning algorithms in information retrieval systems face challenges in effectively creating negative examples, which are crucial for training but often result in lower effectiveness, typically around 80% efficacy.
Innovation Solution
A computer-implemented method and system that generates training sets by retrieving search engine result pages, creating graphs mapping queries and documents, and identifying negative training examples by pairing search queries with documents located a predetermined number of nodes away, using user interaction parameters to select the most relevant documents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If random or simple methods are used to generate negative training examples, then the generation process is simple and fast, but the effectiveness and accuracy of training drops to around 80%
Solution Approach 1:
The patent segments the training example generation process into distinct components: positive example generation from query logs, negative example generation through graph-based methods, and quality filtering. This segmentation allows each component to be optimized independently, improving overall reliability while managing complexity through modular design.
Solution Approach 2:
The patent introduces an intermediary graph structure that maps queries to documents to intermediate nodes representing user interactions. This intermediary layer enables the system to generate high-quality negative examples by tracing paths through user behavior data, bridging the gap between simple query-document pairs and complex relevance judgments.
2Measurement precision
If graph-based methods with user interaction data are used to generate negative training examples, then the accuracy and effectiveness of search result rankings improves, but the processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary actions by pre-building the graph structure from query logs and user interaction data before actual training example generation. This pre-processing step organizes the data in a way that enables efficient querying and path traversal during negative example generation, reducing processing time during the actual training phase.
Solution Approach 2:
The patent applies partial action by generating negative examples only for queries where user interaction data is available and relevant, rather than attempting to generate negative examples for all queries. This selective approach maintains high accuracy where possible while reducing overall processing time and computational resources.
3Reliability
If more comprehensive user interaction data is collected and processed, then the quality of negative training examples improves, but the data processing complexity and storage requirements increase
Solution Approach 1:
The patent extracts only the essential user interaction data needed for negative example generation, such as click-through rates, dwell time, and navigation patterns, rather than storing and processing all possible user interaction metrics. This extraction approach maintains high-quality training examples while reducing data volume and processing complexity.
Solution Approach 2:
The patent applies local quality by focusing user interaction data analysis on specific regions of the graph that are most relevant to negative example generation, such as paths leading to non-clicked results or results with low engagement metrics. This localized focus improves negative example quality while minimizing the overall data processing burden.
Data Source
AI summary
Methods and systems for generating a training set for training a MLA is disclosed. The method comprises: retrieving a plurality of SERPs, generating a graph mapping the plurality of documents and the plurality of queries, such that: a given node of the graph includes a given search query of the plurality of search queries; the given node is connected to a first subset of nodes, each one of the first subset of nodes having a document of the plurality of documents; a specific node of the first subset of nodes being connected to a second subset of nodes; receiving an indication of a search query to generate a negative training example for; identifying a node in the graph associated with the search query; generating the training set including the search query paired with a given document located in another node located by a pre-determined number of nodes away.