Hard Negative Mining for Domain-Specific Re-Ranking Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models struggle to provide accurate responses to queries about private company data due to a lack of domain-specific training, and the challenge of crafting hard negative examples for training retrieval/ranking models is costly and prone to human error.
Innovation Solution
A system and method for automatically generating hard negative samples using a multi-dimensional ensemble approach with graph matching to enhance the embedding space of ranking models with domain knowledge, allowing for effective customization and differentiation between relevant and irrelevant information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If manual approaches are used to craft hard negative examples, then training quality improves, but cost increases and human error occurs
Solution Approach 1:
The system enables automatic self-service generation of hard negative examples through computational algorithms. The automated system processes training data, identifies potential hard negatives, and refines them without human intervention, eliminating manual labor costs and human errors while maintaining training quality through algorithmic precision
Solution Approach 2:
The patent replaces the manual mechanical process of crafting hard negatives with an automated computational system. Machine learning models and algorithms substitute human experts, using computational power to identify and generate hard negative examples automatically, thereby reducing cost and eliminating human error while preserving training effectiveness
2Device complexity
If generic embeddings are used for domain documents, then system complexity reduces, but retrieval accuracy deteriorates
Solution Approach 1:
The system implements local quality by creating domain-specific embedding models tailored to particular enterprise domains. Instead of using a single generic embedding model across all domains, the system generates specialized embedding models for each domain (e.g., healthcare, finance, legal), allowing each domain to have optimized retrieval accuracy while managing complexity through modular domain-specific model creation
Solution Approach 2:
The patent applies parameter changes by adapting embedding model parameters to specific domain characteristics. The system fine-tunes embedding models with domain-specific training data, modifying model parameters and weights to capture domain-specific semantics and relationships, thereby improving retrieval accuracy for each domain while maintaining a standardized base model architecture
3Productivity
If ranking models are trained without hard negatives, then training cost reduces, but differentiation capability between similar passages deteriorates
Solution Approach 1:
The system performs preliminary action by automatically identifying and preparing hard negative examples before the main training process. The automated system pre-processes training data to generate hard negative samples, which are then incorporated into the training pipeline, ensuring that models are exposed to challenging differentiation cases without requiring expensive manual preparation
Solution Approach 2:
The system enables self-service generation of hard negative training examples through automated algorithms. The computational system independently identifies, generates, and refines hard negative samples without human intervention, maintaining training effectiveness by providing sufficient hard negative examples while reducing the cost associated with manual creation
Data Source
AI summary
Techniques for hard negative mining for ranking models are provided. In one technique, an input document that is associated with a query is received and input to an embedding model, which outputs a document embedding (DE). Based on the document embedding, multiple embeddings are identified. Clusters of embeddings are generated from the multiple embeddings. A cluster that includes the DE is identified. Based on the DE, two sets of embeddings are identified in the cluster. For each embedding in a first set of embeddings: (1) a particular embedding (PE) is selecting from the second set of embeddings based on a similarity score between the embedding and the PE; (2) a first document that is associated with the PE is identified; and (3) a training instance that includes the first document is generated and added to training data. A model is trained based on the training data.