Dense Text Retriever Training with Auto-Generated Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training dense retriever models in text retrieval tasks rely heavily on annotated data, which is costly, and many datasets lack associated queries, making unsupervised training necessary.
Innovation Solution
A query generation mechanism that automatically generates queries using training documents, employing methods like query extraction (QEXT) and transferred query generation (TQGEN), combined with a bi-encoder dense retriever model and contrastive learning, to create vector representations and update the model parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If annotated data is used for training dense retriever models, then model accuracy is improved, but training cost increases significantly
Solution Approach 1:
The system generates its own training queries automatically using the training documents themselves, eliminating the need for external human annotators. The query generation mechanism extracts or generates queries from the documents, allowing the system to self-service its training data needs without incurring annotation costs.
Solution Approach 2:
A query generation mechanism acts as an intermediary between the training documents and the dense retriever model. This intermediary automatically creates queries from documents using methods like QEXT (extracting text spans) or TQGEN (using language models), bridging the gap between unannotated documents and the model training process.
2Measurement precision
If human annotators generate queries for documents, then query quality is improved, but productivity decreases due to high cost and time consumption
Solution Approach 1:
The patent replaces the mechanical process of human annotation with automated computational methods. Instead of human annotators manually creating queries, the system uses algorithmic approaches (text span extraction, language model generation) to automatically produce queries, dramatically improving productivity while maintaining acceptable query quality.
Solution Approach 2:
The query generation mechanism performs preliminary actions by creating queries from documents before the actual model training process. This pre-processing step automates what would otherwise require human intervention, allowing the training pipeline to proceed efficiently with automatically generated query-document pairs.
3Reliability
If large amounts of annotated data are collected, then model performance is improved, but loss of time increases due to annotation process
Solution Approach 1:
The system performs self-service by automatically generating queries from its own training documents without external human intervention. This eliminates the time-consuming annotation process entirely, allowing the system to scale to large datasets without proportional increases in annotation time.
Solution Approach 2:
The automated query generation enables continuous processing of training documents without interruption for human annotation. The system can continuously generate queries and train models on large datasets without the breaks and bottlenecks inherent in human annotation workflows, maintaining continuous productive action.
Data Source
AI summary
Embodiments described herein provide systems and methods for training a text retrieval model. A system may generate queries associated with provided documents. The queries may be generated in one or more different manners. Examples of query generation may include extracting relevant spans of text from the documents, prompting a language model for a topic, title, abstractive summary, and/or extractive summary based on the documents. Metadata such as title or other HTML tags may be used as queries. Using the one or more queries, the text retrieval model may be trained using contrastive learning, using the generated query, and positive and negative sample documents. A fine-tuning training phase may be performed using domain-specific data which may also be done with generated query pairs, or may be done in a supervised fashion with provided queries. The text retrieval model may be used to locate documents given an input query.