Prompt-Based Query Generation for Data-Scarce Document Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural retrieval systems struggle with data scarcity, requiring significant amounts of annotated query-document pairs for effective training, especially in diverse retrieval tasks, and existing approaches often incur high inference costs or fail to generalize well to new tasks.
Innovation Solution
PROMPTAGATOR uses large language models to generate task-specific synthetic data through prompt-based query generation, followed by filtering and training a document retrieval model with few examples, reducing the need for heavily engineered architectures and improving retrieval performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If neural retrieval systems are trained on diverse retrieval tasks with few labeled examples, then the system can adapt to new tasks with limited data, but the training effectiveness and retrieval performance deteriorate due to data scarcity
Solution Approach 1:
The patent uses large language models to generate synthetic query-document pairs that copy and replicate the patterns of real retrieval tasks. These synthetic data copies enable the retrieval system to learn from diverse tasks without requiring actual labeled examples for each task, thus improving adaptability while maintaining reliable performance through high-quality synthesized training data.
Solution Approach 2:
The patent introduces large language models as an intermediary between the limited labeled examples and the retrieval system training process. The LLMs act as mediators that transform a small number of labeled examples into large quantities of synthetic training data, enabling effective training on diverse tasks with few actual labeled examples while maintaining high retrieval performance.
2Productivity
If dual encoder models are used for retrieval tasks, then the system achieves efficient retrieval, but the model requires tens of thousands of labeled examples for effective training
Solution Approach 1:
The patent generates synthetic query-document pairs that replicate the structure and patterns of real retrieval data, allowing dual encoder models to be trained on large quantities of synthetic data instead of requiring tens of thousands of actual labeled examples. This maintains retrieval efficiency while dramatically reducing the quantity of human-labeled data needed.
Solution Approach 2:
The patent performs preliminary data synthesis using large language models before training the dual encoder retrieval system. By pre-generating extensive synthetic training data from a small seed of labeled examples, the system prepares sufficient training material in advance, enabling effective dual encoder training without requiring large initial labeled datasets.
3Reliability
If heavily engineered retrieval architectures are used to improve performance on data-scarce tasks, then retrieval accuracy improves, but the system complexity and engineering requirements increase
Solution Approach 1:
Instead of using complex engineered architectures, the patent copies and leverages the powerful representation capabilities of existing large language models to generate synthetic training data. This approach achieves high retrieval accuracy through data synthesis rather than architectural complexity, using the LLM's pre-learned linguistic patterns and relationships to create effective training examples for standard retrieval models.
Data Source
AI summary
An example method for prompt-based query generation is provided. The method includes receiving, by a computing device, at least two prompts associated with a retrieval task to be performed on a corpus of documents associated with the task. The method includes applying, based on the at least two prompts and the corpus of documents, a large language model to generate a synthetic training dataset comprising a plurality of query-document pairs, wherein each query-document pair comprises a synthetically generated query and a document from the corpus of documents. The method includes training, on the plurality of query−document pairs from the synthetic training dataset, a document retrieval model to take an input query associated with the retrieval task and predict an output document retrieved from the corpus of documents. The method includes providing, by the computing device, the trained document retrieval model.