LLM Synthetic Query Expansion for Zero-Shot Information Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing information retrieval systems face challenges in handling highly variable queries without labeled training data, particularly in zero-shot learning scenarios, leading to inefficiencies in retrieving relevant documents.
Innovation Solution
Utilizing a large language model (LLM) to generate synthetic queries related to documents, employing adaptive few-shot prompting to refine queries, and incorporating relevance filtering to enhance query expansion, which includes generating and selecting synthetic queries that align with document content.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If zero-shot learning is used to handle queries without labeled training data, then the system can retrieve documents for variable queries, but the retrieval accuracy and relevance are insufficient
Solution Approach 1:
The system performs preliminary actions by generating synthetic queries from available documents before the actual information retrieval task. These synthetic queries are created using LLMs to simulate various user queries that could be posed against the document corpus, allowing the system to pre-process and adapt to the data landscape without requiring labeled training data.
Solution Approach 2:
The system creates copies of the original queries through synthetic query generation. Multiple variations and related queries are generated that mimic real user queries, allowing the retrieval system to learn patterns and improve accuracy by training on these synthesized examples rather than requiring actual labeled data.
2Adaptability or versatility
If traditional query expansion methods are used, then the system can handle some query variations, but the complexity increases and performance degrades in zero-shot scenarios
Solution Approach 1:
The system introduces an intermediary component - the synthetic query generator using LLMs - that mediates between the original queries and the retrieval system. This intermediary generates expanded query variations automatically, handling the complexity of query expansion internally while presenting simplified, pre-processed queries to the retrieval system, thus reducing overall system complexity.
Solution Approach 2:
The system performs self-service by automatically generating its own training data and query expansions without external intervention or labeled data. The LLM-based synthetic query generator creates its own query variations and the system uses these self-generated examples to improve its retrieval performance, eliminating the need for complex manual annotation processes.
3Adaptability or versatility
If more synthetic queries are generated to improve query expansion, then the coverage of user intent increases, but the processing time and computational resources increase
Solution Approach 1:
The system applies partial action by generating a selective subset of synthetic queries rather than exhaustively generating all possible query variations. The LLM generates queries that are most likely to be relevant based on the document content, focusing computational resources on high-value query expansions that provide the greatest improvement in user intent coverage.
Solution Approach 2:
The system dynamically adjusts parameters such as the number of synthetic queries generated, the diversity of query variations, and the selection criteria based on computational constraints. By changing these parameters adaptively, the system optimizes the balance between query intent coverage and processing time, generating enough synthetic queries to improve performance without excessive computational overhead.
Data Source
AI summary
A computing system uses a large language model (LLM) to generate one or more synthetic queries for each document of a set of documents. For a user query, the computing system: selects one or more of the synthetic queries related to the user query; generates an adaptive few-shot prompt to instruct the LLM to generate a response to the query, wherein the adaptive few-shot prompt comprises an example query-response pair for each of the selected one more synthetic queries; provides the adaptive few-shot prompt to the LLM as an input; and generates an amended query based on the output of the LLM in response to the adaptive few-shot prompt.