Retriever-Reader Latency Reduction via ML Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Open domain query response systems face high latency due to the complexity of filtering large datasets in retriever-reader architectures, which hinders efficient response generation and scalability.
Innovation Solution
Implementing a data filtering circuitry that uses a machine learning model to score and filter out irrelevant data, reducing the data array size by removing low-scoring indices and updating the model based on previous queries to improve response generation speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a retriever-reader architecture is used to retrieve evidence from non-parametric external memory, then response generation capability is improved, but latency increases due to the complexity of filtering large datasets
Solution Approach 1:
The patent segments the filtering process into multiple stages: initial retrieval of candidate evidence, scoring/filtering of retrieved data to identify relevant portions, and then generation of response using only the filtered relevant evidence. This segmentation allows the system to handle large datasets by processing them in manageable chunks rather than analyzing everything at once, thus reducing latency while maintaining response quality.
Solution Approach 2:
The patent implements partial action by retrieving more evidence than strictly necessary initially (excessive retrieval), then filtering down to only the relevant portions needed for response generation. The reader model analyzes only a subset of retrieved evidence rather than all of it, performing partial analysis on the most relevant portions. This approach reduces processing time and latency while still generating accurate responses.
2Measurement precision
If large datasets are filtered to ensure accurate response generation, then response accuracy is improved, but processing complexity and latency increase
Solution Approach 1:
The patent applies preliminary action by having the retriever model pre-process and retrieve candidate evidence before the reader model performs detailed analysis. The scoring mechanism preliminarily evaluates retrieved data to identify relevant portions, preparing the data in advance for the reader's accurate analysis. This preliminary filtering reduces the burden on the reader model and maintains accuracy while reducing complexity.
Solution Approach 2:
The patent introduces an intermediary scoring mechanism that sits between the retriever and reader models. This intermediary component scores and filters retrieved evidence, selecting only the most relevant portions to pass to the reader model. This intermediary layer simplifies the reader's task by pre-filtering data, reducing processing complexity while maintaining response accuracy through selective analysis.
3Reliability
If all retrieved data is analyzed by the reader model, then response accuracy is maintained, but processing time and latency increase
Solution Approach 1:
The patent extracts and removes irrelevant portions of retrieved data before passing it to the reader model for analysis. The scoring mechanism identifies and extracts only the relevant evidence portions, discarding unnecessary data. This extraction process reduces the amount of data the reader must analyze, increasing response generation speed while maintaining accuracy by focusing computational resources on relevant information only.
Solution Approach 2:
The patent implements partial action by having the reader model analyze only a subset of the most relevant retrieved evidence rather than all retrieved data. The scoring mechanism identifies the most pertinent portions, and the reader performs analysis on this reduced set. This partial analysis approach maintains response accuracy for critical information while significantly reducing processing time and increasing productivity.
Data Source
AI summary
Disclosed is a technical solution to reduce latency in reader-retriever architectures of an open domain query response system. An example apparatus is to reduce latency in a retriever-reader architecture comprising at least one memory, machine readable instructions, and processor circuitry to at least one of instantiate or execute the machine readable instructions to identify a query, the query having data associated with the query, create a data array based on the data associated with the query, and deploy a machine learning model. The machine learning model is to assign a score to an index of the data array based on a likelihood that a response to the query exists in the index, determine an action to perform on the index based on a comparison of the score and a ranking threshold, and produce the response from the data array based on the action.