Intermediate Model Output Reuse for Low-Latency Query Responses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative models, due to their large size and complexity, require significant computational resources and result in high latency, making them impractical for on-device use and prolonging user-to-computer interactions.
Innovation Solution
A query database is created to store intermediate model outputs from previous queries, allowing for the selection of precomputed outputs similar to new user queries, reducing the need to process the entire generative model and minimizing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large generative model is used to process user queries, then the model can generate accurate and comprehensive responses, but the computational resources required and latency increase significantly
Solution Approach 1:
The patent segments the generative model into multiple computing layers (e.g., embedding layers, attention layers, feed-forward layers) and stores intermediate outputs from each layer for historical queries. When a new query arrives, the system processes only the relevant layers rather than the entire model, reducing latency while maintaining response accuracy through selective computation.
Solution Approach 2:
The patent precomputes and stores intermediate model outputs for historical queries in a query database. These precomputed intermediate outputs are made available for reuse when similar queries arrive, eliminating the need to reprocess previous computations and significantly reducing latency for repeated or similar queries.
2Ease of operation
If a large generative model is deployed on-device, then response generation can be performed locally, but memory constraints prevent loading the model
Solution Approach 1:
The patent extracts only the necessary intermediate outputs from specific computing layers and stores them in a query database. This allows the system to operate on-device with minimal memory footprint, as only the intermediate outputs rather than the complete model are stored and processed locally.
Solution Approach 2:
Instead of loading the entire generative model onto the device, the patent implements partial model deployment by only storing and processing intermediate outputs from relevant layers. This partial action approach enables on-device operation within memory constraints while maintaining sufficient functionality for query processing.
3Adaptability or versatility
If the complete generative model is processed for every query, then full model capability is utilized, but computational resources are consumed excessively
Solution Approach 1:
The patent applies local quality by processing only the specific computing layers that are relevant to the current query rather than the entire model. Intermediate outputs from different layers are selectively retrieved and processed based on the query characteristics, optimizing computational resource consumption while maintaining adaptability.
Solution Approach 2:
The patent creates a query database that stores copies of intermediate model outputs from historical queries. Instead of reprocessing the entire model for each query, the system retrieves and reuses these precomputed intermediate outputs, significantly reducing computational resource consumption while maintaining model capability through the copied representations.
Data Source
AI summary
Implementations relate to storing historical queries processed using a generative model in association with intermediate model outputs generated using the generative model for each of the historical queries. Implementations further relate to receiving a user query processable using the generative model. In response to receiving the user query, the user query can be compared to the historical queries to identify a particular historical query (e.g., having a similarity score satisfying a similarity threshold) that matches the user query. Particular intermediate model output associated with the particular historical query can be selected from all intermediate model outputs stored in association with the particular historical query, and a response to the user query can be generated based at least on the selected particular intermediate model output associated with the particular historical query.