LLM Response Caching with Input Normalization for Lower Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current server-based large language models (LLMs) incur high costs due to inefficient resource usage, leading to increased network traffic, extended response times, and suboptimal quality of responses, particularly in tone conversion and context understanding, which affects user experience and service efficiency.
Innovation Solution
Implement a smart resource optimization caching method that preprocesses natural language inputs into a normalized format, utilizes a cache for matching responses, and selectively engages the LLM for new responses, thereby reducing redundant processing and enhancing response diversity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a server-based LLM model on a vast scale is used to provide high quality services, then service quality is improved, but operational costs increase enormously
Solution Approach 1:
The system performs preliminary action by converting natural language inputs to normalized format in advance and storing them in a cache database. When a new input arrives, the system first checks the cache for matching normalized inputs before invoking the LLM, thereby avoiding redundant LLM calls and reducing operational costs while maintaining service quality
Solution Approach 2:
The system creates copies of previously processed normalized inputs and their corresponding LLM responses in a cache database. Instead of processing the same inputs repeatedly through the expensive LLM, the system retrieves cached copies, significantly reducing computational resource usage and operational costs
2Ease of operation
If natural language inputs are processed directly by the LLM without normalization, then processing simplicity is maintained, but resource efficiency deteriorates due to redundant processing
Solution Approach 1:
The system performs preliminary normalization of natural language inputs to a standardized format before LLM processing. This pre-processing step converts varied user inputs into a consistent normalized form, enabling efficient cache matching and reducing redundant LLM invocations, thereby improving resource efficiency without significantly complicating the overall process
Solution Approach 2:
The system changes the parameter format of natural language inputs by converting them to a normalized format. This parameter transformation enables the cache to efficiently match semantically equivalent inputs that may have different surface forms, improving resource efficiency through better input standardization
3Measurement precision
If the LLM is invoked for every new NL input, then response accuracy is maintained, but response time increases due to repeated processing
Solution Approach 1:
The system performs preliminary conversion of natural language inputs to normalized format and stores these in a cache. Before invoking the LLM, the system checks whether a normalized version of the input already exists in the cache, allowing rapid response for repeated inputs without sacrificing accuracy
Solution Approach 2:
The system retrieves cached copies of previously processed normalized inputs and their corresponding accurate LLM responses. When a match is found, the cached response is returned immediately, maintaining accuracy while dramatically reducing response time by avoiding redundant LLM processing
Data Source
AI summary
An electronic device comprises a memory and at least one processor coupled to the memory. The memory stores a database of previous natural language (NL) inputs to a first large language model (LLM) in a normalized format, and previous responses from the first LLM that correspond to the previous NL inputs in the normalized format. The at least one processor is configured to receive, from another electronic device, a new NL input for the first LLM, wherein the new NL input is converted into the normalized format by a second LLM, determine whether the database includes a matching previous NL input that matches the new NL input in the normalized format, and, if the database includes the matching previous NL input, send, to the other electronic device, a previous response from the first LLM that corresponds to the matching previous NL input as a response to the new NL input.