LLM Session Context Management via External Repository
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) face challenges due to their short-lived sessions, limited context window, and the time-consuming process of enhancing with custom data, which leads to disjointed responses and inefficient knowledge updates.
Innovation Solution
A system and method that utilize a large language model to efficiently translate data formats by receiving input and output format specifications, searching a repository for associated data, merging specifications, providing prompts to the LLM, and verifying responses, while maintaining context through a repository and embeddings for semantic search.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If LLM sessions are extended to maintain context, then response coherence improves, but session duration and context window limits create bottlenecks
Solution Approach 1:
The patent extracts critical contextual information from LLM interactions and stores it externally in a database, separating the session state from the LLM's internal context window. This allows the LLM to maintain coherence across multiple sessions without being constrained by its limited context window, as important information is retrieved from external storage rather than relying solely on in-memory context.
Solution Approach 2:
The patent introduces an intermediary component (session management system with database storage) that bridges the gap between the LLM's short-term memory and the need for long-term context retention. This intermediary stores and retrieves session information, allowing the LLM to access relevant context across sessions without directly maintaining extended session states.
2Measurement precision
If custom data is integrated into LLM knowledge base, then model accuracy improves, but the enhancement process is time-consuming and computationally intensive
Solution Approach 1:
The patent performs preliminary actions by pre-processing and storing custom data in structured formats in a database before the LLM needs to access it. This preparation work includes organizing data, creating embeddings, and structuring information in advance, so that when the LLM needs to access custom knowledge, the data is already ready for efficient retrieval rather than requiring intensive processing during the enhancement phase.
Solution Approach 2:
The patent creates copies of custom data in optimized formats (such as embeddings and structured representations) that can be quickly retrieved and fed to the LLM. Instead of repeatedly processing the original large datasets, the system maintains pre-processed copies that enable fast access while preserving the essential information needed for accurate responses.
3Manufacturing precision
If data transformation specifications are manually created, then translation accuracy improves, but the process becomes inefficient and labor-intensive
Solution Approach 1:
The patent enables the system to automatically generate data transformation specifications by leveraging the LLM's capabilities and the stored session context. Instead of requiring manual creation of specifications, the system self-generates them based on the data transformation requirements, session history, and learned patterns from previous interactions. This automated approach maintains translation accuracy while dramatically improving efficiency.
Solution Approach 2:
The patent incorporates feedback mechanisms where the LLM's responses and the system's performance are continuously monitored and used to refine data transformation specifications. The session context stores information about successful transformations and patterns, which are fed back into the specification generation process to improve accuracy over time while reducing manual intervention requirements.
Data Source
AI summary
Aspects of the subject disclosure may include, for example, a device having a processing system including a processor; and a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations of: receiving, from a user interface, a sample input format specification and output format specification for transforming data; searching a repository for a tag and associated data; merging or updating the sample input format specification and the output format specification with the associated data responsive to finding the tag in the repository, thereby creating updated data; providing the updated data as a prompt to a large language model; receiving a response to the prompt from the large language model; verifying that the response is satisfactory; and storing a context comprising the tag, the updated data and the response in the repository. Other embodiments are disclosed.