LLM Chat History Synchronization for User Interruption Handling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems fail to maintain synchronicity between the client-side user interface and server-side transcript of Large Language Model (LLM) responses, leading to erroneous or irrelevant outputs due to user interruptions, which can inflate token usage and cause errors in subsequent responses.
Innovation Solution
An active listening module is implemented to detect interruptions in LLM responses and synchronize the chat history by determining the last presented portion of the response, allowing the server to modify the chat history and provide accurate context to the LLM for subsequent prompts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the server continues to generate LLM responses without synchronization, then the response generation speed is maintained, but the chat history becomes out of sync with the user interface, leading to erroneous outputs
Solution Approach 1:
The system implements a feedback mechanism where the client application sends interruption signals back to the server application. When the user interrupts or closes the interface, the client detects this state and communicates it to the server, which then stops generating further LLM responses. This feedback loop ensures synchronization between the client-side user interface and server-side transcript, preventing the chat history from becoming out of sync while maintaining efficient response generation during normal operation.
2Reliability
If the server modifies chat history to maintain synchronicity, then the accuracy of subsequent responses is improved, but additional processing time is required
Solution Approach 1:
The system performs preliminary actions by proactively detecting user interruptions at the client side before they result in synchronization errors. The active listening module continuously monitors user input and detects interruption signals early in the interaction process. By identifying and communicating these interruptions to the server in advance, the system prevents erroneous responses from being generated, eliminating the need for time-consuming post-processing corrections and maintaining response accuracy efficiently.
3Loss of energy
If the LLM generates complete responses without interruption detection, then the token usage is optimized, but the user experience deteriorates when interruptions occur
Solution Approach 1:
The system implements dynamic response generation that adapts to user behavior in real-time. The server application continuously monitors for interruption signals from the client and dynamically adjusts the LLM generation process accordingly. When no interruption is detected, the LLM generates complete responses efficiently. When an interruption signal is received, the server dynamically stops generation and clears the transcript, ensuring the user experience remains responsive and relevant to their current needs while maintaining token efficiency for completed interactions.
Data Source
AI summary
A system and method are provided for synchronizing chat histories used in prompting large language models (LLMs). The method includes receiving an indication of an interruption in a messaging conversation at a client application. The method also includes determining a last presented portion of a response. The response is generated by an LLM for the messaging conversation and provided to the client application in response to prompting the LLM with a prompt based on at least a first input provided to the client application. The method also includes modifying a chat history maintained by a server application based on the last presented portion of the response.