LLM Output Streaming With Sensitive Data Masking and Demasking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems using generative machine learned models (LLMs) experience high perceived latency when outputting responses, leading to a negative user experience due to the delay between user input and complete response delivery.
Innovation Solution
Implement a data streaming component that masks sensitive data, inputs it into an LLM in subsets, and demasks the output in a streaming manner to reduce latency by delivering partial responses as they become available.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system waits for the complete response from the LLM before outputting it to the user, then data security is maintained through proper processing, but the perceived latency increases and user experience deteriorates
Solution Approach 1:
The patent segments the complete LLM response into individual tokens and streams them progressively to the user. Instead of waiting for the entire response to be generated, the system outputs tokens as they become available, reducing perceived latency while maintaining security through controlled demasking operations on the segmented token stream
Solution Approach 2:
The patent performs preliminary masking of sensitive data in the input prompt before submitting to the LLM. This preliminary action ensures that sensitive information is protected from the start of the processing pipeline, allowing the system to safely stream the response without compromising data security
2Reliability
If the system masks sensitive data in the input prompt, then data security is improved, but the complexity of data processing increases
Solution Approach 1:
The patent introduces a masking component as an intermediary between the user input and the LLM. This intermediary automatically masks sensitive data using regex patterns and maintains a mapping between masked and original values, simplifying the overall system architecture while ensuring data security through a dedicated security layer
3Measurement precision
If the system outputs the complete response at once, then data accuracy is ensured, but the user experience suffers due to high latency
Solution Approach 1:
The patent implements continuous streaming of response tokens to the user as they are generated by the LLM. This continuous action maintains data accuracy by preserving the complete response sequence while dramatically improving user experience through immediate feedback and reduced perceived waiting time
Data Source
AI summary
Techniques for streaming generative machine learned model (or LLM) output to a virtual space are described herein. A system may receive a request to perform an action. The system may leverage LLMs to assist in performing aspects of the requested action. The system can generate input data to input into the LLM. When generating the input data, the system can identify sensitive data associated with the request. The system can modify the input data to mask and/or anonymize the sensitive data. The system can input the input data into an LLM trained to output a subset (e.g., less than all) of the response at a time. The system can add the output subset to a buffer and upon identifying the masked data in the buffer, the system can demask the sensitive data and output the sensitive data to the user profile in a streaming manner.