LLM Output Filtering With Synchronous and Asynchronous Blocking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems using generative machine learned models (LLMs) face high perceived latency and display of undesirable content, negatively impacting user experience.
Innovation Solution
A data streaming component that filters out undesirable content using synchronous and asynchronous blocking components, reducing latency by streaming tokens in real-time and blocking undesirable content before or after output to the user device.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Object-affected harmful factors
If synchronous blocking is used to filter content before output, then harmful factors are reduced, but latency increases
Solution Approach 1:
The synchronous blocking component performs preliminary content filtering before the LLM output is sent to the user device. This allows undesirable content to be blocked in advance, preventing it from reaching the user while maintaining a responsive interface. The system prepares filtering rules and blocks content proactively rather than reactively.
2Loss of time
If asynchronous blocking is used to filter content after output, then latency is reduced, but harmful factors may reach the user
Solution Approach 1:
The asynchronous blocking component acts as an intermediary that monitors and filters content after it has been sent to the user device. When undesirable content is detected, the system intervenes by blocking or replacing the content, thus reducing initial latency while still protecting the user from harmful factors through post-output filtering.
3Reliability
If both synchronous and asynchronous blocking are used, then filtering effectiveness is improved, but device complexity increases
Solution Approach 1:
The content filtering system is segmented into two distinct components: synchronous blocking and asynchronous blocking. Each component handles specific filtering tasks at different stages of content delivery. This segmentation allows the system to achieve comprehensive filtering effectiveness while maintaining manageable complexity through modular design, where each component can be independently configured and optimized.
Data Source
AI summary
Techniques for filtering out undesirable generative machine learned model (or LLM) output are discussed herein. A system may receive a subset of an LLM output. That is, the system may stream the LLM output to a user device by receiving one or more tokens from the LLM and outputting such token(s) to a user device. However, prior to outputting the token(s) to the user device, the system may determine whether the token(s) include undesirable content that is to be blocked. The system may use synchronous blocking components (e.g., blocks the undesirable token(s) before such token(s) get output to the user device) and/or asynchronous blocking components (e.g., blocks the undesirable token(s) after the token(s) have been output to the user device) to filter out undesirable content. The synchronous and/or asynchronous blocking components may be designed to block one or more undesirable topics such as hateful speech, profanity, bias, toxicity, factualness, etc.