LLM Text Repetition Removal via Embedding Similarity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large Language Models (LLMs) tend to generate textual content that is repetitious, lacking clarity and grammatical correctness, which can make the output unsuitable for visual consumption.
Innovation Solution
A system and method are provided for identifying and removing repetitiveness in textual content generated by LLMs. This involves generating embeddings for sub-components of the text, computing similarity values, and removing repetitive sub-components. The modified content may then be re-submitted to an LLM for further refinement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If LLM generates textual content based on trained data, then the model can produce natural language answers, but the generated content becomes repetitious and lacks grammatical correctness
Solution Approach 1:
The patent introduces an intermediary processing system between the LLM generation and final output. This system includes repetition detection mechanisms that analyze generated text and identify repetitive patterns, then apply corrections to eliminate repetitions and improve grammatical correctness before presenting the final output to users.
Solution Approach 2:
The system implements feedback loops where generated text is continuously analyzed for repetitions, and correction information is fed back to refine the output. The repetition detection and correction process creates a feedback mechanism that improves grammatical correctness by iteratively identifying and resolving repetitive patterns in the generated content.
2Adaptability or versatility
If LLM generates textual content from terabytes of training data, then diverse content can be produced, but repetitiveness increases in the output
Solution Approach 1:
The patent extracts and removes repetitive elements from the generated text through dedicated detection and removal mechanisms. The system identifies repetitive patterns, phrases, and structures, then extracts only the necessary information while eliminating redundant content, thereby preserving content diversity while reducing repetitiveness.
Solution Approach 2:
The system discards repetitive portions of generated text through detection and removal processes, while recovering and preserving the unique, valuable information. The correction mechanisms discard redundant repetitions but maintain the essential meaning and diversity of the original generated content.
3Manufacturing precision
If repetition removal is applied to LLM-generated text, then clarity improves, but additional processing steps are required
Solution Approach 1:
The patent segments the text processing into distinct modules: generation, repetition detection, analysis, and correction. This segmentation allows each component to specialize in a specific task, improving clarity through targeted processing while managing complexity through modular architecture. The detection and correction processes operate as separate, manageable segments rather than a monolithic complex system.
Data Source
AI summary
Techniques for generating repetition-free text using a large language model (LLM) are provided. In one technique, textual content that was generated by an LLM is accessed, where the textual content comprises a plurality of sub-components including a first sub-component and a second sub-component. A first embedding that represents the first sub-component is generated and a second embedding that represents the second sub-component is generated. Based on a similarity between the first embedding and the second embedding, it is determined whether the second sub-component is repetitious with respect to the first sub-component. In response to determining that the second sub-component is repetitious with respect to the first sub-component, at least a portion of the second sub-component is removed from the textual content.