Summarization Model Reduces LLM Prompt Length
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models require lengthy input prompts, which increase computational resources and costs while maintaining output quality, posing a challenge in balancing detailed descriptions and input length limitations.
Innovation Solution
A method involving a summary model trained using a task-driven attention mechanism to refine original input prompts, generating a condensed version that maximizes similarity scores between original and reduced outputs, thereby reducing prompt text length without compromising output quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If lengthy input prompts are used to maintain output quality, then the quality and accuracy of LLM outputs are improved, but the computational resources and costs increase
Solution Approach 1:
The patent extracts and removes redundant information from the original prompt while retaining only the essential elements needed for the LLM to generate high-quality outputs. The summarization model identifies and eliminates unnecessary tokens, achieving significant prompt length reduction (e.g., 50-70% reduction) while maintaining output quality through selective information retention.
Solution Approach 2:
The patent changes the parameter of prompt length by using a summarization model to transform long prompts into condensed versions. The training process optimizes the summarization model to preserve critical information while reducing token count, effectively changing the length parameter without sacrificing the reliability of the LLM output.
2Measurement precision
If detailed descriptions are provided in prompts, then the accuracy and completeness of LLM responses are improved, but the input prompt length increases
Solution Approach 1:
The summarization model extracts and removes redundant information from detailed prompts while retaining essential elements. It identifies unnecessary descriptions and eliminates them, achieving prompt length reduction while maintaining response accuracy through selective information preservation.
Solution Approach 2:
The summarization model acts as an intermediary between the original detailed prompt and the LLM. It processes the long prompt into a condensed version that preserves critical information, serving as a bridge that maintains accuracy while reducing length before the prompt reaches the LLM.
3Reliability
If comprehensive prompts are used to ensure task completion, then the reliability of results is improved, but the computational cost and processing time increase
Solution Approach 1:
The summarization model extracts and removes redundant information from comprehensive prompts while retaining essential task-related elements. It identifies and eliminates unnecessary tokens that do not contribute to task completion, reducing processing time while maintaining result reliability.
Solution Approach 2:
The patent changes the parameter of prompt length by transforming comprehensive prompts into condensed versions through the summarization model. This parameter change reduces the computational burden and processing time while the training process ensures that task completion reliability is maintained.
Data Source
AI summary
A method, computer program product, and computer system are provided for saving prompt text length for large language models. A large language model is prompted with a first prompt to receive a first result. A summary is generated based on prompting a summary model with the first prompt. The large language model is prompted with the generated summary to receive a second result. The summary model is trained based on maximizing a similarity score between the first result and the second result. A text output associated with the first prompt is generated based on prompting the large language model with a second prompt generated by the trained summary model.