Elidable Text for LLM Prompt Context Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models face challenges when the context window size is limited, as they require a substantial amount of background information to make predictions, which exceeds the size of the context window.
Innovation Solution
The use of elidable text, which prioritizes content within a prompt to fit within a fixed-size context window, by replacing lowest-valued lines of text and source code with a replacement string, such as a bracketed ellipsis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If the context window size is increased to accommodate more background information, then the model can make better predictions, but the computational burden and resource requirements increase
Solution Approach 1:
The patent segments the prompt content into different priority levels (focused lines with high priority values and non-focused lines with lower priority values). This segmentation allows the system to selectively include only the most important information in the context window, thereby reducing the amount of data processed while maintaining prediction quality.
Solution Approach 2:
The patent applies local quality by assigning different priority values to different portions of the source code based on their semantic distance from focused lines. This creates a non-uniform distribution of information importance, allowing the system to concentrate computational resources on processing high-priority regions while reducing or eliminating low-priority content.
2Use of energy by moving object
If the context window size is limited to reduce computational burden, then resource requirements decrease, but the model loses access to necessary background information
Solution Approach 1:
The patent extracts and removes low-priority lines of source code from the prompt based on their priority values. By systematically removing non-focused lines that have lower semantic importance, the system reduces the context window size and computational burden while retaining the essential background information needed for accurate predictions.
3Loss of information
If all lines of source code are included in the prompt to ensure complete context, then information completeness is maintained, but the prompt exceeds the context window size
Solution Approach 1:
The patent applies partial action by including only a subset of source code lines in the prompt - specifically those with high priority values above a certain threshold. This partial inclusion strategy ensures that the most critical context is provided to the model while the prompt size remains within the context window limitations, avoiding the need to include every line of source code.
Data Source
AI summary
An elidable text is constructed that prioritizes the content included in a prompt to a large language model having a fixed-size context window. The elidable text is generated from developer-generated instructions or automatically for source code within a source code editor. A source code editor may include a feature that selects certain lines of code as important or focused which are assigned a high-priority value. A changed line, a line of source code at a current cursor position, lines of source code at the beginning of a file and those that output data are considered focused lines. Non-focused lines are assigned a priority based on a distance from a focused line. The elidable text constrains the data included in a prompt to the context window size by replacing the lowest-valued lines of text and source code with a replacement string.