AST-Based Code Compaction for Token-Limited LLM Editing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) struggle with processing entire source code files due to token limits, leading to inefficient resource consumption and inaccurate edits when provided with syntactically invalid or context-lacking code segments.
Innovation Solution
A system dynamically compacts source code using an abstract syntax tree (AST) to identify and include only the most relevant, syntactically valid code segments within a token limit, ensuring accurate and efficient LLM responses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If the entire source code file is provided to the LLM, then the LLM has complete context information, but the prompt exceeds the token budget and consumes excessive computing resources
Solution Approach 1:
The source code file is segmented into an abstract syntax tree (AST) structure, dividing the code into hierarchical nodes representing different linguistic elements. This segmentation allows selective extraction of relevant code portions while maintaining syntactic structure, enabling the prompt to stay within token limits without losing essential context information.
Solution Approach 2:
The system extracts only the necessary code segments from the full source file by traversing the AST and identifying relevant nodes based on the user's request. This extraction process removes unnecessary portions of the code while preserving the essential context needed for accurate LLM processing, thus reducing token count while maintaining information quality.
2Quantity of substance
If code segments are arbitrarily truncated to fit token limits, then the prompt size is reduced, but syntactic validity and accuracy are compromised
Solution Approach 1:
By segmenting the code into AST nodes, the system can identify complete syntactic units (such as full function definitions, class declarations, or statement blocks) rather than arbitrarily cutting code in the middle. This ensures that extracted code segments maintain syntactic validity while fitting within token limits.
Solution Approach 2:
The AST is constructed in advance before prompt generation, allowing the system to pre-identify valid code boundaries and structural relationships. This preliminary structuring enables the subsequent extraction of syntactically complete segments without requiring complex validation during prompt assembly, thus maintaining reliability while controlling prompt size.
3Productivity
If irrelevant code segments are included in the prompt, then the token budget is wasted, but the LLM may generate inaccurate edits due to lack of focus
Solution Approach 1:
The system applies local quality by tailoring the included code segments specifically to the user's request and the target code portion. Rather than including uniform portions of the source file, the AST traversal selectively includes only those segments that are locally relevant to the editing task, improving both resource efficiency and edit accuracy by focusing the LLM's attention on pertinent code.
Solution Approach 2:
The system uses the user's editing request as feedback to guide the AST traversal and segment selection process. The request information informs which parts of the code tree to prioritize, allowing the system to dynamically adjust the included segments to maximize relevance. This feedback-driven approach ensures that token budget is allocated to the most pertinent code portions, enhancing both efficiency and precision.
Data Source
AI summary
Examples enable dynamic eliding of code using structural information for syntactic validity. An identified code segment from an original document and a request for a large language model (LLM) to perform an action associated with the identified code segment is received. A compacted abstract syntax tree (AST) including removable nodes is generated based on the original document. The removable nodes are scored for relevance to the identified code segment. Code segments corresponding to the most relevant removable nodes are added to a compacted document without exceeding a configurable token limit for prompts to the LLM. A modified prompt including the identified code segment and the most relevant code segments is provided to the LLM. The edits received from the LLM in response to the modified prompt are mapped into the original document to create a syntactically valid edited version of the original source code while minimizing resource usage.