RAG Prompt Compression Using Keyword-Weighted Templates
Solution Overview
Problem
Generative AI systems require large amounts of supplemental data, leading to excessive resource utilization and costly API calls due to lengthy prompts, with existing compression techniques often altering meaning or being uncontrollable.
Innovation Solution
A system that assigns importance scores to keywords in a text fragment, using an abstract prompt template to compress the data based on these scores, ensuring controlled reduction while maintaining accuracy and comprehensibility.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large amounts of supplemental data are provided to the generative AI system, then answer accuracy is improved, but resource utilization and API call costs increase
Solution Approach 1:
The patent extracts and identifies only the most important keywords from the supplemental data using importance scoring, rather than processing all data. This selective extraction maintains answer accuracy while significantly reducing the amount of data transmitted to and processed by the generative AI system, thereby reducing resource utilization and API costs.
Solution Approach 2:
The patent applies different processing treatments to different parts of the supplemental data based on their importance scores. High-importance keywords are retained and emphasized, while low-importance content is compressed or removed. This localized quality differentiation ensures that critical information is preserved for accuracy while non-critical data is reduced to save resources.
2Reliability
If comprehensive supplemental content is transmitted to the generative AI system, then answer quality is improved, but network and storage resources are consumed
Solution Approach 1:
The system extracts essential keywords from supplemental content based on importance scores and transmits only these extracted elements to the generative AI system. This extraction process maintains answer quality by preserving critical information while dramatically reducing the volume of data transmitted over the network and stored in memory.
Solution Approach 2:
Instead of transmitting all supplemental content and letting the AI system filter it, the patent inverts the approach by pre-filtering and compressing the data before transmission. The compression is performed in reverse order of traditional approaches - starting with identifying what to keep (important keywords) rather than what to remove, thereby optimizing data transmission efficiency while maintaining answer quality.
3Loss of energy
If existing compression techniques are applied to reduce prompt size, then resource costs are reduced, but meaning alteration and loss of control occur
Solution Approach 1:
The patent performs preliminary analysis and importance scoring of keywords before compression occurs. By pre-identifying which keywords are most important based on their scores, the system can then compress the prompt while deliberately preserving these high-importance elements. This preliminary action ensures that meaning accuracy is maintained while still achieving cost reduction through compression of less critical content.
Solution Approach 2:
The patent changes the parameter of keyword importance scoring to guide the compression process. By using importance scores as a controlling parameter, the system can adjust the compression level selectively - applying more aggressive compression to low-scoring keywords while preserving high-scoring ones. This parameter-based approach provides controlled compression that reduces API costs without altering the meaning of critical information.
Data Source
AI summary
Systems and methods are disclosed herein for compressing a prompt. In an example system, an importance score listing is obtained that includes a score indicative of an importance of a plurality of dataset keywords. From the importance score listing, a keyword importance score is identified for a plurality of keywords in a current text fragment, such as a text fragment to be compressed. A set of placeholders in an abstract prompt template is populated based on the current text fragment. The current text fragment is compressed based on the importance of the plurality of keywords in the current text fragment to generate a compressed text fragment. In an example, the compressed text fragment is included in the prompt for transmission to a computing entity, such as a large language model of a generative question-answering system.