Key Sentence Classification for Smaller LLM Prompts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Increasing the size of a prompt for a generative language model leads to degraded performance, increased resource consumption, and higher costs, while obtaining robust training examples is resource-intensive and time-consuming.
Innovation Solution
A system that uses a key sentence classifier model to identify key sentences in an input document, generating a compressed document for use as a prompt, thereby reducing the size of the input and improving performance and resource efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the size of the prompt is increased to provide more contextual information, then the language model can process more detailed information, but the performance degrades and resource consumption increases
Solution Approach 1:
The patent extracts only the most critical information (key sentences) from the input document and uses these extracted sentences as the prompt for the language model. This extraction is performed by a key sentence classifier that identifies sentences most likely to be needed for the task, thereby removing unnecessary information while preserving what is essential for maintaining performance.
Solution Approach 2:
The patent segments the input document into individual sentences and evaluates each sentence's importance separately. The key sentence classifier processes sentences one at a time, assigning relevance scores to determine which sentences should be included in the compressed prompt. This segmentation allows selective inclusion of only the most relevant sentences.
2Quantity of substance
If the size of the prompt is increased to include more contextual information, then more detailed processing is possible, but memory and processor resources consumption increases
Solution Approach 1:
The system extracts and includes only the essential key sentences in the prompt, eliminating redundant information. This extraction process is driven by the key sentence classifier that identifies sentences with highest relevance to the task, thereby reducing the amount of data that needs to be processed and stored in memory.
Solution Approach 2:
The patent changes the parameter of prompt size by compressing it from the full document length to a reduced set of key sentences. The key sentence classifier dynamically determines the optimal prompt size based on the document content and task requirements, achieving parameter optimization that balances information completeness with resource efficiency.
3Quantity of substance
If the size of the prompt is increased to provide more context, then more comprehensive answers can be generated, but the latency increases
Solution Approach 1:
The patent extracts only the most critical sentences needed for the task and uses these as the compressed prompt. This extraction is performed efficiently by the key sentence classifier, which processes the document and identifies essential sentences quickly, thereby reducing the prompt size without sacrificing the quality of the language model's response.
Solution Approach 2:
The system performs preliminary processing by pre-identifying and classifying key sentences before the language model generates its response. The key sentence classifier analyzes the document structure and content in advance to determine which sentences are most relevant, so that when the language model receives the compressed prompt, it can process it more quickly with reduced latency.
4Reliability
If manual labeling of training examples is performed to obtain robust training data, then the training examples become more reliable, but the process becomes expensive and time-consuming
Solution Approach 1:
The system uses the key sentence classifier itself to generate training labels automatically. Instead of requiring manual annotation, the classifier processes unlabeled documents and assigns key sentence labels based on its own learned patterns and the relevance of sentences to the task. This self-labeling capability dramatically reduces the time and resources needed for training data preparation.
Solution Approach 2:
The patent performs preliminary labeling of training examples using the key sentence classifier before the actual model training begins. The classifier processes a large corpus of unlabeled documents and pre-identifies key sentences, creating a labeled training set in advance. This preliminary action eliminates the need for time-consuming manual annotation during the training process.
Data Source
AI summary
A technique for interacting with a generative language model includes identifying one or more key sentences in an input document using a key sentence classifier model and/or an entity extraction model. Each key sentence summarizes a part of information conveyed by the input document. The technique further includes generating a compressed document that selectively includes the one or more key sentences. The technique then generates a prompt that includes the compressed document instead of the input document and submits the prompt to the language model. The technique reduces consumption of resources and increases performance by reducing the size of the prompt. A training system produces the key sentence classifier model by first training a pair-comparing model based on a relatively small amount of human-labeled data, and then leveraging the pair-comparing model to produce a synthetic data set on which the key sentence classifier model is trained.