Key Sentence Classifier for Prompt Compression in Language Models

Solution Overview

Problem

Increasing the size of prompts for generative language models leads to performance degradation, increased resource consumption, and higher costs, while obtaining robust training examples is resource-intensive and time-consuming.

Innovation Solution

A system that uses a key sentence classifier model to identify key sentences in an input document, generating a compressed document for use as a prompt, reducing the size of the input while maintaining performance and efficiency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If the size of the prompt is increased to provide more contextual information, then the performance of the language model is improved, but the consumption of memory and processor resources increases

Engineering Contradiction:

Improveperformance of language modelVSAvoidconsumption of memory and processor resources

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The system extracts only the most relevant sentences from the input document using a key sentence classifier model, removing unnecessary information while preserving the essential context needed for language model performance. This extraction approach maintains reliability while reducing resource consumption by including only critical sentences in the compressed document.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The input document is segmented into individual sentences, which are then evaluated and selected by the key sentence classifier model. This segmentation allows the system to process and select only the most relevant portions of the document, reducing the overall prompt size while maintaining performance.

Inventive Principle:

Principle #1Segmentation

2Reliability

If the size of the prompt is increased to provide more contextual information, then the performance of the language model is improved, but the latency is increased

Engineering Contradiction:

Improveperformance of language modelVSAvoidlatency of language model response

Core Design Contradiction:

ReliabilityVSSpeed

Solution Approach 1:

The system extracts only the most relevant sentences from the input document using a key sentence classifier model, removing unnecessary information while preserving the essential context needed for language model performance. This extraction approach maintains reliability while reducing resource consumption by including only critical sentences in the compressed document.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The input document is segmented into individual sentences, which are then evaluated and selected by the key sentence classifier model. This segmentation allows the system to process and select only the most relevant portions of the document, reducing the overall prompt size and latency.

Inventive Principle:

Principle #1Segmentation

3Reliability

If a robust set of training examples is obtained to improve language model performance, then the performance is improved, but the cost and time required is increased

Engineering Contradiction:

Improveperformance of language modelVSAvoidtime and cost to obtain training examples

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system uses the language model itself to generate synthetic training examples by having it rewrite sentences in the training dataset. This self-service approach eliminates the need for manual creation of training examples, significantly reducing time and cost while providing robust training data for improving model performance.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system transforms the training data by having the language model rewrite sentences with different parameters such as vocabulary, style, and structure. This parameter transformation creates diverse synthetic training examples that improve model robustness without requiring manual annotation of additional datasets.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentEP4654074A1Training and applying a key sentence classifier model

Publication Date: 2025.11.26 MICROSOFT TECHNOLOGY LICENSING LLC

Figure 1
Figure 2
Figure 3

AI summary

A technique for interacting with a generative language model includes identifying one or more key sentences in an input document using a key sentence classifier model and/or an entity extraction model. Each key sentence summarizes a part of information conveyed by the input document. The technique further includes generating a compressed document that selectively includes the one or more key sentences. The technique then generates a prompt that includes the compressed document instead of the input document and submits the prompt to the language model. The technique reduces consumption of resources and increases performance by reducing the size of the prompt. A training system produces the key sentence classifier model by first training a pair-comparing model based on a relatively small amount of human-labeled data, and then leveraging the pair-comparing model to produce a synthetic data set on which the key sentence classifier model is trained.