Autoregressive Text Compression Using BERT Deletion Paths
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text compression techniques rely on supervised models that require parallel data inputs and manually devised compression rules, making them inefficient for generating compressed texts of varying styles and lengths, as they often need to be retrained from scratch.
Innovation Solution
An unsupervised text compression model using a pretrained bidirectional language model (BERT) to identify an optimal deletion path through a progressive lookahead greedy tree search, which gradually deletes words from the input sequence to generate coherent and compressed sentences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If supervised models with manual compression rules are used, then text compression quality is improved, but preparation work and retraining requirements increase
Solution Approach 1:
The system performs self-service by automatically learning compression patterns from unlabeled text data without requiring manual annotation or supervision. The unsupervised learning framework enables the model to autonomously identify and apply compression rules, eliminating the need for manual rule creation and reducing preparation workload while maintaining compression quality.
Solution Approach 2:
The system changes the fundamental parameter of learning supervision from supervised to unsupervised mode. This parameter change allows the model to learn compression patterns directly from raw text data without manual labels, significantly reducing the complexity of data preparation and model retraining while preserving compression effectiveness.
2Adaptability or versatility
If supervised models are retrained for different output styles, then compression adaptability is improved, but training time and computational resources increase
Solution Approach 1:
The unsupervised compression model achieves universality by learning general compression patterns that can be applied across multiple output styles and lengths without retraining. The model maintains a single set of learned compression rules that adaptively generate compressed texts in various formats, eliminating the need for separate training processes for different styles.
Solution Approach 2:
The system implements dynamics by enabling the compression model to adaptively adjust its behavior for different output styles through inference-time control rather than retraining. The learned compression patterns can be dynamically applied to generate outputs of varying lengths and styles, allowing the model to respond flexibly to different requirements without computational retraining.
3Manufacturing precision
If manual compression rules are created, then compression precision is improved, but system complexity and maintenance effort increase
Solution Approach 1:
The system replaces the mechanical system of manual rule creation and management with an automated learning system. Instead of manually crafting and maintaining compression rules, the unsupervised model automatically learns compression patterns from data, substituting human cognitive work with machine learning processes that continuously improve without manual intervention.
Solution Approach 2:
The compression system performs self-service by automatically generating and refining its own compression patterns through unsupervised learning. The model continuously improves its compression accuracy by learning from text data without requiring manual rule updates, eliminating the complexity of rule management and maintenance while maintaining high compression precision.
Data Source
AI summary
Embodiments described herein provide a provide a fully unsupervised model for text compression. Specifically, the unsupervised model is configured to identify an optimal deletion path for each input sequence of texts (e.g., a sentence) and words from the input sequence are gradually deleted along the deletion path. To identify the optimal deletion path, the unsupervised model may adopt a pretrained bidirectional language model (BERT) to score each candidate deletion based on the average perplexity of the resulting sentence and performs a simple greedy look-ahead tree search to select the best deletion for each step.