Large-Scale Text Compression via Absolute Language Model Overfitting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods for large-scale text data require prior knowledge or manual organization, leading to potential loss of accuracy and inefficiency in reducing storage space.
Innovation Solution
Utilizing absolute overfitting on pre-trained language models like BERT or GPT-3 to generate unique tokens for each sentence and train a generative model to produce exact sentence matches, storing the token list and model for compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional data compression methods are used on large-scale text data, then storage space is reduced, but accuracy is lost and manual organization is required
Solution Approach 1:
The pre-trained language model automatically learns and organizes patterns in text data without requiring manual organization. The model self-adjusts its parameters through absolute overfitting to perfectly reconstruct the input data, eliminating the need for human intervention while maintaining complete data accuracy during compression and decompression
Solution Approach 2:
The invention changes the parameters of the pre-trained language model through absolute overfitting training, where the model is trained on the specific text data until it can perfectly reconstruct it. This parameter adjustment allows the compressed representation (token list) to lose minimal information while maintaining the ability to accurately reconstruct the original text
2Quantity of substance
If traditional data compression methods are used on large-scale text data, then storage space is reduced, but the process becomes inefficient due to manual organization requirements
Solution Approach 1:
The invention replaces manual organization processes with an automated neural network system. The pre-trained language model automatically processes text data through its attention mechanisms and transformer architecture, substituting human manual organization with efficient computational processes that scale to large datasets without proportional increases in time or effort
Solution Approach 2:
The system performs self-organization through the language model's automatic learning processes. The model independently identifies patterns, structures, and relationships in the text data without external guidance, making the compression process fully automated and highly efficient even for very large datasets
3Manufacturing precision
If unique tokens are generated for each sentence using absolute overfitting, then data accuracy is maintained, but device complexity increases
Solution Approach 1:
A pre-trained language model is used as the starting point, having already learned general language patterns and structures from extensive pre-training on large corpora. This preliminary action means the model already possesses substantial linguistic knowledge before being adapted to the specific compression task, reducing the complexity burden during the absolute overfitting phase while maintaining accuracy
Solution Approach 2:
The text data is segmented into sentences, and unique tokens are generated for each sentence rather than for the entire text corpus. This segmentation reduces the complexity of the absolute overfitting training process by breaking it into smaller, more manageable units, while still maintaining the ability to accurately reconstruct the complete original text
Data Source
AI summary
Embodiments of the present invention provide an approach for compressing data, and more particularly, to large-scale text data encoding and compression using absolute overfitting on pre-trained language models. Large-scale data is parsed into sentences. A unique token is generated for each sentence to form a token list. A generative (or compression) model is trained from the tokens in the token list to produce the corresponding sentence of each token through absolute overfitting of a pre-trained language model. The compressed text data is stored as the token list and generative model, resulting in a storage space savings.