Text Compression with NLG Continuations and Dictionary Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text compression techniques are inefficient in reducing the amount of computer data required to represent digital text, as they fail to effectively leverage redundancy in text data, leading to higher storage and transmission requirements.
Innovation Solution
A method that uses a natural language generation (NLG) model to predict continuations of text characters and a compression dictionary to compress text data, where NLG-type compressed representations are used when predictions match, and dictionary-type compressed representations are used when predictions do not match, iteratively updating the dictionary with new entries based on compressed characters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional text compression techniques are used, then text data can be compressed, but the compression efficiency is low and redundancy is not effectively leveraged
Solution Approach 1:
The system pre-processes text data by identifying and storing redundant patterns in a compression dictionary before actual compression occurs. This preliminary action enables faster and more efficient compression by allowing the system to quickly reference pre-identified patterns during the compression process, rather than analyzing patterns in real-time.
Solution Approach 2:
The invention creates compressed representations by copying and referencing patterns from the compression dictionary. Instead of storing complete text sequences, the system stores references (indices) to previously identified patterns, effectively copying the compression logic rather than the actual text data, thereby reducing storage requirements while maintaining compression efficiency.
2Ease of manufacture
If more text data is stored uncompressed, then text representation is simpler, but storage requirements and transmission bandwidth increase
Solution Approach 1:
The compression system segments text data into identifiable patterns and sequences, storing these segments in a compression dictionary. During compression, the system divides the original text into references to these stored segments, allowing efficient storage while maintaining the ability to reconstruct the original text. This segmentation enables the system to manage complexity through structured organization rather than simple uncompressed storage.
Solution Approach 2:
The invention implements a nested structure where compressed text data contains references to patterns that are themselves references to original text segments. The compression dictionary is nested within the compression system, and individual compressed representations are nested within the overall compressed text stream, creating a hierarchical structure that efficiently packs information.
Data Source
AI summary
A method for text compression comprises recognizing a prefix string of one or more text characters preceding a target string of a plurality of text characters to be compressed. The prefix string is provided to a natural language generation (NLG) model configured to output one or more predicted continuations each having an associated rank. If the one or more predicted continuations include a matching predicted continuation relative to the next one or more text characters of the target string, the next one or more text characters are compressed as an NLG-type compressed representation. If no predicted continuations match the next one or more text characters of the target string, a longest matching entry in a compression dictionary is identified. The next one or more text characters of the target string are compressed as a dictionary-type compressed representation that includes the dictionary index value of the longest matching entry.