Recursive Symbol-Pair Compression for Large Document Sets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text compression techniques are not scalable for large data sets and do not effectively track the frequency of symbol pairs, which limits their ability to achieve optimal space savings and processing efficiency.
Innovation Solution
A method that identifies and counts unique symbol pairs in multiple documents, replacing pairs that exceed a threshold with replacement symbols, and recursively processes documents to optimize compression, using a compression dictionary that sorts entries by frequency for improved lookup times and space savings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If previous text compression techniques are used on single files, then compression is achieved for individual documents, but the techniques do not scale to enormous data sets and do not track symbol pair frequencies
Solution Approach 1:
The patent combines multiple individual document compressions into a unified compression process that operates across entire data sets. By merging the compression operations and sharing the compression dictionary across all documents, the system achieves scalability to enormous data sets while maintaining or improving compression efficiency through frequency tracking of symbol pairs across the entire corpus.
Solution Approach 2:
The compression system is designed to handle both individual documents and enormous data sets with the same core algorithm. The universal compression dictionary can track symbol pair frequencies across any scale of data, making the system multi-functional for different data volumes without requiring separate compression approaches.
2Loss of substance
If symbol pairs are replaced with replacement symbols, then space savings are achieved, but processing time increases due to lookup operations
Solution Approach 1:
The system performs preliminary sorting of the compression dictionary by symbol pair frequency before the actual compression process. By pre-organizing the dictionary with high-frequency symbol pairs in easily accessible memory locations, the system reduces lookup time during compression while maintaining the space savings benefits of symbol pair replacement.
Solution Approach 2:
The compression dictionary is organized with different memory allocation strategies for different frequency categories. High-frequency symbol pairs are placed in faster, more accessible memory locations, while lower-frequency pairs use standard memory allocation. This local quality differentiation optimizes the balance between space savings and processing speed based on actual usage patterns.
3Loss of information
If compression dictionary stores all unique symbol pairs, then complete compression coverage is achieved, but memory usage increases
Solution Approach 1:
The system dynamically adjusts the compression threshold parameter based on available memory resources and data characteristics. By changing the frequency threshold for inclusion in the compression dictionary, the system can optimize between complete compression coverage and memory usage, allowing flexible adaptation to different operational constraints without losing essential compression benefits.
Solution Approach 2:
The system selectively discards low-frequency symbol pairs from the compression dictionary when memory constraints are encountered, while maintaining the ability to recover and include them if needed. This approach allows the system to manage memory usage effectively while preserving the option to achieve complete compression coverage when resources permit.
Data Source
AI summary
A computerized method and apparatus compresses symbolic information, such as text. Symbolic information is compressed by recursively identifying pairs of symbols (e.g., pairs of words or characters) and replacing each pair with a respective replacement symbol. The number of times each symbol pair appears in the uncompressed text is counted, and pairs are only replaced if they appear more than a threshold number of times. In recursive passes, each replaced pair can include a previously substituted replacement symbol. The method and apparatus can achieve high compression especially for large datasets. Metadata, such as the number of times each pair appears, generated during compression of the documents can be used to analyze the documents and find similarities between two documents.