Recursive Symbol-Pair Compression for Large Text Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text compression techniques are not scalable for large data sets and do not effectively track the frequency of symbol pairs, which limits their ability to achieve optimal space savings and processing efficiency.
Innovation Solution
A method that identifies and replaces frequent symbol pairs in multiple documents using a compression dictionary, where each unique symbol pair is associated with a replacement symbol and sorted by frequency, allowing for recursive passes until no further replacements are possible, and using addresses as replacement symbols to optimize memory usage and lookup times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If previous text compression techniques are used on single files, then compression is achieved for individual documents, but the techniques do not scale to enormous data sets and do not track symbol pair frequencies
Solution Approach 1:
The patent merges multiple individual document processing operations into a single unified compression process that handles enormous data sets as a whole. By combining all documents into one processing stream and building a global compression dictionary across the entire data set, the system achieves frequency tracking and compression optimization that spans across all documents simultaneously, rather than treating each document in isolation.
2Loss of substance
If symbol pairs are replaced with replacement symbols, then space savings are achieved, but processing time increases due to dictionary lookups
Solution Approach 1:
The patent performs preliminary action by pre-processing the entire data set to identify and count all symbol pairs before compression. The compression dictionary is built in advance with all symbol pair to replacement symbol mappings, including frequency information. This preliminary frequency analysis and dictionary construction enables optimized compression where the most frequent symbol pairs are replaced first, maximizing space savings while minimizing processing time through efficient lookup operations.
3Loss of substance
If compression is applied to larger data sets, then more symbol pair repetitions occur increasing compression ratio, but the complexity of tracking and managing symbol pairs across multiple documents increases
Solution Approach 1:
The patent implements a universal compression dictionary that serves multiple functions simultaneously: it stores symbol pair to replacement symbol mappings, tracks frequency counts of all symbol pairs across the entire data set, and provides the basis for recursive compression passes. This multi-functional dictionary structure handles the complexity of managing symbol pairs across numerous documents through a single unified data structure, enabling high compression ratios without proportionally increasing system complexity.
4Loss of substance
If recursive passes are performed until no further replacements are possible, then maximum compression is achieved, but processing time increases
Solution Approach 1:
The patent implements feedback through recursive compression passes where the output of each pass becomes the input for the next pass. After each pass, the system checks whether any replacements were made; if no replacements occur in a complete pass, the recursion terminates. This feedback mechanism ensures maximum compression is achieved by continuing passes only as long as improvements are being made, avoiding unnecessary processing time once optimal compression is reached, while still achieving the highest possible compression ratio through multiple iterative optimizations.
Data Source
AI summary
A computerized method and apparatus compresses symbolic information, such as text. Symbolic information is compressed by recursively identifying pairs of symbols (e.g., pairs of words or characters) and replacing each pair with a respective replacement symbol. The number of times each symbol pair appears in the uncompressed text is counted, and pairs are only replaced if they appear more than a threshold number of times. In recursive passes, each replaced pair can include a previously substituted replacement symbol. The method and apparatus can achieve high compression especially for large datasets. Metadata, such as the number of times each pair appears, generated during compression of the documents can be used to analyze the documents and find similarities between two documents.