Symbol-Pair Compression Dictionary for Large Document Sets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression techniques are inefficient for large datasets and do not effectively track the frequency of symbol pairs, which is crucial for improving compression ratios and reducing processing time.
Innovation Solution
A method that identifies and counts unique symbol pairs in documents, replacing pairs that exceed a threshold with replacement symbols, and recursively processes documents to optimize compression, using a compression dictionary that sorts entries by frequency for improved lookup times and space savings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If previous text compression techniques are used, then compression can be applied to individual files, but the techniques do not scale to large datasets and do not track symbol frequency
Solution Approach 1:
The patent segments the compression process into distinct phases: symbol pair identification, frequency counting, dictionary construction, and recursive compression. This segmentation allows the system to handle large datasets by processing them in manageable units while maintaining frequency tracking across the entire dataset through the cumulative dictionary approach.
Solution Approach 2:
The compression dictionary serves multiple functions: it stores symbol pair mappings, tracks frequency counts, and enables both compression and analysis operations. This multi-functionality allows the same data structure to support both compression scalability and frequency analysis without requiring separate systems.
2Loss of substance
If symbol pairs are replaced with replacement symbols, then compression ratio improves, but processing time increases due to dictionary lookups
Solution Approach 1:
The patent performs preliminary actions by pre-processing the data to identify and count all symbol pairs before compression, constructing the frequency dictionary in advance. This preliminary frequency analysis allows the system to optimize the compression process by having replacement symbols ready, reducing the need for complex runtime decisions and improving overall processing efficiency.
Solution Approach 2:
The patent changes the parameter representation by using frequency-based ordering of symbol pairs in the dictionary, where high-frequency pairs are positioned for faster access. This parameter change optimizes the balance between compression ratio and processing time by making the most beneficial replacements more accessible during the compression phase.
3Loss of substance
If compression is applied to multiple documents, then compression ratio improves, but the process requires recursive passes to achieve optimal compression
Solution Approach 1:
The patent maintains continuity of useful action by accumulating symbol pair frequencies across multiple documents in a single cumulative dictionary, rather than resetting for each document. This continuous accumulation allows the system to leverage patterns across the entire dataset while processing documents in sequence, achieving optimal compression without requiring multiple recursive passes through the same data.
Solution Approach 2:
The system performs preliminary frequency counting across all documents before compression, establishing the cumulative dictionary in advance. This preliminary action enables subsequent compression passes to proceed more efficiently by using the pre-computed frequency information to guide replacement decisions across multiple documents simultaneously.
Data Source
AI summary
A computerized system and method of compressing symbolic information organized into a plurality of documents, each document having a plurality of symbols, the system and method including: (i) automatically identifying a plurality of sequential (also referred to as adjacent) and/or non-sequential symbol (also referred to as non-adjacent) pairs in an input document; (ii) counting the number of appearances of each unique symbol pair; and (iii) producing a compressed document that includes a replacement symbol at each position associated with one of the plurality of symbol pairs, at least one of which corresponds to a non-sequential symbol pair. For each non-sequential pair the compressed document includes corresponding indicia indicating a distance between locations of the non-sequential symbols of the pair in the input document.