Dictionary Partitioning for High-Ratio Data Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression methods face challenges in achieving high compression ratios without excessive computational complexity during decompression, and they often waste storage space due to uneven data record lengths in dictionaries, leading to increased processing costs and slow analytical processes.
Innovation Solution
A computer-implemented data compression method that splits a dictionary into a major and minor dictionary based on calculated threshold data record lengths, using mean and standard deviation to optimize memory usage and reduce computational overhead during search operations, allowing for efficient storage and retrieval of data records.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If conventional compression algorithms (Huffman, null suppression) are used to achieve high compression ratios, then compression ratio improves (50-85%), but decompression complexity and computational cost increase significantly
Solution Approach 1:
The patent segments the dictionary into multiple parts (first dictionary portion and second dictionary portion) based on data record length thresholds. This segmentation allows the system to use different storage strategies for different portions: the first portion uses fixed-size memory elements for fast access, while the second portion uses variable-size or compressed storage. By dividing the dictionary into manageable segments with different access patterns, the patent achieves high compression ratios without requiring complex decompression logic for the entire dictionary.
2Volume of stationary object
If memory element size is increased to accommodate the longest data record, then storage capacity improves, but storage space waste increases for shorter records
Solution Approach 1:
The patent applies local quality by assigning different memory element sizes to different portions of the dictionary based on the actual length requirements of data records in each portion. The first dictionary portion (containing shorter records) uses smaller fixed-size memory elements, while the second dictionary portion (containing longer records) uses larger or variable-size memory elements. This localized adaptation of memory allocation eliminates the need to allocate maximum size memory elements for all records, thereby reducing storage space waste while maintaining sufficient capacity for all records.
3Loss of time
If dictionary size is reduced for faster loading, then I/O overhead decreases, but storage of long records becomes problematic
Solution Approach 1:
The patent segments the dictionary into a first portion that fits within available memory for fast loading and a second portion that can be stored externally or in extended storage. The first dictionary portion contains the most frequently accessed or shortest records, enabling quick loading and reducing I/O overhead. The second dictionary portion accommodates longer records that may not fit in memory, maintaining storage flexibility. This segmentation allows the system to optimize for speed where possible while preserving the ability to handle all record types.
4Speed
If fixed memory element size is used for all data records, then access speed improves, but compression ratio decreases due to unused space
Solution Approach 1:
The patent applies local quality by using fixed-size memory elements for the first dictionary portion where they provide fast access, and transitioning to variable-size or compressed storage for the second dictionary portion where compression becomes more important. This localized approach allows the system to maintain fast access speeds for records that benefit from fixed-size allocation while achieving better compression ratios for records where fixed-size allocation would waste significant space. The threshold-based division optimizes the trade-off between access speed and compression ratio for different portions of the data.
Data Source
Figure 1A~1B
Figure 2A~2B
Figure 3A~3B
AI summary
The invention relates o a computer-implemented data compression method comprising: - providing a dictionary (300) comprising a plurality of data records; - computing a mean of the lengths of the data records of the dictionary; - computing a standard deviation of the lengths of the data records of the dictionary; - determining a deviation threshold indicating a maximum allowable deviation of a data record length from a mean data record length; - calculating a threshold data record length from the computed mean, from the computed standard deviation and from the determined deviation threshold, - splitting the dictionary into a minor dictionary (314) and a major dictionary (310), the major dictionary selectively comprising data records whose lengths are below or equal to the calculated threshold data record length, the minor dictionary selectively comprising the ones of the data records whose lengths are larger than the calculated threshold data record length.