Dictionary Compression Match Selection Without Overlapping Strings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing dictionary-based compression methods for large data sets, such as 'gzip', face inefficiencies in selecting non-overlapping character strings, leading to time-consuming processes that hinder compression/decompression throughput and performance.
Innovation Solution
A compression device and method utilizing a substring generator, match information generator, and selector to efficiently process multiple substrings in parallel, employing a hash table and bitmap-based match selection to avoid character string overlaps, followed by entropy coding for optimal data reduction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a plurality of data strings are searched simultaneously in the dictionary to increase compression throughput, then compression/decompression throughput is improved, but the time required to select non-overlapping character strings increases
Solution Approach 1:
The patent segments the data string into multiple substrings that can be processed in parallel. Each substring is searched independently in the dictionary, and match information is generated for each substring. This segmentation allows simultaneous processing of multiple substrings while reducing the complexity of selecting non-overlapping matches, as the selection is performed on a per-substring basis rather than on the entire data string.
2Productivity
If the dictionary is searched for multiple shifted data strings in parallel, then compression performance is improved, but device complexity increases
Solution Approach 1:
The patent introduces a new dimension of processing by dividing the data string into multiple substrings along the positional dimension. This allows parallel processing of substrings at different positions without requiring complex coordination between searches, as each substring is independently processed. The dimensionality change from processing the entire string sequentially to processing substrings in parallel reduces the overall system complexity.
Data Source
AI summary
According to one embodiment, a compression device includes a substring generator and a match information generator. The substring generator receives generates substrings which are stored in a memory. Byte positions of the substrings are different from each other. The match information generator determines a first string, at least part thereof matching at least part of one of the substrings, and outputs match information. The match information includes a position of the memory storing the first string and a length of the at least part of the first string matching the at least part of one of the substrings.