Dictionary Compression Using Indexed Match Positions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Dictionary-based encoding methods face challenges in effectively compressing match positions, leading to inefficient encoding when input data contains continuous repetitions or specific patterns, such as text or log data, where the same match position is not consistently output.
Innovation Solution
A compression device is designed with a dictionary-based encoder, a match position encoding unit, and a compression data generator, which uses a hash calculator, hash table, and history buffer to search for matching data, and encodes match positions using indices or addresses, improving encoding efficiency by storing and comparing match positions and lengths to generate compressed data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If dictionary-based encoding is used to compress data, then data compression is achieved, but match positions cannot be effectively compressed leading to inefficient encoding
Solution Approach 1:
The match position encoding is segmented into two parts: a fixed base address component and a variable offset component. This segmentation allows the fixed base address to be reused for multiple match positions, while only the variable offset needs to be stored, thereby reducing the number of bits required to represent match positions and improving encoding efficiency.
Solution Approach 2:
The patent changes the parameter representation of match positions from absolute addresses to relative offsets from base addresses. By transforming the coordinate system from absolute to relative, the patent reduces the bit width required to represent match positions, especially when matches are clustered around certain base addresses, thus improving compression efficiency.
2Measurement precision
If match positions are stored with full precision, then accurate data reconstruction is achieved, but compression efficiency deteriorates
Solution Approach 1:
The match position is segmented into a base address (which can be shared) and an offset (which varies). This segmentation enables accurate representation of match positions while reducing redundancy, as the base address portion is encoded once and reused, and only the smaller offset portion needs to be stored for each match.
Solution Approach 2:
The patent introduces a new dimensional approach by organizing match positions relative to base addresses rather than using flat absolute addressing. This dimensional transformation allows the system to exploit the spatial clustering of matches and represent them more compactly by encoding the offset from base addresses rather than full absolute positions.
Data Source
AI summary
A compression device includes a dictionary based encoder, a second buffer, a comparator, and a compression data generator. The dictionary based encoder searches for second data at least partially matching first data from a first buffer, and acquires a first match position indicating a position of the second data in the first buffer and a match length indicating a matched length of the first and second data. The second buffer stores the previously acquired second match position with an index. The compression data generator generates first compressed data that includes the index assigned to the second match position in the second buffer and the match length when the first match position matches the second match position in the second buffer.