LZ4 Block Compression Using Dictionaries for Higher Compression Ratios
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing LZ4 compression algorithm achieves low data compression ratios due to its limitations in using dictionaries only on frames of multiple data blocks, rather than individual blocks, and typical distributed storage array designs opt for block format compression for simplicity and cost, which restricts the potential for higher compressibility.
Innovation Solution
The proposed solution involves prepending a dictionary to the uncompressed data block and using it to compress the data block into a compressed block with an offset pointer that points to a matching entry, allowing for the use of dictionaries on individual data blocks and potentially improving compression ratios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If dictionaries are used only on frames of multiple data blocks in LZ4 compression, then device complexity is reduced and ease of operation is improved, but data compression ratio deteriorates
Solution Approach 1:
The patent divides the compression operation into two independent formats: frame format for multiple blocks and block format for individual blocks. This segmentation allows dictionaries to be applied selectively - using frame format when multiple blocks need compression (maintaining simplicity) and block format when individual block compression is needed (achieving higher compression ratios). The system thus resolves the contradiction by allowing both simple operation and high compression ratio in different scenarios.
Solution Approach 2:
The patent introduces dynamic selection between frame format and block format based on the compression needs. The system can dynamically switch between using dictionaries on frames (for simplicity) and using dictionaries on individual blocks (for higher compression ratios). This dynamic approach allows the system to adapt to different requirements, resolving the contradiction between ease of operation and compression ratio.
2Device complexity
If block format compression is used for simplicity and cost, then device complexity is reduced, but compressibility and compression ratios worsen
Solution Approach 1:
The patent segments the compression functionality into frame format operations (simpler, lower complexity) and block format operations (more powerful, higher compressibility). By maintaining both formats in the same system, the patent allows users to choose the appropriate format based on their needs - using frame format when simplicity is paramount and block format when compressibility is the priority, thus resolving the contradiction between device complexity and compressibility.
Solution Approach 2:
The patent makes the LZ4 compression system universal by supporting both frame format and block format with dictionary functionality. The same compression device can handle both simple frame-based compression and more sophisticated block-based compression with dictionaries. This multi-functionality resolves the contradiction by allowing a single system to provide both low complexity and high compressibility as needed.
Data Source
AI summary
An information handling system for compressing data includes a data storage device and a processor. The data storage device stores a dictionary and an uncompressed data block. The processor prepends the dictionary to the uncompressed data block, determines, from the uncompressed data block, a literal data string and a match data string where the match data string is a matching entry of the dictionary, and compresses the uncompressed data block into a compressed data block that includes the literal data string and an offset pointer that points to the matching entry.