Online Data Compression Using Similarity-Based Chunk Grouping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data compression techniques in storage systems are limited by their reliance on small back-reference windows, which restrict the identification and elimination of redundant information, leading to suboptimal compression efficiency due to the random distribution of similar data chunks.
Innovation Solution
The proposed solution involves partitioning data into chunks, determining their similarity based on sketches or features, and reorganizing them to group similar chunks together, allowing for more effective compression by standard compressors, while preserving metadata for reconstruction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If standard compression algorithms are used with default back-reference windows, then the compression process is simple and fast, but compression efficiency is limited due to small window size
Solution Approach 1:
The patent applies preliminary action by reorganizing data chunks before compression based on similarity metrics (sketches). This pre-processing step groups similar data together, enabling standard compression algorithms to achieve better compression ratios without modifying the compression algorithms themselves. The reorganization happens in advance, transforming the data layout to maximize compression effectiveness.
Solution Approach 2:
The patent segments data into chunks and computes sketches for each chunk to determine similarity. This segmentation allows the system to identify and group similar data portions independently, then reorganize them to improve compression. The chunking approach enables parallel processing and maintains simplicity while improving efficiency.
2Productivity
If data chunks are reorganized based on similarity before compression, then compression efficiency improves significantly, but additional computation and memory overhead is introduced
Solution Approach 1:
The patent changes parameters by using compact sketch representations instead of full data comparisons for similarity determination. This parameter change (from comparing entire chunks to comparing sketches) dramatically reduces computation overhead while maintaining the ability to identify similar data chunks effectively. The sketch size is controlled to balance between accuracy and computational cost.
Solution Approach 2:
The patent introduces sketches as an intermediary between raw data and compression algorithms. These sketches serve as lightweight proxies that enable similarity detection without requiring full data comparison. The intermediary layer (sketch computation and matching) reduces the computational burden while preserving the essential similarity information needed for effective reorganization.
3Productivity
If similar data chunks are grouped together for compression, then redundancy elimination is improved, but data access patterns may be disrupted
Solution Approach 1:
The patent creates a reorganized copy of the data with similar chunks grouped together, while maintaining the ability to retrieve original data through metadata mappings. The compression is applied to the reorganized copy, but the system preserves reference information that allows efficient retrieval of the original data sequences. This copying approach enables compression improvement without permanently disrupting data access patterns.
Data Source
AI summary
Techniques for improving data compression of a storage system in an online manner are described herein. According to one embodiment, in response to a sequence of data to be stored, the sequence of data is partitioned into a plurality of data chunks according to a predetermined chunking algorithm. A sketch for each of the data chunks is generated based on one or more features extracted from the data chunk. Each of the data chunks of the sequence of data is associated with one of a plurality of groups based on the sketch, wherein each group is represented by a sketch. The data chunks of each group are compressed and stored in a compression region of the storage systems, such that similar data chunks are compressed and stored in the same compression region.