Storage Data Compression Using Region Sketch Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data compression techniques in storage systems face inefficiencies due to limited window sizes for identifying redundancy, leading to suboptimal compression rates and increased computational overhead, especially when dealing with large datasets.
Innovation Solution
The approach involves partitioning data into chunks, generating similarity-based sketches, and reorganizing these chunks to group similar data together, allowing for more effective compression by standard compressors, while preserving metadata for reconstruction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If the redundancy window size is increased to find more internal redundancy, then compression efficiency is improved, but computation time and memory usage increase
Solution Approach 1:
The patent divides the data stream into fixed-size chunks and processes them independently. Each chunk is compressed using a limited window size, avoiding the need to process large amounts of data in a single pass. This segmentation allows the system to achieve good compression ratios without requiring excessive computation time or memory resources.
Solution Approach 2:
The patent pre-computes and stores metadata about data chunks, including information about potential redundancy patterns. This preliminary processing allows the compression algorithm to make faster decisions during actual compression operations, reducing computation time while maintaining compression efficiency.
2Manufacturing precision
If the redundancy window size is increased to find more internal redundancy, then compression efficiency is improved, but memory usage increases
Solution Approach 1:
By dividing data into smaller chunks with limited overlap regions, the patent reduces the memory footprint of the compression window. Each chunk only needs to maintain a limited history for redundancy detection, rather than storing large amounts of historical data in memory.
Solution Approach 2:
The patent uses a partial window approach where only the necessary portion of historical data is retained in memory for each chunk. Instead of maintaining a large global window, the system uses smaller local windows that are sufficient for achieving good compression ratios without excessive memory consumption.
3Productivity
If data is compressed using traditional methods without reorganization, then processing speed is maintained, but compression ratios are suboptimal
Solution Approach 1:
The patent segments data into chunks and applies compression independently to each chunk. This approach maintains processing speed by avoiding the need to analyze and reorganize large amounts of data globally, while still achieving improved compression ratios through the chunk-based redundancy detection and the use of multiple hash functions to identify patterns within each chunk.
Data Source
AI summary
Techniques for improving data compression of a storage system using coarse and fine grained similarity are described herein. According to one embodiment, region sketches for a plurality of regions of the set of data are generated, each region storing a plurality of data chunks. A region sketch index having a plurality of entries is maintained, each corresponding to one of the region sketches of the regions. The entries of the region sketch index are sorted based on the sketches of the regions, such that regions with an identical region sketch are positioned adjacent to each other within the region sketch index, representing similar regions. The data chunks of the similar regions that are identified based on the sorted entries of the region sketch index are reorganized to improve data compression of the data chunks of the similar regions.