Compression-Aware Chunking for Backup Stream Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud storage providers charge based on incremental chunk sizes, leading to customers being overcharged for data chunks smaller than the nominal size, resulting in inefficient use of storage resources.
Innovation Solution
Implementing a compression-aware chunking process that dynamically adjusts chunk boundaries based on the compressibility of the data stream to create uniform-sized, minimally sized chunks, optimizing storage usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If data is chunked using conventional methods and then compressed, then the backup stream is processed, but the final compressed chunk size is smaller than the nominal storage pricing size, resulting in customers being overcharged
Solution Approach 1:
The chunking process dynamically adjusts chunk boundaries based on the compressibility of the data stream. Instead of using fixed-size chunks, the system continuously monitors compression ratios and modifies chunk sizes in real-time to ensure that compressed chunks meet the minimum storage pricing threshold, thereby optimizing storage cost efficiency while maintaining precision in chunk size consistency
Solution Approach 2:
The system changes the parameter of chunk size based on compression characteristics. By analyzing the compression ratio of data segments, the system adjusts the chunking parameters to ensure that the final compressed chunk size is at least the nominal storage pricing size, transforming the static chunking approach into a dynamic parameter adjustment process that aligns with storage pricing models
2Device complexity
If conventional chunking is used without considering compressibility, then the chunking process is simple, but the compressed chunk size becomes smaller than expected, reducing storage value
Solution Approach 1:
The chunking process incorporates feedback from compression operations to dynamically adjust chunk boundaries. The system monitors the compression ratio of data segments and uses this feedback to modify chunking decisions, ensuring that compressed chunks achieve the desired minimum size. This feedback mechanism increases complexity slightly but significantly improves storage utilization efficiency
Solution Approach 2:
The system performs preliminary analysis of data compressibility before finalizing chunk boundaries. By assessing the compression characteristics of data segments in advance, the system can pre-adjust chunk sizes to ensure that the final compressed output meets storage pricing requirements, thereby improving storage utilization efficiency before the actual compression occurs
Data Source
AI summary
One example method includes reading a data chunk from a data stream, compressing the data chunk, and calculating a chunk delta. When the chunk delta is greater than zero, the compressed data chunk is appended to an incomplete data chunk. When the chunk delta is zero or less, the boundaries of a completed and compressed data chunk having a size at least as large as a minimum size are declared.