Variable-Size Chunking for Data Deduplication and Random Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication methods, such as fixed-size deduplication, suffer from low deduplication ratios and inefficient random access due to their inability to handle variable-sized data chunks effectively, leading to increased storage costs and computational resource usage.
Innovation Solution
A key-value-based storage system that divides data files into variable-sized chunks, which are further divided into blocks of different sizes, with metadata generated to indicate block and chunk sizes, enabling efficient deduplication and random access by leveraging fixed-size deduplication mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If fixed-size deduplication is used to divide data into blocks, then storage space can be reduced by eliminating identical blocks, but the deduplication ratio is low and blocks with similar but not identical data cannot be deduplicated
Solution Approach 1:
The patent segments data into variable-size chunks instead of fixed-size blocks. Each chunk is divided into variable-size sub-chunks based on content boundaries, allowing more granular and accurate deduplication. This segmentation approach enables identification of similar data regions even when they don't align with fixed block boundaries, thereby improving the deduplication ratio while effectively reducing storage space.
2Reliability
If variable-size chunking is used to improve deduplication ratio, then more similar blocks can be identified, but random access becomes more complex and computational resources increase
Solution Approach 1:
The patent performs preliminary actions by pre-calculating and storing metadata about variable-size chunks, including their boundaries, sizes, and hierarchical relationships. This metadata is organized in advance to enable efficient random access without requiring complex computations at access time. The preliminary organization of chunk information allows the system to quickly locate and retrieve specific data regions while maintaining high deduplication ratios.
3Loss of substance
If differential compression is used to achieve better data deduplication, then more similar data can be reduced, but significant computational processing power is required increasing time and resource usage
Solution Approach 1:
The patent uses simple hash functions and lightweight comparison algorithms instead of complex differential compression techniques. By employing computationally inexpensive methods to generate chunk signatures and identify duplicates, the system achieves effective data reduction without requiring significant computational processing power. This approach prioritizes efficiency and low resource consumption while maintaining good deduplication performance.
4Ease of operation
If fixed-size blocks are used for storage, then random access is simple with straightforward key calculation, but deduplication effectiveness is limited when blocks are not identical
Solution Approach 1:
The patent introduces a hierarchical dimension to the storage structure by organizing data into variable-size chunks that contain variable-size sub-chunks. This multi-level hierarchy adds a new dimension to data organization, allowing the system to maintain simple access patterns at each level while achieving effective deduplication across the entire structure. The hierarchical approach enables both simplicity in access and effectiveness in deduplication by operating at multiple granularities simultaneously.
Data Source
AI summary
A method of storing a data file in a key value-based storage system includes dividing the data file into at least two variable size chunks. Each of the variable size chunks is divided into blocks, where the blocks have different sizes depending on the amount of data in each block. A set of metadata is generated for the data file including information indicating the size of each block and the size of each chunk and writing the data file and the set of metadata to the key value-based storage.