Source-Side Deduplication Using Fixed and Variable Data Chunks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data backup methods, such as chunking and generating data chunk identifiers for variable-length chunks, are CPU-intensive and time-consuming, especially for small data changes, leading to increased resource and time costs in data management operations like backup, migration, and replication.
Innovation Solution
Implementing techniques that chunk data into fixed-sized or variable-length data chunks, generate corresponding identifiers, and perform deduplication to reduce resource usage and data transmission, using a tree data structure and metadata to manage and locate data chunks efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is chunked into variable-length chunks and identifiers are generated for deduplication, then data transmission volume is reduced, but CPU resource consumption increases
Solution Approach 1:
The patent divides data into fixed-sized chunks instead of variable-length chunks. This segmentation approach maintains deduplication capabilities while reducing CPU overhead by using uniform chunk sizes that are more efficient to process. The fixed size allows for simpler hashing and comparison operations during the deduplication process.
Solution Approach 2:
The patent changes the parameter of chunk size from variable to fixed. This parameter change optimizes the balance between data reduction and processing efficiency. By using fixed-sized chunks, the system achieves better predictability in CPU usage and faster processing speeds while still maintaining effective deduplication through identifier generation and comparison.
2Reliability
If variable-length chunking is performed for all data, then backup completeness is improved, but processing time increases
Solution Approach 1:
The patent segments data into fixed-sized chunks, which maintains complete data coverage similar to variable-length chunking but with improved processing efficiency. The fixed segmentation allows for parallel processing and reduces the computational overhead associated with determining chunk boundaries and sizes for each data segment.
Solution Approach 2:
The patent applies partial deduplication by generating identifiers for fixed-sized chunks and comparing them against existing storage. This approach processes data in manageable fixed units rather than attempting to optimize every variable-length boundary, achieving sufficient deduplication without the excessive processing time that would result from more aggressive optimization attempts.
3Productivity
If fixed-sized chunking is used instead of variable-length, then processing speed is improved, but data reduction efficiency may be reduced
Solution Approach 1:
The patent changes the chunk size parameter from variable to fixed, which directly improves processing speed by eliminating the computational overhead of dynamic size determination. The fixed size enables more efficient memory allocation, hashing, and comparison operations.
Solution Approach 2:
By segmenting data into fixed-sized chunks, the patent maintains effective deduplication capability. The uniform segmentation ensures that identical data portions are consistently represented across different files and backups, enabling effective identification and elimination of duplicates even with fixed boundaries.
Data Source
AI summary
Range information associated with one or more objects is received from a storage system. One or more missing ranges and/or one or more mismatched ranges associated with the one or more objects is determined based on the received range information. A plurality of data chunk identifiers associated with a plurality of variable-length data chunks included in the one or more determined ranges associated with the one or more objects is provided to the storage system. A response that is used to identify among the plurality of variable-length data chunks, one or more variable-length data chunks not already stored in a storage associated with the storage system is received from the storage system. Content of the identified one or more variable-length data chunks is provided to the storage system.