Data Deduplication via Multi-Level Chunk Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication methods have a fixed deduplication ratio, limiting further reduction in data storage needs and storage costs.
Innovation Solution
A method and system that combines multiple data chunks into a new data object, performing deduplication on this new object to generate new meta data and chunks, which are then stored, allowing for further reduction in storage space by removing duplicated data within the combined object.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data chunk-level deduplication is used, then storage space is reduced, but the deduplication ratio becomes fixed and cannot be improved further
Solution Approach 1:
The patent segments the deduplication process into multiple levels: first dividing data into chunks at the chunk level, then further dividing those chunks into sub-chunks at the sub-chunk level. This multi-level segmentation enables progressive deduplication, where each level can achieve additional deduplication ratios beyond what a single level could achieve alone, thus resolving the contradiction between reducing storage space and maintaining adaptable deduplication ratios.
2Quantity of substance
If multiple data chunks are combined into a new data object, then deduplication ratio is improved, but system complexity increases
Solution Approach 1:
The patent performs preliminary actions by pre-calculating and storing metadata for each data chunk before the actual deduplication process. This includes pre-computing chunk identifiers, maintaining chunk maps, and organizing data structures in advance. When deduplication is performed on combined data objects, the preliminary metadata enables efficient lookup and comparison, reducing the computational complexity during the actual deduplication operation despite the increased system complexity from combining multiple chunks.
Data Source
AI summary
The present disclosure discloses a method and system for data deduplication. The method comprises: acquiring meta data and multiple data chunks corresponding to at least one original data object, which are generated by using a data deduplication method; combining the acquired multiple data chunks into a new data object; performing deduplication on the new data object to generate new meta data and new data chunks corresponding to the new data object; and storing the meta data corresponding to the at least one original data object, the new meta data corresponding to the new data object, and the new data chunks. The method and system can further improve deduplication ratio, lower data storage amount, and save costs.