Variable Size Data Deduplication with Change Indicators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data deduplication methods in primary storage systems face challenges such as low deduplication ratios, inability to deduplicate blocks with similar but not identical data, and inefficient random access of variable-sized chunks.
Innovation Solution
The method involves dividing data storage into variable size chunks and fixed size chunks, with metadata that includes change indicators to track changes in fixed size chunks. When changes exceed a threshold, the fixed size chunks are rechunked into variable sized chunks, enabling high deduplication ratios and accurate random access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If fixed size deduplication is used, then storage space is saved, but deduplication ratio is low and blocks with similar but not identical data cannot be deduplicated
Solution Approach 1:
The patent segments data into variable-size chunks rather than fixed-size blocks. Each chunk is divided into smaller units that can be independently compared, allowing similar but not identical blocks to be recognized as duplicates. This segmentation enables more granular comparison and higher deduplication ratios while maintaining storage space efficiency.
Solution Approach 2:
The patent changes the parameter of block size from fixed to variable. By allowing chunks to have different sizes based on actual data content, the system can adapt to varying data patterns and achieve better deduplication. The variable size parameter enables the system to capture semantic similarity rather than relying solely on exact byte-level matching.
2Reliability
If differential compression is used for data reduction, then data deduplication is improved, but computational processing power requirement increases
Solution Approach 1:
The patent segments data into variable-size chunks that can be independently processed and compared. This segmentation allows for more efficient deduplication by reducing the computational complexity of comparing large blocks of data. Smaller, variable-sized chunks require less processing power to compare while maintaining the ability to identify duplicates effectively.
Solution Approach 2:
The patent applies partial deduplication by only processing and comparing portions of data that are likely to be duplicates, rather than processing entire blocks. This selective approach reduces the overall computational burden while achieving sufficient deduplication results, avoiding the excessive processing required by full differential compression.
3Reliability
If variable size chunking is used, then deduplication accuracy is improved, but random access complexity increases
Solution Approach 1:
The patent performs preliminary indexing of variable-size chunks, creating a mapping structure that records the position and size of each chunk. This preliminary action enables efficient random access by allowing the system to quickly locate and retrieve specific chunks without having to scan through the entire variable-size chunked data structure, thus reducing access complexity.
Solution Approach 2:
The patent introduces an intermediary indexing structure that mediates between the variable-size chunked data and the access operations. This intermediary layer provides a simplified interface for random access, translating complex variable-size chunk addresses into manageable access patterns, thereby reducing the perceived complexity of accessing variable-size chunked data.
Data Source
AI summary
A method of storing data to a data storage and related device are provided, wherein data storage is divided into two or more variable size chunks, and each variable size chunks is divided into blocks, and the data storage comprises set of metadata. The data storage is also divided into two fixed size chunks. The method comprises: writing incoming write IO to a position of the corresponding size chunk in the data storage; setting the change indicator for the corresponding fixed size chunk to indicate that the corresponding fixed size chunk has been changed since it was divided into variable sized chunks. When change indicators indicate that division of the associated fixed size chunk or chunks into variable sized chunks should be changed, rechunking fixed size chunk into variable sized chunks is executed. Because of the change indicator, high deduplication is maintained even for offline deduplication in the data storage.