Arbitrary Block Sub-Block Deduplication Storage System
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication systems face inefficiencies due to the use of fixed or variable chunk sizes, which do not adapt well to changing data patterns across multiple backup streams, leading to sub-optimal performance and increased metadata overhead.
Innovation Solution
A storage system employing block and sub-block deduplication units that divide data into arbitrary-sized blocks and sub-blocks, using content-based addressing for efficient duplicate elimination, with a dynamically updated subchunk deduplication context to handle varying data granularities and patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If content defined chunking is used with fixed chunk sizes, then deduplication can be performed, but the system cannot adapt to changing data patterns across multiple backup streams
Solution Approach 1:
The patent implements dynamic chunk size adjustment by switching between two chunking modes: content-defined chunking for adaptability and fixed-size chunking for performance. The system dynamically selects the appropriate mode based on data characteristics, allowing it to adapt to changing patterns while maintaining operational efficiency.
Solution Approach 2:
The system changes the chunk size parameter dynamically based on data patterns. By monitoring data characteristics and adjusting the chunking approach accordingly, the system optimizes its performance for different data scenarios without requiring complete system redesign.
2Quantity of substance
If smaller chunks are used for deduplication, then deduplication ratio improves, but metadata overhead increases
Solution Approach 1:
The system dynamically adjusts the chunk size parameter to balance between deduplication ratio and metadata overhead. By selecting appropriate chunk sizes based on data characteristics, the system maximizes storage efficiency while minimizing the overhead associated with tracking and managing chunk metadata.
3Adaptability or versatility
If content defined chunking is used, then the system is effective in the presence of data insertions and deletions, but performance may be impacted
Solution Approach 1:
The system dynamically switches between content-defined chunking mode (for handling insertions/deletions) and fixed-size chunking mode (for optimal performance). This dynamic adaptation allows the system to maintain both adaptability to data changes and high processing efficiency.
Solution Approach 2:
The system periodically evaluates data characteristics and switches between chunking modes accordingly. This periodic adaptation ensures that the system maintains optimal performance while remaining effective at handling data modifications.
Data Source
AI summary
Duplicate storage elimination is performed in units of block data generated by dividing a data stream into arbitrary-sized blocks. Further, sub-block data is generated by further dividing the block data into a plurality of pieces of data, and sub-address data based on the data content of each of the pieces of sub-block data is stored in a predetermined storage device. As such, duplicate storage elimination is also performed in sub-block data units based on the sub-address data.