Large Block Deduplication via Mapping Layer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies due to exponential growth of redundant data through backups, where deduplication methods are less effective at larger block sizes, leading to high metadata costs and reduced deduplication rates.
Innovation Solution
Implementing a mapping layer that allows for large block deduplication with an offsetting mechanism, enabling nearly equivalent deduplication rates to small block sizes while significantly reducing metadata costs, by calculating hash values for data chunks and querying a hash table to identify matches across sequential patterns, thereby avoiding duplicate data commitments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If small block size (4K, 8K, 16K) is used for deduplication, then deduplication rate is improved, but metadata cost increases significantly
Solution Approach 1:
The patent divides a large data block into multiple smaller data chunks, allowing deduplication to operate at the chunk level rather than the entire block level. This segmentation enables fine-grained deduplication (improving deduplication rate) while reducing the number of metadata entries needed compared to traditional small-block approaches (reducing metadata cost).
Solution Approach 2:
The patent introduces a new dimension by implementing a mapping layer that maps data chunks to physical data blocks. This mapping mechanism allows the system to achieve small-block-level deduplication effectiveness while operating with large block sizes, effectively adding a dimensional transformation that resolves the contradiction between deduplication rate and metadata cost.
2Quantity of substance
If large block size (128K) is used for storage, then metadata cost is reduced, but deduplication rate decreases
Solution Approach 1:
The patent segments large data blocks into smaller data chunks that can be individually hashed and deduplicated. This allows the system to maintain large block sizes for storage efficiency (low metadata cost) while achieving fine-grained deduplication by operating on the segmented chunks within those blocks.
Solution Approach 2:
The patent introduces an intermediary mapping layer that sits between the large block storage structure and the deduplication process. This mapping layer enables the system to work with large blocks for storage while facilitating chunk-level deduplication operations, effectively mediating between the conflicting requirements of large block size and high deduplication rate.
3Quantity of substance
If large block deduplication is implemented, then metadata cost is reduced, but data reduction ratio is suboptimal
Solution Approach 1:
The patent segments large data blocks into smaller data chunks, enabling deduplication to identify and eliminate redundant data at the chunk level. This segmentation approach achieves optimal data reduction ratios by catching finer-grained duplicates while still operating within the efficient large block storage framework.
Solution Approach 2:
The patent adds a dimensional transformation by introducing a mapping layer that enables chunk-level operations within large blocks. This dimensional change allows the system to achieve high data reduction ratios typically associated with small-block deduplication while maintaining the metadata efficiency of large-block storage.
4Measurement precision
If small block size is used for deduplication, then deduplication rate is improved, but storage fragmentation increases
Solution Approach 1:
The patent segments data into chunks that are processed for deduplication, but these chunks are organized within larger block structures. This segmentation approach enables high deduplication rates while the larger block container structure reduces storage fragmentation by maintaining more stable, larger allocation units.
Solution Approach 2:
The patent introduces a mapping layer that adds a dimensional transformation between logical data chunks and physical block allocations. This allows the system to achieve fine-grained deduplication at the chunk level while maintaining stable, large-block physical allocations, thereby reducing storage fragmentation.
Data Source
AI summary
The system, devices, and methods disclosed herein relate to data ratio reduction technology adapted to reduce storage costs by weeding out duplicative data write operations. The techniques and systems disclosed achieve deduplication benefits of smaller block, finer granularity, deduplication systems while simultaneously reducing the enormous metadata costs traditionally associated with small block deduplication. According to an exemplary 128 K block deduplication system and method, we disclose deduplication rates nearly equivalent to 4 K deduplication systems.