Range-Based Deduplication Hash Table with Address Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face inefficiencies due to duplicate data copies, particularly in systems storing large numbers of similar containerized structures like virtual machines, where conventional approaches lead to substantial storage wastage and inefficiency.
Innovation Solution
A method utilizing a hash table and translation table to identify and map duplicate data blocks to shared instances, allowing for efficient deduplication by replacing entries based on address alignment and extending range detection, thereby optimizing storage space usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If a conventional block-by-block deduplication approach is used, then duplicate data can be identified, but storage efficiency is reduced due to substantial storage wastage and the need to track each individual block
Solution Approach 1:
The patent merges multiple consecutive data blocks into range-based units for deduplication tracking. Instead of treating each block independently, the system identifies and tracks ranges of consecutive blocks that share the same deduplication status, combining multiple tracking operations into unified range operations that reduce overhead while maintaining deduplication effectiveness
Solution Approach 2:
The patent segments the address space into distinct ranges with different deduplication characteristics. By dividing the address space into manageable range segments and tracking them separately, the system reduces the complexity of tracking individual blocks while preserving the ability to identify and eliminate duplicates across the entire address space
2Reliability
If the hash table has entries for all data blocks, then duplicate detection is comprehensive, but the hash table size becomes comparable to the physical volume size, increasing memory usage and processing overhead
Solution Approach 1:
The patent combines multiple hash table entries into single range entries. Instead of maintaining separate entries for each block, the system merges consecutive blocks with identical hash values into unified range entries, dramatically reducing the total number of entries while preserving complete duplicate detection capability across all blocks in the range
Solution Approach 2:
The patent segments the hash table into range-based entries rather than block-based entries. Each entry represents a segment (range) of consecutive blocks rather than individual blocks, reducing the quantity of entries while maintaining the ability to detect duplicates through hash comparison at the range level
3Manufacturing precision
If block-by-block translation is performed, then precise mapping is achieved, but I/O operation efficiency is reduced due to the need to process each block individually
Solution Approach 1:
The patent merges multiple block-level translation operations into unified range-level translation operations. By identifying contiguous ranges of blocks that map to the same physical location, the system performs single translation operations for entire ranges rather than individual block translations, maintaining precise mapping while dramatically improving I/O operation throughput
Solution Approach 2:
The patent segments I/O operations into range-based operations rather than block-based operations. By identifying and processing contiguous ranges of blocks as unified units, the system maintains the precision needed for accurate mapping while reducing the total number of translation operations required, thereby improving overall I/O productivity
Data Source
AI summary
Deduplicated data storage is provided by presenting a virtual volume mapped by a translation table to a physical volume of a physical data storage system. The translation table maps sets of ranges of duplicate data blocks of the virtual volume to corresponding individual ranges of shared data blocks of the physical volume. A hash table for identifying duplicate data is indexed by a portion of a hash value calculated from newly written data blocks, and has entries each identifying an address alignment of the corresponding data block. In operation, existing entries are replaced with new entries for colliding data blocks having better address alignment, promoting wider address-space separation of the entries. Upon occurrence of a hit in the hash table, for a given data block in a range of newly written data blocks, data blocks of the range are compared to corresponding blocks in a range identified by the hit to maximize a size of a region to be identified by the translation table as duplicate data.