Global Deduplication Capacity Determination via Selective Page Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In deduplication systems, determining the exact physical capacity that will be reclaimed upon deletion of an object is challenging due to shared data segments among multiple owners, leading to increased metadata overhead and reduced system performance.
Innovation Solution
A method combining exact and statistical calculations to determine unique capacity, using a selective page selection based on hash values to reduce metadata overhead while maintaining high accuracy, employing a 2sum method for updating and maintaining reference counts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If exact capacity determination is performed for all data segments in a deduplication system, then measurement precision of reclamation capacity is improved, but device complexity and metadata overhead increase significantly
Solution Approach 1:
The patent segments the population of pages into different subsets based on hash value characteristics (e.g., pages with hash values having specific bit patterns). Instead of tracking all pages uniformly, only selected segments are monitored for exact capacity determination. This segmentation reduces metadata overhead while maintaining sufficient accuracy for capacity reclamation decisions.
Solution Approach 2:
The patent applies partial action by determining exact capacity information for only a subset of pages rather than all pages in the deduplication system. By selecting specific pages based on hash value criteria (e.g., pages where certain hash bits equal zero), the system achieves adequate measurement precision without the full complexity of tracking every data segment, thus reducing metadata overhead while maintaining acceptable accuracy.
2Measurement precision
If reference counts are maintained for all data segments to determine exclusive ownership, then measurement precision of unique capacity is improved, but productivity and system performance decrease due to increased processing overhead
Solution Approach 1:
The patent segments the page population and applies reference count tracking only to selected segments based on hash value characteristics. This selective approach maintains measurement precision for capacity determination while reducing the overall processing overhead by avoiding exhaustive tracking of all data segments, thereby preserving system performance.
Solution Approach 2:
The patent implements partial action by maintaining reference counts for only a subset of pages rather than all pages in the system. By selecting pages based on specific hash value criteria, the system achieves sufficient unique capacity determination accuracy without the performance penalty of tracking every data segment, thus balancing measurement precision with productivity.
3Quantity of substance
If global deduplication is implemented across multiple storage entities, then storage efficiency is improved, but device complexity increases due to managing shared data segments among multiple owners
Solution Approach 1:
The patent segments the management of deduplicated data by applying different tracking strategies to different page subsets. Pages are divided into groups based on hash value characteristics, with selective monitoring applied to specific segments. This segmentation reduces the complexity of managing shared data segments across multiple storage entities while maintaining global deduplication benefits.
Solution Approach 2:
The patent applies local quality by implementing different levels of tracking and management for different subsets of data segments. Instead of uniform management across all pages, the system applies enhanced monitoring only to selected pages based on local hash value characteristics, while other pages receive standard management. This approach reduces overall device complexity while maintaining storage efficiency through global deduplication.
Data Source
AI summary
An aspect of determining per volume exclusive capacity in a deduplication system includes setting a percentage of a population of pages for selection. For each of the pages, an aspect includes selecting a page in the population, providing a data segment facilitating multiple references of the segment by at least one storage entity, maintaining counts corresponding with each segment in the page, and determining exclusive ownership of the page based on the counts and a key value of one of a plurality of storage entities.