Post-process deduplication with phase rotation and multi-level hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Post-process deduplication in large-scale storage systems is inefficient due to high performance penalties and resource consumption, making it impractical for reducing data storage footprint effectively.
Innovation Solution
Implementing a phase-rotated, multi-level hashing approach that generates low-level hashes for all data blocks, identifies potential matching candidates, and generates high-level hashes only for those candidates, allowing for incremental processing of datasets through enumeration, commonality, sharing, and index update phases, thereby minimizing resource usage and maximizing deduplication efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If post-process deduplication is performed on large-scale storage systems, then storage space is reduced by eliminating redundancy, but performance penalty increases and processing time becomes excessive
Solution Approach 1:
The patent divides the storage system into multiple storage units and processes datasets in phased increments rather than attempting to process the entire storage system at once. This segmentation allows deduplication to proceed in manageable chunks, reducing the performance impact on any single unit while maintaining overall system functionality.
Solution Approach 2:
The patent performs preliminary actions by first identifying candidate blocks using low-level hashes before committing to full deduplication processing. This preliminary filtering step reduces the scope of subsequent intensive processing, allowing the system to prepare and organize data for deduplication without immediately incurring the full performance penalty.
2Quantity of substance
If comprehensive deduplication processing is performed across the entire storage system, then deduplication effectiveness increases, but resource consumption becomes excessive
Solution Approach 1:
The patent applies partial action by processing only a portion of the storage system (one dataset at a time) rather than attempting comprehensive processing of all data simultaneously. This approach achieves meaningful deduplication results in each phase while consuming manageable computational resources, with the option to continue with additional phases if resources are available.
Solution Approach 2:
The patent introduces an intermediary filtering mechanism using low-level hashes to identify candidate blocks before performing full deduplication analysis. This intermediary step acts as a gatekeeper, reducing the number of blocks that require intensive processing and thereby reducing overall computational resource consumption while maintaining deduplication effectiveness.
3Use of energy by moving object
If multi-level hashing with phase rotation is implemented, then computational resource usage decreases, but processing complexity increases
Solution Approach 1:
The patent segments the hashing process into multiple levels (low-level hashes and high-level hashes) and organizes processing into distinct phases (enumeration, commonality, sharing, index update). This segmentation transforms a single complex operation into multiple simpler, more manageable steps that can be executed incrementally with controlled resource usage at each stage.
Solution Approach 2:
The patent performs preliminary low-level hashing for all blocks before performing more computationally intensive high-level hashing only on candidate blocks. This preliminary action establishes a foundation of basic hash values that simplify subsequent matching operations, reducing overall computational complexity despite the multi-level structure.
Data Source
AI summary
Implementations are provided herein for data deduplication, and more particularly, to post-process data deduplication on a large scale out storage system. Multiple techniques and implementations are disclosed that offer greater efficiency, higher performance, and more stability when performing post-process data deduplication at large scale. Disclosed implementations are based on a process for data deduplication involving four main phases: enumeration, commonality, sharing, and update. Multi-level hashing can be used to identify candidates for deduplication during the enumeration phase, providing a more efficient use of compute resources. In addition, datasets can be phase rotated through the post-process deduplication steps providing a more controllable deduplication environment as well as a more efficient use of resources.