De-duplication Engine Coalescing and Sedimentation Phases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed block storage systems face challenges in scalability and performance due to the inefficiencies of traditional offline de-duplication techniques, particularly in identifying and removing duplicate data fragments in large-scale environments, which leads to significant processing overhead and storage challenges.
Innovation Solution
A method and system for scalable offline de-duplication that involves cycling through coalescing and sedimentation phases, using a counting hash set to select seed blocks and a scalable Bloom filter to efficiently identify and store similar blocks, allowing for the removal of unnecessary data without maintaining reference counts, thereby enhancing storage efficiency and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional offline de-duplication compares every page-sized block to every other block, then duplicate data can be identified, but the system does not scale well to large-scale distributed storage
Solution Approach 1:
The patent segments the de-duplication process into distinct phases: coalescing phase where seed blocks are selected and grouped, and sedimentation phase where non-seed blocks are processed. This segmentation allows the system to handle large-scale data by processing blocks in manageable groups rather than comparing all blocks simultaneously, thus improving scalability while maintaining duplicate identification accuracy.
Solution Approach 2:
The patent performs preliminary action by selecting and designating seed blocks before processing the remaining blocks. These seed blocks serve as reference points for identifying duplicates among non-seed blocks. This preliminary selection reduces the overall comparison workload significantly, as subsequent blocks only need to be compared against seed blocks rather than all previously processed blocks, thereby improving scalability.
2Ease of operation
If reference counts are maintained for every data fragment to enable removal of unnecessary data, then data can be properly managed, but significant performance and storage overhead is incurred
Solution Approach 1:
The patent extracts the reference counting mechanism from the distributed de-duplication system. Instead of maintaining reference counts for every data fragment in the distributed system, the patent uses a centralized garbage collector that periodically identifies and removes unreferenced blocks. This extraction eliminates the performance and storage overhead of distributed reference counting while maintaining the ability to properly manage and remove unnecessary data.
3Measurement precision
If page-sized blocks (4KB) are used for de-duplication, then fine-grained duplicate detection is achieved, but processing overhead increases significantly
Solution Approach 1:
The patent introduces dynamics by making block size adaptable rather than fixed. The system can dynamically adjust block sizes based on workload characteristics and performance requirements. During the coalescing phase, larger blocks may be processed to reduce overhead, while the sedimentation phase maintains fine-grained processing for precise duplicate detection. This dynamic approach allows the system to balance processing efficiency with duplicate detection granularity.
Data Source
AI summary
Disclosed herein are systems and method for building content for a de-duplication engine. In one aspect, an exemplary method comprises periodically receiving instructions for cycling through a coalescing phase and a sedimentation phase of the de-duplication engine, during a first coalescing phase, selecting a set of seed blocks that are similar to each other, when an instruction for proceeding to a next sedimentation phase is received, entering the sedimentation phase during which newly received blocks are processed to be stored near similar seed blocks, and when an instruction to proceed to a next coalescing phase is received, entering the coalescing phase to update the set of seed blocks.