Scalable De-duplication Engine Using Bloom Filters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed block storage systems face challenges in scalability and performance due to the inefficiencies of traditional offline de-duplication techniques, particularly when dealing with large-scale data and maintaining reference counts for data fragments, which leads to significant overhead and complexity in removing unnecessary data.
Innovation Solution
A scalable offline de-duplication method and system that uses a de-duplication engine to store de-duplicated page references by hashes and block descriptors, calculates a translation tolerant hash vector for newly received blocks, and determines similarity using a scalable Bloom filter, allowing for efficient storage and removal of duplicate data without significant performance or storage overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional offline de-duplication compares every page-sized block to every other block, then de-duplication completeness is improved, but system scalability deteriorates
Solution Approach 1:
The patent segments the de-duplication process into two phases: an indexing phase that creates a compact representation of all blocks, and a query phase that uses this index for efficient similarity search. This segmentation allows the system to scale by separating the heavy computation of block analysis from the lighter operation of duplicate detection.
Solution Approach 2:
The patent introduces a Bloom filter as an intermediary data structure between the raw block data and the duplicate detection logic. The Bloom filter provides a space-efficient probabilistic representation that enables fast membership queries without requiring direct comparison of all blocks, thus maintaining scalability while preserving de-duplication effectiveness.
2Reliability
If reference count is maintained for every data fragment to remove unnecessary data, then data management accuracy is improved, but system complexity and overhead deteriorate
Solution Approach 1:
The patent extracts the reference counting functionality from the core de-duplication engine and implements it as a separate garbage collection mechanism that operates independently. This allows the main de-duplication system to focus on identifying duplicates while a specialized subsystem handles reference tracking and data retention decisions, reducing overall system complexity.
Solution Approach 2:
The system implements self-service through automatic garbage collection that periodically cleans up unreferenced data fragments without manual intervention. The garbage collector autonomously identifies and removes data that is no longer needed, eliminating the need for complex manual reference management while maintaining data management accuracy.
3Loss of substance
If online de-duplication is performed during data processing, then storage efficiency is improved, but writing performance deteriorates
Solution Approach 1:
The patent implements periodic offline de-duplication operations that run at scheduled intervals rather than continuously during data ingestion. This periodic approach allows the system to maintain high writing performance during normal operations while periodically performing de-duplication to improve storage efficiency, thus resolving the contradiction between the two goals.
Solution Approach 2:
The system performs preliminary indexing of data blocks during the offline de-duplication process before actual duplicate removal occurs. This preliminary action organizes the data in a way that enables efficient subsequent de-duplication operations without impacting the performance of data writing operations.
4Quantity of substance
If large scale distributed storage is implemented, then storage capacity is improved, but de-duplication scalability deteriorates
Solution Approach 1:
The patent transitions from a single-dimension approach of comparing blocks directly to a multi-dimensional approach using distributed hashing and Bloom filters. By organizing data across multiple dimensions (hash space, distributed nodes, probabilistic filters), the system can scale de-duplication operations to match large distributed storage capacities without quadratic complexity growth.
Data Source
AI summary
Disclosed herein are systems and method for de-duplicating blocks of data. In one aspect, an exemplary method comprises receiving a block of data at a de-duplication engine that comprises a first block node and a first page node, wherein the first block node stores a single block descriptor for at least two identical blocks previously received and wherein the first page node stores single instances of identical pages in the at least two identical blocks. The method comprises comparing the received block with the at least two identical blocks. In response to determining that the received block partially matches the at least two identical blocks, the method comprises storing a block descriptor of the received block in a second block node and storing at least one page that matches between the received block and the at least two identical blocks in a second page node of the de-duplication engine.