Duplicate Data Detection Using Recent-Read Bloom Filter Signatures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies in preventing duplicate data entries, leading to unnecessary storage consumption and potential data redundancy.
Innovation Solution
Implementing a storage array controller that calculates signatures for incoming data and compares them against a recently read signature buffer to identify and prevent duplicate data entries, utilizing hash functions and bit-by-bit comparisons to ensure accurate duplication detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If traditional duplicate detection methods are used, then data redundancy is reduced, but storage performance and latency are degraded due to full buffer scans
Solution Approach 1:
The patent segments the duplicate detection process into two distinct phases: a quick filter phase using a Bloom filter that scans only recently read data, and a verification phase using a full buffer scan. This segmentation allows the system to quickly eliminate obvious non-duplicates without performing complete buffer scans for every write operation, thereby maintaining data redundancy reduction while improving storage performance.
Solution Approach 2:
The patent performs preliminary duplicate detection by maintaining a Bloom filter populated with signatures of recently read data. Before performing a complete buffer scan, the system first checks this pre-prepared Bloom filter to quickly identify potential duplicates. This preliminary action reduces the number of complete buffer scans needed, improving storage performance while still effectively reducing storage redundancy.
2Measurement precision
If full buffer scans are performed for every write operation, then duplicate detection accuracy is improved, but storage latency increases
Solution Approach 1:
The patent segments duplicate detection into a fast preliminary check using the Bloom filter and a slower verification scan. The Bloom filter provides quick, low-latency filtering that catches most non-duplicates without requiring full buffer scans, thereby reducing storage latency while maintaining acceptable duplicate detection accuracy through the subsequent verification phase.
Solution Approach 2:
The patent applies partial action by performing a Bloom filter check on all write operations (excessive for accuracy alone) combined with selective full buffer scans only when the Bloom filter indicates a potential duplicate. This partial approach maintains high duplicate detection accuracy by verifying Bloom filter positives while avoiding unnecessary full scans for clear negatives, thus reducing storage latency.
3Measurement precision
If Bloom filter false positives are investigated with full scans, then duplicate detection precision is improved, but storage performance deteriorates
Solution Approach 1:
The patent segments the investigation of Bloom filter matches into two stages: a quick signature comparison stage that checks only the hash signatures, and a slower full data comparison stage performed only when signatures match. This segmentation ensures that the majority of operations (signature comparisons) remain fast, preserving storage performance, while full scans are reserved only for cases where precision is critical.
Solution Approach 2:
The patent applies partial action by performing full buffer scans only partially - specifically, only when the Bloom filter returns a positive result indicating a potential duplicate. For the majority of cases where the Bloom filter returns negative, no full scan is performed. This selective approach maintains high duplicate detection precision for potential duplicates while preserving overall storage performance by avoiding unnecessary full scans.
Data Source
AI summary
Deduplicating data based on recently reading the data, including: determining whether a calculated signature for write data matches a particular signature corresponding to data that was recently read from the storage device, wherein the signature is calculated using the write data as input; and after determining that the calculated signature for the write data matches the particular signature, obtaining the data that was recently read and comparing the data that was recently read to the write data.