Probabilistic Cardinality Estimator for Data Deduplication Parameter Computation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large data storage systems with multiple data volumes, determining deduplication parameters such as deduplication ratio, reclaimable physical blocks, and similarity between data volumes is challenging due to the complexity of tracking unique data values and managing storage space efficiently while maintaining system availability.
Innovation Solution
The implementation of a probabilistic cardinality estimator, based on algorithms like HyperLogLog, allows for online computation of deduplication parameters by tracking unique data values through fingerprints and SBN hash values, enabling efficient deduplication and storage management without disrupting system operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional methods are used to track unique data values and compute deduplication parameters in large data storage systems, then measurement precision of deduplication parameters can be achieved, but device complexity and difficulty of detecting and measuring increase significantly
Solution Approach 1:
The patent introduces probabilistic cardinality estimators as intermediary components that approximate the number of unique data values without requiring complete tracking. These estimators act as mediators between the raw data streams and the deduplication parameter computations, providing sufficiently accurate measurements while avoiding the complexity of exhaustive tracking mechanisms.
Solution Approach 2:
The patent employs lightweight, disposable data structures such as Bloom filters and probabilistic counters that can be quickly created and discarded. These temporary structures enable efficient online computation of deduplication parameters without requiring persistent, complex tracking infrastructure, thus reducing overall system complexity while maintaining measurement precision.
2Measurement precision
If complete tracking of unique data values is implemented, then measurement precision of deduplication parameters improves, but loss of time and productivity decrease due to computational overhead
Solution Approach 1:
The patent applies partial action by using probabilistic methods that provide sufficiently accurate estimates of unique data values without performing complete exhaustive tracking. The probabilistic cardinality estimators compute approximate counts that are accurate enough for deduplication parameter calculations, avoiding the time-consuming nature of complete tracking while maintaining acceptable precision.
Solution Approach 2:
The patent enables continuous online computation of deduplication parameters by implementing incremental updates to probabilistic estimators as data arrives. This allows the system to continuously maintain accurate (within probabilistic bounds) measurements of deduplication parameters without periodic batch processing interruptions, thus reducing time loss while maintaining precision.
3Productivity
If probabilistic cardinality estimators are used for online computation, then productivity and system availability improve, but measurement precision of deduplication parameters decreases slightly
Solution Approach 1:
The patent adjusts the parameters of probabilistic estimators (such as hash function counts, Bloom filter sizes, and probability thresholds) to optimize the balance between precision and productivity. By carefully tuning these parameters, the system achieves sufficiently accurate deduplication parameter measurements while maintaining high online computation efficiency and system availability.
Solution Approach 2:
The patent implements dynamic adaptation of probabilistic estimator parameters based on observed data characteristics and system performance. The system can adjust the level of probabilistic approximation dynamically, increasing precision when needed and maximizing productivity when data patterns are well-understood, thus optimizing the trade-off between measurement accuracy and computational efficiency.
4Measurement precision
If detailed tracking of data values is performed, then measurement precision improves, but storage overhead increases
Solution Approach 1:
The patent uses lightweight, temporary data structures such as probabilistic counters and compact Bloom filters that occupy minimal storage space. These disposable structures enable tracking of unique data values with acceptable precision while requiring far less storage overhead compared to persistent, detailed tracking mechanisms. The structures can be discarded after computation, freeing storage resources.
Solution Approach 2:
The patent extracts only the essential information needed for deduplication parameter computation from the complete data values. Instead of storing and tracking full data values, the system extracts and processes only sufficient statistics (such as hash values and presence indicators) using probabilistic methods, thereby achieving adequate tracking precision with minimal storage overhead.
Data Source
AI summary
In some examples, a system computes respective values for corresponding data value indicators added to and removed from a deduplication data store in which duplicated data values have been eliminated, where each respective data value indicator of the data value indicators represents presence of a unique data value in the deduplication data store. The system updates an estimator based on the respective values, to reflect an addition of a first data value indicator to the deduplication data store and a removal of a second data value indicator from the deduplication data store. The system computes, using the updated estimator, a parameter relating to data deduplication at the deduplication data store.