Predictive Probabilistic Deduplication for Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication methods in storage systems are resource-intensive due to the high cost of calculating hash values and maintaining key-value tables, as they often write and read data to check for duplicates, leading to unnecessary insertions and deletions.
Innovation Solution
The predictive probabilistic deduplication method samples incoming data blocks to predict whether they are unique or duplicate, allowing only predicted duplicates to be inserted into the key-value table, reducing the need for costly writes and reads, and conserving resources by minimizing updates to the key-value table.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data are written to storage without checking for duplicates first, then write performance is improved, but resource consumption increases due to subsequent hash calculations and key-value table updates
Solution Approach 1:
The patent applies preliminary action by sampling a subset of incoming data blocks before writing them to storage to predict whether they are duplicates. This allows the system to make informed decisions about which blocks need hash calculation and key-value table updates, avoiding unnecessary resource consumption while maintaining write performance for unique blocks.
Solution Approach 2:
The patent uses partial action by sampling only a subset of incoming data blocks rather than checking all blocks. This partial sampling approach reduces the computational overhead while still providing sufficient accuracy to distinguish between unique and duplicate blocks, thereby optimizing the trade-off between write performance and resource consumption.
2Reliability
If all incoming data blocks are checked for duplicates by calculating hashes and updating key-value tables, then deduplication accuracy is improved, but processing time and resource consumption increase
Solution Approach 1:
The patent performs preliminary sampling of incoming data blocks to predict duplicate status before performing expensive hash calculations and key-value table updates. This preliminary action filters out blocks that are likely duplicates, allowing the system to maintain high deduplication accuracy while reducing processing time by avoiding unnecessary operations on blocks that can be confidently identified as unique.
Solution Approach 2:
The patent implements feedback by using the sampling results to guide subsequent processing decisions. The sampling process provides feedback about the likelihood of duplicates in incoming blocks, which then influences whether the system performs full hash calculations and key-value table updates, thereby optimizing the balance between deduplication accuracy and processing time.
3Reliability
If the key-value table is updated frequently to track duplicate blocks, then deduplication tracking is improved, but device complexity and operational overhead increase
Solution Approach 1:
The patent applies partial action by updating the key-value table only for blocks that are predicted to be duplicates based on sampling, rather than updating it for all incoming blocks. This selective updating approach reduces operational overhead and simplifies system operations while maintaining reliable deduplication tracking for the blocks that actually need to be tracked.
Data Source
AI summary
Examples perform predictive probabilistic deduplication of storage, such as virtualized or physical disks. Incoming input/output (I/O) commands include data, which is written to storage and tracked in a key-value store. The key-value store includes a hash of the data as the key, and a reference counter and the address of the data as the value. When a certain percentage of sampled incoming data is found to be duplicate, it is predicted that the I/O commands have become not unique (e.g., duplicate). Based on the prediction, subsequent incoming data is not written to storage, and instead the reference counter associated with the hash of the data is incremented. In this manner, predictions on the uniqueness of future data is made based on previous data, and extraneous writes and deletions from the chunk store are avoided.