Storage System Deduplication Based on Data Duplicability Likelihood
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Systems that perform deduplication operations often consume excessive resources by checking for duplicate data units regardless of their likelihood of duplication, particularly for data units with low duplicability, such as database data and application-generated data.
Innovation Solution
Implementing a system that determines the likelihood of duplicability for data units based on their type and transmits this information with write requests to storage nodes, allowing for targeted deduplication operations only when the likelihood exceeds a threshold, thereby conserving system resources and reducing the need for extensive hash value storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deduplication operations are performed for every write request regardless of data type, then storage efficiency is improved through duplicate detection, but system resource consumption increases excessively
Solution Approach 1:
The patent applies local quality by treating different data types differently in the deduplication process. Data units are classified into categories (e.g., database data, application-generated data, file system data) and each category receives appropriate deduplication treatment based on its duplicability characteristics. This selective approach ensures that resources are concentrated on data types that benefit most from deduplication while avoiding unnecessary processing of data types with low duplicability.
Solution Approach 2:
The patent implements partial action by performing deduplication operations only on a subset of data units that are determined to have high duplicability. The system calculates a duplicability metric for each data unit and selectively applies deduplication only when this metric exceeds a threshold, rather than universally applying deduplication to all data units. This partial application of the deduplication process optimizes the balance between storage efficiency and resource consumption.
2Measurement precision
If hash values for all data units are stored to enable duplicate detection, then duplicate detection accuracy is improved, but storage requirements increase
Solution Approach 1:
The patent applies local quality to the hash storage mechanism by storing hash values selectively based on data unit characteristics. Instead of uniformly storing hashes for all data units, the system stores hashes only for data units that are classified as having high duplicability potential. This selective hash storage maintains duplicate detection accuracy for relevant data types while significantly reducing the overall storage requirements for the hash table.
Solution Approach 2:
The patent implements partial action in hash value storage by calculating and storing hashes only for a portion of data units—specifically those with high duplicability metrics. The system partially applies the hash storage operation based on the duplicability assessment, avoiding the excessive storage of hashes for data units that are unlikely to be duplicated, thereby optimizing the trade-off between detection accuracy and storage consumption.
3Reliability
If deduplication checks are performed for all data units, then storage optimization is improved, but processing time increases
Solution Approach 1:
The patent applies local quality by applying different processing treatments to different data units based on their duplicability characteristics. High-duplicability data units undergo full deduplication checks with hash calculation and comparison, while low-duplicability data units receive streamlined processing or are bypassed entirely. This differentiated approach optimizes storage optimization for data types that benefit from it while minimizing processing time overhead.
Solution Approach 2:
The patent implements partial action by performing complete deduplication checks only on a subset of data units that are identified as having high duplicability. The system partially applies the full deduplication process based on preliminary duplicability assessment, avoiding the time-consuming full checks for data units with low duplicability potential, thus reducing overall processing time while maintaining storage optimization benefits where applicable.
Data Source
AI summary
Deduplication operations can be managed based on a likelihood of duplicability. For example, a computing device can generate, by a container of a storage system, an indication of duplicability corresponding to a likelihood of duplicability for the data unit in the storage system. The computing device can transmit the indication of duplicability to a storage node of the storage system for performing an operation based on the indication of duplicability.