Cloud Storage Deduplication via Hash Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current file tiering methods in cloud storage systems fail to efficiently identify and prevent duplicate files from being stored, leading to wastage of cloud storage space and unnecessary resource consumption.
Innovation Solution
Implementing a deduplication technique that computes and compares hash values of files to identify duplicates, creating a single cloud object for duplicate files and using stub files to point to this object, thereby eliminating redundant storage and optimizing cloud object management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If all files are tiered to cloud storage without deduplication, then cloud storage capacity is consumed, but duplicate files waste crucial cloud storage space
Solution Approach 1:
The system performs preliminary deduplication actions by computing hash values and comparing files before tiering to cloud storage. This preliminary identification of duplicates prevents wasteful storage operations, ensuring that only unique files consume cloud storage capacity.
Solution Approach 2:
The system transforms the file identification approach by using hash value computation as a parameter change mechanism. Instead of comparing entire files, the system computes hash values (parameter transformation) to efficiently identify duplicates, thereby reducing cloud storage consumption without sacrificing data integrity.
2Quantity of substance
If hash values are computed and compared for all files, then duplicate files are identified and prevented from being tiered, but processing time and computational resources are consumed
Solution Approach 1:
The system replaces the mechanical approach of comparing entire file contents with a computational hash value comparison system. This substitution dramatically reduces processing time and computational resources while maintaining accurate duplicate detection, thus improving cloud storage efficiency without excessive time loss.
Solution Approach 2:
The system changes the parameter being processed from entire file data to condensed hash values. This parameter transformation enables rapid comparison operations that identify duplicates efficiently, balancing cloud storage space optimization with acceptable processing time requirements.
Data Source
AI summary
A method comprises computing a hash value for at least one file of a plurality of files in connection with a file tiering operation of the plurality of files from a source storage location to a target storage location. The hash value is compared to one or more hash values corresponding to one or more other files of the plurality of files. Based at least in part on the comparing, a determination is made whether the hash value is the same as any of the one or more hash values. The at least one file is identified as a duplicate file in response to an affirmative determination. The target storage location comprises a cloud storage platform.