Hash-Based De-Duplication for Hybrid Storage Transfer Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Hybrid storage systems face challenges in efficiently transferring data from high-performance upper-level storage devices to lower-performance lower-level storage devices, particularly due to the overhead of deep compression and decompression processes, which can impact data transfer efficiency and storage resource utilization.
Innovation Solution
A method that determines a target physical block in an upper-level storage device for data transfer, calculates the compression ratio of data blocks, and performs de-duplication operations based on hash values and a de-duplication hash table, thereby reducing the amount of data to be transferred and improving storage resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If deep compression is performed on data blocks during transfer from upper-level to lower-level storage devices, then storage space utilization is improved, but data transfer efficiency deteriorates due to processing overhead
Solution Approach 1:
The patent performs de-duplication operations before data transfer by calculating hash values and comparing them with existing data in the lower-level storage device. This preliminary action identifies duplicate data blocks that can be skipped during transfer, reducing the actual data volume to be transferred while avoiding the overhead of compressing and decompressing all data blocks.
2Loss of substance
If de-duplication operations are performed on all data blocks during transfer, then data transfer volume is reduced, but processing time increases
Solution Approach 1:
The patent changes the approach from byte-level compression to hash-based de-duplication, transforming the processing parameter from continuous data transformation to discrete block identification. This allows for faster processing by using hash value comparison instead of deep compression algorithms, reducing both transfer volume and processing time.
3Speed
If hybrid storage systems use high-performance upper-level storage devices, then data access speed is improved, but storage cost increases
Solution Approach 1:
The patent segments the storage system into upper-level high-performance storage devices for frequently accessed data and lower-level cost-effective storage devices for archived or duplicate data. By identifying and eliminating duplicates before transfer, the system optimizes the distribution of data across different storage tiers, maximizing the utilization of expensive high-performance storage while reducing overall storage costs.
Data Source
AI summary
Techniques for de-duplicating data involve: determining a target physical block in a first storage device. The techniques further involve: determining a compression ratio of a target data block in a plurality of data blocks to be transferred. The techniques further involve: determining a target hash value of the target data block in response to the compression ratio being lower than a threshold compression ratio. The techniques further involve: determining a de-duplication operation for the target data block based on the target hash value and a de-duplication hash table, the de-duplication hash table storing hash values of data blocks that have been transferred from the first storage device to the second storage device. Accordingly, the amount of data that needs to be transferred can be reduced, and the storage space of the storage devices can be improved, thus increasing the resource utilization and improving the user experience.