Disk Image Deduplication Using In-Memory Seed Hash Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplication of virtual-machine disk images is resource-intensive due to time-consuming disk accesses and high memory requirements for comparing and storing hashes, which limits the efficiency of data storage and transfer.
Innovation Solution
Selecting 'seed' clusters, typically aligned with file starts, for initial comparison in memory, and using a bitmap or index to identify these clusters, allowing for reduced disk swaps and efficient matching of neighboring clusters with pointers, thereby reducing storage and bandwidth needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all clusters are hashed and stored in memory for comparison, then deduplication accuracy is improved, but memory requirements and disk swaps increase
Solution Approach 1:
The patent segments the disk image into clusters and further segments the comparison process by identifying and comparing only 'seed' clusters (aligned with file starts) initially, then extending to neighboring clusters only when matches are found. This segmentation allows selective memory usage rather than loading all cluster hashes into memory simultaneously.
Solution Approach 2:
The patent applies partial action by hashing and storing only a subset of clusters (seed clusters at file starts and their neighbors) in memory for comparison, rather than all clusters. This partial approach reduces memory requirements while maintaining effective deduplication accuracy through strategic sampling of the most likely candidate clusters.
2Speed
If all clusters are loaded into memory for comparison, then comparison speed is improved, but disk access time increases
Solution Approach 1:
The patent performs preliminary action by pre-identifying seed clusters aligned with file starts and pre-computing their hashes before the main comparison process. This preliminary preparation allows the comparison to start with minimal disk accesses, as only these pre-identified seed clusters need to be loaded into memory for initial comparison, significantly reducing overall disk access time.
Solution Approach 2:
The patent extracts only the essential information needed for comparison (hashes of seed clusters and their neighbors) from the full disk image, rather than loading all cluster data into memory. This extraction approach minimizes the data transferred between disk and memory, reducing disk access time while maintaining comparison effectiveness.
3Reliability
If cluster-by-cluster comparison is performed, then deduplication completeness is improved, but processing resources increase
Solution Approach 1:
The patent applies local quality by focusing comparison resources on specific local areas (seed clusters at file starts and their neighboring clusters) rather than uniformly processing all clusters. This localized approach concentrates processing effort where file boundaries and data repetitions are most likely to occur, maintaining deduplication completeness while improving processing efficiency.
Solution Approach 2:
The patent ensures continuity of useful action by systematically processing seed clusters and their neighbors in a continuous manner, extending comparisons to neighboring clusters whenever matches are found. This continuous processing strategy ensures complete deduplication coverage without requiring random access to all clusters, maintaining both completeness and efficiency.
Data Source
AI summary
Deduplication of virtual-machine disk images and other disk images can involve identifying the first clusters in a file. The clusters are hashed. The first-in-file hashes (generated from first-in-file clusters) are stored in an in-memory index, while the full set of hashes is streamed in order to find matches with the hashes stored in the in-memory index. First-in-file hashes in the stream are compared, while other hashes in the stream are compared only if the immediately preceding hash resulted in a match. Comparing non-first-in-file hashes requires disk accesses, but since such comparisons are conditioned on first-in-file matches, there are relatively likely to result in sequences of matches. The net effect is a relatively fast deduplication with compression approaching that resulting from a full comparison of all hashes.