Similarity Based Deduplication Hybrid Cloud Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current multi-system storage environments lack support for hybrid cloud setups in similarity-based deduplication, leading to inefficient data storage and management across on-premises and off-premises systems.
Innovation Solution
A method and apparatus for similarity-based deduplication in secondary or remote storage systems that utilize rolling hash values and strong hash indices across multiple cloud environments, including public and private clouds, to identify and manage duplicate data, ensuring efficient storage by comparing hash values and migrating or importing data as necessary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If similarity based deduplication systems assume all repository data is on-premises, then the system can implement deduplication efficiently within a single location, but the system cannot support hybrid cloud environments with data distributed across multiple locations
Solution Approach 1:
The system divides the deduplication process into distinct modules: a parse module that generates rolling hash values, a comparison module that queries multiple cloud locations, and a migration module that handles data movement. This segmentation allows the system to support hybrid cloud environments while maintaining manageable complexity through modular design.
Solution Approach 2:
The deduplication system is designed to operate universally across multiple storage locations including on-premises systems, public clouds, and private clouds. The comparison module queries strong hash indices from multiple cloud locations, enabling the system to function as a universal deduplication solution that adapts to hybrid cloud architectures.
2Adaptability or versatility
If data is stored across multiple cloud locations without deduplication, then data accessibility is improved, but storage capacity utilization becomes inefficient due to duplicate data
Solution Approach 1:
The system uses hash value copying and comparison to identify duplicate data across multiple cloud locations. Instead of physically comparing entire data sets, the system copies and compares compact hash representations, enabling efficient detection of duplicates while maintaining data accessibility across distributed locations.
Solution Approach 2:
The system merges the deduplication functionality with multi-location data access by integrating the comparison module that queries strong hash indices from multiple cloud locations. This combination allows the system to simultaneously achieve data accessibility across distributed locations and efficient storage capacity utilization through duplicate identification.
3Measurement precision
If rolling hash values are compared across multiple cloud locations, then duplicate data identification is enabled, but the computational overhead and query complexity increases
Solution Approach 1:
The system uses strong hash indices as intermediary structures stored at each cloud location. Instead of directly comparing rolling hash values across all locations (which would be computationally intensive), the comparison module queries these pre-computed strong hash indices, which act as intermediaries that accelerate the duplicate identification process while maintaining accuracy.
Solution Approach 2:
Strong hash indices are pre-computed and stored at each cloud location before the deduplication query occurs. This preliminary action of pre-computing and storing hash indices enables the comparison module to quickly query and identify duplicates across multiple locations without performing complex real-time computations, thereby reducing query processing complexity.
Data Source
AI summary
For similarity based deduplication of remote data repositories, a parse module generates a rolling hash value based on a portion of an incoming stream of backup data. A comparison module compares the rolling hash value with entries stored in a rolling hash index, and in response to matching the rolling hash value with an entry in the rolling hash index, generates a strong hash value and determines if a match of the strong hash value exists in a first strong hash index. The comparison module, in response to a determination that the match does not exist in the first strong hash index, compares the strong hash value with entries in a second strong hash index in the remote data repository. A migration module, in response to a determination that the strong hash value does not match any hash entries, stores the portion of backup data as new data.