Hash-Based Replication Using Short Hash Handles
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems, particularly primary storage systems, face challenges in supporting content-based storage with its advantages due to intensive computational load, block fragmentation, loss of sequential data location, and algorithmic difficulties in distributing tasks across processing and storage elements, leading to performance degradation.
Innovation Solution
The implementation of a content addressable storage system using short hash handles generated from hash signatures, allowing for efficient data replication and deduplication by separating control and data planes, distributing computational tasks, and optimizing data flow across multiple nodes, thereby maintaining performance and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If content-based storage is implemented, then storage capacity efficiency is improved, but computational load increases
Solution Approach 1:
The patent segments the computational task of content addressing by dividing data into fixed-size blocks and generating hash signatures for each block independently. This allows parallel processing of hash generation across multiple blocks, distributing the computational load while maintaining content-based storage efficiency. The hash signature acts as a compact representation that enables deduplication without requiring comparison of entire data sets.
Solution Approach 2:
The patent introduces hash signatures as an intermediary between the original data and the storage system. Instead of directly comparing data blocks for deduplication, the system uses hash signatures as a lightweight proxy that uniquely represents each data block. This intermediary reduces computational complexity from O(n) data comparison to O(1) hash lookup, significantly lowering the computational burden while preserving storage efficiency.
2Measurement precision
If large blocks are broken into smaller blocks for content addressing, then content addressing granularity is improved, but performance degrades due to block fragmentation
Solution Approach 1:
The patent changes the parameter of block size from variable to fixed, establishing a uniform block size for all data operations. This fixed block size eliminates the performance degradation associated with variable-length block fragmentation while maintaining fine-grained content addressing capability. The system achieves precise content addressing through hash signatures of fixed-size blocks, decoupling addressing granularity from physical block size.
3Speed
If address-based mapping is used, then sequential data location is maintained, but storage capacity efficiency decreases due to duplicate data
Solution Approach 1:
The patent uses hash signatures as copies or representations of the actual data blocks. Instead of storing multiple copies of identical data blocks (which wastes capacity), the system stores a single copy and uses hash signatures to reference it. When data needs to be accessed, the hash signature quickly identifies the unique data block location, maintaining fast access while eliminating redundant storage of duplicate content.
4Adaptability or versatility
If content-based mapping is distributed across multiple processing elements, then scalability is improved, but maintaining single content-addressing space becomes more difficult
Solution Approach 1:
The patent adds a new dimension to the content-addressing space by incorporating hash signature space as a separate layer. Instead of trying to maintain a single unified addressing space across distributed elements, the system uses hash signatures as a universal key space that spans all distributed storage elements. Each processing element manages its own data blocks but all elements participate in the same hash signature namespace, enabling scalable distribution while maintaining a unified content-addressing interface.
Data Source
AI summary
In one aspect, a method includes sending a short hash handle from a production site to a replication site, the short hash handle being generated using a hash signature and being smaller than the hash signature, determining at the replication site whether the short hash handle exists at the replication, reading and sending the hash signature associated with the short hash handle if the short hash handle exists at the replication site, determining at the replication site whether the hash signature exists at the replication and emulating a write operation by associating the hash signature to an address in a second storage volume at the replication site if the hash signature exists at the replication site. The hash signature is generated using a data block from a first storage volume at the production site.