Pre-calculated Hashes in Distributed Datastores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed datastores face inefficiencies in CPU usage and network bandwidth due to the need for hash calculation and data deduplication across multiple layers of storage, particularly in systems like vSAN, where data is not hashed before transfer to the capacity tier, leading to increased CPU costs and network bandwidth consumption during deduplication and replication operations.
Innovation Solution
Implementing an efficient hashing process that calculates hash values for data blocks before storage, generates metadata, and stores both data and metadata in the performance tier, allowing for compressed data to be written as full stripes to the capacity tier, reducing CPU costs and network bandwidth by eliminating the need for individual hashing during deduplication and replication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If data is not hashed before transfer to capacity tier, then network bandwidth is consumed during deduplication operations, but CPU costs increase due to individual hashing during deduplication
Solution Approach 1:
The patent applies preliminary action by calculating and storing hash values for data blocks before they are transferred to the capacity tier. This pre-computation eliminates the need for individual hashing operations during deduplication, reducing both network bandwidth consumption (by enabling efficient hash comparison) and CPU costs (by avoiding redundant hashing). The hash values are stored in a hash table that can be quickly queried during deduplication operations.
2Productivity
If hash values are pre-calculated and stored, then CPU costs and network bandwidth are reduced, but device complexity increases due to additional metadata storage
Solution Approach 1:
The patent introduces an intermediary hash table data structure that stores pre-calculated hash values. This hash table acts as a mediator between the data blocks and the deduplication process, enabling efficient comparison and identification of duplicate blocks without requiring complex real-time hashing operations. The hash table simplifies the deduplication logic while maintaining improved productivity.
Data Source
AI summary
A method for generating one or more hashes for one or more data blocks is provided. The method receives a data block to write on at least one physical disk of a set of physical disks associated with a set of host machines. The method then calculates a hash for the received data block and writes a first entry to a data log in a cache disk, the first entry comprising a first header and data indicative of the received block, the first header comprising the hash. The method further writes the data to the at least one physical disk as part of data blocks of a stripe, and stores the hash in a summary block on the at least one physical disk. The summary block is associated with the data blocks of the stripe stored on the at least one physical disk.