Distributed Storage Sub-block Hashing for Data Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face challenges in providing end-to-end data protection while efficiently handling small reads and writes, as they often require transmitting full blocks of data and recalculating hashes, which increases computation and bandwidth usage.
Innovation Solution
The system splits data blocks into sub-blocks, generates sub-block error detection hashes, and transmits only the necessary sub-blocks and hashes, allowing for efficient data protection by concatenating and recalculating hashes only when necessary, reducing computation and bandwidth requirements for small operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full block transmission is used for small reads and writes, then data integrity is ensured, but bandwidth consumption increases
Solution Approach 1:
The patent divides a data block into multiple sub-blocks and generates separate error detection hashes for each sub-block. When a small read or write operation occurs, only the specific sub-blocks involved in the operation need to be transmitted and verified, rather than transmitting the entire block. This segmentation allows the system to maintain data integrity through hash verification while significantly reducing the amount of data transmitted over the network.
2Reliability
If full block hash re-computation is performed for small updates, then data corruption detection is maintained, but computation overhead increases
Solution Approach 1:
The patent computes error detection hashes at the sub-block level rather than requiring full block hash re-computation. When a small portion of a block is updated, only the hashes of the modified sub-blocks need to be recalculated, while the hashes of unchanged sub-blocks remain valid. This dramatically reduces the computational overhead for small updates while maintaining the ability to detect data corruption through hash verification.
3Productivity
If sub-block level processing is implemented, then computation and bandwidth are reduced for small operations, but system complexity increases
Solution Approach 1:
The patent introduces sub-block segmentation with error detection hashes to enable efficient small read/write operations. The system maintains a mapping between sub-blocks and their corresponding hashes, allowing selective processing. While this adds some complexity in terms of managing sub-block identifiers and partial hash computations, the overall system complexity remains manageable because the fundamental block structure is preserved and the hash computation logic is extended in a straightforward manner to the sub-block level.
Data Source
AI summary
A computer program product, system, and method for generating coded fragments comprises receiving, at a first node of a distributed storage system, an I/O request to write a block; splitting the block into a plurality of sub blocks; generating a sub block error detection hash for each of the sub blocks; sending the block and the sub block error detection hashes to a second node within the distributed storage system; and storing the block and the sub block error detection hashes to a storage device at the second node.