CRC File Verification for Missing Reed-Solomon Code Blocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed file systems using Cauchy-Reed-Solomon encoding, verifying and computing checksums for all blocks can be resource-intensive, especially when blocks are missing or corrupted, requiring extensive data transfer and computation.
Innovation Solution
A method where each storage server sends checksums of underlying data block packets, allowing the curator server to compute missing checksums without reading all surviving blocks, reducing data transfer, memory requirements, and computational overhead by using cyclic redundancy checks (CRCs) and XOR operations on zero-padded CRCs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the curator server reads all surviving blocks to compute checksums, then checksum verification accuracy is improved, but bandwidth consumption and data transfer increase significantly
Solution Approach 1:
The patent extracts only the essential checksum information from the blocks rather than reading the entire block contents. The curator server obtains checksums directly from storage servers, which are pre-computed and stored, allowing verification without transferring actual data blocks across the network.
Solution Approach 2:
Checksums are pre-computed and stored by storage servers before the curator server needs to verify them. This preliminary computation eliminates the need for the curator server to read and compute checksums from actual block data during verification, reducing bandwidth consumption.
2Reliability
If the curator server reads all surviving blocks to compute checksums, then complete verification is improved, but memory requirements increase significantly
Solution Approach 1:
The patent extracts only the checksum values needed for verification from the block data. Instead of loading entire blocks into memory, the system works with compact checksum representations, dramatically reducing memory requirements while maintaining verification completeness.
3Measurement precision
If the curator server computes checksums from all surviving blocks, then verification accuracy is improved, but CPU computation time increases significantly
Solution Approach 1:
Storage servers pre-compute and store checksums for their respective blocks before the curator server needs to perform verification. This shifts the computational burden to a preliminary stage, allowing the curator server to perform only lightweight verification operations, significantly reducing computation time.
4Reliability
If checksums are computed for all blocks including missing ones, then data integrity verification is improved, but data transfer requirements increase
Solution Approach 1:
The patent extracts and transfers only checksum metadata rather than actual block data. The curator server receives checksums from storage servers via efficient key-value store operations, avoiding the need to transfer terabytes of actual data while still enabling complete data integrity verification.
Data Source
AI summary
A system and method are provided for verifying and reconstructing a file using cyclic redundancy checks (CRCs). A writer client sends a file, which includes data chunks including one or more data blocks, and code chunks including one or more code blocks. One or more storage devices store the packets underlying the one or more data blocks and their respective packet CRCs. A curator server stores a first CRC for each complete data chunk and each complete code chunk and determines that an incomplete code chunk has one or more missing code blocks. The curator server generates a second CRC for each of the complete data chunks and the complete code chunks using the packet CRCs, and compares the generated second CRCs with the respective stored first CRCs. The curator server generates a missing chunk CRC for the incomplete code chunk using the packet CRCs.