Dependency Tree for Silent Data Corruption Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data scrubbing techniques in distributed storage systems are inefficient, as they often scrub unnecessary physical locations or duplicate data blocks, leading to wasteful data correction processes.
Innovation Solution
A method that uses a dependency tree to identify and correct corrupted data blocks by sequentially scrubbing snapshots of a file, reducing duplicate scrubbing by only targeting blocks that have changed between snapshots, and propagating corrections through the dependency tree.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all physical locations are scrubbed indiscriminately, then data corruption is detected, but unnecessary physical locations are scrubbed leading to inefficiency
Solution Approach 1:
The patent extracts only the changed data blocks between snapshots using a dependency tree, rather than scrubbing all physical locations. The dependency tree identifies which data blocks differ between snapshots, allowing the system to focus scrubbing efforts exclusively on those changed blocks, thereby eliminating unnecessary scrubbing operations while maintaining corruption detection capability.
2Reliability
If data of every file is scrubbed, then data corruption is detected, but data blocks shared by multiple files are scrubbed multiple times leading to inefficiency
Solution Approach 1:
The patent segments the scrubbing task by file and snapshot, using dependency trees to identify only the specific data blocks that changed in each snapshot. This segmentation prevents duplicate scrubbing of shared data blocks because each block is scrubbed only when it appears in a dependency tree as a changed block, eliminating redundant operations across multiple files.
3Reliability
If conventional file-based scrubbing is used, then data corruption is detected, but the process is inefficient due to duplicate scrubbing of shared blocks
Solution Approach 1:
The patent performs preliminary action by building dependency trees that track which data blocks changed between snapshots before the scrubbing operation. This preliminary structuring of data block relationships allows the scrubbing process to efficiently identify and target only the necessary blocks, reducing both duplicate scrubbing and overall process complexity while maintaining comprehensive error detection.
Data Source
AI summary
A file and a sequence of snapshots of the file are stored in a storage device. The sequence of snapshots includes sequentially captured copies of earlier states of the file. A dependency tree indicating data blocks that are different between a given snapshot and a previous snapshot in the sequence of snapshots is stored in the storage device. The sequence of snapshots is sequentially scrubbed, beginning with an earliest snapshot in the sequence of snapshots. When scrubbing a snapshot, each of the data blocks identified in the dependency tree as being different than the data blocks of a previous snapshot in the sequence of snapshots are scrubbed. If a corrupted data block is detected, a determination of which later snapshots include the corrupt data block is made based on the dependency tree and the corrupted data blocks are corrected.