Hierarchical Object Validation in Data Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems lack effective methods to validate filesystem metadata structures as a whole, as current methods either require reading the entire structure for changes or provide weak protection against errors.
Innovation Solution
Implementing multiple levels of validation by associating each chunk of an object with a chunk validator, using reversible operations to compute and update an object validator, which can be stored and used to validate the object without recalculating the entire structure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a checksum of the entire structure is kept to validate the structure as a whole, then validation coverage is improved, but computational overhead increases because the entire structure must be read whenever any part changes
Solution Approach 1:
The patent divides the filesystem metadata structure into multiple blocks, where each block has its own CRC validator. This segmentation allows individual blocks to be validated and updated independently without requiring reading or validation of the entire structure, thus reducing computational overhead while maintaining validation coverage.
Solution Approach 2:
The patent introduces a hierarchical validation dimension by combining individual block CRCs into a parent CRC for each block group, and ultimately into a root CRC. This multi-level hierarchical structure enables efficient validation at different granularities, allowing validation of the entire structure through the root CRC while enabling selective validation of individual blocks when needed.
2Productivity
If block-level CRC validation is used, then computational efficiency is improved, but validation reliability deteriorates because valid blocks may combine to form an invalid structure
Solution Approach 1:
The patent merges individual block CRC validators into hierarchical parent CRCs that validate combinations of blocks. Each parent CRC is computed from the CRCs of its child blocks, creating a validation hierarchy where the root CRC validates the entire structure. This merging ensures that even if individual blocks are valid, their combination is also validated, preventing invalid structure formation while maintaining computational efficiency through selective validation.
3Measurement precision
If the entire structure is read to calculate a new CRC whenever any part changes, then validation accuracy is improved, but system performance deteriorates due to full recalculations
Solution Approach 1:
The patent performs preliminary validation by pre-computing and storing CRC validators for each block and hierarchical level during structure creation or modification. When changes occur, only the affected block's CRC and the necessary parent CRCs need to be updated, not the entire structure. This preliminary action at the block level enables efficient incremental validation while maintaining accuracy.
Solution Approach 2:
The patent implements dynamic validation where the validation scope adapts to the nature and location of changes. Instead of static full-structure validation, the system dynamically determines the minimal validation scope needed based on which blocks have changed, allowing validation to be as comprehensive as needed while being as efficient as possible.
Data Source
AI summary
Objects stored in a storage system (such as a file server system) are protected by multiple levels of validation. Each chunk of an object is associated with a chunk validator, and an object validator is computed for the object based on the chunk validators. The object validator is stored in the storage system and may be used at various times to validate the object, for example, upon a startup of the storage system, upon taking a checkpoint or “snapshot” of the status of the storage system, or at other appropriate times.