Data Scrubbing in Cluster Storage via Hash Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional RAID solutions for data storage are prone to error propagation and physical damage, leading to data loss and inefficiencies in data protection and recovery.
Innovation Solution
A method for data scrubbing in cluster-based storage systems that involves maintaining multiple replicas of data objects across storage devices, using hash values to detect and correct corruption, and employing computational recovery processes to ensure data integrity and protection against failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional RAID solutions are used to duplicate data across redundant disk drives, then data protection against disk failures is improved, but error propagation across multiple copies worsens data integrity
Solution Approach 1:
The system segments data into multiple independent objects and stores them across different storage devices in a distributed cluster architecture. Each data object is independently managed with its own hash value, preventing error propagation between segments. This segmentation approach replaces traditional RAID's monolithic array structure with a distributed system where failures are isolated to individual devices rather than propagating across the entire array.
Solution Approach 2:
The system introduces hash values as intermediary elements between the stored data and the verification process. These hash values serve as mediators that enable detection of data corruption without requiring direct comparison of the actual data contents. The hash intermediary allows the system to identify and correct errors efficiently while maintaining data protection, resolving the contradiction between redundancy and error propagation.
2Quantity of substance
If data is edited in-place on storage devices, then storage efficiency is improved, but additional errors in data objects and nearby sectors are created
Solution Approach 1:
The system performs preliminary actions by calculating and storing hash values of data objects before they are written to or edited on storage devices. This preliminary hashing allows the system to detect any corruption that occurs during in-place editing operations. By having the hash values available beforehand, the system can identify errors and trigger recovery procedures using replicas from other storage devices, thus protecting against the corruption that inevitably occurs during in-place editing while maintaining storage efficiency.
3Ease of repair
If multiple replicas of data objects are stored across storage devices, then data recovery capability is improved, but system complexity increases
Solution Approach 1:
The system implements self-service through automated hash-based verification and error correction mechanisms. Each storage device independently maintains its own data objects and associated hash values, and the system automatically detects and corrects errors by comparing stored hash values with newly calculated ones. This self-service approach enables data recovery without manual intervention while keeping system complexity manageable through standardized, automated procedures rather than complex manual recovery processes.
Data Source
AI summary
Disclosed is the technology for data scrubbing in a cluster-based storage system. This technology allows protecting data against failures of storage devices by periodically reading data object replicas and data object hashes stored in a plurality of storage devices and rewriting those data object replicas that have errors. The present disclosure addresses aspects of writing data object replicas and hashes, checking validity of data object replicas, and performing data scrubbing based upon results of the checking.