Hash-Based Data Integrity Repair in Storage Services
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems are prone to hardware and software errors leading to data corruption and loss, which can result in significant productivity and monetary losses, and existing redundancy and backup solutions often fail to detect and prevent these issues effectively.
Innovation Solution
A data storage service that utilizes collision-free hash values to detect and repair corrupted or lost data blocks by searching for duplicate blocks in a persisted index, integrating data deduplication and error detection to replace corrupted blocks with correct duplicates without user intervention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If redundancy mechanisms and backup solutions are implemented to prevent data loss, then data protection capability is improved, but system complexity and costs increase
Solution Approach 1:
The patent uses hash values as digital copies of data blocks to identify and track data. Instead of maintaining redundant physical copies of entire datasets, the system creates hash representations that serve as unique identifiers and integrity verification mechanisms, significantly reducing the complexity overhead while maintaining data protection capabilities
Solution Approach 2:
The patent replaces traditional mechanical redundancy mechanisms (physical backup copies, mirror storage) with a computational approach using hash-based identification and verification. This substitution eliminates the need for complex physical redundancy infrastructure while achieving equivalent or superior data protection through cryptographic hashing and index-based tracking
2Ease of repair
If traditional backup and restore mechanisms are used to recover corrupted data, then data recovery capability is improved, but detection precision of data corruption worsens
Solution Approach 1:
The patent implements continuous feedback mechanisms where hash values are computed and compared against stored reference hashes to detect data corruption in real-time. This feedback loop enables precise detection of corruption events by comparing current data state against known good states, significantly improving detection precision over traditional backup methods that only detect issues during restore operations
Solution Approach 2:
The patent performs preliminary hashing and verification actions on data blocks as they are written or accessed, before corruption can propagate. By pre-computing and storing hash values of known good data states, the system is able to detect corruption immediately when it occurs, rather than waiting for backup restore operations to reveal issues
3Quantity of substance
If hash-based data deduplication is implemented to reduce storage space, then storage efficiency is improved, but data integrity verification capability worsens
Solution Approach 1:
The patent combines hash-based deduplication with integrity verification mechanisms by using cryptographic hash functions that provide both compression benefits and verification capabilities. The hash structure serves dual purposes: identifying duplicate blocks for space optimization and verifying data integrity through collision-resistant properties, thereby achieving both storage efficiency and reliability simultaneously
Data Source
AI summary
The subject disclosure is directed towards a data storage service that uses hash values, such as substantially collision-free hash values, to maintain data integrity. These hash values are persisted in the form of mappings corresponding to data blocks in one or more data stores. If a data error is detected, these mappings allow the data storage service to search the one or more data stores for data blocks having matching hash values. If a data block is found that corresponds to a hash value for a corrupted or lost data block, the data storage service uses that data block to repair the corrupted or lost data block.