Deduplicated Block Storage Data Integrity via Instance Redundancy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication solutions fail to effectively mitigate data corruption due to data rot in storage devices, which can lead to incorrect data retrieval and corruption of all files pointing to a corrupted block, especially in long-term storage solutions like data backups.
Innovation Solution
The solution involves maintaining metrics on instances and pointers of deduplicated blocks and applying rules based on these metrics to ensure data integrity by creating multiple instances of blocks, rewriting existing blocks with new copies when necessary, and using integrity rules to determine when to store new instances, thereby mitigating corruption and data loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data deduplication is implemented to improve storage utilization, then storage capacity requirements are reduced, but data integrity is compromised due to data rot affecting all files pointing to a corrupted block
Solution Approach 1:
The patent creates multiple copies (instances) of deduplicated data blocks and stores them across different storage devices. When a block is deduplicated, instead of storing a single reference, the system maintains multiple instances of the actual data block. This allows the system to retain storage efficiency through deduplication while protecting against data rot by having redundant copies available if one instance becomes corrupt.
Solution Approach 2:
The system proactively creates redundant instances of data blocks before corruption occurs. By maintaining multiple copies in advance and monitoring their integrity, the system prepares cushioning measures against potential data rot. When corruption is detected in one instance, the system can immediately switch to a healthy copy without data loss.
2Reliability
If multiple instances of data blocks are created to prevent data corruption, then data integrity is improved, but storage capacity requirements increase
Solution Approach 1:
The patent implements a selective approach where not all deduplicated blocks receive multiple instances. Instead, the system applies instance creation based on risk assessment factors such as the criticality of the data, the age of the stored blocks, and the reliability of the storage devices. This partial action approach maintains data integrity for high-risk blocks while avoiding unnecessary storage overhead for low-risk data.
Solution Approach 2:
The system dynamically adjusts the number of instances created for each data block based on changing parameters such as data importance, storage device health, and time since last verification. Critical data may receive more instances while less critical data receives fewer instances, allowing the system to optimize the balance between data protection and storage capacity utilization.
Data Source
AI summary
Methods and apparatuses for maintaining data integrity in deduplicated storage environments. A processor receives a request to write a first block of data to a storage device. A processor compares the first block of data to a second block of data, wherein the second block is stored on the storage device. A processor writes the first block of data to the storage device based, at least in part, on the first block of data matching the second block of data and an amount of pointers to the second block of data being above a predetermined amount.