Two-Layer Deduplicated Cloud Object Corruption Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deduplicated storage systems lack an efficient mechanism to detect data corruption in cloud storage, leading to potential silent corruption of cloud objects, which can affect hundreds or thousands of files, and existing methods are costly and time-consuming, requiring the reading of large amounts of data from the cloud.
Innovation Solution
A two-layered data corruption detection method is introduced, where the first layer scans metadata areas for corruption, and the second layer scans data areas only if metadata is clean, using a progressive and coverage-based approach to read segments incrementally and stop upon detecting corruption, thereby reducing the amount of data read from the cloud.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional full object scanning methods are used to detect corruption, then detection thoroughness is improved, but time consumption and cloud I/O costs increase significantly
Solution Approach 1:
The patent divides cloud objects into segments and introduces a two-layer detection mechanism: Layer 1 scans only metadata areas of segments to quickly identify potentially corrupted objects, while Layer 2 performs full scanning only on objects flagged by Layer 1. This segmentation approach maintains thorough detection capability while dramatically reducing the time and cloud I/O costs for the majority of clean objects.
2Measurement precision
If full cloud object data is read for corruption detection, then detection accuracy is improved, but cloud egress costs and network bandwidth consumption increase
Solution Approach 1:
The patent extracts and scans only the metadata areas of cloud object segments in Layer 1, rather than reading the entire object data. This extraction approach allows the system to identify potentially corrupted objects with minimal cloud egress, and then perform full accuracy scanning only on the small subset of objects that require it, thereby maintaining detection accuracy while dramatically reducing cloud egress costs.
3Reliability
If comprehensive corruption scanning is performed on all cloud objects, then data integrity assurance is improved, but system productivity and operational efficiency deteriorate
Solution Approach 1:
The patent performs preliminary scanning of metadata areas in Layer 1 before conducting full object scanning. This preliminary action identifies and filters out the vast majority of clean objects, allowing the system to maintain comprehensive integrity assurance for objects that require full scanning while preserving overall system productivity by avoiding unnecessary full scans of clean objects.
Data Source
AI summary
A method of detecting object corruption of deduplicated cloud objects includes a two-layered approach. In a first layer, only metadata areas from segments of a deduplicated cloud object are processed in order to detect corruptions in the metadata. If no corruption is detected in the metadata, a second layer analysis can be performed. The second layer analysis includes a progressive scanning of data only on objects that were not found to be corrupted in the first layer analysis.