Deduplicated Object Storage Verification via Similarity Group Path
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplication systems face inefficiencies in storage reduction due to chunk size trade-offs and file division methods, which affect deduplication efficiency and require costly read operations to verify data integrity.
Innovation Solution
A system and method for detecting corruption in deduplicated object storage systems using a similarity group verification path, minimizing costly read operations by employing a controller and worker node collaboration to verify object, slice recipe, and similarity group relationships, ensuring data integrity without extensive data retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional verification methods are used to verify data integrity in deduplicated storage systems, then data integrity can be confirmed, but costly read operations are required to retrieve and verify all data blocks
Solution Approach 1:
The system performs preliminary verification by checking metadata (object recipes, slice recipes, similarity groups) before actual data retrieval. This preliminary action identifies which data blocks need verification and prepares verification information in advance, reducing the need for costly read operations during actual verification.
Solution Approach 2:
The patent introduces metadata (object recipes, slice recipes, similarity groups) as intermediaries between the verification request and the actual data blocks. These metadata structures contain verification information that can be checked without retrieving the actual data, serving as a mediator that reduces direct read operations on stored objects.
2Quantity of substance
If chunk size is reduced to improve deduplication efficiency, then more duplicates can be identified, but meta-data size, deduplication time, and fragmentation increase
Solution Approach 1:
The patent segments metadata into hierarchical structures (object recipes, slice recipes, similarity groups) that organize verification information in a structured manner. This segmentation allows efficient management and retrieval of verification data without requiring linear increases in metadata size as chunk size decreases.
Solution Approach 2:
The patent introduces a hierarchical dimension to metadata organization, moving from flat metadata structures to multi-level hierarchical structures. This dimensional change allows the system to manage increased metadata complexity through structured organization, enabling efficient verification even with smaller chunk sizes that generate more metadata.
Data Source
AI summary
Described is a system for detecting corruption in a deduplicated object storage system accessible by one or more microservices while minimizing costly read operations on objects. A similarity group verification path is selected by a controller module based upon detection of an object storage memory size condition. The similarity group verification path includes controller phases to verify whether objects have been corrupted without having to incur costly read operations.