Verification Microservice for Deduplicated Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplication systems face inefficiencies in storage reduction due to chunk size trade-offs and file division methods, which affect deduplication efficiency and increase metadata size, fragmentation, and deduplicated object storage systems struggle to verify the integrity of relationships between objects without costly read operations.
Innovation Solution
A system and method for detecting corruption in deduplicated object storage systems using a similarity group verification path, involving a controller module and worker nodes to verify object-to-slice recipe, slice recipe-to-similarity group, and similarity group-to-compression region relationships, minimizing costly read operations by tracking corruptions and validating metadata consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If a smaller chunk size is used in deduplication, then storage reduction is improved, but metadata size and fragmentation increase
Solution Approach 1:
The patent segments the verification process into distinct phases (initialization, verification, completion) and uses separate data structures for tracking corruptions. This segmentation allows the system to manage metadata efficiently by organizing corruption tracking information in a structured manner, reducing the overhead associated with small chunk sizes.
Solution Approach 2:
The patent introduces intermediary data structures such as corruption tracking information and phase markers that mediate between the deduplication process and the metadata storage. These intermediaries enable efficient tracking of corruptions without requiring extensive metadata, thus resolving the contradiction between storage reduction and metadata size.
2Reliability
If comprehensive verification of object relationships is performed, then data integrity is improved, but read operations and processing time increase
Solution Approach 1:
The patent performs preliminary actions by initializing verification phases and tracking corruptions before actual verification operations. Corruption tracking information is prepared in advance, allowing the verification process to proceed efficiently without extensive read operations, thus maintaining data integrity while reducing processing time.
Solution Approach 2:
The patent implements a mechanism to skip unnecessary read operations by using phase markers and corruption tracking information. The verification process can rush through already-verified portions and focus only on areas where corruptions may exist, thereby reducing processing time while maintaining comprehensive verification of object relationships.
3Loss of substance
If deduplication is applied to reduce redundancy, then storage efficiency is improved, but verification of object relationships becomes more complex
Solution Approach 1:
The patent implements self-service mechanisms where the verification system uses its own generated corruption tracking information and phase markers to guide the verification process. This self-service approach simplifies verification of object relationships in deduplicated storage by providing the system with intrinsic information about potential corruptions, reducing the complexity of verifying relationships between deduplicated objects.
Data Source
AI summary
Described is a system for detecting corruption in a deduplicated object storage system accessible by one or more microservices while minimizing costly read operations on objects. A similarity group verification path is selected based upon detection of an object storage memory size condition. The similarity group verification path is performed by one or more worker nodes. The similarity group verification path includes worker node phases that verify whether objects have been corrupted without having to incur costly read operations.