Verification Microservice Engine for Deduplicated Storage Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplication systems face inefficiencies in storage reduction due to the trade-off between chunk size, metadata size, deduplication time, and fragmentation, as well as challenges in verifying the integrity of relationships between objects in deduplicated storage systems, which can lead to object corruption and data redundancy.
Innovation Solution
A verification microservice engine is introduced to detect object corruption in deduplicated object storage systems by generating and deploying a controller module and worker nodes within a cloud computing environment, utilizing similarity group and slice recipe verification paths to minimize costly read operations and ensure data integrity by verifying relationships between objects, similarity groups, and compression regions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If a smaller chunk size is used to identify more duplicates, then storage reduction is improved, but metadata size increases and deduplication time increases
Solution Approach 1:
The patent segments the deduplication verification process into distinct phases: controller module preparation phase and worker node execution phase. The controller module prepares verification paths and distributes them to worker nodes, which then execute the actual verification. This segmentation allows parallel processing and reduces the overall deduplication time while maintaining effective storage reduction.
Solution Approach 2:
The patent introduces a controller module as an intermediary between the deduplication system and worker nodes. The controller module generates verification paths, distributes them to worker nodes, and collects results. This intermediary structure enables coordinated parallel verification across multiple worker nodes, reducing total verification time while maintaining accuracy.
2Reliability
If deduplication verification is performed to detect object corruption, then data integrity is improved, but read operations increase causing performance degradation
Solution Approach 1:
The patent performs preliminary actions by having the controller module generate and prepare verification paths before distributing them to worker nodes. The verification paths are pre-computed and organized, allowing worker nodes to execute verification efficiently without performing costly read operations during the coordination phase. This preliminary preparation reduces the performance impact of verification.
Solution Approach 2:
The patent implements partial verification by having different worker nodes verify different portions of the deduplicated data structure (similarity groups, slice recipes, compression regions). Not all data requires full verification simultaneously, allowing the system to maintain data integrity while limiting the performance impact to only the necessary verification portions.
3Productivity
If multiple worker nodes are deployed to verify similarity groups and slice recipes, then verification efficiency is improved, but system complexity increases
Solution Approach 1:
The controller module serves multiple functions: it prepares verification paths, distributes them to worker nodes, collects verification results, and coordinates the overall process. This multi-functional design reduces system complexity by consolidating coordination logic in a single controller module rather than requiring complex peer-to-peer communication between multiple worker nodes.
Solution Approach 2:
The patent segments the verification workload into distinct tasks distributed to different worker nodes: some nodes verify similarity groups while others verify slice recipes and compression regions. This segmentation allows parallel verification improving efficiency, while the clear task division simplifies the overall system architecture compared to having all nodes perform all verification types.
Data Source
AI summary
Described is a system for a verification microservice engine for generating and deploying a controller module and one or more worker nodes to detect corruption in a deduplicated object storage system accessible by one or more microservices while minimizing costly read operations on objects. A worker node(s) stores slice recipes into worker node memory that correspond to slice recipe names verified by a controller module. For each slice recipe, the worker node(s) detects whether slice recipe corruptions, similarity group corruptions and/or compression region corruptions exist.