Verification Microservice for Deduplicated Storage Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplication systems face inefficiencies in storage reduction due to the trade-off between chunk size, metadata size, deduplication time, and fragmentation, as well as challenges in verifying the validity of relationships between objects in deduplicated storage systems.
Innovation Solution
A verification microservice engine is introduced to detect object corruption in deduplicated object storage systems by generating and deploying a controller module and worker nodes, which execute similarity group and slice recipe verification paths to validate object relationships and metadata integrity, minimizing costly read operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If a smaller chunk size is used in deduplication, then storage reduction is improved, but metadata size increases and deduplication time increases
Solution Approach 1:
The patent segments the deduplication verification process into distinct phases (initialization, data ingestion, deduplication, validation) and distributes verification tasks across multiple worker nodes. Each worker node handles specific verification tasks independently, allowing the system to use smaller chunk sizes for better deduplication ratios without overwhelming the metadata management capacity of a single node.
Solution Approach 2:
The patent introduces a distributed verification architecture that adds a spatial dimension to the deduplication process. By distributing verification across multiple worker nodes arranged in a scalable cluster, the system can handle larger metadata volumes and smaller chunk sizes by parallelizing the verification workload across multiple processing dimensions.
2Loss of substance
If a smaller chunk size is used in deduplication, then storage reduction is improved, but deduplication time increases
Solution Approach 1:
The verification process is segmented into distinct phases (initialization, data ingestion, deduplication, validation) that can be executed in parallel across multiple worker nodes. This segmentation allows smaller chunk sizes to be processed more efficiently by distributing the computational burden across multiple independent verification threads.
Solution Approach 2:
The system performs preliminary verification setup by initializing worker nodes with verification paths and metadata structures before the actual deduplication process begins. This preliminary action ensures that when smaller chunks are processed, the verification infrastructure is already in place, reducing the overall deduplication time despite the increased number of chunks.
3Reliability
If verification of object relationships is performed in deduplicated storage, then data integrity is improved, but read operations increase
Solution Approach 1:
The system performs verification of object relationships during the deduplication process itself rather than as a separate post-processing step. Worker nodes verify metadata integrity and object relationships while data is being ingested and deduplicated, ensuring data integrity without requiring additional read operations after the main processing is complete.
Solution Approach 2:
The verification process runs continuously alongside the deduplication operations rather than interrupting or pausing the main data processing workflow. Multiple worker nodes continuously verify object relationships and metadata integrity as data flows through the system, maintaining continuous useful action without increasing overall read operation burden.
Data Source
AI summary
Described is a system for a verification microservice engine for generating and deploying a controller module and one or more worker nodes to detect corruption in a deduplicated object storage system accessible by one or more microservices while minimizing costly read operations on objects. The controller module builds local versions of slice recipe names based on metadata available object recipes. The controller module verifies the accuracy of the metadata based on whether the locally built slice recipes names match slice recipe names in object storage.