Verification Microservice for Deduplicated Object Storage

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Deduplication systems face inefficiencies in storage reduction due to the trade-off between chunk size and metadata size, as well as challenges in verifying the validity of relationships between objects in deduplicated storage systems, which can lead to storage waste and corruption detection difficulties.

Innovation Solution

A verification microservice engine is deployed to estimate and instantiate worker nodes, selecting between similarity group and slice recipe verification paths based on memory conditions, and uses a testing tool to corrupt and verify objects, ensuring the integrity of object relationships in deduplicated storage systems.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of substance

If a smaller chunk size is used in deduplication, then storage reduction is improved, but metadata size and deduplication time increase

Engineering Contradiction:

Improvestorage reductionVSAvoidmetadata size

Core Design Contradiction:

Loss of substanceVSDevice complexity

Solution Approach 1:

The patent segments the verification process into two distinct paths: similarity group verification path and slice recipe verification path. This segmentation allows the system to handle different verification scenarios efficiently, avoiding the need to load all metadata into memory simultaneously, thus resolving the contradiction between storage reduction and metadata complexity

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements dynamic worker node instantiation based on memory conditions. The verification controller estimates the number of worker nodes needed and instantiates them dynamically, adjusting system resources based on current memory availability and verification requirements, which helps manage metadata processing efficiency

Inventive Principle:

Principle #15Dynamics

2Reliability

If verification of object relationships is performed, then data integrity is improved, but read operations and system resources increase

Engineering Contradiction:

Improvedata integrityVSAvoidread operations

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent implements partial verification by selecting between two verification paths: similarity group verification path (verifying object relationships) and slice recipe verification path (verifying object integrity). The system performs only the necessary verification based on memory conditions, avoiding excessive read operations while maintaining data integrity

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The patent introduces a verification controller as an intermediary that coordinates between the storage system and worker nodes. The controller estimates worker node requirements, manages memory conditions, and directs verification operations, reducing unnecessary read operations while ensuring data integrity through controlled verification processes

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS11340784B1Verification microservice for a deduplicated object storage system with worker node estimation

Publication Date: 2022.05.24 EMC IP HLDG CO LLC

AI summary

Described is a system for detecting corruption in a deduplicated object storage system accessible by one or more microservices while minimizing costly read operations on objects. A controller module and one or more worker nodes execute verification paths in concert with each other to identify object corruptions. The controller module estimates a number of worker nodes required for execution of different types of verification paths.