Verifiable Intra-Cluster Migration for Chunk Storage

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional data storage migration technologies in ECS systems fail to provide verifiable intra-cluster migration, leading to issues like chunk table mismatches and data loss during migration, which can impact data accessibility and integrity.

Innovation Solution

The implementation of an intra-cluster migration component that locks chunks to prevent modifications, allowing for verifiable migration by creating a new chunk table while preserving existing chunk relationships, enabling verification and reconciliation of migrated data without disrupting ongoing data access.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional migration technology is used to move chunks between storage spaces, then migration can be performed, but verification of migration completeness and accuracy becomes impossible due to chunk table updates during migration

Engineering Contradiction:

Improvemigration verificationVSAvoidchunk table management

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system creates a snapshot of the chunk table before migration begins, preserving the pre-migration state for later verification. This preliminary action allows the verification process to compare against a known good state without interfering with the migration process itself.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Instead of modifying the original chunk table during migration, the system creates a copy (snapshot) of the chunk table and uses this copy for verification purposes. This allows the original chunk table to be updated freely during migration while the copy remains unchanged for comparison.

Inventive Principle:

Principle #26Copying

2Productivity

If the chunk table is updated during migration to reflect new chunk locations, then migration progress is tracked, but the ability to verify migration against the original state is lost

Engineering Contradiction:

Improvemigration speedVSAvoidmigration verification accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The snapshot acts as an intermediary between the original chunk table and the verification process. It preserves the original state information needed for verification while allowing the chunk table to be freely updated during migration to maintain productivity.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system uses the snapshot to provide feedback on migration completeness by comparing the current chunk table state against the snapshot. This enables verification without preventing ongoing migration operations.

Inventive Principle:

Principle #23Feedback

3Quantity of substance

If garbage collection is performed during migration to manage storage space, then storage efficiency is improved, but mismatches between migrated chunks and chunk table can occur

Engineering Contradiction:

Improvestorage space utilizationVSAvoidchunk table consistency

Core Design Contradiction:

Quantity of substanceVSReliability

Solution Approach 1:

The snapshot serves as a cushion that protects against inconsistencies caused by garbage collection during migration. By preserving the pre-migration state, it allows the system to tolerate and later reconcile changes made by garbage collection operations.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

4Ease of operation

If recovery operations are performed during migration to handle failed chunks, then data accessibility is maintained, but mismatches between recovered chunks and chunk table can occur

Engineering Contradiction:

Improvedata accessibilityVSAvoidchunk table consistency

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The snapshot provides a reference state that allows recovery operations to proceed without immediately compromising verification. The snapshot captures the state before recovery, enabling later verification that recovery operations were performed correctly.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

Data Source

PatentUS11288229B2Verifiable intra-cluster migration for a chunk storage system

Publication Date: 2022.03.29 EMC IP HLDG CO LLC

AI summary

Verifiable intra-cluster migration (VICM) for a chunk storage system is disclosed. VICM can migrate data from a first portion of a cluster to a second portion of a cluster. VICM can comprise locking a first portion of a cluster and locking a corresponding first cluster table during a preparation phase. Chunks of the first portion can then be migrated, during a migration phase, to the second portion and a second cluster table, corresponding to the second portion, can be updated accordingly. Garbage management operations, including recovery operations, can be performed via the second cluster table and the second portion during the migration phase. Upon completion of the migration phase, a reconciliation phase can comprise verifying chunk relationships of the second cluster table and the second portion based on the first cluster table. Exceptions to the verification can be reported via an exception report.