Mutable Data Set Migration With Validation Copy

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Migrating data between data stores is costly and often results in downtime, making it prohibitive even when significant improvements to the client system or service can be achieved, especially for large-scale systems that maintain large amounts of mutable data.

Innovation Solution

Implementing incremental migration techniques that block updates to select mutable data sets during migration, allowing only read access, and using a validation phase to ensure data consistency and minimize disruption, while utilizing a migration service to intelligently select and manage data sets for migration.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data is migrated between data stores, then performance requirements can be improved, but client system downtime and data consistency issues occur

Engineering Contradiction:

Improveperformance requirementsVSAvoiddata consistency

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent applies preliminary action by creating a validation copy of the data set in the destination data store before completing the migration. This validation copy is used to verify data consistency and performance characteristics in advance, allowing the migration to proceed with minimal disruption and ensured consistency. The client system can validate the migration success before full switching occurs.

Inventive Principle:

Principle #10Preliminary action

2Productivity

If data migration is performed, then improvements to client system can be achieved, but migration costs become prohibitive

Engineering Contradiction:

Improveclient system improvementsVSAvoidmigration costs

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent applies segmentation by dividing the migration process into distinct phases: creating a validation copy in the destination data store, validating the copy against the source, and only then completing the migration. This segmented approach allows for risk mitigation and reduces the cost of failed migrations, making the overall process more economically viable.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent uses copying by creating a validation copy of the data set in the destination data store before finalizing the migration. This copy serves as a testbed to verify data integrity and performance improvements without committing the entire migration, thereby reducing the financial risk and cost of the migration process.

Inventive Principle:

Principle #26Copying

3Adaptability or versatility

If mutable data sets are migrated, then data can be moved to support changed performance requirements, but updates and reads may be disrupted

Engineering Contradiction:

Improveperformance requirementsVSAvoidaccess continuity

Core Design Contradiction:

Adaptability or versatilityVSEase of operation

Solution Approach 1:

The patent applies preliminary action by performing validation operations on the destination data store before completing the migration. This includes executing read requests against the validation copy to ensure data integrity and accessibility, thereby minimizing disruption to client system operations during the migration process.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS10452637B1Migration of mutable data sets between data stores

Publication Date: 2019.10.22 AMAZON TECH INC

AI summary

Migration of mutable data sets between data stores may be implemented. Multiple data sets may be stored at a source data store. Some of the data sets may be identified for migration to a destination data store. Updates to the identified data sets may be blocked while the identified data sets are copied from the source data store to the destination data store. In some embodiments, the copies of the data sets at the destination data store are validated. Updates to the data sets may then be allowed at the destination data store instead of the source data store.