Data Migration System Using Triggers and Replication

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Data migration and schema changes in production datastores are risky and can lead to mission critical system failures, resulting in data loss or corruption, due to the challenges of maintaining high reliability in mission-critical datastores.

Innovation Solution

Implementing a system that uses replication between main and backup datastores, synonyms, and triggers to propagate changes across data structures, allowing for incremental testing and migration while maintaining data consistency, thereby reducing the risk of failure during data migration and schema updates.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If data migration and schema changes are performed directly in production datastores, then data structure updates can be implemented, but system reliability deteriorates due to risk of mission critical failures

Engineering Contradiction:

Improvedata structure update capabilityVSAvoidmission critical system reliability

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent segments the production datastore into multiple independent data structure instances (first data structure and second data structure) that can be updated independently. This allows schema changes to be applied to one instance without affecting the other, enabling safe updates while maintaining system reliability through redundancy.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies schema changes to a second data structure instance before the production first data structure. This preliminary action allows testing and validation of schema changes in a non-production environment, ensuring that updates are verified before being applied to the mission-critical production system.

Inventive Principle:

Principle #10Preliminary action

2Productivity

If data migration is performed by directly modifying production datastores, then migration can be completed, but data loss or corruption risk increases

Engineering Contradiction:

Improvemigration completionVSAvoidmission critical data integrity

Core Design Contradiction:

ProductivityVSLoss of information

Solution Approach 1:

The patent maintains a copy of the production data in a second data structure instance. This copy serves as both a testing ground for schema changes and a safety net that can be used to restore production data if migration errors occur, thereby preventing data loss while enabling migration completion.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent prepares a backup data structure instance with the new schema before performing migration. This beforehand cushioning ensures that if migration fails or corrupts production data, the system can switch to the pre-prepared backup instance, cushioning against data loss.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

3Adaptability or versatility

If schema changes are applied to production datastores, then data structure evolution is achieved, but service downtime may occur

Engineering Contradiction:

Improveschema evolution capabilityVSAvoidservice downtime

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The patent merges multiple data structure instances (first and second data structures) into a single production datastore that can serve both old and new schemas simultaneously. This merging allows the system to support both the original schema and the new schema during transition, eliminating service downtime while enabling schema evolution.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The patent implements dynamic schema management where the production datastore can switch between different data structure versions. This dynamics allows the system to adaptively serve requests based on the current schema version being used, enabling seamless transitions without fixed downtime periods.

Inventive Principle:

Principle #15Dynamics

4Reliability

If replication is implemented between multiple data structures, then data consistency is maintained, but system complexity increases

Engineering Contradiction:

Improvedata consistencyVSAvoidreplication system complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent introduces triggers as intermediary mechanisms that automatically maintain data consistency between the first and second data structures. These triggers act as mediators that listen for changes in one data structure and automatically propagate corresponding changes to the other, simplifying the replication process while ensuring data consistency.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS9607020B1Data migration system

Publication Date: 2017.03.28 AMAZON TECH INC

AI summary

Described are techniques and systems to migrate data from a first data structure to a second data structure stored in datastores which are replicated with one another. A “hot” datastore is replicated with one or more “warm” datastores. Triggers maintain consistency between the first data structure and the second data structure. Migration may involve testing using the second data structure of the “warm” datastore. Data processed by the first data structure on the “hot” datastore is distributed via replication to the “warm” datastore, and then by a trigger to the second data structure. Data processed by the second data structure on the “warm” datastore is distributed via trigger to the first data structure, and then by replication to the “hot” datastore. Once performance of the second data structure is deemed acceptable, the “warm” datastore may be designated as “hot” and synonyms therein may direct data to the second data structure.