Entity-Level Data Migration Framework With Cohort Segmentation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Data migration from an outdated source storage system to a new target storage system is complex, especially when dealing with hundreds of thousands of entities, as it requires minimal downtime and handling of conflicting data across multiple tables with different naming schemes, which existing methods often fail to achieve without data loss.

Innovation Solution

The method involves partitioning entities into cohorts based on usage attributes, copying data from the source to the target storage system without locking, verifying data integrity while locked, updating entity pointers, and unlocking to ensure minimal downtime and prevent data overwrite by migrating on an entity level rather than a table level.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of time

If data is migrated by copying without locking the source storage repository, then downtime is minimized and data access continues, but data integrity verification becomes more complex

Engineering Contradiction:

ImprovedowntimeVSAvoiddata integrity verification complexity

Core Design Contradiction:

Loss of timeVSDevice complexity

Solution Approach 1:

The system performs preliminary actions by copying data without locking the source repository before verification. This allows data access to continue during migration, minimizing downtime. The locking is applied only during the verification phase to ensure data integrity, balancing continuous access with verification requirements.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If entities are migrated individually with locking during verification, then data integrity is ensured, but migration time increases

Engineering Contradiction:

Improvedata integrityVSAvoidmigration time

Core Design Contradiction:

ReliabilityVSDuration of action of moving object

Solution Approach 1:

The migration process is segmented into distinct phases: copying phase (without locking) and verification phase (with locking). By segmenting the process, the system can minimize downtime during copying while ensuring data integrity during verification, reducing the overall impact on migration time.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system uses periodic locking during verification intervals rather than continuous locking. Data is copied continuously without locking, then verification is performed with temporary locking, creating a periodic pattern that balances data integrity with minimal disruption to migration speed.

Inventive Principle:

Principle #19Periodic action

3Productivity

If data is migrated at table level rather than entity level, then migration speed increases, but data loss occurs due to conflicting values across multiple tables

Engineering Contradiction:

Improvemigration speedVSAvoiddata loss

Core Design Contradiction:

ProductivityVSLoss of information

Solution Approach 1:

Instead of migrating at the traditional table level, the system inverts the approach by migrating at the entity level. This inversion allows the system to handle conflicting values across multiple tables by processing each entity as a complete unit, preventing data loss while maintaining acceptable migration speed through parallel processing.

Inventive Principle:

Principle #13The other way round (Inversion)

4Productivity

If multiple cohorts are migrated simultaneously, then productivity increases, but coordination complexity and potential data conflicts increase

Engineering Contradiction:

Improvemigration throughputVSAvoidcoordination complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system segments entities into multiple cohorts that can be migrated in parallel. Each cohort is processed independently with its own locking and verification sequence, which reduces coordination complexity compared to managing all entities simultaneously while maintaining high throughput through concurrent processing.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS11768813B1Data migration framework

Publication Date: 2023.09.26 INTUIT INC

AI summary

A method may include selecting a cohort of entities for migration from a source storage repository to a target storage repository, obtaining a mapping between a source storage schema of the source storage repository to a target storage schema of the target storage repository, and migrating data for the entities in the cohort. Migrating the data of an entity may include copying, without locking the data in the source storage repository and in the target storage repository, the data from the source storage repository to the target storage repository, verifying, while the data is locked, that the data in the source storage repository is the same as the data in the target storage repository, changing, while the data in the source storage repository and the target storage repository is locked, an entity pointer for the entity to the target storage repository based on the verifying, and unlocking the data.