Cross-Account Data Migration for Region Builds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
When a new data center or facility is brought online, integrating it into existing networking and computing infrastructure requires providing network and computing services, which often leads to bottlenecks as databases and associated services are not immediately available, necessitating the use of a temporary proxy data store.
Innovation Solution
A cross-account data copy and migration system is implemented, allowing for the migration of data from a proxy data store to a live data store across different user accounts, utilizing a computing environment that executes program instructions to manage the data migration process, including creating tables, exporting and importing data items, and updating the live data store settings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a temporary proxy data store is used during region build, then development teams can access data stores immediately, but data migration to the live data store causes prolonged downtime and outages
Solution Approach 1:
The system performs table creation and data export from the proxy data store before the actual migration switch. The live data store tables are pre-created with correct schemas, and data is prepared in advance, allowing the migration to be executed quickly with minimal downtime when the switch occurs.
Solution Approach 2:
The proxy data store acts as an intermediary during the transition period. It continues to serve development teams while the live data store is being prepared and populated. The proxy data store mediates between the need for immediate access and the requirement for data completeness, allowing parallel operations during migration.
2Loss of time
If data migration is performed quickly, then downtime is reduced, but data completeness and accuracy may be compromised
Solution Approach 1:
The migration process continues to run in the background while the proxy data store remains operational. Data is continuously copied from proxy to live data store without interrupting development team access. The system maintains continuous data transfer operations, ensuring both speed and completeness by not stopping the migration process.
Solution Approach 2:
The system creates complete copies of all data from the proxy data store to the live data store using structured copying operations. Tables are copied with their schemas, and all data rows are replicated, ensuring data completeness. The copying process includes error handling and validation to maintain data accuracy during rapid migration.
3Reliability
If the proxy data store is brought down for migration, then data can be transferred to the live data store, but service availability is reduced
Solution Approach 1:
The system dynamically switches between proxy and live data stores based on migration progress. During migration, both data stores remain operational and accessible. The system dynamically updates the active data store configuration to point to the live data store once migration is complete, maintaining service availability throughout the process without requiring proxy data store shutdown.
Data Source
AI summary
When a request to transition use of a first data store to a second data store is identified (e.g., when a new regional computing stack is to be brought online in a network), tables in the second data store are created that correspond to respective ones of tables of the first data store. Data items are then accessed from the first data store and translated into a format interpretable by a remote data storage service. Thereafter, a network-based application programming interface (API) call is performed to store the data items in the remote data storage service using the data items in the second format. The data items as stored in the remote storage service are accessed and imported into the second data store.