Serverless Bulk Import with Selective Back Pressure
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing solutions for importing data from legacy database systems to new database systems are inefficient, leading to slower performance, increased costs, and scalability issues, especially when handling large-scale data, which can impact production tables and require additional planning and costs.
Innovation Solution
A serverless data import system that allows direct data import from legacy databases to new databases using a management console or API, eliminating the need for custom development and reducing administrative costs, while maintaining performance and scalability, and consuming minimal to no write capacity, allowing for efficient data import with minimal downtime and no impact on live application traffic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If existing data import solutions are used to handle large-scale data, then data import functionality is achieved, but write capacity is consumed which adversely affects production tables and requires additional planning and costs
Solution Approach 1:
The system segments data import operations into separate, isolated import tables that are distinct from production tables. Import operations are performed on these dedicated import tables, preventing any interference with production table performance while still enabling large-scale data import functionality.
Solution Approach 2:
The system extracts the data import function from the production table environment and relocates it to separate import tables. This extraction removes the harmful interaction between import operations and production workloads, allowing production tables to maintain their performance without being affected by import operations.
2Productivity
If existing data import solutions are used, then data can be imported from legacy systems, but custom development and long-term maintenance are required
Solution Approach 1:
The system provides a universal data import mechanism that works across different legacy database systems through standard import table operations. The same import table infrastructure handles diverse data sources without requiring custom development for each system, eliminating maintenance burdens while maintaining high import speeds.
Solution Approach 2:
The system creates simplified copies of production table schemas as import tables, which can be populated through straightforward import operations. These copy structures enable rapid data ingestion from various legacy systems using consistent, maintained procedures rather than custom development for each source system.
3Reliability
If data import is performed on production tables, then data is imported, but import speed decreases and reliability becomes slower
Solution Approach 1:
By segmenting import operations onto separate import tables, the system eliminates contention and resource conflicts that occur when importing data directly into production tables. This segmentation allows import operations to proceed at full speed without being slowed by production workload interactions, while maintaining reliable data transfer.
Solution Approach 2:
Import tables serve as intermediary structures between legacy systems and production tables. Data flows through these intermediary import tables rather than directly into production tables, enabling high-speed import operations while maintaining data integrity and reliability through the standardized import process.
4Adaptability or versatility
If existing import solutions are used, then data import is possible, but additional planning and costs are required
Solution Approach 1:
The system implements a universal import table mechanism that adapts to various legacy database systems and data formats through standardized operations. This universal approach provides import flexibility across different scenarios while simplifying implementation, as the same import table infrastructure handles diverse requirements without additional planning or complexity.
Data Source
AI summary
A system receives a request to import data file(s) from a source data store into a target database. The system reserves a first portion of computing resources that host the target database to import the data file(s). The reservation of the first portion of computing resources permits the import throughput rate of the data file(s) through the first portion of computing resources while maintaining a second portion of the computing resources to support client access to the target database at an access throughput rate. The system initiates import of the data file(s) from the source data store to the target database through one or more storage nodes at the import throughput rate according to the first portion of computing resources. The target database is able to receive access requests from one or more clients during the import of the data file(s) to the target database.