Restartable Data Transfer via Atomic Partition Transactions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transferring large volumes of data from transactional data sources to partitioned databases in restartable environments requires high data throughput, ensures no data loss, and prevents data duplication, especially during failures and restarts.
Innovation Solution
The method involves reading messages from a transactional data source, identifying the appropriate partition for record insertion, and combining record insertion and message deletion into a single transaction, ensuring that neither operation can commit without the other, thereby preventing data loss and duplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data transfer is performed in a restartable environment, then data throughput is improved, but data loss or data duplication may occur during failures and restarts
Solution Approach 1:
The patent combines the insertion of records into the database partition and the deletion of messages from the transactional data source into a single atomic transaction. This merging ensures that both operations either succeed together or fail together, preventing data loss (if insertion fails, message isn't deleted) and data duplication (if deletion fails, insertion is rolled back). The single transaction approach maintains data integrity while enabling high-throughput real-time data transfer in restartable environments.
2Ease of operation
If separate operations are used for record insertion and message deletion, then operational flexibility is improved, but data consistency deteriorates during failures
Solution Approach 1:
The patent merges record insertion and message deletion into a single atomic transaction to ensure data consistency. By using a single transaction that encompasses both operations, the system guarantees that either both operations complete successfully or both are rolled back in case of failure. This eliminates the risk of partial updates where records might be inserted without deleting the source message, or messages deleted without successful insertion, thereby maintaining referential integrity and data consistency across failures and restarts.
Data Source
AI summary
Method, system, and computer program product for transferring data from transactional data sources to partitioned databases are provided. One or more messages from a transactional data source are read. Each message includes one or more records to be inserted into a database comprising a plurality of partitions. One of the partitions of the database in which the one or more records of each message are to be inserted is then identified. A transaction is initiated to insert the one or more records of each message into the one partition and to delete the one or more messages from the transactional data source. The transaction is committed only when insertion of the one or more records of each message into the one partition and deletion of the one or more messages from the transactional data source succeed.