Parallel Data Table Transfer via Self-Coordinating Segments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database transfer methods face challenges in efficiently transferring electronic data tables, with serial transfers being slow and parallel transfers requiring complex coordination between computation tasks, often necessitating third-party systems to ensure ACID properties.
Innovation Solution
A computation engine partitions electronic data tables into segments and generates self-coordinating tasks that transfer these segments to a staging table, with a last committer task ensuring atomicity, consistency, isolation, and durability without relying on third-party systems or communication between tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If serial transfer method is used, then system complexity is low, but transfer speed is slow
Solution Approach 1:
The patent divides the electronic data table into multiple segments and creates corresponding computation tasks for each segment. Multiple processing nodes can simultaneously transfer different segments to the database in parallel, significantly improving transfer speed while maintaining manageable system complexity through structured task division
2Speed
If parallel transfer method is used, then transfer speed is fast, but coordination complexity increases requiring third-party systems
Solution Approach 1:
Each computation task is designed to be self-coordinating, independently tracking its own segment transfer status and automatically determining when all segments are transferred. The tasks use a shared completion table to signal their status without requiring external coordination, eliminating the need for third-party coordination systems while maintaining fast parallel transfer speeds
Solution Approach 2:
The patent implements a feedback mechanism where computation tasks continuously check a shared table to determine whether all segments have been transferred. When the condition is met, tasks automatically proceed to the next phase, enabling self-coordination through structured feedback loops without external intervention
3Reliability
If parallel transfer with third-party coordination is used, then ACID properties are ensured, but system complexity increases
Solution Approach 1:
The computation tasks inherently ensure ACID properties through their self-coordinating design. By using a shared completion table that all tasks access atomically and by ensuring all segments are transferred before proceeding, the system maintains atomicity, consistency, isolation, and durability without requiring additional third-party coordination systems, thus preserving reliability while reducing complexity
Data Source
AI summary
Examples disclosed herein relate to parallel transfers of electronic data. Some examples disclosed herein may include executing, by a processing node of a computation engine, a computation task among a plurality of computation tasks generated by the computation engine for transferring an electronic data table to a target table. The computation task, when executed by the processing node, may cause the processing node to transfer a segment of the electronic data table to a staging table, update a task status table upon completing the transfer of the segment to the staging table, in response to determining that the plurality of computation tasks have completed, update a last committer table with a task identifier associated with the computation task, and in response to determining that the last committer table includes the task identifier associated with the computation task, transfer the staging table to the target table.