Parallel Data Transfer Using Unique Identifiers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data transfer methods in relational databases are limited by processing resources, often relying on single-core operations and full table scans, which result in slow performance when transferring data between databases.
Innovation Solution
The method involves using unique identifiers to perform multiple partial table scans in parallel across multiple cores, allowing data to be transferred in smaller subsets via a standalone tool that can operate independently of the source database, improving transfer efficiency by completing operations in a fraction of the time required by single-operation approaches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a full table scan is used to transfer data, then all data can be transferred in a single operation, but the transfer speed is limited by single-core processing resources
Solution Approach 1:
The patent divides the data transfer operation into multiple segments by splitting the table scan into multiple partial scans, each handled by a separate thread. This allows parallel processing across multiple cores, significantly improving data transfer speed while maintaining manageable complexity through structured thread management
Solution Approach 2:
The patent introduces a new dimension of parallelism by utilizing multi-threading and multiple CPU cores simultaneously. Instead of sequential single-core processing, the system operates across multiple processing dimensions (threads and cores), enabling concurrent execution of multiple table scan operations
2Loss of time
If multiple cores are utilized for parallel data transfer, then transfer time is reduced, but resource coordination complexity increases
Solution Approach 1:
The patent segments the data transfer task into multiple independent partial table scans that can be executed in parallel by different cores. Each segment is self-contained and can be processed independently, reducing coordination overhead while minimizing transfer time through concurrent execution
Solution Approach 2:
The patent implements self-service mechanisms where each thread manages its own partial table scan operation independently. The system automatically distributes work across available cores and handles resource allocation dynamically, reducing the need for complex external coordination while achieving efficient parallel processing
Data Source
AI summary
The present disclosure is related to devices, systems, and methods for transferring data using unique identifiers of relational databases. An example system can include a first server hosting a relational database and a second server coupled to the first server. The second server can be configured to assign a quantity of unique identifiers of a table of the relational database to each of a plurality of groups, perform a first transfer operation to transfer data corresponding to a first one of the plurality of groups via a first core of processing resources of the first server, and perform a second transfer operation to transfer data corresponding to a second one of the plurality of groups via a second core of the processing resources of the first server. The first transfer operation can be performed in parallel with the second transfer operation.