Parallel Data Transfer Coordination in Database Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face inefficiencies in data transfer during ETL operations, particularly in parallel data loading, which leads to resource fragmentation, slower data loading due to uneven mechanisms, and the need for external staging, resulting in increased costs and error-prone interoperability.
Innovation Solution
Implementing database commands that allow the database system to collect, transform, and store data without external engines, enabling parallel data transfer coordination within the database system, thus eliminating the need for external staging and reducing resource overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If external tools are used to perform ETL operations, then data can be extracted and transformed, but interoperability costs increase and programming complexity grows
Solution Approach 1:
The patent merges the ETL operations directly into the database system by implementing external tables that allow parallel data transfer from external sources. This eliminates the need for separate external tools and reduces interoperability complexity by integrating the extraction and loading functions within the database architecture itself.
Solution Approach 2:
The database system performs ETL operations autonomously through internally implemented parallel transfer mechanisms. The system uses its own resources and commands to extract data from external sources and load it into internal tables, eliminating dependency on external ETL tools and reducing overall system complexity.
2Productivity
If sequential data loading is used, then data can be loaded into the database, but data loading speed decreases
Solution Approach 1:
The patent segments the data loading process into parallel operations that can execute simultaneously. Multiple data sources are loaded in parallel using separate transfer commands, and the database system manages these parallel operations to load data from multiple external tables concurrently, significantly increasing data loading speed.
Solution Approach 2:
The system dynamically manages parallel data transfer operations, allowing the database to coordinate multiple simultaneous loading operations from different external sources. This dynamic parallel processing adapts to different data source characteristics and optimizes loading speed based on real-time system conditions.
3Ease of operation
If external staging is implemented, then data can be staged before loading, but resource overhead increases and costs increase
Solution Approach 1:
The patent extracts the staging function from external infrastructure and integrates it directly into the database system. External tables serve as the staging area within the database, eliminating the need for separate external staging servers or filesystems. This reduces resource overhead while maintaining the ability to stage data before loading.
4Productivity
If parallel data transfer is coordinated externally, then data can be loaded in parallel, but coordination errors and resource fragmentation occur
Solution Approach 1:
The database system acts as the intermediary that coordinates all parallel data transfer operations. The parallel transfer commands are issued through the database's internal command architecture, which manages synchronization and resource allocation. This internal coordination mechanism eliminates coordination errors that would occur with external coordination while maintaining parallel loading efficiency.
Data Source
AI summary
Embodiments of the invention enable data to be moved from one or more external sources into a database system. A plurality of processes are executed concurrently. Each of the processes are executable to move data from at least a portion of the one or more external sources into the database system, so that executing the plurality of processes concurrently moves data from different portions of the one or more external sources into the database system in parallel. Execution of the plurality of processes are coordinated from the database system.