Database Query Pushdown for Parallel Computation Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems face increased query processing times and bottlenecks due to data shuffling between nodes when processing parallel computation tasks, as they need to retrieve data from multiple nodes, leading to inefficiencies in data transfer and computation task processing.
Innovation Solution
Implementing a system where the computation engine generates queries to push down computation commands to the DBMS, allowing for pre-processing of data and reducing data shuffling by utilizing a data locality-aware approach, where processing nodes retrieve data locally from connected database nodes, thereby optimizing data transfer and computation efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is retrieved from multiple database nodes to process parallel computation tasks, then computation completeness is improved, but data transfer time and processing bottlenecks increase
Solution Approach 1:
The patent applies local quality by enabling each processing node to access only the specific data segments stored on its locally connected database node, rather than requiring all nodes to access all data. This localized data access approach maintains computation completeness while eliminating the need for time-consuming data shuffling across the network, as each node processes computations on data already available locally.
Solution Approach 2:
The patent segments both the computation tasks and the database nodes into corresponding groups, where each processing node is connected to a specific database node. This segmentation allows parallel processing of different data segments without requiring inter-node data transfer, as each segment can be processed independently on its associated node pair.
2Adaptability or versatility
If data is shuffled between database nodes to satisfy queries, then query fulfillment is improved, but data transfer overhead and system bottlenecks worsen
Solution Approach 1:
The patent applies preliminary action by pre-establishing local connections between processing nodes and database nodes before computation tasks are executed. Data is made locally available in advance on each database node, eliminating the need for time-consuming data shuffling operations during query processing. This preliminary arrangement enables queries to be fulfilled using only local data access.
Solution Approach 2:
The patent enables each processing node to query and process only the data segments stored on its locally connected database node, eliminating the need for network-based data shuffling. This localized approach maintains full query fulfillment capability for distributed datasets while eliminating the energy overhead of transferring data across the network.
3Productivity
If parallel computation tasks are processed by retrieving data from multiple nodes, then computation throughput is improved, but system bottlenecks increase
Solution Approach 1:
The patent segments the parallel computation system into independent processing units, where each processing node operates with its locally connected database node. This segmentation eliminates system-wide bottlenecks by allowing each segment to process data independently without requiring coordination or data transfer between segments, thereby maintaining high computation throughput with reduced complexity.
Solution Approach 2:
Each processing node serves itself by accessing data locally from its connected database node, without requiring data to be fetched from other nodes. This self-service approach eliminates the coordination overhead and bottlenecks associated with centralized data management, enabling scalable parallel processing.
Data Source
AI summary
Examples disclosed herein relate to accessing electronic databases. Some examples disclosed herein may include partitioning a computation task into subtasks. A processing node of a computation engine may generate a database query for retrieving an electronic data segment associated with at least one of the subtasks from a database. The database query may include pre-processing instructions for a database management system (DBMS) associated with the database to pre-process the electronic data segment before providing the electronic data segment to the processing node. The pre-processing instructions may include at least one of: filtering, projection, join, aggregation, count, and user-defined instructions. The generated query may be provided to the DBMS.