Parallel Query Execution via Dynamic Data Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In parallel computing systems, existing methods for query execution do not effectively leverage data replication to enhance query processing efficiency, particularly for multi-predicate queries, leading to suboptimal performance due to sequential evaluation of predicates on a single data copy.
Innovation Solution
A mechanism for real-time data replication across compute nodes in a parallel computing system, where a database is distributed across a first group of nodes, and a second group of nodes is allocated to replicate data dynamically in response to queries, allowing simultaneous evaluation of query predicates on multiple nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If data is stored on a single compute node for query execution, then device complexity is reduced, but query execution time increases due to sequential predicate evaluation
Solution Approach 1:
The patent creates replicated copies of data partitions across multiple compute nodes. When a query is received, the system identifies which nodes hold copies of the required data partitions and directs query execution to those nodes. This copying approach enables parallel predicate evaluation across multiple nodes simultaneously, reducing overall query execution time while maintaining manageable complexity through controlled replication strategies.
Solution Approach 2:
The patent introduces a new dimension to data storage by organizing data across both compute nodes and storage nodes in a distributed architecture. Data partitions are replicated across compute nodes for query processing and stored on dedicated storage nodes for persistence. This dimensional expansion allows the system to achieve parallel query execution without the full complexity of complete data distribution across all compute nodes.
2Productivity
If data is replicated across multiple compute nodes for parallel query processing, then query execution efficiency improves, but system complexity increases
Solution Approach 1:
The patent divides the database into discrete data partitions that can be independently replicated across compute nodes. Each partition is identified by a partition key, allowing the system to selectively replicate only the necessary partitions rather than duplicating entire datasets. This segmentation reduces replication management complexity while maintaining the ability to execute queries in parallel across multiple nodes that hold relevant partition copies.
Solution Approach 2:
The patent introduces a coordination mechanism that acts as an intermediary between query submission and data execution. This coordinator receives queries, determines which compute nodes hold copies of required data partitions, and orchestrates the parallel execution across those nodes. The intermediary abstracts the complexity of data location and replication management from the query processing logic, enabling efficient parallel execution without requiring each node to independently manage replication complexity.
3Speed
If compute nodes are dedicated to storing database partitions, then data access speed improves, but system flexibility decreases for other computing tasks
Solution Approach 1:
The patent implements a dynamic compute node pool where compute nodes can be allocated to different functions based on workload demands. Compute nodes that store data partitions can dynamically serve both as storage nodes for database operations and as general-purpose compute nodes for other computing tasks. The system monitors workload patterns and dynamically adjusts node allocation, allowing the same physical infrastructure to optimize for data access speed when database operations are required while maintaining flexibility for other computational workloads during off-peak database periods.
Data Source
AI summary
Embodiments of the invention may be used to increase query processing parallelism of an in-memory database stored on a parallel computing system. A group of compute nodes each store a portion of data as part of the in-memory database. Further, a pool of compute nodes may be reserved to create copies of data from the compute nodes of the in-memory database as part of query processing. When a query is received for execution, the query may be evaluated to determine whether portions of in-memory should be duplicated to allow multiple elements of the query (e.g., multiple query predicates) to be evaluated in parallel.