Dynamic Query Execution Model for Scalable Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data systems struggle with efficient query processing due to fixed and static resource assignments, leading to inefficiencies in handling large datasets and inability to dynamically adapt to changes in available resources.
Innovation Solution
A dynamic query execution model that scales out parallel query parts to additional computing resources, coordinated by parent and fragment query coordinators, allowing for flexible resource allocation and improved performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If computing resources are assigned to execute operations in a fixed and static manner, then the system structure is simple and easy to manage, but the system cannot dynamically adapt to changes in available resources and flexibility is reduced
Solution Approach 1:
The patent implements dynamic resource assignment where computing resources are allocated and reassigned based on real-time availability and job requirements. The system transitions from static pre-assigned roles to dynamic on-demand allocation, allowing the workload manager to assign resources as they become available rather than requiring fixed assignments beforehand.
Solution Approach 2:
The system divides computing operations into independent tasks that can be distributed across multiple computing resources. Each task can be independently assigned to different resources based on availability, enabling flexible segmentation of workloads rather than requiring monolithic fixed assignments.
2Speed
If more computing resources are used to execute operations, then query execution speed is improved, but the system cannot track individual resource performance and reliability decreases
Solution Approach 1:
The system implements tracking and monitoring of individual computing resource performance and status. The workload manager can identify which resources have completed tasks successfully and which have failed, using this feedback to make informed decisions about resource allocation and job recovery without requiring all resources to remain active throughout the entire operation.
Solution Approach 2:
By dividing operations into separate tasks assigned to individual resources, the system can isolate failures to specific segments rather than affecting the entire job. If one resource fails, other resources can continue executing their assigned tasks independently.
3Ease of manufacture
If computing resources work together in a fixed process group, then coordination is simplified, but the system cannot independently handle failures of individual resources
Solution Approach 1:
The system segments the computing operation into independent tasks that can be assigned to different resources. This segmentation allows the workload manager to track individual resource status and independently manage failures, as each task is separable and can be reassigned without affecting other resources.
Solution Approach 2:
The workload manager acts as an intermediary between the job and individual computing resources. It maintains control over task assignment and can independently manage resource failures by reallocating tasks to available resources, rather than requiring all resources to function as a unified group.
Data Source
AI summary
Embodiments of the present disclosure may provide a dynamic query execution model. This query execution model may provide acceleration by scaling out parallel parts of a query (also referred to as a fragment) to additional computing resources, for example computing resources leased from a pool of computing resources. Execution of the parts of the query may be coordinated by a parent query coordinator, where the query originated, and a fragment query coordinator.