Graph-Based Query Optimizer for Parallel Job Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Query optimization frameworks like the Cascades framework are not suitable for multi-core architectures, as they can only execute jobs sequentially, limiting the ability to take advantage of increased CPU performance from multiple cores, which is essential for computationally intensive tasks like query optimization.
Innovation Solution
A graph-based query optimizer is introduced that builds a dependency graph to represent and schedule query optimization jobs across multiple cores, allowing for parallel execution of jobs based on encoded dependencies, utilizing a state machine to manage job states and minimize lock contention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a stack-based scheduler is used to manage query optimization jobs, then job execution order is controlled, but parallel execution across multiple cores is prevented
Solution Approach 1:
The patent segments the sequential stack-based scheduler into multiple independent worker threads that can execute jobs in parallel. Each worker thread maintains its own stack and executes jobs independently, allowing the system to utilize multiple CPU cores simultaneously while maintaining the stack-based scheduling logic within each thread.
Solution Approach 2:
The patent transitions from a single-dimensional sequential execution model to a multi-dimensional parallel execution model by introducing multiple worker threads running concurrently. This dimensional change allows jobs to be executed across multiple cores simultaneously, dramatically improving query optimization productivity.
2Productivity
If query optimization jobs are executed sequentially, then resource contention is minimized, but CPU performance from multiple cores is not utilized
Solution Approach 1:
The patent introduces a job queue as an intermediary data structure that manages the distribution of jobs to worker threads. The job queue acts as a mediator that coordinates job distribution while maintaining dependency relationships, allowing parallel execution without losing control over job ordering and dependency management.
Solution Approach 2:
The patent implements dynamic job assignment where worker threads dynamically pick up jobs from the shared job queue based on availability and dependency satisfaction. This dynamic approach allows the system to adapt to varying workloads and efficiently utilize available CPU cores while maintaining proper job execution order through dependency tracking.
Data Source
AI summary
A computer readable storage medium comprises executable instructions to receive a query. A graph is built to represent jobs associated with the query. The jobs are assigned to parallel threads according to the graph.