Database Coroutine Scheduling with Two-Level Task Queues
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database execution engines face challenges in efficiently processing complex and large-scale SQL requests due to limitations in parallel execution and resource management, leading to lock conflicts, cache locality issues, and reduced execution speed.
Innovation Solution
A two-level queue system is implemented, utilizing a global task queue shared by multiple threads and local task queues, where coroutine tasks are scheduled based on time information and CPU resource quotas to prioritize execution, ensuring efficient distribution and interruption of tasks across threads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single-level queue system is used for task scheduling, then the system structure is simple, but task distribution and resource utilization are inefficient
Solution Approach 1:
The patent divides the single-level queue into a two-level queue system consisting of a global task queue and multiple local task queues. The global task queue receives and distributes tasks to local task queues, which then execute tasks on specific threads. This segmentation enables better task distribution and resource utilization while maintaining manageable system complexity through hierarchical organization.
2Productivity
If tasks are executed without interruption, then execution continuity is maintained, but resource overhead increases and concurrency is reduced
Solution Approach 1:
The patent implements periodic interruption of task execution based on time slices. Each thread is allocated a time slice during which it executes tasks from its local task queue. When the time slice expires, the thread is interrupted and the task is moved to the global task queue for redistribution. This periodic action enables better resource management, reduces overhead, and improves concurrency by allowing multiple threads to execute tasks in parallel across different time slices.
3Ease of operation
If tasks are distributed randomly to threads, then distribution is simple, but execution priority and fairness are compromised
Solution Approach 1:
The patent introduces execution parameters to tasks, including cumulative execution time and CPU resource quotas. These parameters are used to calculate execution priorities, where tasks with lower cumulative execution time and higher quotas receive higher priorities. This parameter-based approach maintains ease of operation through automated priority calculation while ensuring reliable and fair task scheduling based on actual task requirements and historical execution data.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
The present disclosure relates to a database-based data processing method, device, medium and electronic apparatus, the method comprising: receiving a query request task to be executed, and determining a plurality of coroutine tasks corresponding to the query request task, wherein the coroutine tasks are first added to a global task queue shared by a plurality of threads, and then distributed to a local task queue of a corresponding thread from the global task queue; in each thread, determining a target coroutine task to be executed according to time information of each coroutine task in the local task queue of the thread; interrupting the target coroutine task and adding the target coroutine task to the global task queue when execution of the target coroutine task is not completed and the target coroutine task has been executed in the thread for a current time slice; according to the global task queue, and the local task queue of the thread, determining a new target coroutine task for the thread, and executing the new target coroutine task in a next time slice.