Database Query Execution via Blocking Channel Flow Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database systems face delays and high memory requirements due to synchronization overheads and buffering of data during query execution, especially when processing non-linear operations, leading to inefficient resource consumption and suboptimal performance.
Innovation Solution
The implementation of blocking channels between operators in a database system allows for streaming data processing, eliminating the need for stage-based coordination and reducing synchronization overheads, enabling threads to wait only when necessary and avoiding unnecessary CPU cycles by ensuring data is available before processing continues.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional stage-based query execution with coordinator processor is used, then query processing can be coordinated systematically, but synchronization delays occur between coordinator processor and other processors
Solution Approach 1:
The patent extracts the coordinator processor from the query execution system and replaces it with autonomous threads that directly coordinate with each other through blocking channels. This removes the central coordination bottleneck and eliminates synchronization delays associated with the coordinator processor model.
Solution Approach 2:
The patent introduces blocking channels as intermediary communication mechanisms between threads. These channels use bounded buffers to mediate data flow and synchronization between producer and consumer threads, replacing the coordinator processor's direct synchronization approach.
2Reliability
If large amount of data is buffered for prolonged periods to ensure availability to subsequent stages, then data availability is guaranteed, but large amount of memory is required
Solution Approach 1:
The patent implements dynamic buffering through blocking channels with bounded buffers. The buffer size is fixed but the data flows dynamically through the pipeline based on consumer readiness. This replaces the static approach of pre-buffering all data with a dynamic flow-based approach that uses minimal memory.
Solution Approach 2:
The patent enables continuous data flow through the query execution pipeline using producer-consumer threading models. Data is processed continuously as it becomes available rather than being buffered in large batches, maintaining continuous useful action while minimizing memory requirements.
Data Source
AI summary
A database system generates an execution plan including multiple operators for processing a database query, for example, a join query or a group by query. The database system allocates a set of threads. Threads communicate with other threads via blocking channels. A blocking channel includes a buffer of a fixed capacity. The database system processes the database query by streaming data through operators of the execution plan. A thread sends data generated by an operator to another thread via the blocking channel if the buffer of the blocking channel has available capacity to store the data, or else the thread blocks until the buffer has capacity to store the data. Similarly, a thread receives data generated by an operator of another thread via the blocking channel if the buffer of the blocking channel has available data, or else the thread blocks until the buffer has data.