Distributed Database Query Engine Digest List Parallelism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current query engines in distributed databases face inefficiencies in processing query requests due to limitations in parallelism and resource management, leading to suboptimal performance and latency in query execution.
Innovation Solution
The implementation of a query engine with a query processor that generates a digest list for each query request, utilizing multiple query worker threads for asynchronous I/O operations and SSD-level parallelism, along with secondary indexes for efficient data retrieval and prioritization, enables real-time execution of queries across multiple data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional query engines are used in distributed databases, then system simplicity is maintained, but query execution efficiency and parallelism are limited
Solution Approach 1:
The query engine is segmented into multiple independent worker threads, each capable of handling query requests autonomously. This segmentation enables parallel query processing across multiple threads while maintaining modular system architecture, resolving the contradiction by improving productivity through parallelism without proportionally increasing overall system complexity.
Solution Approach 2:
The system transitions from sequential single-threaded query processing to multi-dimensional parallel processing across multiple worker threads. This dimensional shift from 1D (single thread) to 2D/3D (multiple concurrent threads) enables simultaneous query execution, dramatically improving query execution efficiency while the modular thread architecture prevents exponential complexity growth.
2Loss of time
If sequential query processing is used, then resource management is simplified, but query latency increases
Solution Approach 1:
Multiple query worker threads operate continuously and concurrently, eliminating idle time between sequential query completions. This continuous parallel action ensures that database resources are constantly utilized across multiple queries simultaneously, reducing overall query latency while the thread pool management framework handles resource allocation systematically.
Solution Approach 2:
The system pre-allocates a pool of query worker threads that are ready to execute queries immediately upon arrival. This preliminary preparation of computational resources eliminates startup overhead for each query and reduces latency, while the pre-configured thread pool simplifies runtime resource management compared to dynamic thread creation.
3Productivity
If limited parallelism is implemented, then system stability is maintained, but query processing throughput is reduced
Solution Approach 1:
The query worker thread pool is designed with dynamic characteristics, allowing the system to adaptively manage parallelism levels based on system state and query workload. This dynamic approach enables high throughput during normal operation while maintaining stability through controlled parallelism, resolving the contradiction by making both throughput and stability adjustable rather than fixed.
Data Source
AI summary
In one embodiment, a computer-implemented method of a distributed database system included the step of providing a query processor to receive a query request and to chart out a query execution plan based on the query request. The method includes the step of providing a data generator to generate a digest list based on the query request, wherein the list of digests comprises each stream or filter applied in servicing the query request. The method includes the step of providing a query engine to obtain a digest list from a multiple-data source as a result of the query request and to filter the digest list. The method includes the step of providing a query worker to obtain the query request and process the query request based on a type of the query request. The method includes the step of providing one or more query worker threads to perform an asynchronous I/O operation for the query request.