Store-Compute Cluster Parallelism for Faster Database Task Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems are limited by hardware, data storage methods, and co-processor options, leading to inefficiencies in processing speed and execution time.
Innovation Solution
A parallelized database system architecture that includes a parallelized data input, store, retrieve, and process sub-system, along with a parallelized query and response system, utilizing a distributed network of computing devices with independent processing core resources to execute queries efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a traditional database system architecture is used, then the system structure is simple, but the processing speed and execution time are limited by hardware and storage methods
Solution Approach 1:
The database system is divided into multiple independent computing devices organized in a distributed network. Each computing device contains independent processing core resources that can execute queries autonomously. This segmentation allows parallel processing of database operations across multiple devices, significantly improving processing speed while distributing system complexity across modular components rather than concentrating it in a single system
2Loss of time
If parallel processing is implemented across multiple computing devices, then execution time is reduced, but system complexity increases
Solution Approach 1:
The system employs dynamic query plan generation that adapts to current system conditions and data characteristics. The query optimization process dynamically determines the most efficient execution strategy by evaluating multiple possible query plans and selecting the one with the lowest estimated cost. This dynamic approach allows the system to efficiently manage distributed parallel processing without requiring static, overly complex architecture design
Solution Approach 2:
The query optimization process incorporates feedback mechanisms where the system evaluates query execution statistics and performance metrics to continuously improve query plan generation. This feedback loop allows the system to learn from past executions and refine its parallel processing strategies, reducing the need for manually configured complex distributed systems while maintaining high execution efficiency
3Productivity
If hardware resources are increased to improve processing speed, then execution time decreases, but hardware costs and system complexity increase
Solution Approach 1:
Each computing device in the distributed network is designed with universal processing capabilities that can handle multiple types of database operations. The independent processing core resources are multi-functional and can execute various query types and data processing tasks. This universality allows the system to achieve high productivity through software-based parallel processing rather than requiring specialized hardware configurations for different functions, thereby reducing hardware complexity and costs
Data Source
AI summary
A store and compute sub-system of a database system includes a computing cluster that is operable to receive a plurality of tasks. The computing cluster is further operable to execute, in a concurrent manner, the plurality of tasks. For a first task, a first lead computing device of the computing cluster is operable to: generate a plurality of first partial tasks based on the first task and allocate the plurality of first partial tasks to the plurality of computing devices, wherein the plurality of computing devices executes the plurality of first partial tasks. For a second task, a second lead computing device of the computing cluster is operable to: generate a plurality of second partial tasks based on the second task; and allocate the plurality of second partial tasks to the plurality of computing devices, wherein the plurality of computing device executes the plurality of second partial tasks.