Distributed Database Subsystems for Parallel Query Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, leading to inefficiencies in data processing and query execution.
Innovation Solution
A database system architecture with parallelized sub-systems for data input, storage, and query processing, utilizing a network of computing devices and optimized query planning to distribute and execute queries across multiple nodes, enhancing data partitioning and storage strategies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single centralized database system is used for data storage and processing, then system simplicity and ease of operation are maintained, but processing speed and productivity are limited by hardware constraints and data storage methods
Solution Approach 1:
The database system is divided into multiple independent subsystems (first subsystem, second subsystem, third subsystem, etc.), each capable of autonomous data storage and query processing. This segmentation allows parallel processing of queries across different subsystems, significantly improving processing speed while maintaining individual subsystem simplicity.
Solution Approach 2:
The patent introduces a new dimension of organization by grouping subsystems into sets and using a coordinator to manage these sets. This dimensional approach enables hierarchical coordination and parallel query execution across multiple subsystems, transforming a single-point bottleneck into a distributed processing architecture.
2Loss of time
If data is stored in a centralized manner, then data consistency and access control are simplified, but query execution time increases due to sequential processing limitations
Solution Approach 1:
Data is segmented and stored across multiple independent subsystems rather than in a single centralized location. Each subsystem maintains its own data partition, enabling parallel query execution and reducing query execution time while the coordinator manages access control across the distributed system.
Solution Approach 2:
A coordinator subsystem is introduced as an intermediary between clients and the distributed subsystems. The coordinator receives queries, distributes them to appropriate subsystems, and collects results, thereby managing access control and coordination without requiring direct client-subsystem interaction complexity.
3Productivity
If hardware resources are increased to improve processing speed, then productivity increases, but system cost and complexity increase
Solution Approach 1:
Instead of increasing a single centralized hardware resource, the system segments processing across multiple subsystems with their own hardware resources. This allows linear scaling of processing throughput by adding more subsystems rather than proportionally increasing individual hardware complexity.
Solution Approach 2:
Each subsystem is designed as a universal, multi-functional unit capable of independent data storage, query processing, and result generation. This universality allows the system to scale by simply adding more identical subsystems rather than configuring complex specialized hardware, maintaining ease of deployment while increasing productivity.
Data Source
AI summary
A database system includes a load and store sub-system including a data input module to ingest data, short term storage, and long term storage. The database system further includes a query execution sub-system having a plurality of local query engines operable to collectively execute a set of local query operational instructions on at least a portion of the ingested data set to produce a local partial query response, a plurality of intermediate query engines operable to collectively execute a set of intermediate query operational instructions on at least a portion of the local partial query response to produce an intermediate query response, and a global query engine operable to execute a set of global query operational instructions on at least a portion of the intermediate partial query response to produce a query result. The database system further includes a query planning subsystem operable to generate and assign the operational instructions.