Database Aggregation Flow Using Parallel IO and Re-Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, particularly when handling large volumes of data.
Innovation Solution
A database system architecture that employs parallelized data input, storage, retrieval, and query processing systems, utilizing a network of computing devices and nodes with independent processing core resources to optimize data partitioning, storage, and query execution, enabling efficient and concurrent handling of massive data sets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in traditional database systems with standard processing methods, then data storage is maintained, but processing speed is limited due to hardware constraints
Solution Approach 1:
The patent segments the database processing system into multiple computing devices, each handling specific portions of data. The data is divided into partitions that can be processed independently across different devices, transforming a single bottlenecked processing system into a distributed parallel processing architecture that overcomes hardware speed limitations.
Solution Approach 2:
The patent transitions from single-device processing to multi-device distributed processing, adding a spatial dimension to the processing architecture. By distributing data across multiple computing devices and utilizing networked connectivity, the system processes data in parallel across different computational nodes, effectively increasing processing speed beyond what a single hardware system could achieve.
2Productivity
If complex database functions are executed using traditional methods, then data processing is performed, but execution efficiency is hindered by restricted co-processing options
Solution Approach 1:
The patent creates a universal processing framework where multiple computing devices can perform various database functions including filtering, aggregation, sorting, and joining operations. Each computing device in the distributed system can execute multiple types of co-processing functions, eliminating the limitations of restricted co-processing options and enabling flexible execution of complex database functions across the network.
3Loss of time
If data is processed in a centralized manner, then processing logic is simplified, but processing time increases for large datasets
Solution Approach 1:
The patent segments both the data and the processing logic across multiple computing devices. Data is partitioned into distributed segments that reside on different devices, and processing operations are distributed to the appropriate devices based on data location. This segmentation enables parallel processing of large datasets, significantly reducing processing time compared to centralized approaches while managing complexity through structured distribution.
Solution Approach 2:
The patent implements preliminary actions by pre-partitioning data across computing devices and pre-positioning processing logic near relevant data segments. This preliminary distribution and organization of resources enables immediate parallel processing when queries are executed, eliminating the need for data consolidation and reducing processing time for large datasets.
Data Source
AI summary
A database system is operable to generate an initial query operator execution that includes an IO operator for execution serially before an aggregation operator indicating performance of an aggregation. The initial query operator execution flow is converted into an updated query operator execution flow semantically equivalent to the initial query operator execution flow based on updating the IO operator to include the performance of the aggregation and adding a re-aggregation operator serially after the IO operator. A corresponding query is executed by applying the updated query operator execution flow based on executing, via each of a plurality of parallelized resources, the IO operator generate corresponding sub-aggregation output via performance of the aggregation, and executing the re-aggregation operator upon data blocks generated based on execution of the IO operator across the plurality of parallelized resources.