Partitioned Database Data Aggregation Without Reshuffling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems struggle to handle large-scale security data streams efficiently, as they require reshuffling of data across nodes for real-time JOIN operations, limiting their ability to process events in excess of one million per second, which is insufficient for modern security analytics.
Innovation Solution
The system partitions database data across multiple nodes and uses a data director to distribute data without reshuffling, enabling real-time filtering, joining, and aggregation by replicating dimensions and storing facts in separate tables, allowing for scalable processing of over two million events per second.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is partitioned across multiple nodes for distributed processing, then processing capacity and scalability are improved, but data reshuffling is required for JOIN operations which limits real-time processing speed
Solution Approach 1:
The patent segments the database into dimension tables and fact tables, with dimension tables replicated across all nodes and fact tables partitioned across nodes. This segmentation allows JOIN operations to be performed without data reshuffling, as each node has access to the necessary dimension data locally.
Solution Approach 2:
Dimension tables are replicated across all nodes in advance before any JOIN operations are needed. This preliminary action ensures that when fact tables from different partitions need to be joined, the dimension data is already available at each node, eliminating the need for runtime data reshuffling and enabling real-time processing.
2Measurement precision
If data reshuffling is performed for JOIN operations across distributed nodes, then accurate query results are achieved, but processing time increases beyond real-time capabilities
Solution Approach 1:
Dimension tables are pre-replicated across all nodes before query execution. This preliminary action ensures that when JOIN operations are needed, the dimension data is already available at each node, eliminating the time-consuming data reshuffling process while maintaining query result accuracy.
Solution Approach 2:
Dimension tables are copied to all nodes in the distributed system. This copying approach allows each node to perform JOIN operations using local dimension data without needing to access or transfer data from other nodes, thereby maintaining accuracy while dramatically reducing processing time.
3Reliability
If a single database node is used for security data stream aggregation, then data integrity is maintained, but the system cannot handle large-scale security data streams exceeding one million events per second
Solution Approach 1:
The database system is segmented into multiple nodes, with dimension tables replicated across all nodes and fact tables partitioned across nodes. This segmentation allows the system to distribute the processing load across multiple nodes while maintaining data integrity through the replicated dimension tables that ensure consistent JOIN operations.
Solution Approach 2:
Dimension tables are replicated across all nodes in advance, enabling each node to independently perform JOIN operations on its local fact table data without compromising data integrity. This preliminary replication allows the system to scale to handle over two million events per second while maintaining the same level of data integrity as a single-node system.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, apparatus, systems and articles of manufacture are disclosed. An example partitioned computer database system includes a plurality of nodes, a data director to distribute a plurality of portions of database data across the plurality of nodes, queriers associated with respective ones of the plurality of nodes, the queriers to execute respective sub-queries of respective portions of the database data, and a coordinator to receive a request to query the database data, and merge results of the plurality of sub-queries to form a response to the request.