Parallel Database Routing with Bloom Filters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large database management systems face challenges in efficiently processing and storing data due to hardware limitations, data storage methods, and restricted co-processing options, leading to suboptimal performance.
Innovation Solution
The implementation of a parallelized database system architecture that includes parallelized data input, storage, retrieval, and query processing subsystems, utilizing a Lazy Auxiliary Pairing Heap data structure and Bloom Filters to improve sorting efficiency and reduce processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a large database management system processes and stores data using traditional sequential methods, then hardware limitations and restricted co-processing options constrain processing speed, but implementing parallel processing architecture increases system complexity
Solution Approach 1:
The patent divides the database management system into multiple parallel processing subsystems including parallelized data input subsystem, parallelized storage subsystem, parallelized retrieval subsystem, and parallelized query processing subsystem. Each subsystem operates independently on different data partitions, enabling concurrent processing while maintaining manageable complexity through modular architecture.
Solution Approach 2:
The patent transitions from sequential single-threaded processing to parallel multi-threaded processing by adding a temporal dimension to data processing. Multiple queries and data operations execute simultaneously across different processing cores and threads, fundamentally changing the processing paradigm from one-dimensional sequential execution to multi-dimensional parallel execution.
2Loss of time
If traditional sorting algorithms are used in database operations, then processing time increases for large datasets, but implementing efficient sorting reduces query execution time
Solution Approach 1:
The patent implements pre-computed Bloom filters for sorting operations. Before actual sorting begins, the system pre-processes data to create Bloom filter structures that enable efficient range queries and sorting operations. This preliminary action significantly reduces the time required for sorting large datasets during query execution.
Solution Approach 2:
The patent creates auxiliary data structures including Bloom filters and sorting indices that are copies or representations of the actual data. These auxiliary structures enable efficient sorting and querying without repeatedly accessing and processing the entire original dataset, thereby reducing query execution time while maintaining data integrity.
3Productivity
If more co-processing options are added to overcome hardware limitations, then system complexity and resource requirements increase, but restricted co-processing options limit performance optimization
Solution Approach 1:
The patent designs parallel processing subsystems that can handle multiple types of database operations simultaneously. The same parallelized storage subsystem can perform both data insertion and retrieval operations, while the query processing subsystem can execute multiple query types concurrently. This multi-functionality increases productivity without proportionally increasing system complexity.
Solution Approach 2:
The patent implements self-optimizing parallel processing where the system automatically manages task distribution, load balancing, and resource allocation across parallel subsystems. The database management system dynamically assigns queries to appropriate processing threads and adjusts parallelism levels based on system state, enabling high throughput without requiring complex external coordination or manual configuration.
Data Source
AI summary
A method for execution, by a first intermediate node of a plurality of nodes in a database management system, includes processing a message that includes data that is being sent in accordance with a routing path from a source node to a destination node, is a first size, and indicates a next node of the routing path, wherein the first intermediate node is limited to communication with a subset of nodes of the plurality of nodes, and wherein the subset of nodes includes the next node. The method further includes maintaining a tracking table that indicates a total amount of data sent to each node of the subset of nodes during a first time period. The method further includes resetting the total amount of data sent to each of the subset of nodes to zero based one or more of a command and an initiation of a second time period.