Parallel Query Processing via Dynamic Fragmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional parallel query processing methods are limited by the need for knowledge of underlying database partitions, making them unsuitable for real-time scalable data integration from heterogeneous databases without partitioning, and they reduce query processing efficiency when partitions overlap.
Innovation Solution
A system and method for parallel query processing that dynamically fragments queries into sub-queries using a query fragment identifier, scheduler, and metadata manager, which estimates the number of sub-queries based on available query evaluator nodes and processing characteristics, aggregates results, and streams them without requiring knowledge of database partitions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional parallel query processing methods use knowledge of underlying database partitions, then query processing speed is improved, but the system becomes unsuitable for databases without partitioning and reduces efficiency when partitions overlap
Solution Approach 1:
The patent introduces a metadata manager as an intermediary layer between the query processing system and the underlying databases. This metadata manager dynamically discovers and manages partitioning information without requiring prior knowledge of the database partitioning scheme. The system uses this intermediary to adapt to different database configurations (partitioned or non-partitioned) while maintaining efficient query processing through dynamic partition detection and query rewriting mechanisms.
Solution Approach 2:
The patent implements dynamic query processing where the system adaptively adjusts its partitioning strategy based on runtime conditions. The query optimizer dynamically determines whether to use partition-aware or partition-agnostic processing paths based on the actual database configuration. This dynamic approach allows the system to switch between different processing modes to maintain both speed and compatibility across various database scenarios.
2Loss of time
If data is integrated from heterogeneous databases in real-time, then data integration timeliness is improved, but computational complexity increases
Solution Approach 1:
The patent segments the complex query processing task into multiple independent sub-tasks that can be executed in parallel. The query is divided into several sub-queries that are distributed across multiple query evaluator nodes. This segmentation reduces the computational burden on individual nodes, enables parallel processing to achieve real-time performance, and simplifies the overall computational complexity by breaking down the heterogeneous data integration problem into manageable chunks.
Solution Approach 2:
The patent combines results from multiple heterogeneous data sources through a unified result aggregation mechanism. The system merges data from different database types and structures by standardizing the query interface and result format. This merging approach allows real-time integration of heterogeneous databases while managing computational complexity through a unified processing framework that handles diversity at the metadata level rather than requiring complex source-specific processing logic.
3Productivity
If queries are fragmented into multiple sub-queries for parallel processing, then processing throughput is improved, but query complexity and coordination overhead increase
Solution Approach 1:
The patent implements feedback mechanisms where the metadata manager continuously monitors the performance and status of query evaluator nodes. Based on this feedback, the system dynamically adjusts the query fragmentation strategy, redistributes sub-queries to optimize load balancing, and adapts the level of parallelism. This feedback-driven approach improves processing throughput by optimizing query distribution while reducing coordination overhead through intelligent, adaptive decision-making rather than rigid pre-planning.
Data Source
AI summary
A method and system for parallel query processing is provided. The method includes estimating a number of sub-queries that may be formed corresponding to an input query. The method further includes fragmenting the input query into a plurality of sub queries by adding one or more filter predicates to the input query without altering the semantics of the input query. Further, processing of the sub queries is scheduled. Thereafter, one or more query execution plans are generated and processed. Following the processing of the query execution plans, the results of the processed sub queries are collated, aggregated and presented in a required format.