Query Processing System for OLAP and OLTP Scale Support
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data query engines struggle to support both On-Line Analysis Processing (OLAP) and On-Line Transaction Processing (OLTP) scale query services in the Internet of Things field, failing to meet the diverse and voluminous data query requirements effectively.
Innovation Solution
A query processing method and system that parses query requests, generates logical and physical plans, segments plans for OLAP queries, and assigns tasks to worker nodes, enabling efficient processing and result aggregation, while supporting OLTP queries through direct assignment and processing, and utilizing a storage node to map heterogeneous data sources into a relational database for joint querying.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing data query engines are used, then simple query processing is supported, but they cannot support both OLAP scale query service and OLTP scale query service simultaneously
Solution Approach 1:
The system dynamically adapts its processing mode based on query characteristics. The server node judges whether a logical plan belongs to OLAP or OLTP scale query service and applies different processing flows accordingly. For OLAP queries, the system segments the logical plan into multiple logical subplans and distributes them to worker nodes for parallel processing. For OLTP queries, the system uses direct assignment to worker nodes. This dynamic adaptation enables the system to support both query scales effectively.
2Quantity of substance
If data volume increases for OLAP queries, then analysis capability improves, but query processing complexity increases
Solution Approach 1:
The system segments the logical plan into multiple logical subplans based on data volume and query characteristics. Each logical subplan is then assigned to different worker nodes for parallel processing. The server node coordinates the segmentation and aggregation of results, while worker nodes independently process their assigned subplans. This segmentation approach enables the system to handle large-scale OLAP queries by distributing the processing load across multiple nodes, thereby managing complexity effectively.
3Adaptability or versatility
If heterogeneous data sources are integrated, then data diversity increases, but system complexity increases
Solution Approach 1:
The system implements a universal query processing architecture that can handle heterogeneous data sources through a common interface. The server node receives query requests and generates logical plans that can accommodate multiple data source types. The coordinator node and worker nodes process queries using a standardized physical plan generation approach, regardless of the underlying data source heterogeneity. This multi-functional design allows the system to integrate diverse data sources without proportionally increasing system complexity.
Data Source
AI summary
A query processing method. A server node parses a query request and generates a corresponding logical plan; judges the logical plan belonging to an OLAP scale query service or an OLTP scale query service; in response to judging that the logical plan belongs to the OLAP scale query service, segmenting the logical plan into logical subplans. A coordinator node generates a first physical plan corresponding to the logical plan, and feeds back the first physical plan to the server node. The server node assigns each of the logical subplans to the corresponding worker node according to the first physical plan. The worker node processes the corresponding logical subplan to obtain a query subresult, and reports the processing subresult layer by layer. The server node summarizes the received processing subresult, generates a final query result, and feeds back the final query result to a user.