Dynamic Routing for Pre-computing Query Engines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The query response speed of pre-computing query systems in distributed computing environments is not ideal, and existing solutions have not effectively addressed this issue.
Innovation Solution
A dynamic routing method and apparatus for a query engine that pre- obtains cube data under a preset dimensional combination, determines the degree of aggregation, and switches between a scatter-and-gather and a total-exchange distributed query engine based on aggregation levels and cost evaluations to optimize query processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a pre-computing query system is used in a distributed computing environment, then computing tasks can be completed in advance using spare resources, but the query response speed is not ideal
Solution Approach 1:
The system dynamically switches between two distributed query engines (scatter-and-gather engine and total-exchange engine) based on the degree of aggregation of cube data. This dynamic adaptation allows the system to optimize query response speed for different data aggregation scenarios while maintaining pre-computing capabilities, thereby resolving the contradiction between productivity and speed.
2Speed
If cube data with high degree of aggregation is processed, then query response speed improves, but data detail and precision are lost
Solution Approach 1:
The system dynamically selects between scatter-and-gather and total-exchange query engines based on the degree of aggregation. For high aggregation scenarios, the scatter-and-gather engine provides fast responses, while for low aggregation scenarios requiring detailed data, the total-exchange engine ensures data precision. This dynamic selection resolves the contradiction between speed and measurement precision.
3Device complexity
If a single distributed query engine is used, then system complexity is reduced, but query processing efficiency for different data aggregation levels is compromised
Solution Approach 1:
The system employs two distributed query engines that serve different functions based on data aggregation levels. The scatter-and-gather engine handles high aggregation queries efficiently, while the total-exchange engine handles low aggregation queries with better data precision. This multi-functional approach improves query processing efficiency for different scenarios without significantly increasing system complexity, as the routing mechanism intelligently directs queries to the appropriate engine.
Data Source
AI summary
The present application discloses a dynamic routing method and apparatus for a query engine in a pre-computing system. The method includes: pre-obtaining cube data under a preset dimensional combination in a pre-computing system; determining a degree of aggregation of the cube data selected as expected under the preset dimensional combination after a query request is received; executing query processing on the query request in a first distributed query engine when the degree of aggregation of the cube data under the preset dimensional combination is high; and switching to a second distributed query engine to execute query processing on the query request when the degree of aggregation of the cube data under the preset dimensional combination is low. The present application solves the technical problem that the query response speed of the pre-computing query system is not ideal. Through the present application, the sub-second high-performance query response can be achieved. At the same time, as a result, higher concurrency can be supported so as to meet business needs, and the stability of the query system is simultaneously guaranteed.