Query Engine SQL Plan Optimization via Internal Feature Library
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current query engines face high labor costs and network communication overheads due to the need for multiple data suppliers and frequent interactions with heterogeneous data sources, leading to inefficient query processing.
Innovation Solution
A query processing method that determines the SQL query manner based on statically configured data source feature information stored in an internal library, reducing the need for real-time data source access and optimizing query operations by decomposing SQL into logical and physical plans using this information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple data suppliers are customized for heterogeneous data sources, then data access capability is improved, but labor costs and system complexity increase
Solution Approach 1:
The patent introduces a universal data source feature library that stores feature information for multiple heterogeneous data sources in a unified manner. This library serves as a common interface for all data sources, eliminating the need to customize separate data suppliers for each data source type. The feature library universally handles features such as data types, storage formats, and access protocols across different data sources, thereby reducing system complexity while maintaining adaptability.
Solution Approach 2:
Instead of creating custom data suppliers for each data source, the patent creates feature information copies or descriptions of each data source's characteristics and stores them in the feature library. These feature copies enable the query engine to understand and access different data sources without requiring actual custom suppliers, reducing the complexity of system implementation.
2Measurement precision
If real-time SQL requests are sent to data sources during query execution, then query accuracy is improved, but network communication overhead increases
Solution Approach 1:
The patent pre-stores data source feature information including data types, storage formats, and access protocols in the feature library before query execution. This preliminary action allows the query engine to retrieve necessary information from the library without sending real-time requests to data sources, significantly reducing network communication overhead while maintaining query accuracy through pre-acquired feature knowledge.
Solution Approach 2:
The data source feature library acts as an intermediary between the query engine and actual data sources. Instead of the query engine directly communicating with data sources for each query, it first consults the feature library for access information, thereby reducing direct network communication and its associated overhead while preserving the ability to accurately access data sources when needed.
3Loss of information
If data source feature information is obtained by accessing data sources in real-time, then information freshness is improved, but query processing time increases
Solution Approach 1:
The system performs preliminary actions by pre-collecting and storing data source feature information in the feature library before actual query processing. This includes data types, storage formats, and other access characteristics. By having this information prepared in advance, the query engine can quickly retrieve it during query processing without real-time data source access, thus reducing query processing time while maintaining sufficient information freshness for effective query optimization.
Data Source
AI summary
A query processing method including decomposing an SQL into logical plans based on data source feature information, to obtain a logical plan set, where the data source feature information is stored in an internal storage space of the query engine; generating physical plans for the logical plan set based on the data source feature information, to obtain a physical plan set; determining query costs of the physical plan set based on the data source feature information, to obtain a physical plan with a highest priority; and executing the physical plan with the highest priority, to obtain a query result queried by a user.