Data query method, database platform, and distributed system
By adjusting the execution order of logical plan nodes in the distributed system, the query plan for data query requests is optimized, solving the problem of low execution efficiency in existing technologies and achieving a significant improvement in data query performance.
Patent Information
- Application Number
- PCT/CN2026/082928
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-13
- Filing Date
- 2026-03-11
- Publication Date
- 2026-11-19
AI Technical Summary
In existing technologies, distributed systems are slow in executing data query requests, which leads to reduced database query performance.
By parsing data query requests to generate logical execution plans, the target logical plan node to be optimized and its associated nodes are identified, and their execution order is adjusted so that the associated logical plan nodes are located upstream of the target logical plan node, thereby optimizing the query plan and improving concurrency and query performance.
The query plan was optimized, query performance was improved, execution efficiency was increased by more than 10 times, and the total amount of network transmission was reduced.
Smart Images

Figure CN2026082928_19112026_PF_FP_ABST
Abstract
Description
Data query methods, database platforms and distributed systems Technical Field
[0001] This disclosure relates to the field of database technology, and in particular to a data query method, database platform and distributed system. Background Technology
[0002] With the rapid development of computer and network technologies, distributed systems have been widely used, and database technology has also developed rapidly. It has the characteristics of abundant hardware resources, high processing performance, and the ability to meet the needs of real-time big data analysis, and is widely used in many industries and fields.
[0003] In existing technologies, various database operators can be used to execute corresponding data query requests. Currently, distributed systems execute data query requests by retrieving the corresponding data according to the query plan, which is relatively slow and reduces the database query performance. Therefore, a more efficient data query method is urgently needed. Summary of the Invention
[0004] In view of this, embodiments of this disclosure provide a data query method. One or more embodiments of this disclosure also relate to a database platform, a distributed system, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.
[0005] According to a first aspect of the present disclosure, a data query method is provided, comprising:
[0006] Parse the data query request to obtain the corresponding logical execution plan, wherein the logical execution plan includes at least one logical plan node;
[0007] From the at least one logical plan node, determine the target logical plan node to be optimized, and determine at least one associated logical plan node that is associated with the target logical plan node;
[0008] Adjust the execution order of the target logical plan node and the at least one associated logical plan node to generate an updated logical execution plan, wherein the execution order of the associated logical plan nodes in the updated logical execution plan is upstream of the target logical plan node, and different associated logical plan nodes are used to process data from different data sources;
[0009] Based on the update logic execution plan, the data query request is executed to obtain the corresponding query results.
[0010] According to a second aspect of the present disclosure, a database platform is provided, comprising:
[0011] The parsing module is configured to parse data query requests and obtain the corresponding logical execution plan, wherein the logical execution plan includes at least one logical plan node;
[0012] The determination module is configured to determine the target logical plan node to be optimized from the at least one logical plan node, and to determine at least one associated logical plan node that is associated with the target logical plan node;
[0013] The generation module is configured to adjust the execution order of the target logical plan node and the at least one associated logical plan node to generate an updated logical execution plan, wherein the execution order of the associated logical plan nodes in the updated logical execution plan is upstream of the target logical plan node, and different associated logical plan nodes are used to process data from different data sources;
[0014] The execution module is configured to execute the data query request based on the update logic execution plan and obtain the corresponding query results.
[0015] According to a third aspect of the present disclosure, a distributed system is provided, including a front-end and a distributed platform, the distributed platform including a task scheduling unit and at least one task execution unit;
[0016] The front end is configured to initiate data query requests;
[0017] The task scheduling unit is configured to parse the data query request to obtain a corresponding logical execution plan, wherein the logical execution plan includes at least one logical plan node; determine a target logical plan node to be optimized from the at least one logical plan node, and determine at least one associated logical plan node that is related to the target logical plan node, wherein the target logical plan node is used to integrate data from multiple data sources in the distributed platform into a data stream; adjust the execution order of the target logical plan node and the at least one associated logical plan node to generate an updated logical execution plan, wherein the execution order of the associated logical plan nodes in the updated logical execution plan is upstream of the target logical plan node, and different associated logical plan nodes are used to process data from different data sources; and generate multiple query tasks based on the updated logical execution plan, and schedule the multiple query tasks to the at least one task execution unit.
[0018] The task execution unit is configured to execute the scheduled query task, access the corresponding data source, obtain the corresponding query result, and then feed the query result back to the front end.
[0019] According to a fourth aspect of the present disclosure, a computing device is provided, comprising:
[0020] Memory and processor;
[0021] The memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the data query method described above.
[0022] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the data query method described above.
[0023] According to a sixth aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data query method.
[0024] One embodiment of this disclosure provides a data query method that parses a data query request into a corresponding logical execution plan. From at least one logical plan node included in the logical execution plan, a target logical plan node requiring optimization is identified, and at least one associated logical plan node related to the target logical plan node is identified. The execution order of the target logical plan node and the at least one associated logical plan node is adjusted to generate an updated logical execution plan. In this updated logical execution plan, the execution order of the associated logical plan nodes is upstream of the target logical plan node, and different associated logical plan nodes are used to process data from different data sources. Based on this updated logical execution plan, the data query request is executed to obtain the corresponding query results. Thus, for the target logical plan node requiring optimization, the execution order of the target logical plan node and at least one associated logical plan node can be adjusted, placing the associated logical plan node upstream of the target logical plan node, directly connecting to at least one data source, concurrently executing data query requests, and obtaining the corresponding query results. This optimizes the execution order of the target logical plan node and the associated logical plan nodes, achieving a better query plan, improving execution concurrency, and enhancing query performance. Attached Figure Description
[0025] Figure 1 is a flowchart of a data query method provided in an embodiment of this disclosure;
[0026] Figure 2 is a schematic diagram of the initial execution sequence in a logical execution plan provided by an embodiment of the present disclosure;
[0027] Figure 3 is a schematic diagram of an adjusted execution order in a logical execution plan provided by an embodiment of the present disclosure;
[0028] Figure 4 is a schematic diagram of the initial execution sequence in another logical execution plan provided by an embodiment of the present disclosure;
[0029] Figure 5 is a schematic diagram of the adjusted execution order in another logical execution plan provided by an embodiment of this disclosure;
[0030] Figure 6 is a flowchart of a data query method provided in an embodiment of this disclosure;
[0031] Figure 7 is a schematic diagram of the structure of a database platform provided in an embodiment of this disclosure;
[0032] Figure 8 is a schematic diagram of the structure of a distributed system provided in an embodiment of this disclosure;
[0033] Figure 9 is a schematic diagram of the processing procedure of a distributed system provided in an embodiment of this disclosure;
[0034] Figure 10 is a structural block diagram of a computing device provided in an embodiment of this disclosure. Detailed Implementation
[0035] Numerous specific details are set forth in the following description to provide a full understanding of this disclosure. However, this disclosure can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this disclosure. Therefore, this disclosure is not limited to the specific implementations disclosed below.
[0036] The terminology used in one or more embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this disclosure. The singular forms “a” and “the” as used in one or more embodiments of this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this disclosure refers to and includes any or all possible combinations of one or more associated listed items.
[0037] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this disclosure, and similarly, second may also be referred to as first. Depending on the context, the word “if” as used herein may be interpreted as “when”, “in response to a determination”, or “when…”.
[0038] First, the terms and concepts involved in one or more embodiments of this disclosure will be explained.
[0039] A database is a repository for organizing, storing, and managing data according to a data structure. In computer science, a database refers to a long-term, organized, and shareable collection of data stored within a computer. Data in a database is organized, described, and stored according to a specific mathematical model, exhibiting low redundancy, high data independence, and easy scalability, and can be shared by various users.
[0040] SQL (Structured Query Language) is a programming language used to manage and manipulate databases. It is mainly used to process data in databases, such as querying, inserting, updating and deleting data.
[0041] Data query statements are standardized instructions used in database systems to retrieve, manipulate, or manage data, typically following specific query language specifications (such as SQL). Their core function is to extract or process information that meets certain conditions from the data store and return the results to the requesting party.
[0042] An Abstract Syntax Tree (AST) is an abstract representation of the syntactic structure of source code. It's an intermediate representation generated by a compiler or interpreter when parsing source code, primarily used for program analysis, transformation, and optimization. An AST is an abstract representation of source code, ignoring syntactic details (such as parentheses and semicolons) and retaining only semantically relevant information. An AST represents the syntactic structure of a programming language in a tree structure. The root node represents the entire program, leaf nodes represent the most basic syntactic units, and intermediate nodes represent complex syntactic structures. Each node in the tree represents a syntactic unit in the source code, and nodes typically contain type information and child nodes.
[0043] Database operators are the basic units of operation used in a database system to execute queries and process data. They are building blocks in the database query execution plan and are responsible for operations such as data access, filtering, joining, and aggregation.
[0044] Logical execution plan: This is an intermediate representation generated by the database management system during query optimization, used to describe the logical operation flow of a query. Logical execution plans are typically represented by a tree structure, where each node represents a logical operation (such as scanning a table, filtering data, joining tables, etc.). The leaf nodes of the tree are usually data sources (such as tables or views), while the internal nodes represent operations on the data.
[0045] OLAP (Online Analytical Processing) is a data analytics technique specifically designed for processing large-scale datasets. It allows users to perform fast, stable, and interactive analysis of data from multiple dimensions. Through pre-computation and multidimensional data structures (such as data cubes), OLAP can quickly respond to complex queries, helping businesses and analysts gain deeper insights into patterns, trends, and relationships within data. It is commonly used in decision support systems, business intelligence, and other fields to provide managers with decision-making support.
[0046] Simple Log Service (SLS) is a one-stop log data solution that enables log data collection, storage, querying, analysis, visualization, and alerting. It supports massive log data writes per second, allowing for rapid processing of large-scale logs; it can collect logs from various data sources such as servers, containers, databases, and applications; it provides SQL-like query methods for quickly locating and analyzing log information; it displays log data in intuitive charts and reports through dashboards and other tools; and it can monitor log data in real time according to preset rules, issuing timely alerts when conditions are met.
[0047] A distributed platform is a database system that distributes data across multiple physical locations connected by a network, with each location possessing its own processing capabilities (such as storage and computation). The goal of a distributed platform is to improve performance, reliability, availability, and scalability by distributing data and computing resources. In a distributed platform, data is partitioned and stored across different nodes (servers or devices), each node can independently manage its local data while collaborating with other nodes to complete global tasks.
[0048] It's important to note that in distributed platforms, the union operator is used to aggregate multiple data sources into a single data stream, making it a crucial database operator. Furthermore, union operators are essential implementation methods in areas such as multi-cluster shared views and materialized view data completion. Single-machine databases, due to limited hardware resources, cannot adequately meet the processing performance demands of big data. Distributed databases often implement union operators in stages, but union operators cannot pass down other downstream operators to their upstream counterparts, resulting in the transmission of unnecessary raw data and slower execution efficiency.
[0049] Therefore, this disclosure provides a data query method that addresses the poor performance of joint operators by constructing a high-performance distributed joint operator pass-through optimization. This pass-through optimizes multiple downstream operators of the joint operator, achieves a better query execution plan, improves execution concurrency, avoids invalid processing of original data, reduces the total network transmission volume, and improves execution performance by more than 10 times.
[0050] This disclosure provides a data query method, and also relates to a database platform, a distributed system, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.
[0051] Referring to Figure 1, Figure 1 shows a flowchart of a data query method provided according to an embodiment of the present disclosure, specifically including the following steps 102 to 108.
[0052] Step 102: Parse the data query request to obtain the corresponding logical execution plan, wherein the logical execution plan includes at least one logical plan node.
[0053] Specifically, a data query request refers to a request initiated by the front end to the database to obtain specific data. For example, in a log service scenario, a user can initiate a query request for certain log data through the front end. In response to the query request, the relevant data is retrieved from at least one database storing the log data and then fed back to the front end.
[0054] It's important to note that a logical execution plan is an intermediate representation generated by the database management system during query optimization, used to describe the logical operation flow of a query. Logical execution plans are typically represented by a tree structure, where each node represents a logical operation (such as scanning a table, filtering data, or joining tables). The leaf nodes of the tree are usually data sources (such as tables or views), while the internal nodes represent operations performed on the data.
[0055] In practice, the front-end can initiate a data query request to the database platform. Upon receiving the request, the database platform can parse it and generate a corresponding logical execution plan. This plan includes at least one logical plan node, which corresponds to different database operators within the platform. Examples of logical plan nodes include: UnionNode for Union, TableScanNode for TableScan, ProjectNode for Project, FilterNode for Filter, TableWriteNode for TableWrite, TopNNode for TopN, DistinctLimitNode for DistinctLimit, and AggregationNode for Aggregation. Additionally, the logical execution plan can also include JoinNodes for Join operators.
[0056] Specifically, the front-end can display an interactive query interface where users can input query parameters and initiate a data query request. This request can include information about the data to be queried, such as the table name, query conditions, and fields to be returned. The database platform receives and parses this query request, converting the text and / or character information into instructions that the database can understand. It analyzes the table name, field names, and query conditions to clarify the specific data to be retrieved. Based on the parsing results, the database platform generates a corresponding logical execution plan, describing how it will retrieve data that meets the query conditions from the database. The query is then executed according to this plan, and the results are returned to the front-end.
[0057] In one optional implementation of this embodiment, the logical execution plan is a tree structure; parsing the data query request to obtain the corresponding logical execution plan includes:
[0058] Based on the data query request, generate the corresponding data query statement;
[0059] Parse the data query statement, perform syntactic structure analysis on the data query statement to obtain the corresponding abstract syntax tree, and generate a logical execution plan based on the abstract syntax tree.
[0060] Specifically, the logical execution plan is a tree structure, and the nodes in this tree structure are logical plan nodes. Different logical plan nodes represent different logical operations.
[0061] In practice, the data query request initiated by the front end may be text and / or character information. The database platform can generate a data query statement in Structured Query Language (SQL) based on this request. Then, using a pre-defined lexical and syntactic analysis tool, the platform performs lexical and syntactic analysis on this data query statement, generating the corresponding Abstract Syntax Tree (AST). This pre-defined lexical and syntactic analysis tool can be any tool capable of performing lexical and syntactic analysis on data query statements, such as Antlr4, which supports target code generation for multiple programming languages and is widely used in building language recognition tools, such as compiler front-ends, parsers, and data format conversion tools.
[0062] It should be noted that when the database platform receives a data query request, it can convert it into a data query statement that conforms to the Structured Query Language (SQL) specification, based on the specific requirements of the request, such as which data to query, what filtering conditions to apply to the data, and how to sort the data, and then generate the corresponding data query statement according to the SQL syntax.
[0063] For example, if the data query request is "information on students in the class whose Chinese scores are greater than 80", the database platform can generate an SQL query statement like "SELECT * FROM score table WHERE Chinese score > 80".
[0064] In practice, data query statements can be decomposed into individual "lexical units" (Tokens), such as keywords, identifiers, and operators. Then, according to the grammatical rules of the language, the lexical units are organized into a tree structure to generate the corresponding AST. Then, based on the semantics of different nodes in the AST, they are converted into different logical plan nodes to obtain the logical execution plan corresponding to the data query request. Subsequently, the query operation can be performed in the database based on the logical execution plan to obtain the data that meets the conditions.
[0065] In this embodiment of the disclosure, users can directly initiate data query requests through the front end. The database platform processes these requests into data query statements, performs syntax structure analysis to obtain the corresponding abstract syntax tree, and generates the corresponding logical execution plan based on the abstract syntax tree to realize data query. This eliminates the need for users to input data query statements that conform to the structured query language, lowering the query threshold and improving query efficiency.
[0066] Step 104: Determine the target logical plan node to be optimized from at least one logical plan node, and determine at least one associated logical plan node that is related to the target logical plan node.
[0067] Specifically, a target logical plan node refers to a logical plan node whose execution order is optimized to improve the execution concurrency of the logical execution plan. For example, the target logical plan node is an integrated type of logical plan node, such as a union node, an intersection node (IntersectNode), a difference node (Minus), etc.
[0068] Associated logical plan nodes are logical plan nodes that are related to the target logical plan node. These associated logical plan nodes can be preprocessing or post-processing nodes of the target logical plan node. The execution order between the target logical plan node and associated logical plan nodes affects the amount of input or output data of the target logical plan node, thus impacting query performance such as execution concurrency and data transfer volume. Therefore, in practical implementation, the target logical plan node to be optimized can be identified from at least one logical plan node, and at least one associated logical plan node can be identified to allow for subsequent optimization of the execution order between the target logical plan node and associated logical plan nodes, thereby improving execution concurrency and query performance.
[0069] In one optional implementation of this embodiment, it is applied to a distributed database, and the target logical plan node is a federated node. The federated node is used to integrate multiple data sources in the distributed database into a single data stream.
[0070] The associated logical plan nodes include at least one of the following: table read node, mapping node, filtering node, table write node, ordered constraint node, deduplication constraint node, and aggregation node.
[0071] In actual implementation, taking the data query method provided in this embodiment of the disclosure as an example, it can be applied to a distributed database, that is, it can query relevant data from databases distributed in multiple physical locations.
[0072] In one optional implementation, taking the target logical plan node as a federated node as an example, the federated node is used to integrate multiple data sources in the distributed database into a data stream. In this case, the associated logical plan node may include at least one of the following: table read node, mapping node, filtering node, table write node, ordered constraint node, deduplication constraint node, and aggregation node.
[0073] Specifically, the union node is a set operator in SQL used to merge the result sets of multiple queries; its mathematical basis is the union operation in set theory. The table read node is the most basic data access operator in the database execution plan, retrieving data by scanning the entire table row by row, without relying on any index structure. The mapping node is a projection operation in relational algebra, responsible for selecting specific columns (subsets of fields) from the input dataset. It is the core implementation operator of the SELECT clause in SQL, capable of performing complex expression operations on a single row of data. The filter node is used to filter out a portion of the input data. The table write node is used to insert the results of the query statement into other tables, especially third-party databases (such as MySQL). This node is often a performance bottleneck; changing its execution order with the union node can increase concurrency and improve interoperability with third-party databases. The sorting node sorts the original data and retrieves the first N rows; only the first N rows of the original data are output. The deduplication node removes duplicates from the original data and retrieves the first N rows; only the first N rows of the original data are output. Aggregation nodes can group and aggregate the original data. If the original data has 10,000 rows but only 10 groups, only 10 results will be output.
[0074] In this embodiment, the input and output data volumes of the associated logical plan nodes, such as table read nodes, mapping nodes, filtering nodes, table write nodes, ordered constraint nodes, deduplication constraint nodes, and aggregation nodes, are disproportionate, with the output data volume being much smaller than the input data volume. If these associated logical plan nodes are located downstream of the joint node, the original data from multiple databases in the distributed database needs to be transmitted to the joint node for processing, potentially transmitting a large amount of unnecessary original data, wasting network bandwidth, greatly increasing the amount of data transmitted over the network, and resulting in slow execution efficiency. However, by adjusting the associated logical plan nodes from downstream to upstream of the joint node, the original data from multiple databases in the distributed database can be filtered separately through these associated logical plan nodes, and the output data can then be transmitted to the joint node for processing. This significantly reduces the amount of data transmitted over the network, improves execution concurrency, and thus improves execution efficiency.
[0075] In one optional implementation of this embodiment, determining the target logical plan node to be optimized from at least one logical plan node includes:
[0076] Determine the logical operation type of at least one logical plan node;
[0077] The logical plan node with the logical operation type of integration is determined as the target logical plan node, where the integration type is used to integrate data from at least one data source.
[0078] In actual implementation, the logical operation type of at least one logical plan node can be determined. This logical operation type is used to indicate what type of operation the corresponding logical plan node is performing. For example, if the logical operation type is integration type, it means that the corresponding logical plan node is used to integrate the data from at least one data source into a data stream output.
[0079] It should be noted that for at least one logical plan node in the logical execution plan, the type of logical operation performed by each node needs to be clearly defined. A logical plan node can be understood as a unit that performs a specific logical operation, and the logical operation type may include different types such as integration and filtering. Specifically, the corresponding logical operation type can be determined based on the node attributes of each logical plan node. For example, if the node attribute of a union node is "find union," its corresponding logical operation type is integration; similarly, if the node attribute of an intersection node is "find intersection," its corresponding logical operation type is also integration.
[0080] In practice, after determining the logical operation type of each logical plan node, the node with the logical operation type "integration type" is identified as the target logical plan node. The "integration type" is a special logical operation type whose function is to integrate data from at least one data source. This data source can be different database tables, files, etc., or different tables, files, etc. in the same database.
[0081] In this embodiment of the disclosure, since the target logical plan node of the integration type can integrate data from at least one data source, the amount of data from at least one data source transmitted to the logical plan node determines the amount of data transmitted over the network, the amount of data that the target logical plan node needs to process, the concurrency, query efficiency, and other query performance. Therefore, based on the logical operation type of each logical plan node, the logical plan node with data integration function can be identified as the target logical plan node to be optimized, and its execution order can be optimized to improve query performance.
[0082] In one optional implementation of this embodiment, determining at least one associated logical plan node that is associated with the target logical plan node includes:
[0083] Identify at least one post-processing node located downstream of the target logical plan node in the logical plan node, and use the post-processing node as an associated logical plan node. The post-processing node is used to process the output of the target logical plan node downstream of the target logical plan node, and the output data volume of the post-processing node is less than the input data volume.
[0084] In actual implementation, at least one post-processing node located downstream of the target logical plan node can be identified in the logical plan node. The post-processing node is regarded as an associated logical plan node. That is, the associated logical plan node should be located downstream of the target logical plan node according to the logical execution plan. It processes the output result of the target logical plan node and outputs the corresponding processing result. The amount of input data and the amount of output data of the associated logical plan node are not proportional, and the amount of output data is much smaller than the amount of input data.
[0085] It should be noted that since the output data volume of the associated logical plan node is much smaller than the input data volume, changing the execution order between the associated logical plan node and the target logical plan node can greatly affect the amount of data transmitted to the target logical plan node, the amount of data that the target logical plan node needs to process, and thus affect query performance such as query efficiency and concurrency.
[0086] In this embodiment of the disclosure, a post-processing node that was originally located downstream of the target logical plan node and whose output data volume is less than the input data volume can be identified. This post-processing node is then used as the target logical plan node, which facilitates subsequent adjustment and optimization of the execution order between the associated logical plan node and the target logical plan node, reduces the amount of data transmitted to the target logical plan node, reduces the amount of data transmitted over the network, and improves concurrency, thereby improving the query performance of the database.
[0087] Step 106: Adjust the execution order of the target logical plan node and at least one associated logical plan node to generate an updated logical execution plan. In the updated logical execution plan, the execution order of the associated logical plan nodes is upstream of the target logical plan node, and different associated logical plan nodes are used to process data from different data sources.
[0088] It should be noted that the execution order between the associated logical plan node and the target logical plan node can greatly affect the amount of data transmitted to the target logical plan node and the amount of data that the target logical plan node needs to process, thereby affecting query efficiency, concurrency, and other query performance. Therefore, the execution order between the target logical plan node and at least one associated logical plan node can be adjusted and optimized to generate an updated logical execution plan, thereby achieving a better query plan, improving the concurrency of execution, and thus improving query performance.
[0089] In actual implementation, initially, the associated logical plan node is located downstream of the target logical plan node. The target logical plan node aggregates data from multiple data sources, and the associated logical plan nodes further process the data output by the target logical plan node. The adjustment and optimization of the execution order can refer to reversing the original execution order of the target logical plan node and at least one associated logical plan node, so that the execution order of the associated logical plan node is upstream of the target logical plan node. Different associated logical plan nodes are used to process data from different data sources to achieve concurrent execution of queries, and then the results are output to the target logical plan node for processing. For example, if the original execution order is target logical plan node - associated logical plan node, then the adjusted and optimized execution order is associated logical plan node - target logical plan node.
[0090] In one optional implementation of this embodiment, adjusting the execution order of the target logical plan node and at least one associated logical plan node to generate an updated logical execution plan includes:
[0091] Determine the node type of at least one associated logical plan node;
[0092] Based on the node type of at least one associated logical plan node, adjust the execution order of the target logical plan node and at least one associated logical plan node to generate an updated logical execution plan.
[0093] It should be noted that the node type of each associated logical plan node can be determined. This node is used to indicate whether changing its execution order will cause errors in the logical execution plan, that is, whether changing its execution order will affect the correct execution of the logical execution plan. Different node types take different execution order adjustment measures to adjust the execution order of the target logical plan node and at least one associated logical plan node, and generate an updated logical execution plan.
[0094] In actual implementation, different related logical plan nodes have different dependencies and execution requirements. Therefore, different execution order adjustment measures are configured for different types of nodes. The node type of each related logical plan node is determined. When adjusting the execution order, the execution order adjustment measures corresponding to that node type are adopted to adjust the execution order of the target logical plan node and related logical plan nodes. This avoids improper execution order adjustment from damaging the dependencies between nodes, thereby avoiding errors in the execution plan caused by adjusting the execution order and affecting the correct execution of the logical execution plan.
[0095] In this embodiment of the disclosure, different node types are distinguished, and different execution order adjustment measures are taken for the associated logical nodes of different node types. On the basis of optimizing the execution order to improve the concurrency and execution performance, errors in the execution plan caused by improper execution order adjustment are avoided, and the logical execution plan is correctly executed during the execution order optimization process.
[0096] In one optional implementation of this embodiment, the initial execution order of at least one associated logical plan node is downstream of the target logical plan node; the execution order of the target logical plan node and at least one associated logical plan node is adjusted according to the node type of the at least one associated logical plan node to generate an updated logical execution plan, including:
[0097] Identify at least one of the associated logical plan nodes whose node type is a non-dependent conflict type as the first associated logical plan node;
[0098] Adjust the execution order of the first associated logical plan node to be upstream of the target logical plan node. The non-dependency conflict type is the node type that will not cause execution errors after the execution order is adjusted.
[0099] In practice, for first associated logical plan nodes of non-dependent conflict type, no execution error will occur after adjusting their execution order. Therefore, for first associated logical plan nodes of non-dependent conflict type that do not cause execution errors after adjusting their execution order, the first associated logical plan node can be directly adjusted from being downstream of the target logical plan node to being upstream of the target logical plan node, so as to reduce the amount of data that needs to be transmitted to the target logical plan node.
[0100] As an example, taking the target logical plan node as the union node, and the associated logical plan nodes including table read nodes, mapping nodes, filtering nodes, table write nodes, ordered constraint nodes, deduplication constraint nodes, and aggregation nodes, the execution order of the mapping node, filtering node, and table write node is adjusted from downstream to upstream of the union node without causing an execution error. Therefore, the mapping node, filtering node, and table write node are determined to be the first associated logical plan node of the non-dependent conflict type.
[0101] It's worth noting that mapping nodes can perform a series of complex expression operations on a single row of data. After being passed up to the upstream of the join node, these operations are executed at the same stage as the table read nodes. Multiple mapping nodes can connect to databases located in different physical locations, process data from different databases separately, and transmit the processing results to the join node for further processing. This increases execution concurrency several times over and significantly reduces the amount of data that needs to be transmitted to the join node. Table write nodes are used to insert the results of query statements into other tables. After being passed up to the upstream of the join node, they can directly connect to databases located in different physical locations, increasing execution concurrency and enabling better integration with third-party databases. Filtering nodes can connect to data located in different physical locations, filter data from different databases separately, pre-filtering out a portion of the data. The filtered data is then transmitted to the join node for further processing, increasing execution concurrency several times over and significantly reducing the amount of data that needs to be transmitted to the join node. By passing through the first associated logical plan node of non-dependent conflict type to the upstream of the target logical plan node, each first associated logical plan node can process data from different databases and then transmit the processing results to the joint node. This not only increases the execution concurrency several times but also reduces the total network transmission volume.
[0102] For example, taking the first associated logical plan node as a filtering node, Figure 2 is a schematic diagram of the initial execution order in a logical execution plan provided by an embodiment of this disclosure. As shown in Figure 2, table read nodes 1 to 3 read data from databases 1 to 3 respectively, and transmit the read data to the joint node. The joint node merges the data from the three databases and then transmits it to the filtering node for filtering. Taking the query "male students' scores in the class" as an example, table read node 1 retrieves the scores of each student in class 1 from database 1, table read node 2 retrieves the scores of each student in class 2 from database 2, and table read node 3 retrieves the scores of each student in class 3 from database 3. The scores of each student in class 1, class 2, and class 3 are transmitted to the joint node for merging, and the scores of each student in each class are output to the filtering node. The filtering node filters out the scores of female students from the scores of each student in each class to obtain the scores of male students.
[0103] Figure 3 is a schematic diagram of the adjusted execution order in a logical execution plan provided by an embodiment of this disclosure. As shown in Figure 3, table read nodes 1 to 3 read data from databases 1 to 3 respectively. The data from database 1 read by table read node 1 is transmitted to filter node 1, which filters the data in database 1 and transmits the filtered data to the join node. The data from database 2 read by table read node 2 is transmitted to filter node 2, which filters the data in database 2 and transmits the filtered data to the join node. The data from database 3 read by table read node 3 is transmitted to filter node 3, which filters the data in database 3 and transmits the filtered data to the join node. The join node processes the filtered data from each database. Taking the query "male students' grades in a class" as an example, the process is as follows: Table read node 1 retrieves the grades of all students in class 1 from database 1. These grades are then input to filter node 1 to remove female students' grades. The filtered male students' grades are then transmitted to the join node. Similarly, table read node 2 retrieves the grades of all students in class 2 from database 2. These grades are then input to filter node 2 to remove female students' grades. The filtered male students' grades are then transmitted to the join node. Finally, table read node 3 retrieves the grades of all students in class 3 from database 3. These grades are then input to filter node 3 to remove female students' grades. The filtered male students' grades are then transmitted to the join node. The join node only needs to merge the male students' grades from each class to obtain the male students' grades, thus improving query concurrency and reducing the total network transmission volume.
[0104] In one optional implementation of this embodiment, the initial execution order of at least one associated logical plan node is downstream of the target logical plan node; the execution order of the target logical plan node and at least one associated logical plan node is adjusted according to the node type of the at least one associated logical plan node to generate an updated logical execution plan, including:
[0105] Identify at least one second associated logical plan node whose node type is a dependency conflict type;
[0106] Add a second associated logical plan node upstream of the target logical plan node and retain the second associated logical plan node downstream of the target logical plan node. The dependency conflict type is the node type that causes an error after adjusting the execution order.
[0107] In actual implementation, for second related logical plan nodes with dependency conflicts, adjusting their execution order will result in execution errors. Therefore, for second related logical plan nodes with dependency conflicts that will result in execution errors after adjusting their execution order, a second related logical plan node can be added upstream of the target logical plan node, and the second related logical plan node can be retained downstream of the target logical plan node to ensure that the logical execution plan can be executed correctly.
[0108] As an example, taking the target logical plan node as the union node, and the associated logical plan nodes including table read nodes, mapping nodes, filtering nodes, table write nodes, ordered constraint nodes, deduplication constraint nodes, and aggregation nodes as an example, if the execution order of the ordered constraint node, deduplication constraint node, and aggregation node is changed from downstream of the union node to upstream of the union node, it will cause an execution error. Therefore, the ordered constraint node, deduplication constraint node, and aggregation node are determined to be the second associated logical plan nodes of the dependency conflict type.
[0109] It's important to note that the ordered constraint node sorts the original data and retrieves the first N rows; only the first N rows will be output. The deduplication constraint node removes duplicate data and retrieves the first N rows; only the first N rows will be output. The aggregation node groups and aggregates the original data; if the original data has 10,000 rows but only 10 groups, only 10 results will be output. By passing the ordered constraint node, deduplication constraint node, and aggregation node upstream of the join node for execution, and executing them at the same stage as the table read node, the execution concurrency is increased several times, and the amount of output data during the data source execution stage is significantly reduced.
[0110] For example, taking the second associated logical plan node as an ordered constraint node, Figure 4 is a schematic diagram of the initial execution order in another logical execution plan provided by an embodiment of this disclosure. As shown in Figure 4, table read nodes 1 to 3 read data from databases 1 to 3 respectively, and transmit the read data to the joint node. The joint node merges the data from the three databases and then transmits it to the ordered constraint node for filtering. Taking the query of "top 10 student scores" as an example, table read node 1 retrieves the scores of each student in class 1 from database 1, table read node 2 retrieves the scores of each student in class 2 from database 2, and table read node 3 retrieves the scores of each student in class 3 from database 3. The scores of each student in class 1, class 2, and class 3 are transmitted to the joint node for merging, and the scores of each student in each class are output to the ordered constraint node. The ordered constraint node selects the top 10 student scores from the scores of each student in each class.
[0111] Figure 5 is a schematic diagram of the adjusted execution order in another logical execution plan provided in an embodiment of this disclosure. As shown in Figure 5, table read nodes 1 to 3 read data from databases 1 to 3 respectively. The data from database 1 read by table read node 1 is transmitted to ordered constraint node 1. Ordered constraint node 1 sorts the data in database 1 and outputs the top N data, which are then transmitted to the joint node. The data from database 2 read by table read node 2 is transmitted to ordered constraint node 2. Ordered constraint node 2 sorts the data in database 2 and outputs the top N data, which are then transmitted to the joint node. The data from database 3 read by table read node 3 is transmitted to ordered constraint node 3. Ordered constraint node 3 sorts the data in database 3 and outputs the top N data, which are then transmitted to the joint node. The joint node merges the top N data from each database and outputs the merged data to ordered constraint node 4. Ordered constraint node 4 sorts the merged data from the joint node and outputs the top N data. Taking the query "Top 10 student scores" as an example, the system retrieves the scores of each student in class 1 from database 1 via table read node 1, inputs these scores into ordered constraint node 1, sorts the scores, and outputs the scores of the top 10 students in class 1. Similarly, it retrieves the scores of each student in class 2 from database 2 via table read node 2, inputs these scores into ordered constraint node 2, sorts these scores, and outputs the scores of the top 10 students in class 2. Finally, it retrieves the scores of each student in class 3 from database 3 via table read node 3, inputs these scores into ordered constraint node 3, sorts these scores, and outputs the scores of the top 10 students in class 3. The scores of the top 10 students in class 1, class 2, and class 3 are transmitted to the joint node. The joint node merges these scores to obtain 30 student scores, which are then transmitted to the ordered constraint node 4. The ordered constraint node 4 sorts these 30 student scores and outputs the scores of the top 10 students. The joint node only needs to merge the scores of the top 10 students from each class, improving query concurrency and reducing the total network transmission volume.
[0112] In this embodiment, for a first associated logical plan node of non-dependent conflict type, the first associated logical plan node is directly adjusted from being downstream of the target logical plan node to being upstream of the target logical plan node; for a second associated logical plan node of dependent conflict type, a second associated logical plan node is added upstream of the target logical plan node, and the second associated logical plan node is retained downstream of the target logical plan node. Different testing and optimization of the execution order are adopted for associated logical plan nodes of non-dependent conflict type and dependent conflict type, respectively. While ensuring the correctness of the execution logic, the execution order is optimized, avoiding the transmission of invalid data, reducing network bandwidth, improving the concurrency of the original node execution, and improving execution performance by several times.
[0113] Step 108: Based on the update logic execution plan, execute the data query request and obtain the corresponding query results.
[0114] In actual implementation, the database platform can execute data query requests in various data sources (such as different database tables, files, etc.) based on the updated logical execution plan obtained after adjusting the execution order between the target logical plan node and the associated logical plan node, perform data retrieval and processing, return the query results that meet the conditions to the database platform, obtain the corresponding query results, and feed them back to the front end. The front end can be the user's application, web interface, etc., for the user to view and use.
[0115] In one optional implementation of this embodiment, based on the update logic execution plan, a data query request is executed to obtain the corresponding query results, including:
[0116] Based on at least one associated logical plan node in the update logical execution plan, preprocess the data from at least one data source respectively;
[0117] At least one preprocessed data source is transmitted to the target logical plan node. Based on the target logical plan node, the preprocessed data from at least one data source is integrated to obtain the query results corresponding to the data query request.
[0118] In actual implementation, the update logic execution plan adjusts the associated logic plan nodes located downstream of the target logic plan node to upstream of the target logic plan node, directly preprocesses the data from at least one data source, and then transmits the preprocessed data to the target logic plan node. Based on the target logic plan node, the preprocessed data from at least one data source is integrated to obtain the query results corresponding to the data query request.
[0119] Continuing with the previous example, as shown in Figure 3, table read nodes 1 through 3 read data from databases 1 through 3 respectively. Data from database 1 read by table read node 1 is transmitted to filter node 1, which preprocesses the data (filtering out irrelevant raw data) and transmits the preprocessed data to the join node. Data from database 2 read by table read node 2 is transmitted to filter node 2, which also preprocesses the data (filtering out irrelevant raw data) and transmits the preprocessed data to the join node. Data from database 3 read by table read node 3 is transmitted to filter node 3, which preprocesses the data (filtering out irrelevant raw data) and transmits the preprocessed data to the join node. The join node merges the preprocessed data from each database. If other logical plan nodes exist, execution continues until the query result corresponding to the data query request is obtained. If no other logical plan nodes exist, the output of the join node is used as the query result corresponding to the data query request.
[0120] In practice, the query results corresponding to the obtained data query requests can also be fed back to the front end, where the data query results can be visualized and processed in other ways, such as displaying alarms or issuing alarm prompts when certain set conditions are met.
[0121] It should be noted that by first filtering out a portion of unnecessary raw data from the data source through the upstream associated logical plan nodes of the target logical plan node before transmitting it to the target logical plan node, the amount of data to be transmitted is greatly reduced, saving network bandwidth. Furthermore, by preprocessing the data from at least one data source concurrently through multiple associated logical plan nodes, the concurrency of executing query tasks can be greatly improved, thus increasing query efficiency.
[0122] One embodiment of this disclosure provides a data query method. For a target logical plan node that needs optimization, the execution order of the target logical plan node and at least one associated logical plan node can be adjusted. The associated logical plan node is moved upstream of the target logical plan node, directly connected to at least one data source, and data query requests are executed concurrently to obtain the corresponding query results. This optimizes the execution order of the target logical plan node and the associated logical plan node, achieving a better query plan, thereby improving the concurrency of execution and query performance.
[0123] The following description, in conjunction with Figure 6, uses the application of the data query method provided in this disclosure under a log service as an example to further illustrate the data query method. Figure 6 shows a flowchart of the processing procedure of a data query method provided in an embodiment of this disclosure, which can be applied to an OLAP platform to provide log query services in a distributed database, and specifically includes the following steps.
[0124] Step 602: Receive the log query request initiated by the user through the front end, and generate a corresponding logical execution plan based on the log query request. The logical execution plan includes at least one logical plan node, which includes at least a union node, a table read node, a mapping node, a filtering node, a table write node, an ordered constraint node, a deduplication constraint node, and an aggregation node. The above nodes in the logical execution plan are initially executed downstream of the union node.
[0125] Step 604: Directly pass through the mapping node, filtering node, and table write node to the upstream of the joint node, pass through the ordered constraint node, deduplication constraint node, and aggregation node to the upstream of the joint node, and retain the node downstream of the joint node, and generate an update logic execution plan.
[0126] Step 606: Execute the update logic execution plan. Taking the filter node in the update logic execution plan as an example, table read nodes 1 to N read log data from databases 1 to N respectively. The log data of database 1 read by table read node 1 is transmitted to filter node 1. Filter node 1 filters the log data of database 1, removing irrelevant original data, and transmits the filtered log data to the join node; ...; The log data of database N read by table read node N is transmitted to filter node N. Filter node N filters the log data of database N, removing irrelevant original data, and transmits the filtered log data to the join node. The join node merges the preprocessed log data of each database. If other logic plan nodes exist later, execution continues until the log query result corresponding to the log query request is obtained; if no other logic plan nodes exist later, the output of the join node is used as the log query result corresponding to the log query request.
[0127] Step 608: Feed back the log query results to the front end for visualization.
[0128] One embodiment of this disclosure provides a data query method. For a target logical plan node that needs optimization, the execution order of the target logical plan node and at least one associated logical plan node can be adjusted. The associated logical plan node is moved upstream of the target logical plan node, directly connecting to at least one data source, and executing data query requests concurrently. By having the associated logical plan node upstream of the target logical plan node first filter out a portion of unnecessary raw data from the log data before transmitting it to the target logical plan node for processing, the amount of data to be transmitted is greatly reduced, saving network bandwidth. Furthermore, by having multiple associated logical plan nodes process at least one log data concurrently, the concurrency of executing query tasks can be greatly improved, query efficiency can be increased, and query performance can be significantly enhanced.
[0129] Corresponding to the above method embodiments, this disclosure also provides a database platform embodiment. Figure 7 shows a schematic diagram of the structure of a database platform provided in one embodiment of this disclosure. As shown in Figure 7, the database platform includes:
[0130] The parsing module 702 is configured to parse data query requests and obtain the corresponding logical execution plan, wherein the logical execution plan includes at least one logical plan node;
[0131] The determination module 704 is configured to determine the target logical plan node to be optimized from at least one logical plan node, and to determine at least one associated logical plan node that is associated with the target logical plan node.
[0132] The generation module 706 is configured to adjust the execution order of the target logical plan node and at least one associated logical plan node to generate an updated logical execution plan, wherein the execution order of the associated logical plan nodes in the updated logical execution plan is upstream of the target logical plan node, and different associated logical plan nodes are used to process data from different data sources;
[0133] Execution module 708 is configured to execute data query requests based on the update logic execution plan and obtain the corresponding query results.
[0134] Optionally, the logical execution plan is a tree structure; the parsing module 702 is further configured as follows:
[0135] Based on the data query request, generate the corresponding data query statement;
[0136] Parse the data query statement, perform syntactic structure analysis on the data query statement to obtain the corresponding abstract syntax tree, and generate a logical execution plan based on the abstract syntax tree.
[0137] Optionally, module 704 is further configured as follows:
[0138] Determine the logical operation type of at least one logical plan node;
[0139] The logical plan node with the logical operation type of integration is determined as the target logical plan node, where the integration type is used to integrate data from at least one data source.
[0140] Optionally, module 704 is further configured as follows:
[0141] Identify at least one post-processing node located downstream of the target logical plan node in the logical plan node, and use the post-processing node as an associated logical plan node. The post-processing node is used to process the output of the target logical plan node downstream of the target logical plan node, and the output data volume of the post-processing node is less than the input data volume.
[0142] Optionally, the generation module 706 is further configured as follows:
[0143] Determine the node type of at least one associated logical plan node;
[0144] Based on the node type of at least one associated logical plan node, adjust the execution order of the target logical plan node and at least one associated logical plan node to generate an updated logical execution plan.
[0145] Optionally, the initial execution order of at least one associated logical plan node is downstream of the target logical plan node; the generation module 706 is further configured to:
[0146] Identify at least one of the associated logical plan nodes whose node type is a non-dependent conflict type as the first associated logical plan node;
[0147] Adjust the execution order of the first associated logical plan node to be upstream of the target logical plan node. The non-dependency conflict type is the node type that will not cause execution errors after the execution order is adjusted.
[0148] Optionally, the initial execution order of at least one associated logical plan node is downstream of the target logical plan node; the generation module 706 is further configured to:
[0149] Identify at least one second associated logical plan node whose node type is a dependency conflict type;
[0150] Add a second associated logical plan node upstream of the target logical plan node and retain the second associated logical plan node downstream of the target logical plan node. The dependency conflict type is the node type that causes an error after adjusting the execution order.
[0151] Optionally, execution module 708 is further configured as follows:
[0152] Based on at least one associated logical plan node in the update logical execution plan, preprocess the data from at least one data source respectively;
[0153] At least one preprocessed data source is transmitted to the target logical plan node. Based on the target logical plan node, the preprocessed data from at least one data source is integrated to obtain the query results corresponding to the data query request.
[0154] Optionally, when applied to a distributed database, the target logical plan node is a federated node, which is used to integrate multiple data sources in the distributed database into a single data stream;
[0155] The associated logical plan nodes include at least one of the following: table read node, mapping node, filtering node, table write node, ordered constraint node, deduplication constraint node, and aggregation node.
[0156] One embodiment of this disclosure provides a database platform, including a parsing module, a determining module, a generating module, and an execution module. For a target logical plan node requiring optimization, the execution order of the target logical plan node and at least one associated logical plan node can be adjusted. The associated logical plan node is moved upstream of the target logical plan node, directly connecting to at least one data source, concurrently executing data query requests, and obtaining corresponding query results. This optimizes the execution order of the target logical plan node and associated logical plan nodes, achieving a better query plan, thereby improving execution concurrency and query performance.
[0157] The above is an illustrative scheme of a database platform according to this embodiment. It should be noted that the technical solution of this database platform and the technical solution of the data query method described above belong to the same concept. For details not described in detail in the technical solution of the database platform, please refer to the description of the technical solution of the data query method described above.
[0158] Corresponding to the above method embodiments, this disclosure also provides a distributed system embodiment. Figure 8 shows a schematic diagram of the structure of a distributed system provided by an embodiment of this disclosure. As shown in Figure 8, the distributed system includes a front-end 802 and a distributed platform 804. The distributed platform 804 includes a task scheduling unit 8042 and at least one task execution unit 8044.
[0159] The frontend returned an 802 error, configured to initiate a data query request;
[0160] The task scheduling unit 8042 is configured to parse data query requests and obtain corresponding logical execution plans, wherein the logical execution plans include at least one logical plan node; determine the target logical plan node to be optimized from the at least one logical plan node, and determine at least one associated logical plan node that is related to the target logical plan node, wherein the target logical plan node is used to integrate data from multiple data sources in the distributed platform into a data stream; adjust the execution order of the target logical plan node and at least one associated logical plan node to generate an updated logical execution plan, wherein the execution order of the associated logical plan nodes in the updated logical execution plan is upstream of the target logical plan node, and different associated logical plan nodes are used to process data from different data sources; based on the updated logical execution plan, generate multiple query tasks, and schedule the multiple query tasks to at least one task execution unit;
[0161] The task execution unit is configured to execute the scheduled query task, access the corresponding data source, obtain the corresponding query results, and then send the query results back to the front end.
[0162] Figure 9 illustrates a schematic diagram of the processing procedure of a distributed system according to an embodiment of this disclosure. As shown in Figure 9, the distributed system includes a front-end and a distributed platform. The distributed platform includes a task scheduling unit and a task execution unit. The task scheduling unit includes modules such as a lexical analysis / semantic verification module, a query optimizer, and a task scheduler. The front-end initiates a data query request to the task scheduling unit of the distributed platform. The lexical analysis / semantic verification module in the task scheduling unit performs lexical analysis / semantic verification on the data query request and generates a corresponding logical execution plan. The query optimizer determines the target logical plan node to be optimized from at least one logical plan node in the logical execution plan, and determines at least one associated logical plan node that is related to the target logical plan node. The target logical plan node is used to integrate data from multiple data sources in the distributed platform into a data stream. The execution order of the target logical plan node and at least one associated logical plan node is adjusted to generate an updated logical execution plan. In this updated logical execution plan, the execution order of the associated logical plan nodes is upstream of the target logical plan node, and different associated logical plan nodes are used to process data from different data sources.
[0163] The query optimizer transmits the updated logical execution plan to the task scheduler. The task scheduler generates multiple query tasks 0 to m based on the results of lexical analysis / semantic verification, the generated updated logical execution plan, and data source information. These tasks are then scheduled to task execution units 1 to m. Task execution units 1 to m concurrently process the data from their respective data sources, execute the corresponding query tasks, and transmit the processing results to task execution unit n. Task execution unit n executes query task n and feeds the query results back to the front end.
[0164] Taking Figure 5 above as an example, four query tasks will be generated and scheduled to four task execution units. Task execution units 1 to 3 will execute query tasks 1 to 3 (ordered limited tasks) respectively, access the corresponding databases in parallel, obtain the first N data from each database, and then transmit them to task execution unit 4. Task execution unit 4 will execute query task 4, merge the first N data from each database, sort the merged data, output the first N data, and return them to the front end.
[0165] One embodiment of this disclosure provides a distributed system, including a front-end and a distributed platform. The distributed platform includes a task scheduling unit and a task execution unit. The front-end can initiate data query requests. In the distributed platform, the task scheduling unit can adjust the execution order of the target logical plan node and at least one associated logical plan node. Then, the task execution unit executes the data query request to obtain the corresponding query results. The target logical plan node is used to integrate data from multiple databases in the distributed database into a single data stream. Optimizing the execution order of the target logical plan node and associated logical plan nodes achieves a better query plan, thereby improving execution concurrency and query performance.
[0166] The above is an illustrative scheme of a distributed system according to this embodiment. It should be noted that the technical solution of this distributed system and the technical solution of the data query method described above belong to the same concept. For details not described in detail in the technical solution of the distributed system, please refer to the description of the technical solution of the data query method described above.
[0167] Figure 10 shows a structural block diagram of a computing device according to an embodiment of the present disclosure. The components of the computing device 1000 include, but are not limited to, a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 via a bus 1030, and a database 1050 is used to store data.
[0168] The computing device 1000 also includes an access device 1040, which enables the computing device 1000 to communicate via one or more networks 1060. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1040 may include one or more of any type of wired or wireless network interface (e.g., a network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.
[0169] In one embodiment of this disclosure, the aforementioned components of the computing device 1000, as well as other components not shown in FIG. 10, may be interconnected, for example, via a bus. It should be understood that the computing device block diagram shown in FIG. 10 is merely for illustrative purposes and is not intended to limit the scope of this disclosure. Those skilled in the art can add or replace other components as needed.
[0170] The computing device 1000 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1000 can also be a mobile or stationary server.
[0171] The processor 1020 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data query method.
[0172] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data query method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data query method described above.
[0173] An embodiment of this disclosure also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described data query method.
[0174] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data query method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the data query method described above.
[0175] An embodiment of this disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data query method.
[0176] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data query method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data query method described above.
[0177] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0178] Computer instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in computer-readable media can be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.
[0179] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this disclosure are not limited to the described order of actions, because according to the embodiments of this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this disclosure.
[0180] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0181] The preferred embodiments disclosed above are merely illustrative of this disclosure. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments of this disclosure. These embodiments are selected and specifically described in this disclosure to better explain the principles and practical applications of the embodiments of this disclosure, thereby enabling those skilled in the art to better understand and utilize this disclosure. This disclosure is limited only by the claims and their full scope and equivalents.
Claims
1. A data query method, comprising: Parse the data query request to obtain the corresponding logical execution plan, wherein the logical execution plan includes at least one logical plan node; From the at least one logical plan node, determine the target logical plan node to be optimized, and determine at least one associated logical plan node that is associated with the target logical plan node; Adjust the execution order of the target logical plan node and the at least one associated logical plan node to generate an updated logical execution plan, wherein the execution order of the associated logical plan nodes in the updated logical execution plan is upstream of the target logical plan node, and different associated logical plan nodes are used to process data from different data sources; Based on the update logic execution plan, the data query request is executed to obtain the corresponding query results.
2. The data query method of claim 1, the logical execution plan is a tree structure. The parsing of the data query request yields the corresponding logical execution plan, including: Based on the data query request, generate the corresponding data query statement; The data query statement is parsed, its syntax structure is analyzed to obtain the corresponding abstract syntax tree, and the logical execution plan is generated based on the abstract syntax tree.
3. The data query method according to claim 1, wherein determining the target logical plan node to be optimized from the at least one logical plan node includes: Determine the logical operation type of the at least one logical plan node; The logical plan node with the logical operation type of integration is determined as the target logical plan node, wherein the integration type is used to integrate data from at least one data source.
4. The data query method according to claim 3, wherein determining at least one associated logical plan node that is associated with the target logical plan node includes: A post-processing node located downstream of the target logical plan node is determined among the at least one logical plan node, and the post-processing node is used as the associated logical plan node. The post-processing node is used to process the output result of the target logical plan node downstream of the target logical plan node, and the output data volume of the post-processing node is less than the input data volume.
5. The data query method according to any one of claims 1 to 4, wherein adjusting the execution order of the target logical plan node and the at least one associated logical plan node to generate an updated logical execution plan includes: Determine the node type of the at least one associated logical plan node; Based on the node type of the at least one associated logical plan node, the execution order of the target logical plan node and the at least one associated logical plan node is adjusted to generate the update logical execution plan.
6. The data query method according to claim 5, wherein the initial execution order of the at least one associated logical plan node is downstream of the target logical plan node; the step of adjusting the execution order of the target logical plan node and the at least one associated logical plan node according to the node type of the at least one associated logical plan node to generate the update logical execution plan includes: Determine the first associated logical plan node among the at least one associated logical plan nodes whose node type is a non-dependency conflict type; The execution order of the first associated logical plan node is adjusted to be upstream of the target logical plan node, wherein the non-dependency conflict type is the node type that does not produce execution errors after the execution order is adjusted.
7. The data query method according to claim 5, wherein the initial execution order of the at least one associated logical plan node is downstream of the target logical plan node; the step of adjusting the execution order of the target logical plan node and the at least one associated logical plan node according to the node type of the at least one associated logical plan node to generate the update logical execution plan includes: Identify a second associated logical plan node among the at least one associated logical plan node whose node type is a dependency conflict type; Add a second associated logical plan node upstream of the target logical plan node and retain the second associated logical plan node downstream of the target logical plan node, wherein the dependency conflict type is the node type that causes an error after adjusting the execution order.
8. The data query method according to any one of claims 1 to 4, wherein executing the data query request based on the update logic execution plan to obtain the corresponding query result includes: Based on at least one associated logical plan node in the update logical execution plan, data from at least one data source is preprocessed respectively; The preprocessed data from the at least one data source is transmitted to the target logical plan node. Based on the target logical plan node, the preprocessed data from the at least one data source is integrated to obtain the query result corresponding to the data query request.
9. The data query method according to any one of claims 1 to 4, applied to a distributed database, wherein the target logical plan node is a federated node, and the federated node is used to integrate multiple data sources in the distributed database into a single data stream; The associated logical plan nodes include at least one of the following: table read node, mapping node, filtering node, table write node, ordered constraint node, deduplication constraint node, and aggregation node.
10. The data query method of claim 3, wherein, The logical plan nodes corresponding to the integration type include at least one of the following: joint nodes, intersection nodes, and difference nodes.
11. The data query method of claim 1, wherein, The associated logic plan node includes a mapping node, which is used to select a specific column from the input dataset or perform expression operations.
12. In the data query method according to claim 6, after adjusting the execution order of the first associated logical plan nodes to be upstream of the target logical plan node, multiple first associated logical plan nodes process data from different data sources in parallel to obtain processing results, and transmit the processing results to the target logical plan node.
13. The data query method of claim 9, the join logical plan node comprising the map node, the filter node, and the table write node, wherein, The mapping node, the filtering node, and the table writing node are the first associated logical plan nodes of the non-dependent conflict type.
14. The data query method according to claim 8, based on at least one associated logical plan node in the update logical execution plan, preprocesses data from at least one data source, including: The data from the data source is filtered by the associated logical plan node to reduce the amount of data transmitted to the target logical plan node.
15. The data query method according to claim 1, wherein the data query request is a log query request, the data source is distributed storage log data, and the data query method is applied to a log service platform.
16. A database platform, comprising: The parsing module is configured to parse data query requests and obtain the corresponding logical execution plan, wherein the logical execution plan includes at least one logical plan node; The determination module is configured to determine the target logical plan node to be optimized from the at least one logical plan node, and to determine at least one associated logical plan node that is associated with the target logical plan node; The generation module is configured to adjust the execution order of the target logical plan node and the at least one associated logical plan node to generate an updated logical execution plan, wherein the execution order of the associated logical plan nodes in the updated logical execution plan is upstream of the target logical plan node, and different associated logical plan nodes are used to process data from different data sources; The execution module is configured to execute the data query request based on the update logic execution plan and obtain the corresponding query results.
17. A distributed system, comprising a front-end and a distributed platform, the distributed platform comprising a task scheduling unit and at least one task execution unit; The front end is configured to initiate data query requests; The task scheduling unit is configured to parse the data query request and obtain the corresponding logical execution plan, wherein... The logical execution plan includes at least one logical plan node; From the at least one logical plan node, a target logical plan node to be optimized is determined, and at least one associated logical plan node is determined that is related to the target logical plan node. The target logical plan node is used to integrate data from multiple data sources in the distributed platform into a single data stream. The execution order of the target logical plan node and the at least one associated logical plan node is adjusted to generate an updated logical execution plan. In the updated logical execution plan, the execution order of the associated logical plan nodes is upstream of the target logical plan node, and different associated logical plan nodes are used to process data from different data sources. Based on the updated logical execution plan, multiple query tasks are generated, and the multiple query tasks are scheduled to the at least one task execution unit. The task execution unit is configured to execute the scheduled query task, access the corresponding data source, obtain the corresponding query result, and then feed the query result back to the front end.
18. A computing device, comprising: Memory and processor; The memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the data query method according to any one of claims 1 to 15.
19. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the data query method according to any one of claims 1 to 15.
20. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the data query method according to any one of claims 1 to 15.
