Data processing method and system, and electronic device and storage medium

By generating a set of candidate plans and using a target execution cost prediction model for accurate cost prediction, the problem of low efficiency and high resource consumption of AI query optimizers when handling massive data query requests is solved, achieving efficient data processing and resource utilization.

WO2026237525A1PCT designated stage Publication Date: 2026-11-19ALIBABA (CHINA) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2026/084112
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-12
Filing Date
2026-03-17
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing AI query optimizers are inefficient and resource-intensive when handling massive data query requests, and they ignore project characteristics and differences in resource allocation, resulting in high training and deployment resource costs and insufficient model generalization ability.

Method used

By receiving data query requests, a candidate plan set is generated, and a target execution cost prediction model is used to predict costs. The target execution plan is selected, and the model trained using the target project's training plan is used to perform accurate cost prediction and quickly generate query responses. This ensures the high quality and relevance of the model training data and avoids resource waste.

Benefits of technology

It significantly improved query optimizer performance, optimized resource allocation, accelerated query response speed, improved data processing efficiency and user experience in multi-tenant distributed data warehouses, and reduced overall costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2026084112_19112026_PF_FP_ABST
    Figure CN2026084112_19112026_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure relates to a data processing method and system, and an electronic device and a storage medium. The method comprises: receiving a data query request; generating a first candidate plan set corresponding to the data query request; using a target execution cost prediction model to perform cost prediction on the first candidate plan set, and selecting a target execution plan, wherein the target execution cost prediction model is obtained by performing training by means of a training plan corresponding to a target project, the target project is selected from candidate projects, the target project comprises a plurality of historical queries, and the training plan comprises historical default execution plans and candidate query plans respectively corresponding to the plurality of historical queries; and on the basis of the target execution plan, generating a data query reply corresponding to the data query request. The present disclosure solves the technical problems in the related art of low data processing efficiency and high resource consumption costs when massive data query requests are processed.
Need to check novelty before this filing date? Find Prior Art

Description

Data processing methods, systems, electronic devices and storage media Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more specifically, to a data processing method, system, electronic device, and storage medium. Background Technology

[0002] With the advent of the big data era, the demand for big data processing is increasing, leading to a surge in computational resource consumption challenges for multi-tenant distributed data warehouses due to massive data query requests. As a core component of database management systems, the performance of the query optimizer directly impacts query execution efficiency and resource costs. Traditional database query optimizers typically rely on manually designed rules and cardinality estimation based on independence assumptions to select query plans. However, this approach often results in suboptimal query plans. Artificial Intelligence (AI)-based query optimizers, with their advanced execution cost prediction and plan generation capabilities, have become a key technology for improving query performance. However, these AI query optimizers require collecting execution costs by executing numerous candidate query plans during training, leading to high resource and time costs and severely impacting training efficiency and system resource management. Furthermore, AI query optimizers are often deployed uniformly across all projects, ignoring the differences in project characteristics and resource allocation. This not only increases the resource overhead of training and deployment but also leads to insufficient model generalization ability due to inconsistent data distribution between offline training and online inference phases, further affecting data processing efficiency.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This disclosure provides a data processing method, system, electronic device, and storage medium to at least solve the technical problems of low data processing efficiency and high resource consumption cost in the related art when processing massive data query requests.

[0005] According to one aspect of the present disclosure, a data processing method is provided, comprising: receiving a data query request; generating a first candidate plan set corresponding to the data query request; performing cost prediction on the first candidate plan set using a target execution cost prediction model, and selecting a target execution plan, wherein the target execution cost prediction model is trained using a training plan corresponding to a target item, the target item is selected from the candidate items, the target item includes: multiple historical queries, and the training plan includes: historical default execution plans and candidate query plans corresponding to each of the multiple historical queries; and generating a data query response corresponding to the data query request based on the target execution plan.

[0006] According to another aspect of the embodiments of this disclosure, a data processing method is also provided, comprising: receiving e-commerce platform order query data; generating a first candidate plan set corresponding to the e-commerce platform order query data; performing cost prediction on the first candidate plan set using a target execution cost prediction model, and selecting a target execution plan, wherein the target execution cost prediction model is trained using a training plan corresponding to a target item, the target item is selected from the candidate items, the target item includes: multiple historical queries, and the training plan includes: historical default execution plans and candidate query plans corresponding to each of the multiple historical queries; and generating an e-commerce platform order statistics response corresponding to the e-commerce platform order query data based on the target execution plan.

[0007] According to another aspect of the embodiments of this disclosure, a data processing method is also provided, comprising: obtaining a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes a data query request; and returning a data processing response through a second application programming interface, wherein the response data carried in the data processing response includes a data query reply corresponding to the data query request, wherein the data query reply is obtained according to any one of the data processing methods in the embodiments of this disclosure.

[0008] According to another aspect of the embodiments of this disclosure, a data processing method is also provided, comprising: acquiring a currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: a data query request; responding to the data processing dialogue request and returning a data processing dialogue response, wherein the information carried in the data processing dialogue response includes: a data query response corresponding to the data query request, the data query response being obtained according to any one of the data processing methods in the embodiments of this disclosure; and displaying the data query response in a graphical user interface.

[0009] According to another aspect of the embodiments of this disclosure, a data processing method is also provided, comprising: displaying a data query request on the operation interface in response to an input instruction applied to the operation interface; and displaying a data query response corresponding to the data query request on the operation interface in response to a processing instruction applied to the operation interface; wherein the data query response is obtained according to any one of the data processing methods in the embodiments of this disclosure.

[0010] According to another aspect of the embodiments of this disclosure, a data processing system is also provided, including: a client for sending a data query request; a server connected to the client for generating a first candidate plan set corresponding to the data query request, performing cost prediction on the first candidate plan set using a target execution cost prediction model to select a target execution plan, and generating a data query response corresponding to the data query request based on the target execution plan, wherein the target execution cost prediction model is trained using a training plan corresponding to a target item, the target item is selected from the candidate items, the target item includes: multiple historical queries, and the training plan includes: historical default execution plans and candidate query plans corresponding to each of the multiple historical queries; the client is also used to output the data query response.

[0011] According to another aspect of the present disclosure, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of the present disclosure when it runs.

[0012] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the methods of the various embodiments of the present disclosure.

[0013] According to another aspect of the embodiments of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this disclosure.

[0014] According to another aspect of the embodiments of this disclosure, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods of various embodiments of this disclosure.

[0015] According to another aspect of the embodiments of this disclosure, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this disclosure.

[0016] In this embodiment, by receiving a data query request, a first candidate plan set corresponding to the data query request is generated. Then, a target execution cost prediction model is used to predict the cost of the first candidate plan set, and a target execution plan is selected. The target execution cost prediction model is trained using the training plan corresponding to the target item. The target item is selected from the candidate items, including multiple historical queries. The training plan includes the historical default execution plan and candidate query plan corresponding to each of the multiple historical queries. Finally, a data query response corresponding to the data query request is generated based on the target execution plan. This significantly improves the performance of the query optimizer by intelligently selecting items, customizing the training of the target execution cost prediction model, accurately predicting execution costs, and quickly generating query responses. It further optimizes resource allocation, accelerates query response speed, and ultimately improves the data processing efficiency and user experience of the multi-tenant distributed data warehouse.

[0017] By intelligently selecting target projects from candidate projects and using the training plan corresponding to the target project for model training, the high quality and relevance of model training data can be ensured, which is conducive to building a more accurate target execution cost prediction model that is more adapted to specific scenarios. The target execution cost prediction model can quickly and accurately predict the cost of each plan in the first candidate plan set, thereby rapidly filtering out the target execution plan from the first candidate plan set. This avoids the waste of resources in actually executing all candidate query plans to determine the target execution plan, effectively reducing the overall cost of query processing and improving resource utilization. After selecting the target execution plan, the data query response corresponding to the data query request can be quickly generated based on the target execution plan, thereby accelerating the response speed of query processing in large-scale, high-concurrency query scenarios and improving the user experience. In summary, the embodiments of this disclosure achieve the technical effects of improving data processing efficiency and reducing resource consumption costs, thereby solving the technical problems of low data processing efficiency and high resource consumption costs in related technologies when processing massive data query requests.

[0018] It is worth noting that the above general description and the following detailed description are merely for illustrative and explanatory purposes and do not constitute a limitation thereof. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0020] Figure 1 is a schematic diagram of an application scenario of a data processing method according to an embodiment of the present disclosure;

[0021] Figure 2 is a structural block diagram of a computing environment for a data processing method according to an embodiment of the present disclosure;

[0022] Figure 3 is a flowchart of a data processing method according to an embodiment of the present disclosure;

[0023] Figure 4 is a schematic diagram of a data processing method according to an embodiment of the present disclosure;

[0024] Figure 5 is a schematic diagram of another data processing method according to an embodiment of the present disclosure;

[0025] Figure 6 is a flowchart of another data processing method according to an embodiment of the present disclosure;

[0026] Figure 7 is a flowchart of another data processing method according to an embodiment of the present disclosure;

[0027] Figure 8 is a flowchart of another data processing method according to an embodiment of the present disclosure;

[0028] Figure 9 is a flowchart of another data processing method according to an embodiment of the present disclosure;

[0029] Figure 10 is a structural block diagram of a data processing apparatus according to an embodiment of the present disclosure;

[0030] Figure 11 is a structural block diagram of another data processing apparatus according to an embodiment of the present disclosure;

[0031] Figure 12 is a structural block diagram of another data processing apparatus according to an embodiment of the present disclosure;

[0032] Figure 13 is a structural block diagram of another data processing apparatus according to an embodiment of the present disclosure;

[0033] Figure 14 is a structural block diagram of another data processing apparatus according to an embodiment of the present disclosure;

[0034] Figure 15 is a structural block diagram of a computer terminal according to an embodiment of the present disclosure. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] According to embodiments of this disclosure, a data processing method is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0038] Considering the significant computational resource consumption of machine learning models on mobile terminals, the method provided in this disclosure can be applied to the application scenario shown in Figure 1, but is not limited thereto. In the application scenario shown in Figure 1, the machine learning model is deployed on server 10. Server 10 can connect to one or more clients 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. Clients 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Clients 20 can interact with users through a graphical user interface to invoke the large model, thereby implementing the method provided in this disclosure.

[0039] It should be noted that, provided that the operating resources of the client device can meet the deployment and operation conditions of the large model, the embodiments of this disclosure can be performed on the client device.

[0040] Figure 2 is a structural block diagram of a computing environment for a data processing method according to an embodiment of the present disclosure. As shown in Figure 2, the computing environment 201 includes multiple computing nodes (such as servers) running on a distributed network (shown as 210-1, 210-2, ... in the figure). Each computing node contains local processing and memory resources, and the end user 202 can remotely run applications or store data within the computing environment 201. Applications can be provided as multiple services 220-1, 220-2, 220-3, and 220-4 within the computing environment 201, representing services "A", "D", "E", and "H", respectively.

[0041] End user 202 can provide and access services through a web browser or other software application on a client. In some embodiments, the provisioning and / or requests of end user 202 can be provided to ingress gateway 230. Ingress gateway 230 may include a corresponding agent to handle the provisioning and / or requests for services (one or more services provided in computing environment 201).

[0042] The services are provided or deployed based on various virtualization technologies supported by the computing environment 201. In some embodiments, services may be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. VM-based virtualization can simulate a real computer by initializing a virtual machine, executing programs and applications without directly accessing any actual hardware resources. While the machine is virtualized by a virtual machine, container-based virtualization can launch containers to virtualize an entire operating system (OS), allowing multiple workloads to run on a single OS instance.

[0043] In one embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). For example, as shown in Figure 2, service 220-2 can be equipped with one or more Pods 240-1, 240-2, ..., 240-N (collectively referred to as Pods). A Pod can include a proxy 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively referred to as containers). One or more containers in a Pod handle requests related to one or more corresponding functions of the service. The proxy 245 typically controls service-related network functions such as routing and load balancing. Other services can also be equipped with similar Pods.

[0044] During operation, executing a user request from end user 202 may require calling one or more services in computing environment 201. Executing one or more functions of one service requires calling one or more functions of another service. As shown in Figure 2, service "A" 220-1 receives a user request from end user 202 from ingress gateway 230. Service "A" 220-1 can call service "D" 220-2, and service "D" 220-2 can request service "E" 220-3 to execute one or more functions.

[0045] The aforementioned computing environment can be a cloud computing environment, where resource allocation is managed by cloud services, allowing functionality development without needing to consider implementation, adjustment, or server scaling. This computing environment allows developers to execute event-responsive code without building or maintaining complex infrastructure. Services can be partitioned into a set of functions that can automatically and independently scale, rather than scaling a single hardware device to handle potential loads.

[0046] In the above operating environment, this disclosure provides a data processing method as shown in Figure 3. It should be noted that the data processing method of this embodiment can be executed by the server in the embodiment shown in Figure 1. Figure 3 is a flowchart of a data processing method according to an embodiment of this disclosure. As shown in Figure 3, the method may include the following steps:

[0047] Step S31: Receive a data query request;

[0048] Step S32: Generate the first candidate plan set corresponding to the data query request;

[0049] Step S33: Use the target execution cost prediction model to predict the cost of the first candidate plan set and select the target execution plan. The target execution cost prediction model is trained using the training plan corresponding to the target item. The target item is selected from the candidate items. The target item includes: multiple historical queries. The training plan includes: the historical default execution plan and candidate query plan corresponding to each of the multiple historical queries.

[0050] Step S34: Generate a data query response corresponding to the data query request based on the target execution plan.

[0051] The data query requests described above can be data query requests submitted by users in a multi-tenant distributed data warehouse. Data query requests typically contain Structured Query Language (SQL) statements used to retrieve, analyze, or manipulate data from the data warehouse. Data query requests can also involve simple data retrieval or complex operations such as multi-table joins and data aggregation.

[0052] When receiving data query requests, this can be achieved through the front-end query interface of the multi-tenant distributed data warehouse. Specific methods include, but are not limited to, Application Programming Interface (API) calls, web pages, command-line interfaces, and database client applications. Specifically, the multi-tenant distributed data warehouse provides APIs for applications or users to directly call and submit query requests. Users can construct query requests using programming languages ​​and send them to the multi-tenant distributed data warehouse server via specific communication protocols. The web interface provides a graphical user interface, allowing users to directly enter SQL queries or use predefined query templates through a web browser. The data warehouse can also provide command-line tools, allowing users to directly enter SQL query commands in the terminal or command line. Users can use dedicated database client applications, which provide an interface for interacting with the data warehouse, where users can write and execute queries.

[0053] After receiving a data query request, a first candidate plan set is generated, which includes multiple different candidate query plans. Upon receiving the data query request, the query optimizer can generate a series of possible execution plans based on the structure of the data query request and the rules of the multi-tenant distributed data warehouse, thus obtaining the first candidate plan set. The candidate query plans in the first candidate plan set can be multiple execution paths generated by the query optimizer based on different strategies and algorithms after initial analysis of the user query request, aiming to select a more efficient or lower-cost plan to execute the query.

[0054] Furthermore, a target execution cost prediction model is used to predict the cost of the first candidate plan set and select the target execution plan. During cost prediction, the target execution cost prediction model can predict the resources and time required for each plan in the first candidate plan set during actual execution. The target execution cost prediction model is trained using the training plan corresponding to the target project. The training plan for the target project includes: historical default execution plans and candidate query plans corresponding to multiple historical queries. The target project can be selected from a series of candidate projects pre-selected through rule filtering and AI-based ranking. The target project has a sufficient amount of historical data and stable query patterns, making it suitable for training the target execution cost prediction model. The selection of the target project ensures the quality and representativeness of the model training data, helping the target execution cost prediction model learn how to more accurately predict the cost of the execution plan.

[0055] For example, once the target execution cost prediction model has completed cost predictions for all candidate query plans in the first candidate plan set, the next step is to select an execution plan with lower cost or better expected performance as the target execution plan. The selection of the target execution plan can be based on the estimated execution cost. The target execution cost prediction model can evaluate factors such as plan structure and resource requirements to select the target execution plan from multiple candidate query plans.

[0056] For example, after selecting a target execution plan, necessary data access operations, such as table scans and index lookups, can be performed according to the target execution plan to retrieve the data query response corresponding to the data query request from the storage system. Further, the data is processed according to the instructions in the target execution plan, such as performing hash joins, merge joins, filtering, and aggregation to obtain the expected query results. After obtaining the expected query results, sorting, formatting, and other operations can be performed on the results for display to the user.

[0057] Based on steps S31 to S34 above, by receiving a data query request, a first candidate plan set corresponding to the data query request is generated. Then, the target execution cost prediction model is used to predict the cost of the first candidate plan set, and a target execution plan is selected. The target execution cost prediction model is trained using the training plan corresponding to the target item. The target item is selected from the candidate items. The target item includes multiple historical queries. The training plan includes the historical default execution plan and candidate query plan corresponding to each of the multiple historical queries. Finally, a data query response corresponding to the data query request is generated based on the target execution plan. In this way, by intelligently selecting items, customizing the training of the target execution cost prediction model, accurately predicting the execution cost, and quickly generating query responses, the performance of the query optimizer is significantly improved, resource allocation is further optimized, the query response speed is accelerated, and the data processing efficiency and user experience of the multi-tenant distributed data warehouse are ultimately improved.

[0058] By intelligently selecting target projects from candidate projects and using the training plan corresponding to the target project for model training, the high quality and relevance of model training data can be ensured, which is conducive to building a more accurate target execution cost prediction model that is more adapted to specific scenarios. The target execution cost prediction model can quickly and accurately predict the cost of each plan in the first candidate plan set, thereby rapidly filtering out the target execution plan from the first candidate plan set. This avoids the waste of resources in actually executing all candidate query plans to determine the target execution plan, effectively reducing the overall cost of query processing and improving resource utilization. After selecting the target execution plan, the data query response corresponding to the data query request can be quickly generated based on the target execution plan, thereby accelerating the response speed of query processing in large-scale, high-concurrency query scenarios and improving the user experience. In summary, the embodiments of this disclosure achieve the technical effects of improving data processing efficiency and reducing resource consumption costs, thereby solving the technical problems of low data processing efficiency and high resource consumption costs in related technologies when processing massive data query requests.

[0059] The data processing methods in the embodiments of this disclosure will be further described below.

[0060] In one optional embodiment, the data processing method in this disclosure further includes:

[0061] Based on the training plan, the initial execution cost prediction model is adversarially trained to generate domain classification loss and execution cost prediction loss;

[0062] Based on the domain classification loss and execution cost prediction loss, the model parameters of the initial execution cost prediction model are updated to obtain the target execution cost prediction model.

[0063] The training plan described above includes historical default execution plans and candidate query plans for each of the multiple historical queries. The initial execution cost prediction model can predict the execution cost of the training plan. The goal of adversarial training is to make the feature representations of the initial execution cost prediction model more consistent across different domains, thereby improving the generalization ability of the initial execution cost prediction model.

[0064] In related technologies, since only historical data of each project is used for model training during the training phase, while the execution cost of a completely new candidate query plan generated by the candidate query plan generator needs to be predicted during the deployment phase, there is a distribution offset problem between the training set and the test set. To solve the distribution offset between the training set and the test set, embodiments of this disclosure can improve the generalization ability of the initial execution cost prediction model by using adversarial training to make the feature representation of the initial execution cost prediction model more consistent across different domains.

[0065] Adversarial training can be performed within the domain adaptation module, which includes a gradient reversal layer (GRL) and a domain classifier. The core idea of ​​the domain adaptation module is to use an adversarial training mechanism to enable better distribution alignment of features generated by the feature extractor between the source and target domains.

[0066] During adversarial training, the domain classifier distinguishes whether the input data comes from the source or target domain, while the initial execution cost prediction model predicts the execution cost. Specifically, in each training iteration, the initial execution cost prediction model calculates the predicted execution cost for each training plan based on the current model parameters and compares it with the actual execution cost, calculating the execution cost prediction loss. Simultaneously, the domain classifier evaluates whether the features generated by the initial execution cost prediction model can distinguish between the source and target domains, calculating the domain classification loss. By introducing a gradient flipping layer, the gradient is maintained normally during forward propagation and its direction is flipped during backpropagation. The initial execution cost prediction model can simultaneously minimize the execution cost prediction loss to improve the accuracy of cost prediction and maximize the domain classification loss, thereby making the feature representations of the source and target domains as similar as possible.

[0067] For example, model parameter updates can follow the gradient descent principle. By calculating the gradient of the loss function with respect to the model parameters and adjusting the model parameters of the initial execution cost prediction model according to the gradient direction, the cross-domain generalization ability of the initial execution cost prediction model can be improved while maintaining high prediction accuracy. Model parameter updates can be repeated across the entire training set until the initial execution cost prediction model reaches a predetermined performance metric or convergence criterion. The resulting target execution cost prediction model possesses both high-precision prediction capabilities on source domain data and can also effectively handle target domain data, even if the data distribution differs from the training data.

[0068] Based on the above optional embodiments, adversarial training is performed on the initial execution cost prediction model according to the training plan to generate domain classification loss and execution cost prediction loss. Then, based on the domain classification loss and execution cost prediction loss, the model parameters of the initial execution cost prediction model are updated to obtain the target execution cost prediction model. The target execution cost prediction model can better cope with unknown query types and data distributions, reduce the risk of decreased prediction accuracy in new environments, and enhance the overall performance and practicality of the query optimizer.

[0069] In one optional embodiment, the data processing method in this disclosure further includes:

[0070] Using the project selector, target projects are selected from candidate projects according to preset filtering rules. The preset filtering rules are used to quantify the applicability of candidate projects from multiple dimensions.

[0071] Specifically, in the design and implementation of AI-based query optimizers, the project selector plays a crucial role. It selects target projects from numerous candidate projects, i.e., projects suitable for applying AI query optimization techniques. Preset filtering rules ensure that the selected projects meet the needs of AI optimization across multiple dimensions, thereby ensuring efficient resource utilization.

[0072] The aforementioned preset filtering rules typically involve multiple dimensions. These dimensions are designed to quantify the applicability of candidate projects, ensuring that the selected target projects maximize the benefits of the query optimizer. These dimensions include, but are not limited to: data scale and query complexity, query stability, richness of historical execution data, potential benefit assessment, and resource consumption and cost considerations. Specifically, the data scale and query complexity dimension ensures that the target project has a sufficiently large amount of data and complex query patterns to support the effective learning and generalization of the target execution cost prediction model. The query stability dimension excludes candidate projects with unstable query patterns or frequently changing data distribution. The richness of historical execution data ensures that the target project has sufficient historical query and execution data to train the initial execution cost prediction model. The potential benefit assessment dimension allows for a preliminary evaluation to determine the benefits that applying the AI ​​optimizer to the candidate project will bring, excluding low-benefit projects that are unlikely to achieve significant performance improvements even with optimization. The resource consumption and cost considerations dimension assesses the cost of applying the AI ​​optimizer to the candidate project, including additional computational resource consumption, ensuring that the benefits of optimization outweigh its costs.

[0073] Based on the above optional embodiments, by using a project selector, target projects are selected from candidate projects according to preset filtering rules. The preset filtering rules are used to quantify the applicability of candidate projects from multiple dimensions, which can avoid wasting resources on inefficient or unsuitable candidate projects. The accurate selection of target projects helps to improve query efficiency, reduce resource consumption, and achieve more intelligent and economical data warehouse management.

[0074] In one optional embodiment, the project selector includes: a project filter and a project sorter. The project selector filters target projects from candidate projects according to preset filtering rules, including:

[0075] Using the project filter, candidate projects are filtered according to preset filtering rules to obtain the remaining projects;

[0076] Using a project sorter, the remaining projects are sorted according to their estimated revenue in order to filter out target projects.

[0077] Specifically, the project filter selects candidate projects based on preset filtering rules. These rules aim to quantify the applicability of candidate projects from multiple dimensions, including but not limited to the number of historical queries, the characteristics of the tables involved, and the structural diversity of the query plans. If a candidate project does not meet any of the preset filtering rules, such as having too little data, an overly simple query pattern, or scarce historical data, it will be excluded and no longer considered as a training or application target for AI query optimization. The remaining projects are those with sufficient data and query complexity, and rich historical execution records, which are more suitable as training data sources or application targets for the AI ​​optimizer after the project filter has been used.

[0078] For the remaining items filtered by the item filter, the item sorter will calculate the potential benefits of each remaining item based on a more refined set of evaluation metrics, including but not limited to: estimates of performance improvements based on historical queries, potential resource savings, and cost-benefit analysis of model training and deployment.

[0079] The project sorter ranks all remaining projects based on estimated benefits. The goal of this sorting is to identify the remaining projects that will benefit most from AI query optimization, thus quickly filtering out target projects. After sorting, starting from the top of the list, N projects with the highest estimated benefits can be selected as target projects. N can be a preset value or dynamically determined based on cost-benefit analysis. These target projects will be used for subsequent training data preparation, model training, and the final deployment of the AI ​​query optimizer.

[0080] Based on the above optional embodiments, by using a project filter to filter candidate projects according to preset filtering rules to obtain the remaining projects, and then using a project sorter to sort the remaining projects according to the estimated benefits of the remaining projects, the target projects can be selected. This not only improves resource utilization, but also ensures the quality and representativeness of model training data, thereby helping to build a more accurate and more generalizable target execution cost prediction model.

[0081] In one optional embodiment, the preset filtering rules include: historical query quantity dimension, table feature dimension, and query plan structure dimension. Using an item filter, candidate items are filtered according to the preset filtering rules to obtain the remaining items, which include:

[0082] Using a project filter, candidate projects are quantified from the dimensions of historical query volume, table characteristics, and query plan structure. Projects that do not meet the preset measurement conditions are excluded from the candidate projects, resulting in the remaining projects.

[0083] Specifically, when quantifying candidate items using the historical query volume dimension, the total number of query requests for each candidate item over a past period can be counted and compared with a preset historical query volume. If the total number of query requests is lower than the preset historical query volume, the candidate item is determined to be unsuitable for AI optimizer training due to insufficient data and is thus excluded. The item filter will identify candidate items with frequent query activity and sufficient data volume, which can serve as good training data sources, based on the historical query volume dimension, while candidate items with low activity or scarce data will be filtered out.

[0084] Table feature dimensions examine the table features involved in candidate projects, including but not limited to table size, number of columns, column data types, and partitioning strategies, ensuring that these features are suitable for training and applying the AI ​​query optimizer. Project filters analyze the metadata of all tables in candidate projects to evaluate whether table features contribute to model learning and query plan optimization. For example, candidate project A contains 10 tables, each with 20 columns, diverse data types, and multiple partitioning strategies. Query patterns in candidate project A cover most table operations, including joins, aggregations, and filtering, but instances of each query pattern are relatively few. Candidate project B contains 5 tables, each with 10 columns, with relatively uniform data types but sufficient diversity. While candidate project B's query pattern coverage is not as broad as project A's, it offers richer instances of table operations, with each query pattern appearing multiple times. Although candidate project A provides richer information on table structure and data types, from a model training perspective, candidate project B, with its smaller number of tables and columns and higher query diversity, is a better choice. The features of candidate item B help the initial execution cost prediction model quickly grasp basic query optimization strategies in the early stages of learning. At the same time, its rich query examples can enhance the initial execution cost prediction model's understanding of various operation combinations and improve the generalization ability of the initial execution cost prediction model.

[0085] The query plan structure dimension examines the structural diversity of historical query plans in candidate projects, ensuring that the initial execution cost prediction model can learn a rich variety of query patterns and optimization strategies. By analyzing the execution plans of historical queries in candidate projects, the project filter identifies query plans with complex join, aggregation, sorting, and filtering operations. Query plans with more complex structures and richer patterns help improve the generalization ability of the initial execution cost prediction model.

[0086] The candidate projects that do not meet the preset measurement conditions are excluded. The preset measurement conditions can be any of the following dimensions: historical query quantity, table feature, and query plan structure. All remaining projects after filtering meet the preset measurement conditions.

[0087] Based on the above optional embodiments, by using a project filter, candidate projects are quantified from the dimensions of historical query volume, table features, and query plan structure, respectively. Some projects that do not meet the preset measurement conditions are excluded from the candidate projects, and the remaining projects are obtained. Thus, by using a project filter to perform quantitative evaluation and screening from multiple dimensions, it can be ensured that the final selected remaining projects not only have sufficient query data and rich table features, but also provide diverse query plan structures, further ensuring that the AI-based query optimizer can be deployed efficiently and accurately.

[0088] In one optional embodiment, the data processing method in this disclosure further includes:

[0089] Select a subset of projects from the remaining projects;

[0090] Generate a second candidate plan set for some projects;

[0091] The project sorter is trained based on the default execution plan and the second candidate plan set.

[0092] Specifically, filtering some projects can further focus on those that may benefit from AI query optimization. Even among the remaining projects after passing through the project filter, there may still be differences in benefits between projects. Therefore, it is necessary to further evaluate each remaining project, such as through preliminary cost-benefit analysis, to identify a subset of projects that can bring performance improvements or resource savings.

[0093] For example, during the selection of some projects, lightweight algorithms or preliminary model predictions can be used to estimate the expected benefits of each remaining project, including but not limited to: reductions in resource consumption, shorter query response times, and other metrics. Simultaneously, the cost of applying the AI ​​query optimizer to the remaining projects also needs to be assessed, such as additional computational resource consumption and model training time. Based on a comprehensive consideration of benefits and costs, a subset of projects is selected from the remaining projects.

[0094] Generating a second candidate plan set for some projects enriches the model training materials, ensuring the initial execution cost prediction model learns more optimization strategies and execution plan patterns. Specifically, multiple possible execution plans are generated for queries in some projects, exploring different join orders, index usage, parallelism settings, etc., to reflect the diversity and flexibility of query execution strategies. Besides historical execution data based on projects, new execution plans can be generated through simulation or heuristic algorithms, helping the initial execution cost prediction model learn to make reasonable predictions even without historical precedents. Furthermore, after obtaining the second candidate plan set, a project sorter is trained based on the default execution plan and the second candidate plan set.

[0095] Based on the above optional embodiments, by selecting some projects from the remaining projects, a second candidate plan set is generated for some projects. Finally, a project sorter is trained based on the default execution plan and the second candidate plan set, which further ensures that the AI-based query optimizer can be deployed efficiently and accurately, thereby improving resource utilization and overall performance.

[0096] In an optional embodiment, the item sorter is trained based on the default execution plan and a second set of candidate plans, including:

[0097] Obtain the first execution cost of the default execution plan and the second execution cost of the second candidate plan set;

[0098] Based on the first execution cost and the second execution cost, determine the estimated revenue of each query included in a portion of the project;

[0099] A project sorter is trained based on the estimated returns.

[0100] The first execution cost mentioned above represents the actual execution cost of the default execution plan for each project. It typically involves running the query in the data warehouse system, recording the time and resource consumption required for completion. This first execution cost reflects the query performance under the current system configuration. The observable characteristics of the default execution plan usually reflect the improvement potential of the AI ​​query optimizer in its optimization strategies. For example, when the default execution plan contains multiple nested join operations and the execution cost is abnormally high, it usually indicates inefficient join order and significant performance optimization potential. The project sorter can estimate the potential optimization space of the default execution plan by learning its characteristics. The second execution cost mentioned above reflects the performance of querying data under different execution strategies.

[0101] Furthermore, based on the first execution cost and the second execution cost, the estimated revenue of each query included in a subset of items is determined, and an item ranking algorithm is trained based on the estimated revenue. The first and second execution costs of each query, along with the calculated estimated revenue, are used as training data. Through multiple rounds of training and parameter tuning, the item ranking algorithm is continuously optimized until it can accurately predict the estimated revenue of a query.

[0102] Based on the above optional embodiments, by obtaining the first execution cost of the default execution plan and the second execution cost of the second candidate plan set, and then determining the estimated benefits of each query contained in a partial project based on the first execution cost and the second execution cost, a project sorter is finally trained based on the estimated benefits. The project sorter can effectively improve the performance and resource utilization efficiency of the AI-based query optimizer in a multi-tenant distributed data warehouse based on a deep understanding of the first execution cost, the second execution cost, and the estimated benefits.

[0103] In one optional embodiment, the estimated revenue of each query included in a subset of projects is determined based on a first execution cost and a second execution cost, including:

[0104] Based on the first execution cost and the second execution cost, the potential optimization space of the default execution plan relative to the optimal plan is estimated, where the optimal plan is the plan with the minimum execution cost in the second candidate plan set;

[0105] Based on the potential optimization space, a subset of projects is scored to determine the estimated revenue of each query contained within those projects.

[0106] Specifically, the first step is to obtain the actual execution cost of each query under the default execution plan, i.e., the first execution cost. The first execution cost can represent the performance benchmark of the current database management system when processing the query. In the second candidate plan set, the plan with the minimum execution cost, i.e., the optimal plan, is found and identified. The first execution cost of the default execution plan is compared with the second execution cost of the optimal plan, and the difference or ratio between the two is calculated. This yields the potential optimization space of the default execution plan relative to the optimal plan. The larger the value of the potential optimization space, the more significant the performance improvement or resource saving that can be achieved through the AI ​​query optimizer.

[0107] Each query is executed multiple times to smooth out the impact of the system environment and to calculate the potential optimization space. The potential optimization space represents the performance ratio of the execution cost of the default query plan relative to the optimal plan found by the candidate query plan generator. Based on the estimated potential optimization space, some items are scored. The score can be a direct reflection of the potential optimization space or a weighted value to consider other factors such as query frequency and resource consumption. The estimated benefits of all queries within the same item are aggregated, which can be done by averaging, medianing, or weighted averaging, to determine the estimated benefit at the overall item level. Based on the estimated benefits, some items are ranked, and the top N items are selected to deploy the AI-based query optimizer.

[0108] Based on the above optional embodiments, by estimating the potential optimization space of the default execution plan relative to the optimal plan based on the first execution cost and the second execution cost, and then scoring some projects based on the potential optimization space, the estimated benefits of each query contained in some projects are determined, thereby providing an important basis for subsequent project sorting and selection, ensuring that resources are prioritized for projects with greater optimization potential and higher estimated benefits, thereby achieving more efficient and economical resource allocation and query processing optimization.

[0109] In one optional embodiment, a project sorter is used to sort the remaining projects based on their estimated revenue, in order to filter out target projects, including:

[0110] The project sorter is used to obtain the estimated revenue of the remaining projects. The remaining projects are sorted according to the estimated revenue to obtain the sorting results. Based on the sorting results, the top-ranked projects are selected as target projects.

[0111] Specifically, the estimated benefits of all queries under the same project are aggregated to reflect the overall benefit that the entire project may obtain after optimization by the AI ​​query optimizer. This is achieved by averaging, weighted averaging, or other statistical methods. A project sorter is then used to rank the estimated benefits of each remaining project. The sorting process produces a list in which the remaining projects are ranked from highest to lowest based on their estimated benefits. Projects ranked higher indicate that they are expected to bring more significant performance improvements or resource savings under the optimization of the AI ​​optimizer.

[0112] Furthermore, based on resource constraints, optimization goals, or service requirements, a threshold is set, or the top N items are directly selected as target items. The value of N can be determined based on cost-benefit analysis, system resource availability, or service priority. Items with higher estimated returns are selected from the ranking results as target items, becoming the objects of further optimization by the AI ​​query optimizer.

[0113] Based on the above optional embodiments, by using a project sorter to obtain the estimated revenue of the remaining projects, sorting the remaining projects according to the estimated revenue to obtain the sorting results, and selecting the top-ranked projects as target projects based on the sorting results, the accuracy and efficiency of optimization can be significantly improved, unnecessary resource waste can be reduced, and thus effectively control costs while ensuring query performance.

[0114] In one optional embodiment, the data processing method in this disclosure further includes:

[0115] Deploy candidate projects in a preset storage area, which is a multi-tenant distributed data warehouse.

[0116] A pre-defined storage area can be a resource area reserved for a specific project or tenant to support query processing, data storage, and computation. The pre-defined storage area has sufficient computing resources (such as a Central Processing Unit (CPU) and memory), storage capacity, and network bandwidth to ensure that candidate projects can run smoothly and perform efficient data processing and query optimization. In a multi-tenant environment, the pre-defined storage area needs to implement resource isolation to ensure that a tenant's query operations do not affect other tenants, maintaining system stability and fairness.

[0117] During the deployment of candidate projects, corresponding computing and storage resources can be allocated to each project in a preset storage area based on the estimated resource requirements and expected query load of the candidate projects. This includes, but is not limited to, setting CPU and memory quotas, disk space size, and network bandwidth. Resource allocation must consider the differences in requirements and priorities among candidate projects to ensure that high-priority projects receive sufficient resource support. The AI ​​query optimizer is configured and adjusted in the preset storage area to adapt to the query characteristics and data patterns of the candidate projects. This includes, but is not limited to, fine-tuning model parameters, selecting optimization strategies, and configuring data loading and preprocessing. The goal is to ensure that the optimizer can generate effective query plans for each candidate project to reduce execution costs and improve performance. Furthermore, it is necessary to ensure that the deployment of each candidate project does not violate data security and resource isolation policies in a multi-tenant environment. Data access control can be implemented to ensure that each tenant can only access its authorized data. Resource quotas and runtime isolation technologies can also be used to prevent resource contention and performance interference.

[0118] Based on the above embodiments, by deploying candidate projects in a preset storage area, which is a multi-tenant distributed data warehouse, not only is the query performance of candidate projects improved, but the stable operation of the entire multi-tenant distributed data warehouse and the effective utilization of resources can also be ensured.

[0119] In one optional embodiment, the initial execution cost prediction model is adversarially trained according to the training plan to generate a domain classification loss and an execution cost prediction loss, including:

[0120] The initial execution cost prediction model is used to generate the encoding results of the training plan, and the domain classification and cost prediction are performed based on the encoding results to obtain the classification results and the predicted execution cost, respectively.

[0121] The domain classification loss is calculated based on the classification results and domain labels, and the execution cost prediction loss is calculated based on the predicted execution cost and the actual execution cost. In the cost prediction process, the default execution plan is a labeled plan and the candidate query plan is an unlabeled plan. In the domain classification process, the domain labels include: the candidate plan domain label assigned to the candidate query plan and the default plan domain label assigned to the default execution plan.

[0122] Specifically, the training plan is encoded into a feature vector through an initial execution cost prediction model. The encoding process can be implemented through tree convolution, graph neural networks, or other machine learning techniques suitable for processing structured data to capture the structural features, operational attributes, and environmental information of the training plan.

[0123] A domain classifier is used to classify the encoding results, distinguishing whether the training plan belongs to the default plan domain or the candidate plan domain. The domain classifier aims to identify the source of the training plan, i.e., whether the training plan is a default plan trained based on historical data or a candidate query plan generated based on the current query. A domain classification loss is calculated based on the classification results of the domain classifier and predefined domain labels. The domain classification loss reflects the accuracy of the domain classifier in distinguishing plans from different domains. During the domain classification process, domain labels include: a candidate plan domain label assigned to candidate query plans, a default plan domain label assigned to the default execution plan, a candidate plan domain label of 1, and a default plan domain label of 0.

[0124] The initial execution cost prediction model can also predict the execution cost based on the encoding results, and calculate the execution cost prediction loss based on the predicted execution cost and the actual execution cost. For labeled default execution plans, the difference between the predicted execution cost and the actual execution cost can be used to calculate the loss; while for unlabeled candidate query plans, there is no need to calculate the execution cost prediction loss, only the domain classification loss.

[0125] Based on the above optional embodiments, the encoding results of the training plan are generated by using the initial execution cost prediction model, and the domain classification and cost prediction are performed based on the encoding results to obtain the classification results and predicted execution costs, respectively. Then, the domain classification loss is calculated based on the classification results and domain labels, and the execution cost prediction loss is calculated based on the predicted execution cost and the actual execution cost. These are used to update the model parameters of the initial execution cost prediction model, thereby obtaining the target execution cost prediction model. The target execution cost prediction model can better cope with unknown query types and data distributions, reduce the risk of decreased prediction accuracy in new environments, and enhance the overall performance and practicality of the query optimizer.

[0126] In one optional embodiment, the model parameters of the initial execution cost prediction model are updated based on the domain classification loss and the execution cost prediction loss to obtain the target execution cost prediction model, including:

[0127] Gradient flipping is applied to the domain classification loss to obtain the flipped result;

[0128] Based on the flipping results and the execution cost prediction loss, the model parameters of the initial execution cost prediction model are updated to obtain the target execution cost prediction model.

[0129] Specifically, during backpropagation, the gradient of the domain classification loss is flipped to ensure that the initial execution cost prediction model can gradually eliminate the differences between the source and target domains while optimizing the cost prediction capability.

[0130] The execution cost prediction loss and the gradient flipping result are combined to form a composite loss function. The model parameters are then updated using the gradient of this composite loss function via backpropagation. The goal of this update is to minimize the execution cost prediction loss while maximizing the domain classification loss through gradient flipping, thereby improving the model's generalization ability.

[0131] After multiple rounds of training and parameter updates, the initial execution cost prediction model gradually converges, becoming a target execution cost prediction model that can accurately predict execution costs and has good generalization ability. When faced with an unknown query plan, the target execution cost prediction model can make more accurate cost predictions, thus providing a more reliable decision-making basis for the query optimizer.

[0132] Based on the above optional embodiments, by performing gradient flipping on the domain classification loss, a flipping result is obtained. Then, based on the flipping result and the execution cost prediction loss, the model parameters of the initial execution cost prediction model are updated to obtain the target execution cost prediction model. The target execution cost prediction model can overcome the distribution differences between the source domain and the target domain while maintaining high prediction accuracy. Ultimately, it becomes a query optimization tool with better performance and stronger generalization ability in a multi-tenant distributed data warehouse environment, thereby further improving the efficiency of query data processing.

[0133] In an optional embodiment, step S32, generating the first candidate plan set corresponding to the data query request includes:

[0134] The query optimizer generates a first candidate plan set corresponding to the data query request using at least one plan generation method, wherein the at least one plan generation method includes: a plan generation method that scales the cardinality estimated during the generation process of the query optimizer, and a plan generation method that enables or disables at least one optimization option in the query optimizer.

[0135] In query plan generation methods that scale the estimated cardinality during the query optimizer's process, the optimization typically relies on an estimate of the data volume (cardinality) to select operations and arrange their order. By adjusting the estimated cardinality, different execution plans can be generated. Scale operations can involve increasing or decreasing the cardinality of a specific table or operation to explore different data access patterns and join strategies. The aim is to generate a set of candidate query plans covering multiple execution strategies, thereby increasing plan diversity and providing richer training and selection criteria for AI models. Through these scaling operations, the query optimizer generates multiple execution plans, each containing different operation orders, join methods, index usage, and other details, forming a first set of candidate plans.

[0136] In a query plan generation method that enables or disables at least one optimization option in the query optimizer, the optimizer typically includes multiple optimization options, such as whether to use hash joins, whether to enable parallel processing, and whether to apply index scans. By selectively enabling or disabling these options, variations of the execution plan can be generated. By designing strategies to systematically enable or disable different optimization options, or by using random selection methods to increase plan diversity, the aim is to explore possible optimization paths and generate candidate query plans with different characteristics and optimization strategies. Based on maintaining the initial candidate plan set, more candidate query plans are generated by enabling or disabling optimization options to enrich the candidate plan set and improve the flexibility and possibilities for subsequent optimization and selection.

[0137] Based on the above optional embodiments, the query optimizer generates a first candidate plan set corresponding to the data query request using at least one plan generation method. The query optimizer can generate a candidate plan set covering multiple execution strategies, providing a rich selection space for subsequent AI-based optimization processes, thereby achieving more efficient and intelligent query processing capabilities. It is suitable for multi-tenant distributed data warehouse environments and can effectively improve overall query performance and resource utilization efficiency.

[0138] In an optional embodiment, in step S33, the target execution cost prediction model is used to predict the cost of the first candidate plan set, and the selection of the target execution plan includes:

[0139] The target execution cost prediction model is used to predict the cost of the first candidate plan set to obtain the prediction results, and the target execution plan that meets the preset execution cost conditions is selected from the first candidate plan set based on the prediction results.

[0140] Specifically, each candidate query plan in the first candidate plan set is input into the target execution cost prediction model. The model uses its learned patterns and knowledge to predict the execution cost of each plan, yielding a preset result. The target execution cost prediction model uses the parsed plan's structural information, operation type, connection method, filtering conditions, and environmental information (such as CPU utilization, input / output wait times, etc.) as prediction inputs. The model outputs the predicted execution cost for each plan, reflecting the resource consumption and execution time of the candidate query plan under the expected environment. The predicted execution costs of all candidate query plans are collected to form a list or dataset containing the cost predictions for each plan.

[0141] Furthermore, based on the prediction results, a target execution plan that meets the preset execution cost conditions is selected from the first candidate plan set. The preset execution cost conditions can be used to select the query plan with the lowest execution cost as the final target execution plan.

[0142] Based on the above optional embodiments, the first candidate plan set is predicted by using a target execution cost prediction model to obtain the prediction result, and the target execution plan that meets the preset execution cost conditions is selected from the first candidate plan set based on the prediction result. This enables the multi-tenant distributed data warehouse to process various queries more efficiently, thereby improving the overall data processing efficiency and user experience.

[0143] In an optional embodiment, the cost prediction of the first candidate plan set using a target execution cost prediction model to obtain the prediction result includes:

[0144] The target execution cost prediction model is used to encode the first candidate plan set to obtain the encoding result, and the execution cost prediction is performed on the encoding result to obtain the prediction result. The encoding result is obtained by encoding the structural information and environmental information of the first candidate plan set. The structural information is used to determine the operation type of the plan node contained in the first candidate plan set and the multi-dimensional operation information associated with the first candidate plan set. The environmental information is used to determine the actual execution environment of the first candidate plan set.

[0145] The aforementioned structural information is used to determine the operation types of the plan nodes contained in the first candidate plan set and the multi-dimensional operation information associated with the first candidate plan set. The operation types of plan nodes include, but are not limited to: table scan, join, aggregation, and filter. Table operation information includes, but is not limited to, table name and partition column information. Join operation information includes, but is not limited to, the join method and join object of the join operator. Aggregation operation information includes, but is not limited to, the aggregation column and aggregation method. Filter operation information includes, but is not limited to, the key-value pair conditions of the filter operator. The aforementioned environmental information includes system-level environmental characteristics such as CPU idle rate, I / O wait rate, and overall load rate.

[0146] A target execution cost prediction model is used to encode the features of the first candidate plan set to obtain the encoding result. The feature encoding results of structural information and environmental information are fused together and provided as the input feature matrix to the target execution cost prediction model. The target execution cost prediction model receives the encoded feature matrix and uses its internal weights and calculation logic to predict the execution cost of each candidate query plan to obtain the prediction result.

[0147] Based on the above optional embodiments, by using a target execution cost prediction model to encode the first candidate plan set to obtain the encoding result, and by performing execution cost prediction on the encoding result to obtain the prediction result, the execution efficiency of different candidate query plans can be effectively evaluated, and adaptive predictions can be made according to the actual environment, thereby improving query processing efficiency under limited resources.

[0148] Figure 4 is a schematic diagram of a data processing method according to an embodiment of the present disclosure. As shown in Figure 4, a data query request is received, and a first candidate plan set corresponding to the data query request is generated. Then, a target execution cost prediction model is used to predict the cost of the first candidate plan set, and a target execution plan is selected. The target execution cost prediction model is trained using a training plan corresponding to the target project. The target project is selected from the candidate projects. The target project includes multiple historical queries. The training plan includes the historical default execution plan and candidate query plan corresponding to each of the multiple historical queries. Finally, a data query response corresponding to the data query request is generated based on the target execution plan. In the process of selecting target projects, a project filter can be used to filter candidate projects according to preset filtering rules to obtain the remaining projects. Then, a project sorter is used to sort the remaining projects according to their estimated benefits to select the target project.

[0149] Figure 5 is a schematic diagram of another data processing method according to an embodiment of the present disclosure. As shown in Figure 5, the encoding result of the training plan is generated using the initial execution cost prediction model, and domain classification and cost prediction are performed based on the encoding result to obtain the classification result and the predicted execution cost, respectively. The domain classification loss is calculated based on the classification result and the domain label, and the execution cost prediction loss is calculated based on the predicted execution cost and the actual execution cost. In the cost prediction process, the default execution plan is a labeled plan, and the candidate query plan is an unlabeled plan. In the domain classification process, the domain label includes: the candidate plan domain label assigned to the candidate query plan and the default plan domain label assigned to the default execution plan. The gradient of the domain classification loss is flipped to obtain the flipped result. Based on the flipped result and the execution cost prediction loss, the model parameters of the initial execution cost prediction model are updated to obtain the target execution cost prediction model.

[0150] Figure 6 is a flowchart of another data processing method according to an embodiment of the present disclosure. As shown in Figure 6, the method may include the following steps:

[0151] Step S61: Receive order query data from the e-commerce platform;

[0152] Step S62: Generate the first candidate plan set corresponding to the order query data of the e-commerce platform;

[0153] Step S63: Use the target execution cost prediction model to predict the cost of the first candidate plan set and select the target execution plan. The target execution cost prediction model is trained using the training plan corresponding to the target item. The target item is selected from the candidate items. The target item includes: multiple historical queries. The training plan includes: the historical default execution plan and candidate query plan corresponding to each of the multiple historical queries.

[0154] Step S64: Generate e-commerce platform order statistics response corresponding to e-commerce platform order query data based on the target execution plan.

[0155] Based on steps S61 to S64 above, by receiving order query data from e-commerce platforms, a first candidate plan set corresponding to the e-commerce platform order query data is generated. Then, a target execution cost prediction model is used to predict the cost of the first candidate plan set, and a target execution plan is selected. The target execution cost prediction model is trained using the training plan corresponding to the target item. The target item is selected from the candidate items, including multiple historical queries. The training plan includes the historical default execution plan and candidate query plan corresponding to each of the multiple historical queries. Finally, a data query response corresponding to the e-commerce platform order query data is generated based on the target execution plan. This significantly improves the performance of the query optimizer by intelligently selecting items, customizing the training of the target execution cost prediction model, accurately predicting execution costs, and quickly generating query responses. It further optimizes resource allocation, accelerates query response speed, and ultimately improves the data processing efficiency and user experience of the multi-tenant distributed data warehouse. By intelligently selecting target items from the candidate items and using the training plan corresponding to the target items for model training, the high quality and relevance of the model training data can be ensured, which is conducive to building a more accurate target execution cost prediction model that is more adapted to specific scenarios. The target execution cost prediction model can quickly and accurately predict the cost of each plan in the first candidate plan set, thereby rapidly selecting the target execution plan from the first candidate plan set. This avoids the waste of resources that would otherwise be incurred by actually executing all candidate query plans to determine the target execution plan, effectively reducing the overall cost of query processing and improving resource utilization. After selecting the target execution plan, the model can quickly generate data query responses corresponding to e-commerce platform order query data based on the target execution plan, thereby accelerating the response speed of query processing in large-scale, high-concurrency query scenarios and improving user experience. In summary, the embodiments of this disclosure achieve the technical effects of improving data processing efficiency and reducing resource consumption costs, thus solving the technical problems of low data processing efficiency and high resource consumption costs in related technologies when handling massive data query requests.

[0156] Figure 7 is a flowchart of another data processing method according to an embodiment of the present disclosure. As shown in Figure 7, the method may include the following steps:

[0157] Step S71: Obtain a data processing request through the first application programming interface, wherein the request data carried in the data processing request includes: a data query request;

[0158] Step S72: A data processing response is returned through the second application programming interface. The response data carried in the data processing response includes: a data query reply corresponding to the data query request, which is obtained according to the data processing method of any one of the embodiments of this disclosure.

[0159] Based on the above steps S71 to S72, a data processing request is obtained through the first application programming interface. The request data carried in the data processing request includes a data query request. Then, a data processing response is returned through the second application programming interface. The response data carried in the data processing response includes a data query reply corresponding to the data query request. This enables efficient and accurate generation of the data query reply corresponding to the data query request.

[0160] In the process of generating a data query response, a first candidate plan set corresponding to the data query request is generated by receiving the data query request. Then, the target execution cost prediction model is used to predict the cost of the first candidate plan set and select the target execution plan. The target execution cost prediction model is trained using the training plan corresponding to the target item. The target item is selected from the candidate items, which include multiple historical queries. The training plan includes the historical default execution plan and candidate query plan corresponding to each of the multiple historical queries. Finally, the data query response corresponding to the data query request is generated based on the target execution plan. In this way, by intelligently selecting items, customizing the training of the target execution cost prediction model, accurately predicting the execution cost, and quickly generating query responses, the performance of the query optimizer is significantly improved, resource allocation is further optimized, query response speed is accelerated, and the data processing efficiency and user experience of the multi-tenant distributed data warehouse are ultimately improved.

[0161] By intelligently selecting target projects from candidate projects and using the training plan corresponding to the target project for model training, the high quality and relevance of model training data can be ensured, which is conducive to building a more accurate target execution cost prediction model that is more adapted to specific scenarios. The target execution cost prediction model can quickly and accurately predict the cost of each plan in the first candidate plan set, thereby rapidly filtering out the target execution plan from the first candidate plan set. This avoids the waste of resources in actually executing all candidate query plans to determine the target execution plan, effectively reducing the overall cost of query processing and improving resource utilization. After selecting the target execution plan, the data query response corresponding to the data query request can be quickly generated based on the target execution plan, thereby accelerating the response speed of query processing in large-scale, high-concurrency query scenarios and improving the user experience. In summary, the embodiments of this disclosure achieve the technical effects of improving data processing efficiency and reducing resource consumption costs, thereby solving the technical problems of low data processing efficiency and high resource consumption costs in related technologies when processing massive data query requests.

[0162] Figure 8 is a flowchart of another data processing method according to an embodiment of the present disclosure. As shown in Figure 8, the method may include the following steps:

[0163] Step S81: Obtain the currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: a data query request;

[0164] Step S82: In response to the data processing dialogue request, a data processing dialogue response is returned, wherein the information carried in the data processing dialogue response includes: a data query response corresponding to the data query request, and the data query response is obtained according to any one of the data processing methods in the embodiments of this disclosure;

[0165] Step S83: Display the data query response within the graphical user interface.

[0166] Based on the above steps S81 to S83, by obtaining the currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: a data query request, and then responding to the data processing dialogue request, returning a data processing dialogue response, wherein the information carried in the data processing dialogue response includes: the data query response corresponding to the data query request, and finally displaying the data query response in the graphical user interface, the data query response corresponding to the data query request can be generated efficiently and accurately.

[0167] In the process of generating a data query response, a first candidate plan set corresponding to the data query request is generated by receiving the data query request. Then, the target execution cost prediction model is used to predict the cost of the first candidate plan set and select the target execution plan. The target execution cost prediction model is trained using the training plan corresponding to the target item. The target item is selected from the candidate items, which include multiple historical queries. The training plan includes the historical default execution plan and candidate query plan corresponding to each of the multiple historical queries. Finally, the data query response corresponding to the data query request is generated based on the target execution plan. In this way, by intelligently selecting items, customizing the training of the target execution cost prediction model, accurately predicting the execution cost, and quickly generating query responses, the performance of the query optimizer is significantly improved, resource allocation is further optimized, query response speed is accelerated, and the data processing efficiency and user experience of the multi-tenant distributed data warehouse are ultimately improved.

[0168] By intelligently selecting target projects from candidate projects and using the training plan corresponding to the target project for model training, the high quality and relevance of model training data can be ensured, which is conducive to building a more accurate target execution cost prediction model that is more adapted to specific scenarios. The target execution cost prediction model can quickly and accurately predict the cost of each plan in the first candidate plan set, thereby rapidly filtering out the target execution plan from the first candidate plan set. This avoids the waste of resources in actually executing all candidate query plans to determine the target execution plan, effectively reducing the overall cost of query processing and improving resource utilization. After selecting the target execution plan, the data query response corresponding to the data query request can be quickly generated based on the target execution plan, thereby accelerating the response speed of query processing in large-scale, high-concurrency query scenarios and improving the user experience. In summary, the embodiments of this disclosure achieve the technical effects of improving data processing efficiency and reducing resource consumption costs, thereby solving the technical problems of low data processing efficiency and high resource consumption costs in related technologies when processing massive data query requests.

[0169] Figure 9 is a flowchart of another data processing method according to an embodiment of the present disclosure. As shown in Figure 9, the method may include the following steps:

[0170] Step S91: In response to the input command applied to the operation interface, display the data query request on the operation interface;

[0171] Step S92: In response to the processing command applied to the operation interface, display the data query response corresponding to the data query request on the operation interface; wherein, the data query response is obtained according to the data processing method of any one of the embodiments of this disclosure.

[0172] Based on the above steps S91 to S92, by responding to the input command applied to the operation interface, the data query request is displayed on the operation interface, and then by responding to the processing command applied to the operation interface, the data query response corresponding to the data query request is displayed on the operation interface. Thus, the data query response corresponding to the data query request can be generated efficiently and accurately.

[0173] In the process of generating a data query response, a first candidate plan set corresponding to the data query request is generated by receiving the data query request. Then, the target execution cost prediction model is used to predict the cost of the first candidate plan set and select the target execution plan. The target execution cost prediction model is trained using the training plan corresponding to the target item. The target item is selected from the candidate items, which include multiple historical queries. The training plan includes the historical default execution plan and candidate query plan corresponding to each of the multiple historical queries. Finally, the data query response corresponding to the data query request is generated based on the target execution plan. In this way, by intelligently selecting items, customizing the training of the target execution cost prediction model, accurately predicting the execution cost, and quickly generating query responses, the performance of the query optimizer is significantly improved, resource allocation is further optimized, query response speed is accelerated, and the data processing efficiency and user experience of the multi-tenant distributed data warehouse are ultimately improved.

[0174] By intelligently selecting target projects from candidate projects and using the training plan corresponding to the target project for model training, the high quality and relevance of model training data can be ensured, which is conducive to building a more accurate target execution cost prediction model that is more adapted to specific scenarios. The target execution cost prediction model can quickly and accurately predict the cost of each plan in the first candidate plan set, thereby rapidly filtering out the target execution plan from the first candidate plan set. This avoids the waste of resources in actually executing all candidate query plans to determine the target execution plan, effectively reducing the overall cost of query processing and improving resource utilization. After selecting the target execution plan, the data query response corresponding to the data query request can be quickly generated based on the target execution plan, thereby accelerating the response speed of query processing in large-scale, high-concurrency query scenarios and improving the user experience. In summary, the embodiments of this disclosure achieve the technical effects of improving data processing efficiency and reducing resource consumption costs, thereby solving the technical problems of low data processing efficiency and high resource consumption costs in related technologies when processing massive data query requests.

[0175] It should be noted that the preferred implementation schemes involved in the above embodiments of this disclosure are the same as the schemes, application scenarios and implementation processes provided in the above embodiments, but are not limited to the schemes provided in the above embodiments.

[0176] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0177] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0178] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solutions of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0179] According to embodiments of this disclosure, a data processing apparatus for implementing the above-described data processing method is also provided. FIG10 is a structural block diagram of a data processing apparatus according to an embodiment of this disclosure. As shown in FIG10, the apparatus includes:

[0180] The receiving module 1001 is configured to receive data query requests;

[0181] The first generation module 1002 is configured to generate the first candidate plan set corresponding to the data query request;

[0182] Processing module 1003 is configured to use a target execution cost prediction model to predict the cost of the first candidate plan set and select a target execution plan. The target execution cost prediction model is trained using the training plan corresponding to the target item. The target item is selected from the candidate items. The target item includes multiple historical queries. The training plan includes the historical default execution plan and candidate query plan corresponding to each of the multiple historical queries.

[0183] The second generation module 1004 is configured to generate data query responses corresponding to data query requests based on the target execution plan.

[0184] Optionally, the data processing device further includes: a training module 1005, configured to perform adversarial training on the initial execution cost prediction model according to the training plan, generating a domain classification loss and an execution cost prediction loss; and an update module 1006, configured to update the model parameters of the initial execution cost prediction model based on the domain classification loss and the execution cost prediction loss, to obtain a target execution cost prediction model.

[0185] Optionally, the data processing device further includes a filtering module 1007, configured to use an item selector to filter target items from candidate items according to preset filtering rules, wherein the preset filtering rules are used to quantify the applicability of candidate items from multiple dimensions.

[0186] Optionally, the filtering module 1007 is further configured to: use a project filter to filter candidate projects according to preset filtering rules to obtain remaining projects; and use a project sorter to sort the remaining projects according to their estimated revenue to select target projects.

[0187] Optionally, the filtering module 1007 is also configured to: use the project filter to quantify the candidate projects from the dimensions of historical query quantity, table feature and query plan structure, respectively, and exclude some projects that do not meet the preset measurement conditions from the candidate projects to obtain the remaining projects.

[0188] Optionally, the filtering module 1007 is further configured to filter out a subset of items from the remaining items; the data processing device further includes: a third generation module 1008, configured to generate a second candidate plan set for the subset of items; and a training module 1005, configured to train an item sorter based on the default execution plan and the second candidate plan set.

[0189] Optionally, the training module 1005 is further configured to: obtain the first execution cost of the default execution plan and the second execution cost of the second candidate plan set; determine the estimated revenue of each query contained in a subset of items based on the first execution cost and the second execution cost; and train an item sorter based on the estimated revenue.

[0190] Optionally, the training module 1005 is further configured to: estimate the potential optimization space of the default execution plan relative to the optimal plan based on the first execution cost and the second execution cost, wherein the optimal plan is the plan with the minimum execution cost in the second candidate plan set; score some items based on the potential optimization space, and determine the estimated revenue of each query contained in the some items.

[0191] Optionally, the filtering module 1007 is further configured to: obtain the estimated revenue of the remaining projects using a project sorter, sort the remaining projects according to the estimated revenue to obtain a sorting result, and select the top-ranked projects as target projects based on the sorting result.

[0192] Optionally, the data processing apparatus further includes a deployment module 1009, configured to deploy candidate projects in a preset storage area, wherein the preset storage area is a multi-tenant distributed data warehouse.

[0193] Optionally, the training module 1005 is further configured to: generate the encoding result of the training plan using the initial execution cost prediction model, and perform domain classification and cost prediction based on the encoding result to obtain the classification result and the predicted execution cost, respectively; calculate the domain classification loss based on the classification result and the domain label, and calculate the execution cost prediction loss based on the predicted execution cost and the actual execution cost, wherein, in the cost prediction process, the default execution plan is a labeled plan and the candidate query plan is an unlabeled plan, and in the domain classification process, the domain label includes: the candidate plan domain label assigned to the candidate query plan and the default plan domain label assigned to the default execution plan.

[0194] Optionally, the update module 1006 is further configured to: perform gradient flipping on the domain classification loss to obtain the flipping result; and update the model parameters of the initial execution cost prediction model based on the flipping result and the execution cost prediction loss to obtain the target execution cost prediction model.

[0195] Optionally, the first generation module 1002 is further configured to: generate a first candidate plan set corresponding to the data query request by means of a query optimizer using at least one plan generation method, wherein the at least one plan generation method includes: a plan generation method that scales the cardinality estimated during the generation process of the query optimizer, and a plan generation method that enables or disables at least one optimization option in the query optimizer.

[0196] Optionally, the processing module 1003 is further configured to: use a target execution cost prediction model to predict the cost of the first candidate plan set to obtain the prediction result, and select a target execution plan that meets the preset execution cost conditions from the first candidate plan set based on the prediction result.

[0197] Optionally, the processing module 1003 is further configured to: use a target execution cost prediction model to perform feature encoding on the first candidate plan set to obtain an encoding result, and perform execution cost prediction on the encoding result to obtain a prediction result, wherein the encoding result is obtained by feature encoding the structural information and environmental information of the first candidate plan set, the structural information is used to determine the operation type of the plan nodes contained in the first candidate plan set and the multi-dimensional operation information associated with the first candidate plan set, and the environmental information is used to determine the actual execution environment of the first candidate plan set.

[0198] It should be noted that the receiving module 1001, the first generating module 1002, the processing module 1003, and the second generating module 1004 correspond to steps S31 to S34 in the above embodiments. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the electronic device provided in the above embodiments.

[0199] Figure 11 is a structural block diagram of another data processing apparatus according to an embodiment of the present disclosure. As shown in Figure 11, the apparatus includes:

[0200] The receiving module 1101 is configured to receive order query data from e-commerce platforms;

[0201] The first generation module 1102 is configured to generate the first candidate plan set corresponding to the order query data of the e-commerce platform;

[0202] Processing module 1103 is configured to use a target execution cost prediction model to predict the cost of the first candidate plan set and select a target execution plan. The target execution cost prediction model is trained using the training plan corresponding to the target item. The target item is selected from the candidate items. The target item includes multiple historical queries. The training plan includes the historical default execution plan and candidate query plan corresponding to each of the multiple historical queries.

[0203] The second generation module 1104 is configured to generate e-commerce platform order statistics responses corresponding to e-commerce platform order query data based on the target execution plan.

[0204] It should be noted that the receiving module 1101, the first generating module 1102, the processing module 1103, and the second generating module 1104 correspond to steps S61 to S64 in the above embodiments. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the electronic device provided in the above embodiments.

[0205] Figure 12 is a structural block diagram of another data processing apparatus according to an embodiment of the present disclosure. As shown in Figure 12, the apparatus includes:

[0206] The acquisition module 1201 is configured to acquire a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: a data query request;

[0207] The return module 1202 is configured to return a data processing response through a second application programming interface, wherein the response data carried in the data processing response includes: a data query response corresponding to the data query request, and the data query response is obtained according to any one of the data processing methods in the embodiments of this disclosure.

[0208] It should be noted that the acquisition module 1201 and the return module 1202 correspond to steps S71 to S72 in the above embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the electronic device provided in the above embodiments.

[0209] Figure 13 is a structural block diagram of another data processing apparatus according to an embodiment of the present disclosure. As shown in Figure 13, the apparatus includes:

[0210] The acquisition module 1301 is configured to acquire the currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: a data query request;

[0211] Return module 1302 is configured to return a data processing dialogue response in response to a data processing dialogue request. The data processing dialogue response carries information including: a data query response corresponding to the data query request, wherein the data query response is obtained according to any one of the data processing methods in the embodiments of this disclosure.

[0212] Display module 1303 is configured to display data query responses within a graphical user interface.

[0213] It should be noted that the acquisition module 1301, return module 1302, and display module 1303 correspond to steps S81 to S83 in the above embodiments. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the electronic device provided in the above embodiments.

[0214] Figure 14 is a structural block diagram of another data processing apparatus according to an embodiment of the present disclosure. As shown in Figure 14, the apparatus includes:

[0215] The first display module 1401 is configured to display a data query request on the operation interface in response to input commands applied to the operation interface.

[0216] The second display module 1402 is configured to respond to processing instructions applied to the operation interface and display the data query response corresponding to the data query request on the operation interface; wherein the data query response is obtained according to the data processing method of any one of the embodiments of this disclosure.

[0217] It should be noted that the first display module 1401 and the second display module 1402 mentioned above correspond to steps S91 to S92 in the above embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the electronic device provided in the above embodiments.

[0218] It should be noted that the preferred implementation schemes involved in the above embodiments of this disclosure are the same as the schemes, application scenarios and implementation processes provided in the above embodiments, but are not limited to the schemes provided in the above embodiments.

[0219] Embodiments of this disclosure can provide a data processing system, including: a client for sending a data query request; a server connected to the client for generating a first candidate plan set corresponding to the data query request, using a target execution cost prediction model to predict the cost of the first candidate plan set to select a target execution plan, and generating a data query response corresponding to the data query request based on the target execution plan, wherein the target execution cost prediction model is trained using a training plan corresponding to a target item, the target item is selected from the candidate items, the target item includes: multiple historical queries, and the training plan includes: historical default execution plans and candidate query plans corresponding to each of the multiple historical queries; the client is also used to output the data query response.

[0220] Embodiments of this disclosure may provide a computer terminal, which may include a server and a client. The server may be any one of the servers in a server device group or a cloud server.

[0221] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0222] In this embodiment, the computer terminal described above can execute the program code in the method.

[0223] Optionally, FIG15 is a structural block diagram of a computer terminal according to an embodiment of the present disclosure. As shown in FIG15, the computer terminal may include: one or more (only one is shown in the figure) processors 152, memory 154, memory controller, and peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0224] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0225] The processor can invoke an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.

[0226] Those skilled in the art will understand that the structure shown in the figure above is merely illustrative, and the computer terminal may also be a smartphone, tablet computer, PDA, mobile internet device (MID), PAD, or other terminal device. The figure above does not limit the structure of the aforementioned electronic device. For example, a computer terminal may include more or fewer components (such as network interfaces, display devices, etc.) than shown in the figure above, or may have a different configuration than shown in the figure above.

[0227] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0228] Embodiments of this disclosure also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store program code executed by the method provided in the above embodiments.

[0229] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a computer terminal cluster, or in any mobile terminal in a mobile terminal cluster.

[0230] Optionally, in this embodiment, the computer-readable storage medium is configured to store an executable program, which, when the executable program is running, controls the device where the computer-readable storage medium is located to execute the method described in any of the above embodiments.

[0231] Embodiments of this disclosure also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.

[0232] Embodiments of this disclosure also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which can be used to store a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.

[0233] Embodiments of this disclosure also provide a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the method provided in the above embodiments.

[0234] In the above embodiments of this disclosure, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0235] In the several embodiments provided in this disclosure, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0236] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0237] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0238] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0239] The above description is only a preferred embodiment of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of this disclosure, and these improvements and modifications should also be considered within the scope of protection of this disclosure.

Claims

1. A data processing method, comprising: Receive data query requests; Generate the first candidate plan set corresponding to the data query request; A target execution cost prediction model is used to predict the cost of the first candidate plan set and select a target execution plan. The target execution cost prediction model is trained using a training plan corresponding to the target item. The target item is selected from the candidate items. The target item includes multiple historical queries. The training plan includes the historical default execution plan and candidate query plan corresponding to each of the multiple historical queries. The data query response corresponding to the data query request is generated based on the target execution plan.

2. The data processing method according to claim 1, wherein, The data processing method further includes: According to the training plan, the initial execution cost prediction model is subjected to adversarial training to generate domain classification loss and execution cost prediction loss; Based on the domain classification loss and the execution cost prediction loss, the model parameters of the initial execution cost prediction model are updated to obtain the target execution cost prediction model.

3. The data processing method according to claim 2, wherein, The data processing method further includes: Using a project selector, the target project is selected from the candidate projects according to preset filtering rules, wherein the preset filtering rules are used to quantify the applicability of the candidate projects from multiple dimensions.

4. The data processing method according to claim 3, wherein, The project selector includes a project filter and a project sorter. Using the project selector, the target project is selected from the candidate projects according to the preset filtering rules, including: Using the project filter, the candidate projects are filtered according to the preset filtering rules to obtain the remaining projects; The project sorter is used to sort the remaining projects according to their estimated revenue in order to select the target project.

5. The data processing method according to claim 4, wherein, The preset filtering rules include: historical query quantity dimension, table feature dimension, and query plan structure dimension. Using the item filter, the candidate items are filtered according to the preset filtering rules to obtain the remaining items, which include: Using the project filter, the candidate projects are quantified from the dimensions of historical query quantity, table feature, and query plan structure, respectively. Some projects that do not meet the preset measurement conditions are excluded from the candidate projects to obtain the remaining projects.

6. The data processing method according to claim 4, wherein, The data processing method further includes: Select a subset of projects from the remaining projects; Generate a second candidate plan set for the aforementioned projects; The project sorter is trained based on the default execution plan and the second candidate plan set.

7. The data processing method according to claim 6, wherein, The project sorter, trained based on the default execution plan and the second candidate plan set, includes: Obtain the first execution cost of the default execution plan and the second execution cost of the second candidate plan set; Based on the first execution cost and the second execution cost, determine the estimated revenue of each query included in the partial project; The project sorter is trained based on the estimated revenue.

8. The data processing method according to claim 7, wherein, Based on the first execution cost and the second execution cost, the estimated revenue for each query included in the partial project is determined to include: Based on the first execution cost and the second execution cost, the potential optimization space of the default execution plan relative to the optimal plan is estimated, wherein the optimal plan is the plan with the minimum execution cost in the second candidate plan set; The projects are scored based on the potential optimization space to determine the estimated revenue for each query included in the projects.

9. The data processing method according to claim 4, wherein, Using the project sorter, the remaining projects are sorted according to their estimated revenue to filter out the target projects, including: The estimated revenue of the remaining projects is obtained using the project sorter, the remaining projects are sorted according to the estimated revenue to obtain a sorting result, and the top-ranked projects are selected as the target projects based on the sorting result.

10. The data processing method according to claim 1, wherein, The data processing method further includes: The candidate projects are deployed in a preset storage area, wherein the preset storage area is a multi-tenant distributed data warehouse.

11. The data processing method according to claim 2, wherein, According to the training plan, the initial execution cost prediction model is adversarially trained to generate the domain classification loss and the execution cost prediction loss, including: The initial execution cost prediction model is used to generate the encoding result of the training plan, and the domain classification and cost prediction are performed based on the encoding result to obtain the classification result and the predicted execution cost, respectively. The domain classification loss is calculated based on the classification results and domain labels, and the execution cost prediction loss is calculated based on the predicted execution cost and the actual execution cost. In the cost prediction process, the default execution plan is a labeled plan and the candidate query plan is an unlabeled plan. In the domain classification process, the domain labels include: the candidate plan domain label assigned to the candidate query plan and the default plan domain label assigned to the default execution plan.

12. The data processing method according to claim 2, wherein, Based on the domain classification loss and the execution cost prediction loss, the model parameters of the initial execution cost prediction model are updated to obtain the target execution cost prediction model, including: Gradient flipping is applied to the domain classification loss to obtain the flipped result; Based on the flipping result and the execution cost prediction loss, the model parameters of the initial execution cost prediction model are updated to obtain the target execution cost prediction model.

13. The data processing method according to claim 1, wherein, The first candidate plan set corresponding to the data query request includes: The query optimizer generates the first candidate plan set corresponding to the data query request using at least one plan generation method, wherein the at least one plan generation method includes: a plan generation method that scales the cardinality estimated during the generation process of the query optimizer, and a plan generation method that enables or disables at least one optimization option in the query optimizer.

14. The data processing method according to claim 1, wherein, The target execution plan is selected by using the target execution cost prediction model to predict the cost of the first candidate plan set, including: The target execution cost prediction model is used to predict the cost of the first candidate plan set to obtain the prediction result, and the target execution plan that meets the preset execution cost condition is selected from the first candidate plan set based on the prediction result.

15. The data processing method according to claim 14, wherein, The cost prediction of the first candidate plan set using the target execution cost prediction model to obtain the prediction result includes: The target execution cost prediction model is used to perform feature encoding on the first candidate plan set to obtain the encoding result, and the execution cost is predicted on the encoding result to obtain the prediction result. The encoding result is obtained by feature encoding the structural information and environmental information of the first candidate plan set. The structural information is used to determine the operation type of the plan nodes included in the first candidate plan set and the multiple dimensions of operation information associated with the first candidate plan set. The environmental information is used to determine the actual execution environment of the first candidate plan set.

16. A data processing method, comprising: Receive order query data from e-commerce platforms; Generate the first candidate plan set corresponding to the order query data of the e-commerce platform; A target execution cost prediction model is used to predict the cost of the first candidate plan set and select a target execution plan. The target execution cost prediction model is trained using a training plan corresponding to the target item. The target item is selected from the candidate items. The target item includes multiple historical queries. The training plan includes the historical default execution plan and candidate query plan corresponding to each of the multiple historical queries. Based on the target execution plan, generate the e-commerce platform order statistics response corresponding to the e-commerce platform order query data.

17. A data processing method, comprising: A data processing request is obtained through a first application programming interface, wherein the request data carried in the data processing request includes: a data query request; A data processing response is returned through a second application programming interface, wherein the response data carried in the data processing response includes: a data query reply corresponding to the data query request, wherein the data query reply is obtained according to the data processing method of any one of claims 1 to 15.

18. A data processing method, comprising: Obtain the currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: a data query request; In response to the data processing dialogue request, a data processing dialogue response is returned, wherein the information carried in the data processing dialogue response includes: a data query response corresponding to the data query request, wherein the data query response is obtained according to the data processing method of any one of claims 1 to 15; The data query response is displayed within the graphical user interface.

19. A data processing method, comprising: In response to input commands applied to the user interface, a data query request is displayed on the user interface. In response to the processing instructions applied to the operation interface, the data query response corresponding to the data query request is displayed on the operation interface; The data query response is obtained according to the data processing method described in any one of claims 1 to 15.

20. A data processing system, comprising: The client is used to send data query requests; The server, connected to the client, is configured to generate a first candidate plan set corresponding to the data query request, use a target execution cost prediction model to predict the cost of the first candidate plan set to select a target execution plan, and generate a data query response corresponding to the data query request based on the target execution plan. The target execution cost prediction model is trained using a training plan corresponding to a target item, the target item is selected from the candidate items, and the target item includes multiple historical queries. The training plan includes the historical default execution plan and candidate query plan corresponding to each of the multiple historical queries. The client is also used to output the data query response.

21. An electronic device, comprising: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the data processing method according to any one of claims 1 to 19.

22. A computer-readable storage medium comprising a stored executable program, wherein, When the executable program is executed, it controls the device containing the computer-readable storage medium to perform the data processing method according to any one of claims 1 to 19.