Model performance prediction method and apparatus, and performance prediction model construction method and apparatus

Abstract

The present description relates to the technical field of financial technology, and in particular relates to a model performance prediction method and apparatus, and a performance prediction model construction method and apparatus. The model performance prediction method comprises: determining influencing factors for a model to be subjected to prediction; inputting the influencing factors into a performance prediction model to obtain a key predicted value of resource consumption and a key predicted value of average time consumption that are required for invoking the model to be subjected to prediction, wherein the performance prediction model is obtained by constructing a training sample set from a key predicted value of sample resource consumption, a key predicted value of sample average time consumption and a plurality of influencing factors for each sample model, and training a plurality of initial prediction models; and on the basis of the key predicted value of resource consumption and the key predicted value of average time consumption, determining an overall predicted value of resource consumption and an overall predicted value of average time consumption for the model to be subjected to prediction.

Description

Model performance prediction method, performance prediction model construction method and apparatus

[0001] Related applications

[0002] This application claims Chinese Patent Application No. 202510215736.9, filed on February 26, 2025, and incorporates the disclosure of the aforementioned patent application as part of this application. Technical Field

[0003] This specification relates to the field of financial technology, particularly to model performance prediction methods, performance prediction model construction methods, and apparatus. Background Technology

[0004] In recent years, the fields of big data and machine learning have experienced rapid development, driving innovation and transformation across multiple industries. In machine learning, balancing model effectiveness with model performance (e.g., computational efficiency, resource consumption) is a crucial issue, especially when deploying and running models in production environments. Model effectiveness primarily refers to the model's predictive ability, typically measured by evaluation metrics such as accuracy, recall, precision, F1 score (the harmonic mean of precision and recall), and the area under the ROC (Receiver Operating Characteristic Curve) curve (AUC). Better model effectiveness indicates a higher degree of fit to the data in the task. Model performance refers to the model's efficiency during real-time computation, involving inference time, memory usage, and computational resource consumption. A high-performance model should be able to complete inference within reasonable time and resource consumption. Finding the appropriate balance requires comprehensive consideration of various factors, including business needs, hardware limitations, and response time requirements.

[0005] Before a model goes live, it undergoes stress testing and evaluation. Only models that meet certain performance standards are allowed to go live. If a model fails the stress test, analysts often need to readjust the model, including modifying model parameters and retraining it, which greatly increases the workload at each stage of model production and significantly impacts delivery efficiency. Summary of the Invention

[0006] To address the problems of existing technologies, this specification provides a method for predicting model performance, a method for constructing a performance prediction model, and an apparatus.

[0007] This specification provides a model performance prediction method, which includes: determining the influencing factors of the model to be predicted; inputting the influencing factors into the performance prediction model to obtain the key predicted values ​​of resource consumption and average time consumption required to call the model to be predicted, wherein the performance prediction model is obtained by constructing a training sample set from the key values ​​of sample resource consumption, the key values ​​of sample average time consumption, and multiple influencing factors of the sample model, and training multiple initial prediction models; and determining the overall predicted value of resource consumption and the overall predicted value of average time consumption of the model to be predicted based on the key values ​​of resource consumption and average time consumption.

[0008] According to one aspect of the embodiments of this specification, the method further includes: determining the theoretical value of resource consumption prediction and the theoretical value of average time prediction of the model to be predicted based on the sum of resource consumption when a single feature set in at least one original feature set on which the model to be predicted depends is invoked, the sum of resource consumption when a single derived feature set in at least one derived feature set on which the model to be predicted depends is invoked, the maximum value of average time consumption when a single feature set in at least one original feature set on which the model to be predicted depends is invoked, and the maximum value of average time consumption when a single derived feature set in at least one derived feature set on which the model to be predicted depends is invoked; determining whether the overall predicted value of resource consumption of the model to be predicted is greater than the theoretical value of resource consumption prediction, and whether the overall predicted value of average time consumption of the model to be predicted is greater than the theoretical value of average time consumption prediction; determining the final predicted amount of resource consumption and the final predicted amount of average time consumption of the model to be predicted based on the comparison result between the overall predicted value of resource consumption and the theoretical value of resource consumption prediction, and the comparison result between the overall predicted value of average time consumption and the theoretical value of average time consumption prediction; and ending the performance prediction of the model to be predicted if the final predicted amount of resource consumption satisfies a first preset condition and the final predicted amount of average time consumption satisfies a second preset condition.

[0009] According to one aspect of the embodiments of this specification, determining the final resource consumption forecast and the final average time forecast based on the comparison result of the overall predicted value of resource consumption and the theoretical predicted value of resource consumption, and the comparison result of the overall predicted value of average time consumption and the theoretical predicted value of average time consumption, includes: determining whether the overall predicted value of resource consumption of the model to be predicted is greater than the theoretical predicted value of resource consumption, and whether the overall predicted value of average time consumption of the model to be predicted is greater than the theoretical predicted value of average time consumption; if not, determining the theoretical predicted value of resource consumption as the final resource consumption forecast, and determining the theoretical predicted value of average time consumption as the final average time forecast; if yes, determining the overall predicted value of resource consumption as the final resource consumption forecast, and determining the overall predicted value of average time consumption of the model to be predicted as the final average time forecast.

[0010] According to one aspect of the embodiments of this specification, the first preset condition includes that the final resource consumption prediction does not exceed a preset resource consumption threshold; the second preset condition includes that the final average time prediction does not exceed a preset average time threshold.

[0011] This specification discloses a method for constructing a performance prediction model. The performance prediction model is constructed as follows: parsing the sample feature set dependencies of a sample model; initiating a call to the sample model based on the sample feature set dependencies, wherein the sample feature set dependencies include at least an original sample feature set and a derived sample feature set; determining key values ​​of sample resource consumption and key values ​​of sample average time consumption generated during the sample model call by calling the original sample feature set and the derived sample feature set; constructing a training sample set using the key values ​​of sample resource consumption, key values ​​of sample average time consumption, and multiple influencing factors; and training multiple initial prediction models using the training sample set to obtain a performance prediction model; wherein the multiple initial prediction models include: an initial prediction model for predicting resource consumption and an initial prediction model for predicting average time consumption.

[0012] According to one aspect of the embodiments of this specification, parsing the sample feature set dependency relationship of a sample model and initiating a call to the sample model based on the sample feature set dependency relationship includes: initiating a call to the original sample feature set to obtain a first call result; and initiating a call to the derived sample feature set based on the first call result to obtain a second call result.

[0013] According to one aspect of the embodiments of this specification, determining the key values ​​of sample resource consumption and sample average time consumption generated during the sample model invocation process by invoking the original sample feature set and the derived sample feature set includes: initiating an invocation on the original sample feature set to obtain the maximum values ​​of the first resource consumption and the first average time consumption of the first invocation result; initiating an invocation on the derived sample feature set to obtain the maximum values ​​of the second resource consumption and the second average time consumption of the second invocation result; determining the key value of sample resource consumption based on the first resource consumption and the second resource consumption; and determining the key value of sample average time consumption based on the maximum values ​​of the first average time consumption and the second average time consumption.

[0014] According to one aspect of the embodiments of this specification, a training sample set is constructed using the key values ​​of sample resource consumption, the key values ​​of sample average time consumption, and multiple influencing factors. Multiple initial prediction models are trained using the training sample set to obtain a performance prediction model, comprising: calculating multiple influencing factors based on the original sample feature set and the derived sample feature set; inputting the multiple influencing factors into the multiple initial prediction models respectively to obtain the output results of each initial prediction model; calculating a loss value based on the labels of each initial prediction model and the output results of each initial prediction model; iteratively updating the model parameters of each initial prediction model using the loss value until the performance prediction model is obtained; wherein the maximum values ​​of the first resource consumption, the second resource consumption, the first average time consumption, and the second average time consumption are respectively used as the labels of each initial prediction model.

[0015] This specification provides a model performance prediction device, comprising: an influencing factor determination unit, configured to determine influencing factors of the model to be predicted; and a performance prediction model, configured to input the influencing factors into the performance prediction model to obtain key predicted values ​​of resource consumption and average time consumption required to call the model to be predicted, wherein the performance prediction model is obtained by constructing a training sample set from the key predicted values ​​of sample resource consumption, the key predicted values ​​of sample average time consumption, and multiple influencing factors, and training multiple initial prediction models; and a prediction value determination unit, configured to determine the overall predicted value of resource consumption and the overall predicted value of average time consumption of the model to be predicted based on the key predicted values ​​of resource consumption and the key predicted values ​​of average time consumption.

[0016] This specification provides a performance prediction model construction apparatus, comprising: a calling unit, configured to parse the sample feature set dependency relationship of a sample model and initiate a call to the sample model according to the sample feature set dependency relationship, wherein the sample feature set dependency relationship includes at least an original sample feature set and a derived sample feature set; a key value determination unit, configured to determine the sample resource consumption key value and the sample average time consumption key value generated during the sample model call by calling the original sample feature set and the derived sample feature set; and a training unit, configured to construct a training sample set using the sample resource consumption key value, the sample average time consumption key value and multiple influencing factors, and train multiple initial prediction models using the training sample set to obtain a performance prediction model; wherein the multiple initial prediction models include: an initial prediction model for predicting resource consumption and an initial prediction model for predicting average time consumption.

[0017] This specification also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the model performance prediction and performance prediction model construction method.

[0018] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the model performance prediction and performance prediction model construction method.

[0019] This solution can predict the resource consumption and performance of a model before it goes live, reducing manual workload, avoiding rework, and ensuring efficient model delivery. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 shows a flowchart of a model performance prediction method according to an embodiment of this specification;

[0022] Figure 2a shows a flowchart of a method for determining the final resource consumption forecast and the final average time consumption forecast according to an embodiment of this specification.

[0023] Figure 2b is a flowchart of a method for determining the final resource consumption forecast and the final average time consumption forecast according to an embodiment of this specification.

[0024] Figure 3 is a flowchart of a performance prediction model construction method according to an embodiment of this specification.

[0025] Figure 4 is a flowchart of a specific process of calling a model according to an embodiment of this specification;

[0026] Figure 5 shows a flowchart of a method for determining key values ​​of sample resource consumption and key values ​​of average sample time generated during the sample model invocation process according to an embodiment of this specification.

[0027] Figure 6 is a flowchart of a method for constructing a performance prediction model according to an embodiment of this specification;

[0028] Figure 7 is a schematic diagram of the structure of a model performance prediction device according to an embodiment of this specification;

[0029] Figure 8 is a schematic diagram of a performance prediction model construction device according to an embodiment of this specification.

[0030] Figure 9 shows a schematic diagram of a model deployment architecture according to an embodiment of this specification;

[0031] Figure 10 is a schematic diagram of the calling order of the dependent feature set of a model to be predicted according to an embodiment of this specification;

[0032] Figure 11 shows a schematic diagram of the structure of a computer device according to an embodiment of this specification. Detailed Implementation

[0033] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0035] This specification provides the operational steps of the methods described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods shown in the embodiments or drawings can be executed sequentially or in parallel.

[0036] In related technologies, when using a trained model as an API (Application Programming Interface) service, the service is typically provided externally through the API interface, allowing users to obtain model prediction results by calling the API. However, during service provision, the service provider processes input data in real time and returns prediction results, focusing more on the model training process and model performance. There is no direct and effective means to control the resource consumption of the features the model relies on during processing, or the overall model performance during the model call process, making it impossible to achieve a balance between model performance and model effectiveness. Therefore, this specification provides a model performance prediction method, a performance prediction model construction method, and an apparatus to solve the above-mentioned technical problems.

[0037] It should be noted that the model performance prediction method, performance prediction model construction method and device described in this specification can be used in the field of financial technology, as well as in any other field besides financial technology, including data analysis, software development, information technology and other fields. This specification does not limit the application field of the model performance prediction method, performance prediction model construction method and device.

[0038] All information, data, and signals disclosed herein are authorized by the user or by all parties, and the collection, use, and processing of such data comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0039] Figure 1 shows a flowchart of a method for determining the overall predicted value of resource consumption and the overall predicted value of average time consumption of a model to be predicted according to an embodiment of this specification. The method includes at least the following steps 101-103, which are executed by the model performance prediction device.

[0040] Step 101: Determine the influencing factors of the model to be predicted. This step requires identifying the influencing factors of the model to be predicted in order to predict its resource consumption and average processing time.

[0041] In the embodiments described in this specification, in a real production environment, influencing factors are those factors that cause resource competition when the feature sets of multiple models to be predicted are called simultaneously. In some examples, influencing factors include, but are not limited to, resource consumption and average execution time.

[0042] In the embodiments of this specification, resource contention refers to the struggle for computing resources among multiple processes or threads during execution. For example, CPU (Central Processing Unit) resource contention specifically refers to the competition for computing resources by the CPU among multiple processes or threads during execution. Since the CPU's processing power is limited, contention arises when multiple processes simultaneously request CPU computation, potentially leading to decreased system performance and increased response time. In multitasking operating systems, the CPU can allocate CPU time (i.e., the time spent by the computer processor executing a specific program) to different processes or threads through time-slice round-robin or other scheduling algorithms to minimize the impact of resource contention and improve the overall system efficiency and responsiveness.

[0043] This step calculates the influencing factors of the model to be predicted by monitoring parameters such as CPU consumption and average execution time when a single feature set in the original feature set and derived feature set (if any) on which the model to be predicted depends is invoked. Based on data similar to Tables 1 and 2 in this specification, the CPU consumption and average execution time when a single feature set in the original feature set and derived feature set on which the model to be predicted depends is determined. In the embodiments of this specification, the feature set performance tables corresponding to Tables 1 and 2 record parameters such as CPU consumption and average execution time when a single feature set in all candidate original and derived feature sets is invoked; therefore, the parameters can be directly obtained from the tables to calculate the influencing factors.

[0044] In some embodiments, influencing factors include, but are not limited to: the number of original feature sets on which the model to be predicted depends (denoted as num_fc), the sum of CPU consumption when a single feature set in the original feature set on which the model to be predicted is called (denoted as sum_cpu_fc), the maximum average time spent when a single feature set in the original feature set on which the model to be predicted is called (denoted as max_avg_cost_fc), the number of secondary derived feature sets on which the model to be predicted depends (denoted as num_sfc), the sum of CPU consumption when a single feature set in the secondary derived feature set on which the model to be predicted is called (denoted as sum_cpu_sfc), the maximum average time spent when a single feature set in the secondary derived feature set on which the model to be predicted is called (denoted as max_avg_cost_sfc), the coefficient of variation of CPU consumption when a single feature set in the original feature set on which the model to be predicted is called (denoted as cv_cpu_fc), the range of CPU consumption when a single feature set in the feature set on which the model to be predicted is called (denoted as range_cpu_fc), and the variation of average time spent when a single feature set in the feature set on which the model depends is called. The coefficient of variation (denoted as cv_avg_cost_fc), the range of average time spent calling a single feature set in the feature set that the model depends on (denoted as range_avg_cost_fc), the range of CPU consumption when a single feature set in the feature set that the model depends on is called (denoted as range_cpu_fc), the coefficient of variation of CPU consumption when a single feature set in the secondary derived feature set that the model depends on is called (denoted as cv_cpu_sfc), the range of CPU consumption when a single feature set in the secondary derived feature set that the model depends on is called (denoted as range_cpu_sfc), the coefficient of variation of average time spent calling a single feature set in the secondary derived feature set that the model depends on is called (denoted as cv_avg_cost_sfc), the range of average time spent calling a single feature set in the secondary derived feature set that the model depends on is called (denoted as range_avg_cost_sfc), the total CPU resources of the relevant production server (denoted as sum_cpu_total), and the number of concurrent requests corresponding to the performance indicators (the number of requests that can be initiated by clients at the same time) (denoted as num_concurrency), etc. The method for determining the influencing factors of the model to be predicted in this step is similar to the scheme in the embodiment shown in Figure 6 later, and will not be described in detail here.

[0045] Step 102: Input the influencing factors into the performance prediction model to obtain the key predicted values ​​of resource consumption and average time consumption required to call the model to be predicted. The performance prediction model is constructed by building a training sample set from the key predicted values ​​of sample resource consumption, the key predicted values ​​of sample average time consumption, and multiple influencing factors of the sample model, and then training multiple initial prediction models.

[0046] In this specification, the performance prediction model is used to predict the resource consumption and average time required to call the model before it goes live, thereby determining the model performance. For a detailed description of the construction of the performance prediction model, please refer to the performance prediction model construction method shown in Figures 3-6 in the following embodiments.

[0047] In this step, the influencing factors of the model to be predicted are used as feature data of the performance prediction model. After being input into the performance prediction model, the performance prediction model outputs prediction results related to the resource consumption and time consumption that the model to be predicted will occupy. That is, the prediction results are the predicted values ​​of resource consumption and average time consumption for the model to be predicted. Specifically, the performance prediction model includes four prediction models: the first prediction model, the second prediction model, the third prediction model, and the fourth prediction model. It should be noted that the first prediction model, the second prediction model, the third prediction model, and the fourth prediction model can all be any artificial intelligence model. In this step, the influencing factors of the model to be predicted are input into these four prediction models respectively, and the output results of the four prediction models are as follows: the output of the first prediction model (denoted as sum_cpu_fc_Y), the output of the second prediction model (denoted as sum_cpu_sfc_Y), the output of the third prediction model (denoted as max_avg_cost_fc_Y), and the output of the fourth prediction model (denoted as max_avg_cost_sfc_Y). In the output of the four prediction models, sum_cpu_fc_Y and sum_cpu_sfc_Y are the key predicted values ​​of resource consumption, while max_avg_cost_fc_Y and max_avg_cost_sfc_Y are the key predicted values ​​of average time consumption.

[0048] Step 103: Based on the key predicted values ​​of resource consumption and average time consumption, determine the overall predicted value of resource consumption and the overall predicted value of average time consumption for the model to be predicted.

[0049] In the embodiments of this specification, according to the sequential flow of the model invocation, the overall predicted value of the resource consumption (cpu_predict) of the model to be predicted includes: the CPU consumption generated by the simultaneous invocation of at least one original feature set that the model ultimately depends on, the CPU consumption generated by the simultaneous invocation of at least one derived feature set, and the sum of the CPU consumption generated by its own algorithm and the API entry point; the overall predicted value of the average time consumption of the model to be predicted is the sum of the maximum average time consumption generated by the simultaneous invocation of at least one original feature set that the model ultimately depends on, the maximum average time consumption generated by the simultaneous invocation of at least one derived feature set, the average time consumption of its own algorithm, and the average time consumption of the API entry point. Key predicted values ​​of resource consumption include the CPU consumption generated by the simultaneous invocation of at least one original feature set that the model ultimately depends on, and the CPU consumption generated by the simultaneous invocation of at least one derived feature set that the model depends on. Key predicted values ​​of average time consumption include the maximum average time consumption generated by the simultaneous invocation of at least one original feature set that the model ultimately depends on, and the maximum average time consumption generated by the simultaneous invocation of the derived feature set that the model depends on.

[0050] In this step, determining the overall predicted value of resource consumption and the overall predicted value of average time for the model to be predicted, based on the key values ​​of resource consumption and average time, specifically includes: inputting the key values ​​of resource consumption and average time into the overall predicted value of resource consumption and the overall predicted value of average time, respectively, to determine the overall predicted value of resource consumption and the overall predicted value of average time. The overall predicted value of resource consumption for the model to be predicted can be represented as follows:

[0051] cpu_predict = SUM(

[0052] SUM(cpu(the original feature set on which the model to be predicted depends)),

[0053] SUM(cpu(derived feature set on which the model to be predicted depends)),

[0054] OneOf(lr_cpu, rf_cpu, dt_cpu),

[0055] api_cpu

[0056] );

[0057] Here, SUM() represents summation, cpu() represents listing the resource consumption of multiple objects within the parentheses, cpu(original feature set dependent on the model to be predicted) is the CPU consumption generated when at least one original feature set ultimately dependent on the model to be predicted is called simultaneously, SUM(cpu(original feature set dependent on the model to be predicted)) is the sum of CPU consumption generated when at least one original feature set ultimately dependent on the model is called simultaneously, cpu(derived feature set dependent on the model to be predicted) is the CPU consumption generated when at least one derived feature set dependent on the model is called simultaneously, and SUM(cpu(derived feature set dependent on the model to be predicted)) is the sum of CPU consumption generated when at least one derived feature set dependent on the model is called simultaneously.

[0058] The key predicted values ​​for resource consumption in this step include SUM(cpu(original feature set on which the model to be predicted depends)) and SUM(cpu(derived feature set on which the model to be predicted depends)); OneOf() represents the CPU consumption generated by the algorithm of the model to be predicted itself, and the specific constant value is selected according to the different algorithms used by the model to be predicted; api_cpu represents the CPU consumption generated by the API interface when it calls the model to be predicted with the original feature set and the derived feature set as parameters.

[0059] In this step, the average time taken by the model to be predicted can be represented as follows:

[0060] avg_cost_predict = SUM(

[0061] MAX(avg_cost(the original feature set on which the model to be predicted depends)),

[0062] MAX(avg_cost(the derived feature set on which the model to be predicted depends)),

[0063] OneOf(lr_cost,rf_cost,dt_cost),

[0064] api_cost

[0065] )

[0066] In this context, MAX() represents finding the maximum value, avg_cost() represents listing the average time of multiple objects within the parentheses, avg_cost(original feature set on which the model to be predicted depends) is the average time generated when at least one original feature set on which the model to be predicted ultimately depends is called simultaneously, MAX(avg_cost(original feature set on which the model to be predicted depends)) is the average time generated when at least one original feature set on which the model to be predicted ultimately depends is called simultaneously, avg_cost(derived feature set on which the model to be predicted depends) is the maximum of the average time generated when at least one derived feature set on which the model to be predicted depends is called simultaneously, and MAX(avg_cost(derived feature set on which the model to be predicted depends)) is the maximum of the average time generated when at least one derived feature set on which the model to be predicted depends is called simultaneously.

[0067] The key predicted values ​​for average time consumption in this step include MAX(avg_cost(the original feature set on which the model to be predicted depends)) and MAX(avg_cost(the derived feature set on which the model to be predicted depends)); OneOf() represents the average time consumption generated by the algorithm of the model to be predicted itself, and the specific constant value is selected according to the different algorithms used by the model to be predicted; api_cost represents the average time consumption generated by the API interface calling the model to be predicted with the original feature set and the derived feature set as parameters.

[0068] This step substitutes the key predicted values ​​of resource consumption into the overall resource consumption prediction formula of the model to be predicted, thereby determining the overall predicted value of resource consumption for the model. The overall resource consumption prediction process for the model to be predicted is as follows:

[0069] cpu_predict_model = SUM(

[0070] sum_cpu_fc_Y,

[0071] sum_cpu_sfc_Y,

[0072] OneOf(lr_cpu, rf_cpu, dt_cpu),

[0073] api_cpu

[0074] ); where cpu_predict_model represents the overall predicted value of resource consumption of the model to be predicted, sum_cpu_fc_Y represents the output of the first prediction model, sum_cpu_sfc_Y represents the output of the second prediction model, the output of the first prediction model and the output of the second prediction model constitute the key predicted value of resource consumption, OneOf(lr_cpu,rf_cpu,dt_cpu) represents the CPU consumption generated by the algorithm selected by the model to be predicted, and api_cpu represents the CPU consumption generated by the API interface when it calls the model to be predicted with the original feature set and the derived feature set as parameters.

[0075] This step substitutes the key predicted value of average time consumption into the overall prediction formula for the average time consumption of the model to be predicted, thereby determining the key predicted value of the average time consumption of the model to be predicted. The overall prediction process for the average time consumption of the model to be predicted is as follows:

[0076] avg_cost_predict_model = SUM(

[0077] max_avg_cost_fc_Y,

[0078] max_avg_cost_sfc_Y,

[0079] OneOf(lr_cost,rf_cost,dt_cost),

[0080] api_cost

[0081] ).

[0082] Wherein, avg_cost_predict_model represents the overall predicted value of the average time of the model to be predicted, max_avg_cost_fc_Y represents the output of the third prediction model, max_avg_cost_sfc_Y represents the output of the fourth prediction model, max_avg_cost_fc_Y and max_avg_cost_sfc_Y constitute the key predicted value of the average time, and api_cost represents the average time generated when the API interface calls the model to be predicted with the original feature set and the derived feature set as parameters.

[0083] Therefore, this step sums the key predicted value of resource consumption, the resource consumption determined by its own algorithm, and the resource consumption of the API entry point to determine the overall predicted value of resource consumption of the model to be predicted; and sums the key predicted value of average time consumption, the average time consumption determined by its own algorithm, and the average time consumption of the API entry point to determine the overall predicted value of average time consumption of the model to be predicted.

[0084] Furthermore, after determining the overall predicted value of resource consumption and the overall predicted value of average time consumption for the model to be predicted, this method may also include inputting the overall predicted value of resource consumption and the overall predicted value of average time consumption to the resource scheduler. The scheduler combines the real-time resource level of the current server cluster, the resource occupancy baseline of the deployed models, and the priority of the business to which each model belongs to, and performs multi-model packaging, resource quota setting, and elastic scaling decisions. This is to schedule models with complementary resource consumption characteristics to the same node to reuse hot resources such as CPU and GPU during off-peak hours, and / or dynamically set the resource limit of the container (such as computing power share, memory limit) according to the prediction results, so as to ensure that the average time consumption of each model under resource competition conditions can still meet the service level agreement (SLA) requirements. At the same time, it automatically triggers horizontal expansion or vertical upgrade for models whose predicted time consumption exceeds the threshold, thereby maximizing the overall resource utilization of the cluster while ensuring service quality.

[0085] This manual predicts the resource consumption and average time consumption of each model to be predicted after going online under resource contention, and determines how to better allocate server resources to each model to be predicted. This optimizes the model performance and model effect of each model, ensuring that each model meets the resource budget and service throughput expectations, and achieves a balance between model effect and model performance. This greatly reduces the workload of each stage of model production and significantly improves delivery efficiency.

[0086] Figure 2a shows a flowchart of a method for determining the final resource consumption forecast and the final average time forecast according to an embodiment of this specification. In this specification, the accuracy of the calculated overall resource consumption forecast and overall average time forecast of the model to be predicted still needs to be determined. Therefore, it is necessary to combine the theoretical values ​​of resource consumption forecast and average time forecast when there is no resource competition for judgment. The method in Figure 2a includes at least the following steps 201-207.

[0087] Step 201: Based on the total resource consumption when a single feature set in at least one original feature set on which the model to be predicted is dependent is invoked, the total resource consumption when a single derived feature set in at least one derived feature set on which the model to be predicted is dependent is invoked, the maximum average time when a single feature set in at least one original feature set on which the model to be predicted is dependent is invoked, and the maximum average time when a single derived feature set in at least one derived feature set on which the model to be predicted is dependent is invoked, determine the theoretical value of the predicted resource consumption and the theoretical value of the predicted average time of the model to be predicted.

[0088] In the embodiments of this specification, without considering resource contention, the CPU resource consumption and average time consumed when each feature set or derived sample feature set in all original sample feature sets and derived sample feature sets that can be used to construct the prediction model is called individually using monitoring tools in advance, and the data is pre-stored in the system, as shown in Example Tables 1 and 2, where Table 1 is the performance table of the original sample feature set and Table 2 is the performance table of the secondary derived sample feature set. Subsequently, analysts select a subset of original and derived sample feature sets from all the original and derived sample feature sets recorded in the table to construct the model to be predicted. After determining the dependent features of the model to be predicted, the system calculates the sum of the resource consumption (denoted as sum_cpu_fc in this specification) and the maximum average time of each call to all original feature sets that the model to be predicted depends on, as well as the sum of the resource consumption and the maximum average time of each call to all derived feature sets that the model to be predicted ultimately depends on. These data are then input into the overall resource consumption prediction model and the overall average time prediction model to obtain the theoretical values ​​of resource consumption prediction and average time prediction, denoted as cpu_predict_fallback and avg_cost_predict_fallback, respectively. In this specification, different models to be predicted correspond to different theoretical values ​​of resource consumption prediction and average time prediction. Therefore, the theoretical values ​​of resource consumption prediction and average time prediction in this specification can be understood as dynamically changing, not constant values.

[0089] Table 1

[0090] Table 2

[0091] In this specification, the data monitored by the monitoring tool represents data from individual feature set calls (e.g., when each feature set is deployed on different servers). There is no resource contention between individual feature sets. However, in actual model calls, multiple original and derived feature sets inevitably reside on the same server. Therefore, resource contention is unavoidable during actual model calls. In other words, considering resource contention, the resource consumption of simultaneously calling each feature set will be higher than that of calling a single feature set due to resource contention. For example, when multiple feature sets deployed on the same set of servers are called simultaneously, they compete for the CPU resources of that set, increasing the call time for each feature set and thus increasing CPU resource consumption. Therefore, without considering resource contention, the CPU resource consumption and average time consumed when each feature set or derived sample feature set in all the original sample feature sets and derived sample feature sets used to construct the model to be predicted is individually called, as the theoretical value of resource consumption prediction and the theoretical value of average time prediction. The theoretical value of resource consumption prediction and the theoretical value of average time prediction can be used as the theoretical value of resource consumption prediction and the theoretical value of average time prediction (the theoretical value can also be the lower limit value) to determine whether the overall predicted value of resource consumption and the overall predicted value of average time of the model to be predicted are within the normal range, thereby determining whether the performance prediction result of the model to be predicted is within the normal range.

[0092] As shown below, the parameters on the right side of the equal signs in the four equations are all data obtained by the monitoring tool, and the parameters on the right side of the equal signs are the parameters that will be input into the formula for calculating the theoretical value.

[0093] SUM(cpu(the original feature set on which the model to be predicted depends)) = sum_cpu_fc

[0094] SUM(cpu(derived feature set on which the model to be predicted depends)) = sum_cpu_sfc

[0095] MAX(avg_cost(the original feature set on which the model to be predicted depends)) = max_avg_cost_fc

[0096] MAX(avg_cost(derived feature set on which the model to be predicted depends)) = max_avg_cost_sfc

[0097] cpu_predict_fallback = SUM(

[0098] sum_cpu_fc,

[0099] sum_cpu_sfc,

[0100] OneOf(lr_cpu, rf_cpu, dt_cpu),

[0101] api_cpu

[0102] );

[0103] Wherein, cpu_predict_fallback represents the theoretical value of the resource consumption prediction of the model to be predicted, sum_cpu_fc represents the total CPU consumption when a single original feature set in the original feature set on which the model to be predicted is called without resource contention, sum_cpu_sfc represents the total CPU consumption when a single derived feature set in the derived feature set on which the model to be predicted is called without resource contention, OneOf(lr_cpu,rf_cpu,dt_cpu) represents the total CPU consumption of the algorithm of the model to be predicted itself without resource contention, and api_cpu represents the total CPU consumption of the API interface without resource contention.

[0104] avg_cost_predict_fallback=SUM(

[0105] max_avg_cost_fc,

[0106] max_avg_cost_sfc,

[0107] OneOf(lr_cost,rf_cost,dt_cost),

[0108] api_cost

[0109] ).

[0110] Here, `avg_cost_predict_fallback` represents the theoretical value of the average time prediction for the model to be predicted. The theoretical values ​​of resource consumption prediction and average time prediction are used to evaluate whether the performance prediction results of the model to be predicted are within the normal range. `max_avg_cost_fc` represents the maximum average time when a single original feature set in the original feature set on which the model to be predicted is called is called when there is no resource contention. `max_avg_cost_sfc` represents the maximum average time when a single derived feature set in the derived feature set on which the model to be predicted is called is called when there is no resource contention. `OneOf(lr_cost,rf_cost,dt_cost)` represents the average time of the algorithm of the model to be predicted itself when there is no resource contention. `api_cost` represents the average time of the API interface when there is no resource contention.

[0111] Step 202: Based on the comparison results of the overall predicted value of resource consumption and the theoretical value of resource consumption, as well as the comparison results of the overall predicted value of average time consumption and the theoretical value of average time consumption, determine the final predicted value of resource consumption and the final predicted value of average time consumption for the model to be predicted.

[0112] The process of determining the final resource consumption forecast and the final average time consumption forecast of the model to be predicted is shown in Figure 2b. Figure 2b is a flowchart of a method for determining the final resource consumption forecast and the final average time consumption forecast according to an embodiment of this specification, which includes at least steps 2021 to 2023.

[0113] Step 2021: Determine whether the overall predicted value of resource consumption of the model to be predicted is greater than the theoretical value of resource consumption prediction, and whether the overall predicted value of average time consumption of the model to be predicted is greater than the theoretical value of average time consumption prediction.

[0114] As mentioned above, in practice, considering resource contention, the resource consumption of each feature set will be higher than when calling a single feature set due to resource contention. Therefore, in this step, the overall predicted value of resource consumption of the model to be predicted, calculated in step 103, is compared with the theoretical value of resource consumption prediction, i.e., the magnitudes of cpu_predict_model and cpu_predict_fallback are determined. The overall predicted value of average time consumption of the model to be predicted, calculated in step 103, is also compared with the theoretical value of average time consumption prediction, i.e., the magnitudes of avg_cost_predict_model and avg_cost_predict_fallback are determined.

[0115] Step 2022: If not, determine the theoretical value of resource consumption forecast as the final resource consumption forecast, and determine the theoretical value of average time consumption forecast as the final average time consumption forecast.

[0116] Step 2023: If yes, determine the overall predicted value of resource consumption as the final predicted value of resource consumption, and determine the overall predicted value of the average time consumption of the model to be predicted as the final predicted value of average time consumption.

[0117] In this specification, if `cpu_predict_model` is greater than `cpu_predict_fallback`, then `cpu_predict_model` is the final predicted CPU resource consumption, i.e., the final resource consumption prediction, indicating that the performance prediction model's prediction result conforms to common sense regarding resource contention. Otherwise, the theoretical value of `cpu_predict_fallback` is used as the final resource consumption prediction.

[0118] In this specification, if avg_cost_predict_model is greater than avg_cost_predict_fallback, then avg_cost_predict_model is the final average time prediction.

[0119] Step 203: If the final resource consumption prediction meets the first preset condition and the final average time prediction meets the second preset condition, then the performance prediction of the model to be predicted ends.

[0120] The first preset condition includes that the final predicted resource consumption does not exceed the preset resource consumption threshold; the second preset condition includes that the final predicted average time consumption does not exceed the preset average time consumption threshold.

[0121] In this step, the final predicted resource consumption is compared with the system's preset resource consumption threshold to determine if it meets the first preset condition. The final predicted average time consumption is then compared with the system's preset resource consumption threshold to determine if it meets the second preset condition. If both conditions are met, it indicates that the model under test performs well in terms of resource consumption and average time consumption, and no adjustment to the model's feature set is needed, thus completing the performance prediction of the model. Analysts can then use the current model to conduct subsequent work. If the conditions are not met, it indicates that the model's performance in terms of resource consumption and average time consumption has not met expectations, and further adjustments to the original or derived feature sets on which the model relies are needed for re-prediction until the prediction results meet the performance standards.

[0122] In this manual, the preset resource consumption threshold can be set to 1000 millicores, and the preset average latency threshold can be set to 200 milliseconds. Both thresholds can be constant standards. Of course, the thresholds can be adapted to different systems or scenarios, and this manual does not impose any restrictions.

[0123] In this specification, if the final predicted resource consumption and the final predicted average execution time of the model meet the system's preset standards, it indicates that the model's performance and efficiency meet the expected service throughput capacity. The model can then proceed with subsequent work after performance prediction is complete. Otherwise, it is necessary to adjust the original feature set list and the derived feature set list of the model, reselect a suitable original feature set or derived feature set, repeat the steps described above, and re-predict the model until the prediction results meet the performance standards. This specification, by predicting the resource overhead and performance of the model before its development and after its deployment, helps in accurate resource budgeting and service throughput capacity expectations, reduces the burden on analysts, and improves model optimization efficiency.

[0124] In this manual, the system can also use the model to predict and determine the newly added model that finally meets the standard for online deployment after the model to be predicted is launched. The newly added model can be called the incremental model. The system can also periodically capture the actual data of the incremental model performance test, evaluate the effect of the four prediction models, and add the actual data of the incremental model to the training sample set of the prediction model to continuously optimize the prediction model.

[0125] Figure 3 is a flowchart of a performance prediction model construction method according to an embodiment of this specification, wherein the method includes at least steps 301-303. The method described in Figure 3 is executed by a performance prediction model construction device in the system.

[0126] Step 301: parse the sample feature set dependency relationship of the sample model, and initiate a call to the sample model according to the sample feature set dependency relationship. The sample feature set dependency relationship includes at least the original sample feature set and the derived sample feature set.

[0127] In the embodiments of this specification, the sample model is a pre-trained computational model. By parsing the configuration of the sample model, the dependency relationships of the sample feature sets can be determined. The original sample feature set and the derived sample feature set in this step are both sample feature datasets. The dependency relationships of the sample feature sets include: the original sample feature set and the derived sample feature set, which are the feature data used by the sample model during training. The feature parameters of the sample model come from the original sample feature set and the derived sample feature set. The sample feature set dependency relationship records the dependency relationship between the sample model and the original sample feature set, the dependency relationship between the sample model and the derived sample feature set, and the dependency relationship between the original sample feature set and the derived sample feature set.

[0128] This process involves using a pre-developed derivation algorithm to process a series of raw data that changes over time in real time, resulting in a set of raw features. These raw features describe certain attributes or characteristics of the data, and the set of these features is called the raw feature set. Then, the pre-developed derivation algorithm is used to process several raw feature sets in real time, resulting in a series of new features. This new feature set is called the secondary derivation feature set. The derivation sample feature set can include multiple derivation sample feature sets, such as secondary derivation sample feature sets, tertiary derivation sample feature sets, etc. This specification uses a secondary derivation sample feature set as an example to represent the derivation sample feature set.

[0129] In machine learning models, the selection and retrieval of feature data are crucial, directly impacting model performance and accuracy. In this embodiment, we assume three sample models: "Model 1," "Model 2," and "Model 3." The customer's requirement is to access "Model 2." First, by analyzing the feature set dependencies of Model 2, we can determine that "Model 2" depends on the original feature set 1, the original feature set N, the secondary derived feature set 2, and the secondary derived feature set N. Further, based on these dependencies, we can determine that the secondary derived feature set 2 depends on the original feature set 1 and the original feature set N, and the secondary derived feature set N depends on the original feature set 1 and the original feature set 2. Therefore, "Model 2" ultimately depends on the original feature set 1, the original feature set 2, the original feature set N, the secondary derived feature set 2, and the secondary derived feature set N. Figure 10 illustrates the order in which the service provider calls the dependent feature sets of Model 2 when the customer calls the API to obtain the model prediction results.

[0130] Step 302: By calling the original sample feature set and the derived sample feature set, determine the key values ​​of sample resource consumption and the key values ​​of average sample time consumption generated during the sample model calling process.

[0131] In the embodiments of this specification, initiating a call to the sample model is essentially initiating a call to the feature data of the sample model, specifically: initiating a call to the original sample feature set and the derived sample feature set of the sample model. Based on the embodiment in step 101, the process of calling "Model 2" actually includes the process of calling the original sample feature set 1, the original sample feature set 2, the original sample feature set N, the secondary derived sample feature set 2, and the secondary derived sample feature set N.

[0132] In this specification, calling the sample model includes multiple steps, including at least: calling the original sample feature set; calling the secondary derived sample feature set based on the result of calling the original sample feature set; and calling the model based on the result of calling the secondary derived sample feature set and the result of calling the original sample feature set.

[0133] Correspondingly, the resource consumption of "Model 2" is the sum of the CPU consumption of its final dependent original sample feature set being called simultaneously, the secondary derived sample feature set being called simultaneously, its own algorithm, and the API entry point; the average time of "Model 2" is the sum of the maximum average time of the simultaneous calling of its final dependent original sample feature set, the maximum average time of the simultaneous calling of the secondary derived sample feature set, the average time of its own algorithm, and the average time of the API entry point. Among these, the maximum value of the resource consumption and average time of the simultaneous calling of the final dependent original sample feature set and the secondary derived sample feature set is difficult to predict. Therefore, it is necessary to statistically determine the key values ​​of sample resource consumption and average time corresponding to these two key steps for subsequent construction of the performance prediction model.

[0134] Step 303: Construct a training sample set using key values ​​of sample resource consumption, key values ​​of average sample time consumption, and multiple influencing factors. Use the training sample set to train multiple initial prediction models to obtain a performance prediction model.

[0135] The initial prediction models include: an initial prediction model for predicting resource consumption and an initial prediction model for predicting average execution time. In this specification, the purpose of building performance prediction models is to predict the resource consumption and average execution time required to invoke a model before it goes live, thereby determining the model's performance. This allows analysts to initially select model features that will enable the model to meet performance standards, and then focus on improving model performance using these selected features. This further reduces the workload of analysts, avoids rework and re-prediction due to substandard performance, improves model delivery efficiency, and achieves a balance between model performance and model effectiveness.

[0136] After determining the key values ​​of sample resource consumption and average time consumption for critical steps in the sample model invocation process in step 102, these key values, along with multiple influencing factors of the sample model, constitute a training sample set. This training sample set is then used to train multiple initial prediction models, thereby constructing a performance prediction model. Once the prediction model is deployed in production, the feature set and performance data upon which it depends can be further used to train the performance prediction model.

[0137] Referring to Figure 10, the following provides a specific flow of the dependent feature set invocation order of an embodiment of this specification. Taking the embodiment of "Model 2" in step 301 as an example, after the client initiates a request to invoke Model 2, it includes multiple invocation sub-steps, as follows:

[0138] Sub-step 1: The client initiates a call to "Model 2";

[0139] Sub-step 2: After the service provider's API entry parses the dependency of "Model 2", it simultaneously initiates calls to the original sample feature set 1, the original sample feature set 2, and the original sample feature set N.

[0140] Sub-step 3: After obtaining the call results of the original sample feature set 1, the original sample feature set 2, and the original sample feature set N from the service provider's API entry, the service provider simultaneously initiates calls to the secondary derived sample feature set 2 and the secondary derived sample feature set N.

[0141] Sub-step 4: The service provider's API entry uses the obtained results of the original sample feature set 1, the original sample feature set N, the secondary derived sample feature set 2, and the secondary derived sample feature set N as parameters to initiate a call to model 2.

[0142] Sub-step 5: The service provider will return the results of Model 2 to the customer.

[0143] As described above, for various types of models, the maximum resource consumption and average time consumption of sub-steps 2 and 3 involved in calling the model are difficult to predict. Therefore, this specification needs to statistically analyze the resource consumption and average time consumption of sub-step 2 and sub-step 3 during the sample model call process to construct a predictive model and provide predictive services for more types of models to be predicted.

[0144] In another embodiment, the process of calling the model can be implemented through the embodiment shown in FIG4, which includes at least the following steps 401-402.

[0145] Step 401: Initiate a call to the original sample feature set to obtain the first call result. This step corresponds to sub-step 2. After the service provider's API entry parses the dependency of "Model 2", it simultaneously initiates calls to original sample feature set 1, original sample feature set 2, and original sample feature set N. In this step, some input parameters can be used to obtain raw data, and the raw data can be analyzed and processed to obtain the original sample feature set. For example, if "Model 2" is a credit-related model, inputting user information (such as account, contact information, etc.) will retrieve the corresponding user's raw transaction data, operation records, etc. Further analysis and calculation of the raw transaction data, operation records, etc., can yield the user's user information features (such as operation frequency, consumption habits, etc.), which belong to the original sample feature set in this specification. The call to the original sample feature set, and the resulting call result, is referred to as the first call result.

[0146] Step 402: Based on the first call result, initiate a call to the derived sample feature set to obtain the second call result. This step corresponds to sub-step 3, where a call is simultaneously initiated to the secondary derived sample feature set based on the first call result. The resulting second call result is the result after analyzing the secondary derived feature set. For example, the first call result includes: the number of transactions a user makes every 7 days and the number of transactions a user makes every month, thus allowing for the calculation of trends and coefficients of variation in the number of transactions a user makes every 7 days and every month within a time window.

[0147] In this embodiment of the specification, the model invocation process further includes: initiating an invocation of the sample model based on the first invocation result and the second invocation result, and returning the final invocation result. In this step, the results obtained from the invocations of the original sample feature set and the derived sample feature set in the preceding steps are combined to initiate an invocation of the model. For example, the results obtained from the invocations of the original sample feature set and the derived sample feature set can be used as parameters to initiate an invocation of the model, resulting in the final invocation result. In this embodiment of the specification, the original sample feature set and the secondary derived sample feature set have a significant impact on the accuracy of the prediction of the subsequent prediction model to be constructed; therefore, it is necessary to record the resource consumption and average time consumed during the invocation process using monitoring tools.

[0148] Figure 5 shows a flowchart of a method for determining key values ​​of sample resource consumption and key values ​​of average sample time generated during the sample model call process according to an embodiment of this specification, specifically including the following steps 501-503.

[0149] According to the embodiment in Figure 4, the resource consumption of sample model 2 is the sum of the CPU consumption of the original sample feature set on which sample model 2 depends being called simultaneously, the secondary derived sample feature set on which sample model 2 depends being called simultaneously, the algorithm of sample model 2 itself, and the API entry point for calling sample model 2. The average time consumption is the sum of the maximum average time consumption of the original sample feature set on which sample model 2 depends being called simultaneously, the maximum average time consumption of the secondary derived sample feature set on which sample model 2 depends being called simultaneously, the average time consumption of the algorithm of sample model 2 itself, and the average time consumption of the API entry point for calling sample model 2. The resource consumption and average time consumption generated by sub-step 4 in calling sample model 2 can be classified as constants in common machine learning algorithms. The resource consumption and average time consumption generated by sub-step 5 in calling sample model 2 can also be summarized as constant resource consumption and time consumption in actual production processes. Furthermore, the inventors have found that the proportion of sub-steps 4 and 5 in the resource consumption and average time consumption of the prediction target is much smaller than that of the original sample feature set and the secondary derived sample feature set, and has minimal impact on the accuracy of the prediction. Therefore, the embodiments of this specification only need to statistically analyze the resource consumption and average time consumption corresponding to sub-step 2 and sub-step 3 during the sample model call process, and refer to the statistical results as the key values ​​of sample resource consumption and the key values ​​of sample average time consumption.

[0150] Step 501: Initiate a call to the original sample feature set to obtain the maximum value of the first resource consumption and the first average time consumption of the first call result.

[0151] In this specification, the total resources consumed when simultaneously calling each original sample feature set on which the sample model depends are used, through monitoring tools, are defined as the first resource consumption. On the other hand, the average time consumed by each original sample feature set when simultaneously calling them is also calculated. In some embodiments, the maximum average time is selected from multiple average times as the maximum value of the first average time.

[0152] Step 502: Initiate a call to the derived sample feature set to obtain the maximum value of the second resource consumption and the second average time consumption of the second call result. In this specification, the total resources consumed when simultaneously calling each derived sample feature set on which the sample model depends are used by a monitoring tool, and this is taken as the second resource consumption. On the other hand, the average time consumption of each derived sample feature set when simultaneously calling each derived sample feature set is calculated. In some embodiments, the maximum average time consumption is selected from multiple average times consumption as the maximum value of the second average time consumption. In this specification, the first resource consumption and the second resource consumption include CPU resources consumed, such as CPU time.

[0153] Step 503: Based on the first resource consumption and the second resource consumption, determine the key value of sample resource consumption; and based on the maximum value of the first average time consumption and the maximum value of the second average time consumption, determine the key value of sample average time consumption. In this step, the key value of sample resource consumption is the resource consumption corresponding to sub-steps 2 and 3 in the embodiment of Figure 4. The key value of sample average time consumption in this step is the average time corresponding to sub-steps 2 and 3 in the embodiment of Figure 4. These two types of data are statistically obtained as known parameters and used as the training sample set for training the initial prediction model. This guides the iterative update of the initial prediction model, ultimately constructing a performance prediction model. This forms an effective tool for predicting the resource overhead of the model before its deployment, achieving a balance between model effectiveness and model performance.

[0154] Figure 6 shows a flowchart of a method for constructing a performance prediction model according to an embodiment of this specification, which includes at least the following steps 601-604.

[0155] Step 601: Calculate multiple influencing factors based on the original sample feature set and the derived sample feature set.

[0156] In the embodiments of this specification, in a real production environment, since the calculation models for calling the original sample feature set and the secondary derived sample feature set are deployed in the same group of servers, the four indicators corresponding to the key values ​​of sample resource consumption and the key values ​​of sample average time consumption will be affected by resource competition. Therefore, it is necessary to determine the factors that affect resource competition when multiple feature sets are called at the same time, which are the influencing factors.

[0157] In this specification, the factors influencing resource competition for each indicator include, but are not limited to: the number of original sample feature sets on which the sample model depends (denoted as num_fc), the total CPU consumption of a single call to the original sample feature sets on which the model depends (denoted as sum_cpu_fc), the coefficient of variation of CPU consumption of a single call to the original sample feature sets on which the sample model depends (denoted as cv_cpu_fc), the range of CPU consumption of a single call to the original sample feature sets on which the sample model depends (denoted as range_cpu_fc), the maximum average time of a single call to the original sample feature sets on which the sample model depends (denoted as max_avg_cost_fc), the coefficient of variation of the average time of a single call to the original sample feature sets on which the sample model depends (denoted as cv_avg_cost_fc), the range of the average time of a single call to the original sample feature sets on which the sample model depends (denoted as range_avg_cost_fc), the number of secondary derived sample feature sets on which the sample model depends (denoted as num_sfc), and the number of sample model dependencies. The total CPU consumption of a single call to the secondary derived sample feature set of Lai (denoted as sum_cpu_sfc), the coefficient of variation of CPU consumption of a single call to the secondary derived sample feature set on which the sample model depends (denoted as cv_cpu_sfc), the range of CPU consumption of a single call to the secondary derived sample feature set on which the sample model depends (denoted as range_cpu_sfc), the maximum average time of a single call to the secondary derived sample feature set on which the sample model depends (denoted as max_avg_cost_sfc), the coefficient of variation of the average time of a single call to the secondary derived sample feature set on which the sample model depends (denoted as cv_avg_cost_sfc), the range of the average time of a single call to the secondary derived sample feature set on which the sample model depends (denoted as range_avg_cost_sfc), the total CPU resources of the relevant production server (denoted as sum_cpu_total), and the concurrency corresponding to the performance indicators (the number of requests that can be initiated by clients at the same time) (denoted as num_concurrency).

[0158] In the embodiments of this specification, in order to determine the sample resource consumption and average time consumption of the sample model Score1 during the call process, the data in Tables 1 and 2 below (which have already appeared in step 201) are obtained through performance testing when the model is deployed. Table 1 is the performance table of the original sample feature set, and Table 2 is the performance table of the secondary derived sample feature set. In Tables 1 and 2, the CPU corresponding to the feature set refers to CPU resource consumption, and average cost refers to the average time consumption. These parameters are obtained through monitoring tools and represent the CPU consumption and average time consumption of a single call to the original sample feature set, the average time consumption of a single call to the original sample feature set, and the CPU consumption and average time consumption of a single call to the secondary derived sample feature set.

[0159] Table 1

[0160] Table 2

[0161] Taking the sample model Score1 as an example, analyzing the dependency relationships of the sample feature sets of sample model Score1 reveals that the sample feature sets that sample model Score1 depends on include: original sample feature set 1, original sample feature set 5, original sample feature set 13, original sample feature set 22, secondary derived sample feature set 2, and secondary derived sample feature set 12. Further, a list of secondary derived sample feature sets that sample model Score1 depends on is shown in Table 3 below.

[0162] Table 3

[0163] As can be seen, the union of the original sample feature sets id that the two secondary derived sample feature sets (secondary derived sample feature set 2 and secondary derived sample feature set 12) depend on is {5,6,7,9,22}. The union of this union with the list of original sample feature sets {1,5,13,22} that the sample model Score1 depends on is {1,5,6,7,9,13,22}. That is, the final list of original sample feature sets that the sample model Score1 depends on is shown in Table 4 below. Table 4 is the final list of original sample feature sets that it depends on.

[0164] Table 4

[0165] In this step, based on the data in Tables 3 and 4, the specific values ​​of the 16 factors affecting resource competition in the sample model Score1 are calculated and used as feature data for training the prediction model. See below:

[0166] num_fc: 7 indicates that the sample model Score1 depends on 7 original sample feature sets;

[0167] sum_cpu_fc: 22+16+32+28+17+38+25=178, which represents the total CPU consumption of all original sample feature sets that the sample model Score1 depends on when a single call is made;

[0168] cv_cpu_fc: Standard deviation / mean (178 / 7≈25.43), representing the coefficient of variation of CPU consumption per call to the original sample feature set on which the sample model Score1 depends;

[0169] It is denoted as cv(22,16,32,28,17,38,25)≈0.312845.

[0170] range_cpu_fc: 38-16=22, which represents the range of CPU consumption when the original sample feature set on which the sample model Score1 depends is called.

[0171] max_avg_cost_fc: 45, represents the maximum average time taken per call to the original sample feature set that the sample model Score1 depends on;

[0172] cv_avg_cost_fc: cv(26,18,35,34,20,45,28)≈0.318679, representing the coefficient of variation of the average time taken for a single call to the secondary derived sample feature set on which the sample model Score1 depends.

[0173] range_avg_cost_fc: 45-18=27, representing the range of average time taken when a single call is made to the original sample feature set on which the sample model depends;

[0174] num_sfc: 2, indicating that the number of secondary derived sample feature sets on which the sample model depends is 2;

[0175] sum_cpu_sfc: 7+23=30, representing the total CPU consumption of a single call to the secondary derived sample feature set on which the sample model depends;

[0176] cv_cpu_sfc: cv(7,23)≈0.754247, representing the coefficient of variation of CPU consumption when a single call is made to the secondary derived sample feature set on which the sample model depends;

[0177] range_cpu_sfc: 23-7=16, which represents the range of CPU consumption when a single call is made to the secondary derived sample feature set on which the sample model depends;

[0178] max_avg_cost_sfc: 25, represents the maximum average time taken per call to the secondary derived sample feature set that the sample model depends on;

[0179] cv_avg_cost_sfc: cv(7,25)≈0.795495, representing the coefficient of variation of the average time taken for a single call to the secondary derived sample feature set on which the sample model depends;

[0180] range_avg_cost_sfc: 25-7=18, representing the range of average time taken when a single call is made to the secondary derived sample feature set on which the sample model depends.

[0181] During performance testing of the sample model Score1, the remaining two influencing factors were recorded:

[0182] sum_cpu_total: 1280000, the total CPU resources of the production server.

[0183] And num_concurrency: 50, the number of concurrent connections corresponding to the performance metric.

[0184] Step 602: Input multiple influencing factors into multiple initial prediction models to obtain the output results of each initial prediction model. In the embodiments of this specification, the initial prediction models can be selected from linear regression models and multinomial regression models. Each initial prediction model uses the influencing factors as sample feature data, Y1, Y2, Y3, and Y4 as labels, and selects a regression algorithm (e.g., multinomial regression algorithm) for training.

[0185] In this step, the 16 influencing factors of the sample model Score1 are input into the four initial prediction models, and the four initial prediction models are trained using the multinomial regression algorithm. Substituting the influencing factors further yields the output results of the four initial prediction models.

[0186] The output of the first initial prediction model is shown below:

[0187] sum_cpu_fc_Y=model1(num_fc,sum_cpu_fc,cv_cpu_fc,range_cpu_fc,

[0188] max_avg_cost_fc,cv_avg_cost_fc,range_avg_cost_fc,num_sfc,

[0189] sum_cpu_sfc,cv_cpu_sfc,range_cpu_sfc,max_avg_cost_sfc,

[0190] cv_avg_cost_sfc,range_avg_cost_sfc,sum_cpu_total,

[0191] num_concurrency

[0192] )

[0193] The output of the second initial prediction model is shown below:

[0194] sum_cpu_sfc_Y=model2(num_fc,sum_cpu_fc,cv_cpu_fc,range_cpu_fc,

[0195] max_avg_cost_fc,cv_avg_cost_fc,range_avg_cost_fc,num_sfc,

[0196] sum_cpu_sfc,cv_cpu_sfc,range_cpu_sfc,max_avg_cost_sfc,

[0197] cv_avg_cost_sfc,range_avg_cost_sfc,sum_cpu_total,

[0198] num_concurrency

[0199] )

[0200] The output of the third initial prediction model is shown below:

[0201] max_avg_cost_fc_Y=model3(num_fc,sum_cpu_fc,cv_cpu_fc,range_cpu_fc,

[0202] max_avg_cost_fc,cv_avg_cost_fc,range_avg_cost_fc,num_sfc,

[0203] sum_cpu_sfc,cv_cpu_sfc,range_cpu_sfc,max_avg_cost_sfc,

[0204] cv_avg_cost_sfc,range_avg_cost_sfc,sum_cpu_total,

[0205] num_concurrency

[0206] )

[0207] The output of the fourth initial prediction model is shown below:

[0208] max_avg_cost_sfc_Y=model4(num_fc,sum_cpu_fc,cv_cpu_fc,range_cpu_fc,

[0209] max_avg_cost_fc,cv_avg_cost_fc,range_avg_cost_fc,num_sfc,

[0210] sum_cpu_sfc,cv_cpu_sfc,range_cpu_sfc,max_avg_cost_sfc,

[0211] cv_avg_cost_sfc,range_avg_cost_sfc,sum_cpu_total,

[0212] num_concurrency

[0213] )

[0214] Step 603: Calculate the loss value based on the label and output of each initial prediction model. Iterate the model parameters of each initial prediction model using the loss value until the preset conditions are met to obtain the performance prediction model. The maximum values ​​of the first resource consumption, the second resource consumption, the first average time consumption, and the second average time consumption are respectively used as the labels of each initial prediction model.

[0215] In this step, the process of constructing each prediction model includes: training a first prediction model based on the output of the first initial prediction model and the first resource consumption; training a second prediction model based on the output of the second initial prediction model and the second resource consumption; training a third prediction model based on the output of the third initial prediction model and the maximum value of the first average time consumption; and training a fourth prediction model based on the output of the fourth initial prediction model and the maximum value of the second average time consumption. Specifically, the output of the first initial prediction model is sum_cpu_fc_Y. A loss function is calculated based on the output of the first initial prediction model and the first resource consumption label Y1. The model parameters of the first initial prediction model are iterated using the value of the loss function corresponding to the current training until the value of the loss function converges to a preset threshold, thus constructing the first prediction model. The output of the second initial prediction model is sum_cpu_sfc_Y. A loss function is calculated based on the output of the second initial prediction model and the second resource consumption label Y2. The model parameters of the second initial prediction model are iterated using the value of the loss function corresponding to the current training until the value of the loss function converges to a preset threshold, thus constructing the second prediction model. Similarly, the output of the third initial prediction model is max_avg_cost_fc_Y. Based on the output of the third initial prediction model and the maximum value label Y3 of the first average time consumption, the third prediction model is constructed. The output of the fourth initial prediction model is max_avg_cost_sfc_Y. Based on the output of the fourth initial prediction model and the maximum value label Y3 of the second average time consumption, the fourth prediction model is constructed.

[0216] In this specification, the first resource consumption is the metric obtained by monitoring the performance of the sample model in step 501: SUM(cpu(original sample feature set on which the model depends)), the second resource consumption is the metric: SUM(cpu(derived sample feature set on which the model depends)), the maximum value of the first average time consumption is: MAX(avg_cost(original sample feature set on which the model depends)), and the maximum value of the second average time consumption is: MAX(avg_cost(derived sample feature set on which the model depends)). These four metrics are used as the labels Y1, Y2, Y3, and Y4 predicted by the model, respectively.

[0217] Specifically, the first resource consumption index is used as the label Y1 of the first initial prediction model, the second resource consumption index is used as the label Y2 of the second initial prediction model, the maximum value of the first average time consumption is used as the label Y3 of the third initial prediction model, and the maximum value of the second average time consumption is used as the label Y4 of the fourth initial prediction model.

[0218] In summary, after constructing the first, second, third, and fourth prediction models, the performance prediction model described in this specification is obtained. This performance prediction model serves as an effective tool for predicting production environment performance and resource costs before the model goes live. It helps analysts predict the final resource costs of the model before training it, thus achieving a balance between model effectiveness and performance before the model goes live.

[0219] Figure 7 is a schematic diagram of a model performance prediction device according to an embodiment of this specification. The figure illustrates the basic structure of the model performance prediction device. The functional units and modules can be implemented in software, or model performance prediction can be achieved using general-purpose chips or specific chips. The device specifically includes:

[0220] Influencing factor determination unit 701 is used to determine the influencing factors of the model to be predicted;

[0221] The prediction unit 702 is used to input influencing factors into the performance prediction model to obtain the key predicted values ​​of resource consumption and average time consumption required to call the model to be predicted. The performance prediction model is constructed by building a training sample set from the key predicted values ​​of sample resource consumption, the key predicted values ​​of sample average time consumption, and multiple influencing factors of the sample model, and then training multiple initial prediction models.

[0222] The prediction value determination unit 703 is used to determine the overall predicted value of resource consumption and the overall predicted value of average time consumption of the model to be predicted based on the key predicted value of resource consumption and the key predicted value of average time consumption.

[0223] This specification also discloses the structure of another model performance prediction device, which specifically includes:

[0224] The theoretical value determination unit is used to determine the theoretical value of resource consumption and the theoretical value of average time consumption of the model to be predicted based on the total resource consumption when a single feature set in at least one original feature set on which the model to be predicted is dependent is called, the total resource consumption when a single derived feature set in at least one derived feature set on which the model to be predicted is dependent is called, the maximum average time consumption when a single feature set in at least one original feature set on which the model to be predicted is dependent is called, and the maximum average time consumption when a single derived feature set in at least one derived feature set on which the model to be predicted is dependent is called.

[0225] The final value determination unit is used to determine the final resource consumption forecast and the final average time forecast of the model to be predicted based on the comparison results between the overall predicted value of resource consumption and the theoretical value of resource consumption forecast, as well as the comparison results between the overall predicted value of average time and the theoretical value of average time forecast.

[0226] The condition judgment unit is used to end the performance prediction of the model to be predicted if the final resource consumption prediction meets the first preset condition and the final average time prediction meets the second preset condition.

[0227] This specification discloses a theoretical value determination unit, which specifically includes:

[0228] The judgment module is used to determine whether the overall predicted value of resource consumption of the model to be predicted is greater than the theoretical value of resource consumption prediction, and whether the overall predicted value of average time consumption of the model to be predicted is greater than the theoretical value of average time consumption prediction.

[0229] The first determining module is used to determine, if not, the theoretical value of the resource consumption forecast as the final resource consumption forecast, and to determine the theoretical value of the average time consumption forecast as the final average time consumption forecast.

[0230] The second determining module is used to determine, if yes, the overall predicted value of resource consumption as the final predicted value of resource consumption, and to determine the overall predicted value of the average time consumption of the model to be predicted as the final predicted value of average time consumption.

[0231] Figure 8 is a schematic diagram of a performance prediction model building device according to an embodiment of this specification. The figure illustrates the basic structure of the performance prediction model building device. The functional units and modules can be implemented in software, or the performance prediction model can be built using a general-purpose chip or a specific chip. The device specifically includes:

[0232] Calling unit 801 is used to parse the sample feature set dependency relationship of the sample model and initiate a call to the sample model according to the sample feature set dependency relationship. The sample feature set dependency relationship includes at least the original sample feature set and the derived sample feature set.

[0233] The key value determination unit 802 is used to determine the key values ​​of sample resource consumption and the key values ​​of average sample time generated during the sample model calling process by calling the original sample feature set and the derived sample feature set.

[0234] Training unit 803 is used to construct a training sample set using key values ​​of sample resource consumption, key values ​​of sample average time consumption, and multiple influencing factors. Multiple initial prediction models are trained using the training sample set to obtain a performance prediction model. Among them, the multiple initial prediction models include: an initial prediction model for predicting resource consumption and an initial prediction model for predicting average time consumption.

[0235] This specification also discloses a structure for a calling unit, which specifically includes:

[0236] The first calling module is used to initiate a call to the original sample feature set and obtain the first calling result;

[0237] The second calling module is used to initiate a call to the derived sample feature set based on the first calling result, and obtain the second calling result.

[0238] This specification also discloses a structure for a key value determination unit, which specifically includes:

[0239] The first acquisition module is used to simultaneously initiate calls to the original sample feature set and obtain the maximum value of the first resource consumption and the first average time consumption of the first call result.

[0240] The second acquisition module is used to simultaneously initiate calls to the derived sample feature set and obtain the maximum value of the second resource consumption and the second average time consumption of the second call result.

[0241] The key value determination module is used to determine the key value of sample resource consumption based on the first resource consumption and the second resource consumption, and to determine the key value of sample average time consumption based on the maximum value of the first average time consumption and the maximum value of the second average time consumption.

[0242] This specification also discloses a structure for a training unit, which specifically includes:

[0243] The influencing factor determination module is used to calculate multiple influencing factors based on the original sample feature set and the derived sample feature set;

[0244] The prediction result acquisition module is used to input multiple influencing factors into multiple initial prediction models and obtain the output results of each initial prediction model.

[0245] The iterative update module is used to calculate the loss value based on the label and output of each initial prediction model, and iteratively update the model parameters of each initial prediction model through the loss value until the preset conditions are met to obtain the performance prediction model; wherein, the maximum value of the first resource consumption, the second resource consumption, the first average time consumption, and the second average time consumption are respectively used as the label of each initial prediction model.

[0246] Figure 9 shows a schematic diagram of a model deployment architecture according to an embodiment of this specification. The model, feature set, and secondary derived feature set are all sources of resource overhead and computation time when the model cloud service provides services. A series of original data that changes over time are processed in real-time using a pre-developed derivative algorithm to obtain a series of features; this feature set is called the feature set. Several feature sets are then processed in real-time using a pre-developed derivative algorithm to obtain a series of new features; this new feature set is called the secondary derived feature set.

[0247] Figure 10 shows a schematic diagram of the calling order of the dependent feature sets of a model to be predicted according to an embodiment of this specification. As shown in Figure 10, by analyzing the dependency relationship of the sample feature sets of Model 2, it can be determined that "Model 2" depends on the original sample feature set 1, the original sample feature set N, the secondary derived sample feature set 2, and the secondary derived sample feature set N. Furthermore, it can be determined that the secondary derived sample feature set 2 depends on the original sample feature set 1 and the original sample feature set N, and the secondary derived sample feature set N depends on the original sample feature set 1 and the original sample feature set 2. Therefore, "Model 2" ultimately depends on the original sample feature set 1, the original sample feature set 2, the original sample feature set N, the secondary derived sample feature set 2, and the secondary derived sample feature set N. For a detailed description of Figure 10, please refer to Figure 4.

[0248] Figure 11 illustrates a computer device provided in an embodiment of this specification. The model performance prediction method described in this application can be applied to a computer device. The computer device 1102 may include one or more processors 1104, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The computer device 1102 may also include any memory 1106 for storing information of any kind, such as code, settings, data, etc. Non-limitingly, for example, the memory 1106 may include any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Furthermore, any memory may provide volatile or non-volatile retention of information. Furthermore, any memory may represent a fixed or removable component of the computer device 1102. In one case, when the processor 1104 executes associated instructions stored in any memory or combination of memories, the computer device 1102 may perform any operation of the associated instructions. The computer device 1102 also includes one or more drive mechanisms 1108 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0249] Computer device 1102 may also include an input / output module 1110 (I / O) for receiving various inputs (via input device 1112) and providing various outputs (via output device 1114). A specific output mechanism may include a presentation device 1116 and an associated graphical user interface (GUI) 1118. In other embodiments, the input / output module 1110 (I / O), input device 1112, and output device 1114 may be omitted, and the device may function solely as a computer device within a network. Computer device 1102 may also include one or more network interfaces 1120 for exchanging data with other devices via one or more communication links 1122. One or more communication buses 1124 couple the components described above together.

[0250] Communication link 1122 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 1122 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0251] Corresponding to the methods in Figures 1 to 6, embodiments of this specification also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the above-described methods.

[0252] This specification also provides computer-readable instructions, wherein when a processor executes the instructions, the program therein causes the processor to perform the methods shown in Figures 1 to 6.

[0253] It should be understood that in the various embodiments of this specification, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0254] It should also be understood that, in the embodiments of this specification, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this specification generally indicates that the preceding and following related objects have an "or" relationship.

[0255] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this specification.

[0256] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0257] In the several embodiments provided in this specification, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.

[0258] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described in this specification, depending on actual needs.

[0259] Furthermore, the functional units in the various embodiments of this specification can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0260] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this specification. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0261] This specification uses specific embodiments to illustrate the principles and implementation methods of this specification. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this specification. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this specification. Therefore, the content of this specification should not be construed as a limitation of this specification.

Claims

1. A method for predicting model performance, characterized in that, The method includes: Identify the influencing factors of the model to be predicted; The influencing factors are input into the performance prediction model to obtain the key predicted values ​​of resource consumption and average time consumption required to invoke the model to be predicted. The performance prediction model is constructed from the key predicted values ​​of sample resource consumption, the key predicted values ​​of sample average time consumption, and multiple influencing factors to build a training sample set, and then trained with multiple initial prediction models. Based on the key predicted values ​​of resource consumption and average time consumption, the overall predicted values ​​of resource consumption and average time consumption of the model to be predicted are determined.

2. The method according to claim 1, characterized in that, The method further includes: Based on the total resource consumption when a single feature set in at least one original feature set on which the model to be predicted is dependent is invoked, the total resource consumption when a single derived feature set in at least one derived feature set on which the model to be predicted is dependent is invoked, the maximum average time when a single feature set in at least one original feature set on which the model to be predicted is dependent is invoked, and the maximum average time when a single derived feature set in at least one derived feature set on which the model to be predicted is dependent is invoked, the theoretical value of the predicted resource consumption and the theoretical value of the predicted average time of the model to be predicted are determined. Based on the comparison results between the overall predicted resource consumption and the theoretical predicted resource consumption, and the comparison results between the overall predicted average time and the theoretical predicted average time, the final predicted resource consumption and the final predicted average time of the model to be predicted are determined; and If the final resource consumption prediction meets the first preset condition and the final average time prediction meets the second preset condition, then the performance prediction of the model to be predicted ends.

3. The method according to claim 2, characterized in that, The step of determining the final resource consumption forecast and the final average time forecast based on the comparison results between the overall predicted resource consumption and the theoretical predicted resource consumption, and the comparison results between the overall predicted average time and the theoretical predicted average time, includes: Determine whether the overall predicted value of resource consumption of the model to be predicted is greater than the theoretical value of resource consumption prediction, and whether the overall predicted value of average time consumption of the model to be predicted is greater than the theoretical value of average time consumption prediction. If not, determine that the theoretical value of the resource consumption forecast is the final resource consumption forecast, and determine that the theoretical value of the average time consumption forecast is the final average time consumption forecast; and If so, the overall predicted value of resource consumption is determined to be the final predicted value of resource consumption, and the overall predicted value of the average time consumption of the model to be predicted is determined to be the final predicted value of average time consumption.

4. The method according to claim 2 or 3, characterized in that, The first preset condition includes that the final predicted resource consumption does not exceed a preset resource consumption threshold; and The second preset condition includes that the predicted final average time consumption does not exceed the preset average time consumption threshold.

5. A method for constructing a performance prediction model, characterized in that, The performance prediction model is constructed in the following manner: The sample feature set dependency relationship of the sample model is parsed, and the sample model is invoked according to the sample feature set dependency relationship, wherein the sample feature set dependency relationship includes at least the original sample feature set and the derived sample feature set; By calling the original sample feature set and the derived sample feature set, the key values ​​of sample resource consumption and the key value of average sample time consumption generated during the sample model calling process are determined; and A training sample set is constructed using the key values ​​of sample resource consumption, the key values ​​of sample average time consumption, and multiple influencing factors. Multiple initial prediction models are trained using the training sample set to obtain a performance prediction model. The multiple initial prediction models include an initial prediction model for predicting resource consumption and an initial prediction model for predicting average time consumption.

6. The method according to claim 5, characterized in that, Parsing the sample feature set dependencies of the sample model and initiating a call to the sample model based on the sample feature set dependencies includes: An invocation is made to the original sample feature set, and a first invocation result is obtained; and Based on the first call result, a call is initiated to the derived sample feature set to obtain the second call result.

7. The method according to claim 6, characterized in that, By calling the original sample feature set and the derived sample feature set, the key values ​​of sample resource consumption and average sample time consumed during the sample model calling process are determined, including: Initiate a call to the original sample feature set to obtain the maximum value of the first resource consumption and the first average time consumption of the first call result; Initiating a call to the derived sample feature set yields the maximum value of the second resource consumption and the second average time consumption of the second call result; and Based on the first resource consumption and the second resource consumption, the key value of the sample resource consumption is determined, and based on the maximum value of the first average time consumption and the maximum value of the second average time consumption, the key value of the sample average time consumption is determined.

8. The method according to claim 7, characterized in that, A training sample set is constructed using the key values ​​of sample resource consumption, the key values ​​of average sample time consumption, and multiple influencing factors. Multiple initial prediction models are trained using this training sample set to obtain performance prediction models, including: Based on the original sample feature set and the derived sample feature set, multiple influencing factors are calculated; Multiple influencing factors are input into multiple initial prediction models, and the output results of each initial prediction model are obtained; and Based on the labels and output results of each initial prediction model, a loss value is calculated. The model parameters of each initial prediction model are iteratively updated using the loss value until a preset condition is met, thus obtaining the performance prediction model. The maximum values ​​of the first resource consumption, the second resource consumption, the first average time consumption, and the second average time consumption are respectively used as the labels of each initial prediction model.

9. A model performance prediction device, characterized in that, The device includes: The influencing factor determination unit is used to determine the influencing factors of the model to be predicted. The prediction unit is used to input the influencing factors into the performance prediction model to obtain the key predicted values ​​of resource consumption and average time consumption required to invoke the model to be predicted. The performance prediction model is obtained by constructing a training sample set from the key predicted values ​​of sample resource consumption, the key predicted values ​​of sample average time consumption, and multiple influencing factors, and training multiple initial prediction models. The prediction value determination unit is used to determine the overall predicted value of resource consumption and the overall predicted value of average time consumption of the model to be predicted based on the key predicted values ​​of resource consumption and the key predicted values ​​of average time consumption.

10. A performance prediction model construction device, characterized in that, The device includes: The calling unit is used to parse the sample feature set dependency relationship of the sample model and initiate a call to the sample model according to the sample feature set dependency relationship, wherein the sample feature set dependency relationship includes at least the original sample feature set and the derived sample feature set; The key value determination unit is used to determine the key values ​​of sample resource consumption and average sample time consumption generated during the sample model invocation process by calling the original sample feature set and the derived sample feature set; and The training unit is used to construct a training sample set using the key values ​​of sample resource consumption, the key values ​​of sample average time consumption, and multiple influencing factors, and to train multiple initial prediction models using the training sample set to obtain a performance prediction model; wherein, the multiple initial prediction models include: an initial prediction model for predicting resource consumption and an initial prediction model for predicting average time consumption.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images