Method and apparatus for creating ai model, and database system

Abstract

The present disclosure relates to the technical field of computers. Provided are a method and apparatus for creating an AI model, and a database system. In the present disclosure, filtering processing is performed on a first attribute set carried in a model creation instruction. In this way, since attributes used for creating an AI model are reduced by means of filtering processing, during the creation of the AI model, the amount of data that a database server needs to transmit to an AI server is reduced, such that the data transmission efficiency is improved, thereby improving the efficiency of AI model creation.

Description

Methods, apparatus and database systems for creating AI models

[0001] This application claims priority to Chinese Patent Application No. 202411526539.0, filed on October 29, 2024, entitled "Method, Apparatus and Database System for Creating AI Models", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of computer technology, and in particular to a method, apparatus and database system for creating AI models. Background Technology

[0003] With the development of artificial intelligence (AI) and database technologies, the integration of AI and databases has become an important trend in technological development. Many database systems have already integrated AI-related operations, such as creating AI models and using AI models for reasoning.

[0004] After receiving an instruction to create an AI model, the database server can locally read data from several attributes specified in the instruction and transmit the corresponding data to the AI ​​server. The AI ​​server uses this data to train the model, obtains the AI ​​model, and returns it to the database server. Users can then use this AI model for inference within the database server.

[0005] However, the process of creating an AI model generally requires a large amount of data for model training, which means that the database server needs to send a large amount of data to the AI ​​server, resulting in low efficiency in creating AI models. Summary of the Invention

[0006] This disclosure provides a method, apparatus, and database system for creating AI models, which reduces the amount of data transmitted and improves data transmission efficiency.

[0007] Firstly, a method for creating an AI model is provided, which is applied to a database server. The method includes: firstly, receiving a first model creation instruction carrying a first set of attributes; then, based on the data corresponding to the first set of attributes, performing a filtering process on multiple attributes included in the first set of attributes to obtain a second set of attributes; then, sending the data corresponding to the second set of attributes stored in the database server to an AI server; finally, the AI ​​server creates an AI model based on the data corresponding to the second set of attributes and feeds the AI ​​model back to the database server.

[0008] The first attribute set includes multiple attributes. The first model creation instruction is used to instruct the creation of an AI model based on the data corresponding to the first attribute set stored in the database server. The AI ​​server is used to create an AI model based on the data corresponding to the second attribute set.

[0009] In this way, because the attributes used to create the AI ​​model are reduced through the filtering process, the amount of data that the database server needs to transmit to the AI ​​server during the AI ​​model creation process is reduced, improving data transmission efficiency and thus improving the efficiency of creating the AI ​​model.

[0010] In one possible implementation, the filtering process can be as follows: filtering out the target attribute from the first attribute set to obtain a second attribute set, wherein the distribution characteristics of the target attribute data satisfy the distribution filtering condition, and / or, the association characteristics of the target attribute data with the data of other attributes satisfy the association filtering condition.

[0011] In this way, by distributing and / or associating screening conditions, some attributes can be selectively screened out, which effectively reduces the amount of data transmission and improves the overall performance of the system while ensuring the performance of the AI ​​model.

[0012] In one possible implementation, the distribution screening conditions include at least one of the following screening conditions: the proportion of null values ​​in the data of the target attribute reaches a first proportion threshold; the number of different values ​​in the data of the target attribute is less than a number threshold.

[0013] In this way, by setting the filtering condition "the proportion of null values ​​in the attribute data reaches the first proportion threshold", those containing a large number of null values ​​can be effectively filtered out. By setting the filtering condition "the number of different values ​​in the attribute data is less than the number threshold", those attributes with overly simplistic values ​​can be effectively filtered out. These attributes do not contribute much to improving the performance of the AI ​​model during training. Therefore, setting such filtering conditions can reduce unnecessary data transmission and improve the system's processing efficiency without affecting the performance of the AI ​​model.

[0014] In one possible implementation, the association filtering conditions include at least one of the following filtering conditions: the correlation between the data of the target attribute and the data of other attributes is greater than a first correlation threshold; the correlation between the data of the target attribute and the data of other attributes (such as label attributes) is less than a second correlation threshold. Wherein, the other attributes belong to a first set of attributes.

[0015] In this way, by setting the correlation between the target attribute's data and other attribute data as a condition for association filtering, attributes that are highly correlated with other attributes or almost uncorrelated with the label attributes can be filtered out. This avoids introducing redundant or useless information during model training. Similarly, while ensuring the performance of the AI ​​model, it effectively reduces the amount of data transmission and improves the overall performance of the system.

[0016] In one possible implementation, for AI model retraining, a set of third attributes can be maintained. The corresponding processing includes: receiving instruction information from the AI ​​server after creating the AI ​​model, and generating and recording the set of third attributes based on this instruction information. Based on this set of third attributes, the corresponding retraining process includes: after receiving a model retraining instruction corresponding to the AI ​​model, sending the data corresponding to the set of third attributes stored in the database server to the AI ​​server; the AI ​​server retraining the AI ​​model based on the data corresponding to the set of third attributes, and then feeding back the retrained AI model to the database server.

[0017] In this way, by generating a third attribute set and transmitting only the data corresponding to this third attribute set to the AI ​​server during retraining, instead of transmitting all the data selected by the user, the amount of data transmitted can be significantly reduced. Since the third attribute set is filtered and optimized, containing the most valuable parts for model retraining, the overhead of data transmission is reduced while ensuring model performance.

[0018] In one possible implementation, for AI model retraining, a third attribute set can be maintained. The corresponding process includes: first, sampling a specified number of data points corresponding to the first attribute set and sending the data to the AI ​​server; the AI ​​server determining instruction information based on the specified number of data points corresponding to the first attribute set and feeding the instruction information back to the database server; then, the database server generating and recording the third attribute set based on the instruction information. Based on this third attribute set, the corresponding retraining process includes: after receiving a model retraining instruction corresponding to the AI ​​model, sending the data corresponding to the third attribute set stored in the database server to the AI ​​server; the AI ​​server retraining the AI ​​model based on the data corresponding to the third attribute set and feeding the retrained AI model back to the database server.

[0019] In this way, by sending a portion of the sampled data to the AI ​​server, the AI ​​server can generate indications based on the sampled data, potentially resulting in more targeted and effective indications. By generating a third attribute set and transmitting only the data corresponding to this attribute set to the AI ​​server during retraining, instead of transmitting all data, the amount of data transmitted can be significantly reduced. Since the third attribute set is filtered and optimized, containing the most valuable parts for model retraining, data transmission overhead is reduced while maintaining model performance.

[0020] In one possible implementation, considering that the data in the database server may change, the third attribute set used for retraining the AI ​​model can be dynamically updated. A trigger condition can be set for the process of "sending a specified number of data points corresponding to the first attribute set to the AI ​​server." The trigger condition could be: the data corresponding to the first attribute set has changed compared to the last time the third attribute set was recorded, and the change ratio reaches a second ratio threshold.

[0021] By checking whether the proportion of data change reaches the second proportion threshold, it can be ensured that the third attribute set is only regenerated when the data changes significantly, thereby avoiding unnecessary calculations and data transmission.

[0022] In one possible implementation, the instruction information includes attributes selected by the AI ​​server for model training. Correspondingly, the third set of attributes consists of the attributes selected by the AI ​​server for model training.

[0023] In this way, when retraining the AI ​​model, only the attributes used in the process of creating the AI ​​model are used to transmit the corresponding attribute data to the AI ​​server, thus reducing the data transmission overhead while ensuring model performance.

[0024] In one possible implementation, the indication information includes importance values ​​corresponding to at least one attribute. Correspondingly, the third attribute set consists of attributes whose importance values ​​are greater than an importance threshold.

[0025] In this way, filtering based on the importance value of attributes can reduce the amount of data transmission that has a minor impact on model performance while ensuring model performance, thereby reducing the computational burden and improving the overall efficiency of the system.

[0026] In one possible implementation, firstly, a second model creation instruction carrying a second set of attributes is generated, and then, in response to the second model creation instruction, the data corresponding to the second set of attributes stored in the database server is sent to the AI ​​server.

[0027] In this way, the second model creation instruction is a separate command that carries information about the second attribute set. This means that regardless of the platform on which the instruction is executed, as long as the platform supports the instruction format, the second attribute set can be correctly parsed and the corresponding data sent to the AI ​​server. This improves the system's cross-platform compatibility.

[0028] In one possible implementation, firstly, the target attributes used for training the AI ​​model are received from the AI ​​server. Subsequently, when receiving the model inference instruction corresponding to the AI ​​model, the data corresponding to the model inference instruction and the target attributes stored on the database server can be input into the AI ​​model for model inference processing to obtain the result data.

[0029] This ensures that the AI ​​model uses the same attributes during training and inference, guaranteeing that the inference process proceeds normally.

[0030] In a second aspect, a data query apparatus is provided, the apparatus comprising at least one module for implementing the method provided in the first aspect or any possible implementation thereof.

[0031] Thirdly, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the method provided by the first aspect or any possible implementation thereof.

[0032] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the method provided by the first aspect or any possible implementation thereof.

[0033] Fifthly, a computer program product containing instructions is provided, which, when executed by a cluster of computing devices, causes the cluster of computing devices to perform the method provided by the first aspect or any possible implementation thereof. Attached Figure Description

[0034] Figure 1 is a schematic diagram of a process for creating an AI model according to an embodiment of this disclosure;

[0035] Figure 2 is a schematic diagram of a process for executing a model creation request according to an embodiment of this disclosure;

[0036] Figure 3 is a schematic diagram of a process for creating an AI model according to an embodiment of this disclosure;

[0037] Figure 4 is a schematic diagram of a screening condition provided in an embodiment of this disclosure;

[0038] Figure 5 is a schematic diagram of a screening condition provided in an embodiment of this disclosure;

[0039] Figure 6 is a schematic diagram of a screening condition provided in an embodiment of this disclosure;

[0040] Figure 7 is a schematic diagram of a screening condition provided in an embodiment of this disclosure;

[0041] Figure 8 is a schematic flowchart of a method for determining a third attribute set according to an embodiment of this disclosure;

[0042] Figure 9 is a schematic diagram of a process for determining a third attribute set according to an embodiment of this disclosure;

[0043] Figure 10 is a schematic diagram of a process for determining a third attribute set according to an embodiment of this disclosure;

[0044] Figure 11 is a schematic flowchart of a method for determining a third attribute set according to an embodiment of this disclosure;

[0045] Figure 12 is a schematic diagram of a data change ratio provided in an embodiment of this disclosure;

[0046] Figure 13 is a schematic diagram of a process for determining a third attribute set according to an embodiment of this disclosure;

[0047] Figure 14 is a schematic diagram of a process for determining a third attribute set according to an embodiment of this disclosure;

[0048] Figure 15 is a schematic diagram of a model retraining process provided in an embodiment of this disclosure;

[0049] Figure 16 is a schematic flowchart of an embodiment of the present disclosure for executing a model retraining request;

[0050] Figure 17 is a schematic diagram of a model retraining process provided in an embodiment of this disclosure;

[0051] Figure 18 is a schematic diagram of a model inference process provided in an embodiment of this disclosure;

[0052] Figure 19 is a schematic diagram of a process for determining a target attribute set according to an embodiment of this disclosure;

[0053] Figure 20 is a schematic diagram of a model reasoning process provided in an embodiment of this disclosure;

[0054] Figure 21 is a schematic diagram of the structure of an apparatus for creating an AI model according to an embodiment of this disclosure;

[0055] Figure 22 is a schematic diagram of a computing device provided in an embodiment of this disclosure;

[0056] Figure 23 is a schematic diagram of a computing device cluster provided in an embodiment of this disclosure;

[0057] Figure 24 is a schematic diagram of a computing device cluster provided in an embodiment of this disclosure. Detailed Implementation

[0058] database

[0059] A database can be considered a warehouse for storing data according to certain rules, built using computers. It has a very large storage capacity, capable of holding millions, tens of millions, hundreds of millions, or even more data entries. The data in a database is generally stored on computer storage media (such as hard drives).

[0060] Databases are generally divided into relational databases and non-relational databases. Relational databases typically store data in tables. Non-relational databases typically store data in non-tabular formats, such as key-value pairs or graph structures.

[0061] Database server

[0062] A database server is a computer system specifically designed for managing, storing, and providing data services; it can be considered the computer on which a database is built. A database server can be a physical machine, a cluster of physical machines, a virtual machine, a cluster of virtual machines, a container, a container cluster, or even a single process.

[0063] AI server

[0064] An AI server is a computer system or cluster specifically designed to run artificial intelligence and machine learning tasks. It typically features high-performance processors, ample memory, high-speed storage devices, and acceleration hardware such as graphics processing units (GPUs). The implementation of an AI server can be compared to that of a database server.

[0065] AI model

[0066] AI models are mathematical models trained using machine learning algorithms that can make predictions or decisions on unknown data. For example, AI models can be used to identify objects in images (such as cars, trees, and flowers). From a functional perspective, AI models can be categorized into classifiers, regressors, and clusterers. For instance, classifiers are used in image recognition (such as identifying cats and dogs in photos), text classification (such as spam detection), and medical diagnosis (such as cancer diagnosis); regressors are used in scenarios such as house price prediction, sales forecasting, and temperature prediction; and clusterers are used in image segmentation (such as dividing a photo containing apples and oranges into two parts, one containing only apples and the other only oranges) and customer segmentation (such as dividing a company's customer base into several categories, such as "frequent shoppers" and "occasional shoppers"). From an algorithmic perspective, AI models can be categorized into: supervised learning models (such as linear regression, logistic regression, and random forests), unsupervised learning models (such as K-means, hierarchical clustering, and principal component analysis), and deep learning models (convolutional neural networks, recurrent neural networks, and long short-term memory networks).

[0067] Creating AI Models

[0068] Creating an AI model refers to the entire process from data preparation to model training and evaluation. This process includes a series of steps such as data collection, data preprocessing, feature engineering, model selection, model training, model evaluation, and model optimization. The ultimate goal is to build a machine learning model that can accurately predict or make decisions based on unknown data.

[0069] The aforementioned "features" are the input or output data of the AI ​​model, such as petal size, user age, and stock trading volume. In this embodiment, "features" correspond to "attributes" in the database, meaning that data with specified attributes is used as feature data for training the AI ​​model. Feature engineering mainly involves feature selection.

[0070] Training Model

[0071] Training a model refers to adjusting the parameters of an AI model based on sample data, so that the AI ​​model can better fit the training data and improve the accuracy of the AI ​​model's output results.

[0072] Model Inference

[0073] Model inference refers to the process of using a pre-trained AI model to predict or make decisions based on new, unknown data. For example, by inputting relevant parameters of a flower (such as petal size, stamen size, etc.) into an AI model, the AI ​​model outputs the variety of the flower.

[0074] Retraining

[0075] Retraining refers to the process of retraining an AI model with new data after the model has been trained, so that the AI ​​model can adapt to the new data distribution and improve the model performance.

[0076] property

[0077] An attribute refers to a column name in a data table. Each attribute represents a specific data characteristic. Examples include product name, store name, product number, price, and stock trading volume. In a data table, each attribute can correspond to specific data, and its data type can be defined, such as numeric, character, or date.

[0078] Correlation

[0079] Correlation refers to the statistical relationship between multiple variables. It can be calculated based on various specific values ​​of two variables, and corresponding algorithms include Pearson correlation coefficient, Spearman rank correlation coefficient, Kendall rank correlation coefficient, mutual information, and chi-squared test. A higher correlation between two variables means that based on the value of one variable, it is easier and more accurate to obtain the value of the other; a lower correlation means that based on the value of one variable, it is less accurate to obtain the value of the other. For example, if a high Pearson correlation coefficient is found between a student's math and physics scores, then we can assume that based on a student's performance in math, we can predict their performance in physics with relatively high accuracy. Conversely, if there is a very low correlation between math and physical education scores, it means that we cannot accurately predict a student's physical education performance based on their math scores.

[0080] This disclosure embodiment determines the correlation between different attributes based on the data of different attributes in the data table, and then performs some attribute filtering processing based on the correlation.

[0081] Importance value

[0082] Importance value is a quantitative indicator that measures the degree of influence of a feature (corresponding to "attribute" in this embodiment) on the predictive ability of the model. In machine learning, feature importance values ​​can help identify which features have the greatest impact on the model.

[0083] With the development of AI and database technologies, the integration of AI and databases has become a significant trend in technological advancement. Performing AI-related processing within a database system is not a simple data computation process, but rather a complex process combining database-related operations with AI-related operations. Many database systems have already integrated AI-related operations, such as creating AI models and using these models for inference.

[0084] After receiving an instruction to create an AI model (hereinafter referred to as a model creation instruction), the database server can read the data of several attributes specified in the instruction locally and transmit the corresponding data to the AI ​​server. The AI ​​server uses this data to train the model, obtains the AI ​​model, and returns it to the database server. Users can then use the AI ​​model for inference within the database server.

[0085] However, the process of creating an AI model generally requires a large amount of data for model training, which means that the database server needs to send a large amount of data to the AI ​​server, resulting in low efficiency in creating AI models.

[0086] This disclosure provides a method for creating an AI model. Based on data stored in a database and the relationships between that data, this method performs attribute filtering on the attribute set carried by the user-input model creation command. Attribute filtering reduces the number of attributes used to train the AI ​​model, aiming to decrease the amount of data transmission between the database server and the AI ​​server and improve the execution efficiency of the model creation command. The processing flow of this method is shown in Figure 1, including the following steps:

[0087] Step 101: The database server receives a first model creation instruction carrying a first set of attributes.

[0088] The first attribute set includes multiple attributes, which can be input by the user. The first model creation instruction is used to instruct the creation of an AI model based on the data corresponding to the first attribute set stored in the database server.

[0089] During database usage, users send operation commands to the database server through a client. These commands can be basic operations such as adding, deleting, modifying, and querying data within the database, or extended commands related to AI, such as creating AI models and using AI models for inference. When a user wants to create an AI model in the database, they can enter a first model creation command in the client and have the client send this command to the database server. The first model creation command instructs the creation of an AI model based on data corresponding to certain attributes in one or more data tables.

[0090] For example, the first model creation command entered by the user could be "Create model xxx as select{y1,y2,y3,y4}from table z". This first model creation command specifies that an AI model named "xxx" be created based on the data corresponding to the four attributes y1, y2, y3, and y4 in the table "table z". Here, {y1,y2,y3,y4} is the first set of attributes.

[0091] Users can also specify parameters in the first model creation command using fields like "with()" within parentheses. These parameters can include instructions to use a specific attribute as a label attribute, which corresponds to the AI ​​model's output data and serves as training calibration data (i.e., the reference truth value for the output data) during training. Attributes selected as label attributes are not discarded. Parameters can also include AI model configuration parameters, such as training mode and model algorithm type. If the user does not input label attributes and model configuration parameters through the aforementioned fields, these can be pre-set uniformly. For example, the first attribute in the first attribute set can be uniformly set as the label attribute, and the training mode can be pre-set to asynchronous or synchronous mode. Label attributes can also be pre-set to null values, i.e., no label attribute is specified. Model configuration parameters can also be determined using automated machine learning techniques. For example, the model algorithm type can be automatically determined based on the data type in the data table. The automated machine learning techniques used can be automated machine learning (AutoML) or smart machine learning (SmartML), etc.

[0092] Step 102: Based on the data corresponding to the first attribute set, the database server performs a filtering process on the multiple attributes included in the first attribute set to obtain the second attribute set.

[0093] After parsing the first model creation instruction, the database server can obtain the first attribute set carried therein. Then, for each attribute in the first attribute set, it is determined whether the data of the attribute meets the screening conditions. Here, one or more screening conditions can be set. If one of the screening conditions is met, the attribute is screened out in the first attribute set. The screening conditions can be divided into multiple types, including distribution screening conditions and association screening conditions, etc. The distribution screening conditions are used to determine the distribution characteristics of the data, and the association screening conditions are used to determine the association characteristics between the data of different attributes (the present disclosure provides various feasible distribution screening conditions and association screening conditions, and the corresponding screening conditions will be described in detail in the subsequent content). After all the attributes (attributes other than the label attribute) in the first attribute set have been judged, a second attribute set can be obtained. The second attribute set consists of the attributes in the first attribute set that have not been screened out.

[0094] The database server can then replace the first attribute set in the first model creation instruction with the second attribute set to generate a second model creation instruction.

[0095] Step 103, the database server sends the data corresponding to the stored second attribute set to the AI server.

[0096] In response to the second model creation instruction, the database server reads data locally based on each attribute in the second attribute set to obtain the data corresponding to these attributes. Subsequently, the data can be provided to the AI server and the AI server is instructed to create an AI model. There are multiple possibilities for the corresponding processing methods, which are introduced separately below:

[0097] Processing method 1

[0098] The database server directly sends a model creation request to the AI server, and carries the read data in the model creation request to instruct the AI server to create an AI model based on this data. The model creation request can also carry the above-mentioned label attribute, model configuration parameters, etc.

[0099] Processing method 2

[0100] The database server sends the read data to a specified device (which can be a storage device provided by a third party, or a storage device supporting the AI server, or a storage device supporting the database server). Then, the database server sends a model creation request to the AI server, which can carry the storage location information of the data, such as the address of this specified device, the storage path of the data, etc., and can also carry the above-mentioned label attribute, model configuration parameters, etc. The AI server can read the corresponding data in the specified device based on the storage location information.

[0101] The above-mentioned method of generating and executing a second model creation instruction based on a second attribute set is equivalent to optimizing the received model creation instruction and then executing the optimized instruction, thus eliminating the need to modify the database kernel logic. Alternatively, in another method described in steps 102-103 above, there is no need to generate and execute the second model creation instruction; that is, attribute filtering logic is added to the response logic of the model creation instruction.

[0102] Step 104: The AI ​​server creates an AI model based on the received data.

[0103] The AI ​​server executes the model creation request. The execution process can be illustrated in Figure 2, including the following steps:

[0104] Step 1041: The AI ​​server preprocesses the acquired data.

[0105] Preprocessing may include one or more of the following:

[0106] Preprocessing 1: The AI ​​server cleans the acquired data.

[0107] Data cleaning can include removing outliers and filling in missing values.

[0108] Removing outliers can be done as follows: For multiple data points corresponding to a certain attribute (i.e., a column of data), if a certain data point differs too much from the other data points, then that data point is considered an outlier. The outlier can be replaced with a normal value (for example, the normal value can be the average of the other data points in the column), or the entire row of data corresponding to that outlier can be deleted.

[0109] Filling in missing values ​​can be done as follows: For multiple data points corresponding to a certain attribute, if some data points are missing, a supplementary value can be determined and filled into the position of the missing data. The supplementary value can be a preset value, the average value of the data points corresponding to the attribute, or a value randomly selected from the data points corresponding to the attribute, and so on.

[0110] Preprocessing 2: The AI ​​server performs data transformation on the acquired data, such as normalizing or standardizing numerical data, and encoding categorical data (e.g., one-hot encoding, label encoding, etc.).

[0111] In the third preprocessing step, the AI ​​server performs feature selection (in the current application scenario, "feature" is equivalent to "attribute") on the acquired data, filtering out certain features based on certain filtering conditions. The filtering conditions used by the AI ​​server and the database server can be different, or they can use similar filtering conditions but with different specific parameters.

[0112] Step 1042, the AI server initializes the model to obtain an initial AI model.

[0113] The model creation request may carry the above model configuration parameters, and the AI server can initialize the model based on the model configuration parameters. The obtained initial AI model is an AI model that has not been trained and tuned, and the model parameters therein are all initial values (such as preset fixed values or random values).

[0114] Step 1043, the AI server uses the feature-processed data to train the AI model.

[0115] During the training process, for the case where there are label attributes (i.e., supervised training), the data corresponding to the label attributes can be used as the reference data for the output results of the initial AI model, and the data corresponding to other attributes except the label attributes can be used as the input data of the initial AI model. Continuously adjust the model parameters of the initial AI model to find a set of model mathematical parameters that make the loss function as small as possible. When the training reaches a certain number of rounds or the model performance no longer improves significantly, the model training is completed. During the training process, the database server can periodically send a status query request to the AI server, and the AI server can feedback the training status to the database server, such as in training, training completed, etc.

[0116] Step 105, the AI server feedbacks the trained AI model to the database server.

[0117] The AI server feedbacks the AI model to the database server. The feedback data includes the basic data of the AI model, that is, the model algorithm type (also called the model architecture), model parameters, etc. In addition to the basic information, the feedback data can also include other relevant information of the AI model, for example, model performance information, the attributes finally selected by the AI server for training the AI model, etc. The various feedback data can be concentrated in one file and feedback to the database server, or can be feedback to the database server as different files respectively.

[0118] [[ID=1​​Figure 3 provides a visual example of the processing flow shown in Figure 1. The database server receives a first model creation instruction, where the first attribute set is {y1, y2, y3, y4}. After filtering, it obtains a second attribute set {y1, y3, y4}, and then updates the first model creation instruction to the second model creation instruction. The second model creation instruction is executed, retrieving the data corresponding to {y1, y3, y4} and sending it to the AI ​​server. The AI ​​server creates an AI model based on this data and feeds it back to the database server.

[0120] In this embodiment, the first set of attributes carried in the model creation instruction is filtered out. Because the attributes used to create the AI ​​model are reduced through this filtering process, the amount of data that the database server needs to transmit to the AI ​​server during the AI ​​model creation process is reduced, improving data transmission efficiency and thus increasing the efficiency of AI model creation.

[0121] The following explains the screening conditions involved in step 102 above. A set of screening conditions can be set, which may include several screening conditions, including distribution screening conditions and / or correlation screening conditions. During screening, data of a certain attribute (an attribute other than the tag attribute) can be screened out as long as it meets one of the screening conditions in the set of screening conditions. Several screening conditions that can be used are introduced below, and the corresponding set of screening conditions may include at least one of the following screening conditions. Conditions one and two are distribution screening conditions, and conditions three and four are correlation screening conditions.

[0122] Condition 1: The proportion of null values ​​in the data of this attribute reaches the first proportion threshold.

[0123] The null value ratio refers to the proportion of null values ​​(null or NaN) in the data corresponding to a given attribute to the total number of data points corresponding to that attribute (the sum of null values ​​and non-null values). The null value ratio is a distribution characteristic of the data.

[0124] The following is a specific example to illustrate this: As shown in Figure 4, suppose attribute a has a total of 8 records, of which 5 records are null values. The first proportion threshold is set to 0.25 (25%). The null value proportion is calculated as 5 / 8 = 0.625. Since 0.625 is greater than 0.25, attribute a will be filtered out.

[0125] Condition 2: The number of different values ​​in the data of this attribute is less than the number threshold.

[0126] The number of different values ​​in data is a distribution characteristic of the data.

[0127] The threshold for the number of values ​​can be a pre-set value. In the first case, different attributes can correspond to different thresholds; in the second case, different attributes can correspond to a uniform threshold. For the first case, the threshold can also be a calculated value. The calculation method could be: record the number of possible values ​​for each attribute (e.g., the number of possible values ​​for the "month" attribute is 12, and the number of possible values ​​for the "gender" attribute is 2), and set a uniform percentage threshold (e.g., 25%). For any attribute, multiply the number of possible values ​​for that attribute by the uniform percentage threshold to obtain the threshold for that attribute. The above uses a uniform percentage threshold for different attributes. Of course, in some application scenarios, different percentage thresholds can be used for different attributes. Whether a uniform or different percentage thresholds are used, they can be set based on the training requirements for data diversity.

[0128] The following is a concrete example: As shown in Figure 5, suppose an attribute has 8 data points, with 2 different values, and a threshold of 5. The number of different values ​​is 2, which is less than 5, so this attribute will be eliminated. The possible values ​​for this attribute are 10, and the uniform proportion threshold is 50%. Multiplying these two values ​​together gives a threshold of 5.

[0129] The purpose of this screening criterion is to determine whether an attribute has sufficient diversity by counting the number of different values. If an attribute has a small number of different values, the feature may not be representative enough and has little value in AI model training.

[0130] Condition 3: The correlation between the data of this attribute and the data of other attributes is greater than the first correlation threshold.

[0131] Among these, other attributes belong to the first attribute set. These other attributes are not label attributes, meaning they are not the attributes specified in the first model creation instruction that correspond to the output data of the AI ​​model. The correlation between data of different attributes is a data association feature.

[0132] The primary relevance threshold can be a pre-set fixed value. A larger primary relevance threshold may improve the training convergence speed and accuracy of the AI ​​model, but may decrease creation efficiency. Conversely, a smaller primary relevance threshold may improve creation efficiency, but may decrease the training convergence speed and accuracy of the AI ​​model. Technical personnel or users can weigh the creation efficiency, training convergence speed, and accuracy of the AI ​​model when setting the primary relevance threshold.

[0133] The following is a specific example to illustrate this: As shown in Figure 6, based on the values ​​corresponding to attribute c and attribute d, it can be seen that these two attributes are completely correlated (i.e., the correlation is equal to 1. This result can be obtained through many correlation calculation algorithms, and the specific calculation process will not be explained).

[0134] The purpose of this screening criterion is to filter out attributes that are highly correlated with other attributes by calculating the correlation between attributes, thereby avoiding low data utilization and multicollinearity problems. Multicollinearity refers to a strong correlation between two or more input features, causing the model to become unstable or difficult to interpret during training.

[0135] To illustrate the purpose of this screening condition more clearly, let's look at a concrete example: Suppose the first attribute set includes two attributes: age and years of work experience. Generally, older age is likely to be accompanied by a higher number of years of work experience, and these two attributes may be highly correlated. In this case, one of the attributes will be screened out (if the calculated correlation is below the first correlation threshold).

[0136] There are many methods to calculate the correlation between two attributes. Commonly used correlation calculation methods include Pearson correlation coefficient, Spearman rank correlation coefficient, Kendall rank correlation coefficient, mutual information, and chi-square test.

[0137] When multiple data points are highly correlated, the specific attribute to be removed can be selected by comparing from other perspectives. For example, when the correlation between two attributes is greater than the first correlation threshold, the attribute with a higher proportion of null values ​​can be removed, or the attribute with a lower number of different values ​​can be removed, and so on.

[0138] Condition 4: The correlation between the data of this attribute and the data of other attributes is less than the second correlation threshold.

[0139] Among these, the other attributes belong to the first attribute set. These other attributes can be label attributes, that is, attributes specified by the first model creation instruction that correspond to the output data of the AI ​​model.

[0140] The second relevance threshold can be a pre-set fixed value. A smaller second relevance threshold may improve the training convergence speed and accuracy of the AI ​​model, but decrease creation efficiency. Conversely, a larger second relevance threshold may improve creation efficiency, but decrease the training convergence speed and accuracy of the AI ​​model. Technical personnel or users can weigh the creation efficiency, training convergence speed, and accuracy of the AI ​​model when setting the second relevance threshold.

[0141] The following is a concrete example to illustrate this: As shown in Figure 7, based on the values ​​corresponding to attributes e and f, it can be seen that these two attributes are completely uncorrelated (i.e., the correlation is equal to 0; this result can be obtained through many correlation calculation algorithms, the specific calculation process of which will not be explained). Here, attribute f corresponds to the output data of the AI ​​model.

[0142] The primary purpose of this filtering criterion is to remove features that have a low correlation with the specified attribute (i.e., the label attribute). The goal is to ensure that the retained attributes contribute significantly to the prediction of the specified attribute, thereby improving the utilization of the training data.

[0143] To illustrate the purpose of this filtering condition more clearly, let's look at a specific example: Suppose the first set of attributes includes two attributes, namely weather temperature and stock price. Normally, there should be no obvious correlation between weather temperature and stock price. When the specified attribute is stock price, the weather temperature attribute will be filtered out (if the calculated correlation is lower than the second correlation threshold).

[0144] In practical applications, the data stored in databases is frequently updated. To adapt to these changes, AI models also need to be updated accordingly. Model retraining is one method developed to address this need. For AI models already created in the database server, to improve the efficiency of subsequent AI model retraining, the database server can predetermine a set of attributes (referred to as the third attribute set) for each AI model. Several methods for determining the third attribute set are given below:

[0145] The processing flow for method one is shown in Figure 8, and includes the following steps:

[0146] Step 801: After creating the AI ​​model, the AI ​​server sends instruction information back to the database server.

[0147] The indication information is used to directly or indirectly indicate the importance of at least one attribute in the second attribute set to model training. The indication information may include the attribute selected by the AI ​​server for model training (referred to as Content 1), and / or the importance value corresponding to at least one attribute (referred to as Content 2).

[0148] Importance score is a quantitative indicator that measures the contribution of each attribute to the model's predictive ability. It determines which attributes are most important by analyzing the impact of features on the model's prediction results. There are many methods for calculating importance scores, and the specific method chosen depends on the model used.

[0149] For example, in regressors (such as linear regression and logistic regression), the importance of each attribute can be evaluated using the weight coefficients corresponding to each attribute in the AI ​​model. The magnitude and sign of the weight coefficients reflect the influence of each attribute on the model output. The larger the absolute value of the weight coefficient, the higher the importance of the attribute. That is, the importance value of an attribute is equal to the absolute value of its corresponding weight coefficient.

[0150] To ensure the stability or interpretability of the model, regularization methods or manual settings can be used to constrain the range of weight coefficients, for example, limiting the weight coefficients to a certain range (e.g., [-1, 1]). The corresponding importance value range is [0, 1].

[0151] For example, in classifiers, the importance value for decision tree models and their derivative models (such as random forests, gradient boosting trees, etc.) can be calculated using metrics such as the Gini index or information gain.

[0152] The Gini index measures the impurity of a node, representing the probability that a sample is misclassified. A lower Gini index for each node is better. A lower Gini index means that the samples in the node tend to belong to the same class, making the node more "pure." A higher Gini index means that the samples in the node are more unevenly distributed, with similar probabilities of belonging to multiple classes.

[0153] The formula for calculating the Gini index G(t) is as follows:

[0154] Where C is the number of categories, p i It is the sample proportion of category i.

[0155] The method for calculating the importance value based on the Gini index is as follows:

[0156] For a given feature, each time it's used for splitting in a decision tree, the change in the Gini index during that split is calculated. These changes are then summed to obtain the feature's total Gini importance, which is its significance value. Finally, optionally, the feature importance values ​​can be normalized so that the sum of all feature importance values ​​is 1. This normalization not only makes feature importance easier to understand but also facilitates comparisons of the importance of different features.

[0157] Information gain measures the degree to which the uncertainty of a system is reduced when a certain feature is split at a node. Information gain is primarily calculated based on entropy. The formula for calculating information gain H(t) is as follows:

[0158] Where C is the number of categories, p i It is the probability of category i.

[0159] The method for calculating the importance value based on information gain is similar to the method for calculating the importance value based on the Gini index, so it will not be repeated here.

[0160] Besides the aforementioned methods for calculating feature importance values ​​specific to certain models, there are also some general methods for evaluating feature importance values. For example, the feature permutation method involves randomly shuffling the data for a particular feature in the dataset (while keeping the data for other features unchanged), using the dataset before and after the shuffling to perform model inference, and observing the change in model performance. The greater the decrease in performance, the more important the feature is.

[0161] The content of the instruction message can be as follows:

[0162] For example, the instruction information may include the model name "xxx" and the attributes {y1, y4}, meaning "the attributes used to train the xxx model are {y1, y4}".

[0163] Alternatively, the indication information could be the importance values ​​of each attribute in the second set of attributes. For example, the importance value of attribute y1 is 0.8, the importance value of attribute y3 is 0.15, and the importance value of attribute y4 is 0.73.

[0164] Step 802: The database server generates a third attribute set based on the instruction information and records the third attribute set.

[0165] Specifically, for the case where the indication information is as described in point one, the third attribute set can consist of all attributes selected by the AI ​​server for model training. For the case where the indication information is as described in point two, the third attribute set can consist of attributes whose importance values ​​are greater than an importance threshold. The importance threshold can be a preset fixed value or it can be calculated. For different models, the importance threshold can be the same or different.

[0166] For example, if the instruction information indicates that "the attributes used to train the xxx model are {y1, y4}", then {y1, y4} is used as the third attribute set. The purpose is to transmit only the data corresponding to the attributes used by the AI ​​server when training the model, thereby reducing the amount of data transmission.

[0167] If the indication information is the importance values ​​of each attribute in the second attribute set, then the third attribute set is composed of attributes whose importance values ​​are greater than the importance threshold. Assuming the importance threshold is 0.2, the importance value of attribute y1 is 0.8, the importance value of attribute y3 is 0.15, and the importance value of attribute y4 is 0.73. Specifically, y1, with an importance value of 0.8, is greater than the importance threshold of 0.2 and is added to the third attribute set; y3, with an importance value of 0.15, is less than the importance threshold of 0.2 and is not added to the third attribute set; and y4, with an importance value of 0.73, is greater than the importance threshold of 0.2 and is added to the third attribute set. This generates the third attribute set {y1, y4}.

[0168] The database server records the generated set of third attributes and the corresponding model name. This information is used during subsequent model retraining to determine the data to be transmitted to the AI ​​server. A mapping table between model names and third attribute sets can be created within the database server to maintain the third attribute set for each AI model.

[0169] Figure 9 provides a visual example of the processing flow shown in Figure 8. The database server receives the instruction information from the AI ​​server, which indicates that "the attributes used for training the xxx model are {y1, y4}". {y1, y4} is used as the third attribute set, and the third attribute set and the corresponding model name "xxx" are recorded.

[0170] Figure 10 provides another visual example of the processing flow shown in Figure 8. The database server receives instruction information from the AI ​​server, where the instruction information is the importance value of each attribute in the second attribute set. For example, the importance value of attribute y1 is 0.8, the importance value of attribute y3 is 0.15, and the importance value of attribute y4 is 0.73. The importance threshold is 0.2. Attributes y1 and y4 with importance values ​​greater than 0.2 are used to form the third attribute set, i.e., {y1, y4}, and the generated third attribute set and the corresponding model name "xxx" are recorded.

[0171] The second processing method, as shown in Figure 11, includes the following steps:

[0172] For this processing flow, certain trigger conditions can be preset. Whenever the corresponding trigger condition is met, the following steps can be executed. The trigger condition can be that the data corresponding to the first attribute set has changed relative to the last time the third attribute set was recorded, and the change ratio reaches a second ratio threshold.

[0173] The change ratio can be the ratio of the number of rows that changed to the number of rows in the original data, or the ratio of the number of data items that changed to the number of data items in the original data. For example, as shown in Figure 12, assuming the original data has 9 rows and 4 columns, with a total of 36 data items, and one data item was modified, the change ratio is 1 / 9 based on the number of rows and 1 / 36 based on the number of data items.

[0174] The second ratio threshold can be a pre-set fixed value. A smaller second ratio threshold allows for more timely retraining, enabling the AI ​​model to better maintain its alignment with the data, but this consumes more processing and transmission resources. Conversely, a larger second ratio threshold results in poorer timeliness of retraining, making it harder for the AI ​​model to maintain its alignment with the data, but this consumes fewer processing and transmission resources. Therefore, technical personnel can weigh the requirements of timeliness, the alignment between the AI ​​model and the data, and the consumption of processing and transmission resources when setting the second ratio threshold.

[0175] The third attribute set can be recorded for the first time using the processing method described above, which occurs when the AI ​​model is created. Subsequent updates to the recorded third attribute set can be made using processing method two; that is, this process can be executed each time the aforementioned triggering conditions are met to update the recorded third attribute set.

[0176] Step 1101: The database server sends a specified number of data corresponding to the first set of attributes stored to the AI ​​server.

[0177] The database server can extract data row by row from the data corresponding to the first attribute set. This extraction can be done by randomly selecting a specified number of rows, or by extracting according to certain rules. For example, all rows can be divided into N groups in sequence, and the first M rows, the middle M rows, the last M rows, or random M rows can be extracted from each group. The product of N and M is equal to the specified number. Another example is that a specified number of rows can be extracted from all rows at fixed line spacing.

[0178] Step 1102: The AI ​​server determines the indication information based on a specified number of data corresponding to the first attribute set.

[0179] The content of the instruction information can be found in the description in step 801.

[0180] When the AI ​​server determines the indication information based on the data sent by the database server (i.e., the data corresponding to the first attribute set), it can use this data to perform a complete AI model creation process (see step 104), using the attributes used for model training as indication information, or calculating the importance value of each attribute as indication information. Alternatively, the AI ​​server can only perform the data preprocessing part of the AI ​​model creation process (see step 1041), using the remaining attributes after preprocessing as indication information, or calculating the importance value of each attribute as indication information.

[0181] Step 1103: The AI ​​server sends instruction information back to the database server.

[0182] Step 1104: The database server generates a third attribute set based on the instruction information and records the third attribute set.

[0183] This step is basically the same as step 802, and you can refer to the description of step 802. It will not be repeated here.

[0184] After determining the set of third attributes corresponding to the AI ​​model, the set of third attributes corresponding to the AI ​​model can be replaced with the currently determined set of third attributes in the above correspondence table between model names and sets of third attributes. For example, in the correspondence table, the model name "xxx" originally corresponds to {y1, y4}, and the currently determined set of third attributes is {y1, y3, y4}, then {y1, y4} in the correspondence table can be replaced with {y1, y3, y4}.

[0185] Figure 13 provides a visual example of the processing flow shown in Figure 11. The database server samples a small amount of data (called sampled data) corresponding to the first attribute set and sends the sampled data to the AI ​​server. The AI ​​server then sends feedback information to the database server, indicating that "the attributes used to train the xxx model are {y1, y4}". {y1, y4} is used as the third attribute set, and the third attribute set and the corresponding model name "xxx" are recorded.

[0186] Figure 14 provides another visual example of the processing flow shown in Figure 8. The database server samples a small amount of data (called sampled data) corresponding to the first attribute set and sends it to the AI ​​server. The AI ​​server then sends feedback information to the database server, where the feedback information is the importance value of each attribute in the first attribute set. For example, the importance value of attribute y1 is 0.8, the importance value of attribute y2 is 0.35, the importance value of attribute y3 is 0.15, and the importance value of attribute y4 is 0.73. The importance threshold is 0.2. The attributes y1, y2, and y4 with importance values ​​greater than 0.2 are used to form a third set, i.e., {y1, y2, y4}, as the third attribute set. The generated third attribute set and the corresponding model name "xxx" are recorded.

[0187] Based on the third attribute set of the above records, the process for model retraining can be shown in Figure 15, including the following steps:

[0188] Step 1501: The database server receives the model retraining instruction for the corresponding AI model.

[0189] The command could be "retrain model xxx", directly instructing the AI ​​model named "xxx" to be retrained. Alternatively, it could be a re-entry of the model creation command, specifying the same model name and attribute set. In this case, the database server would also recognize the model creation command as retraining the "xxx" model. For example, "Create model xxx as select{y1, y2, y3, y4} from table z".

[0190] Step 1502: The database server sends the data corresponding to the stored third attribute set to the AI ​​server.

[0191] After the database server executes the model retraining instruction, it can obtain the corresponding set of third attributes based on the model name carried in the retraining instruction. Then, based on each attribute in the set of third attributes, it reads data locally to obtain the data corresponding to these attributes. Subsequently, the data can be provided to the AI ​​server, along with the model file of the AI ​​model, instructing the AI ​​server to perform model retraining. See the relevant explanation in step 103; the only difference is that step 103 sends a model creation request, while step 1502 sends a model retraining request.

[0192] Step 1503: The AI ​​server retrains the AI ​​model based on the data corresponding to the third attribute set.

[0193] The retraining process can be illustrated in Figure 16, and includes the following steps:

[0194] Step 15031: The AI ​​server preprocesses the acquired data.

[0195] This step is basically the same as the process in step 1041. Please refer to the description of step 1041 for details. It will not be repeated here.

[0196] Step 15032: The AI ​​server loads the AI ​​model (which is a previously trained AI model) based on the model file of the AI ​​model.

[0197] Step 15033: The AI ​​server retrains the AI ​​model using the pre-processed data.

[0198] Step 1504: The AI ​​server sends the retrained AI model back to the database server.

[0199] Figure 17 provides a visual example of the processing flow shown in Figure 15. The database server receives the model retraining instruction, executes it, obtains the third attribute set {y1, y4} corresponding to the AI ​​model "xxx", and then retrieves the corresponding data through this third attribute set, sending it to the AI ​​server. The AI ​​server retrains the model based on this data and provides feedback to the database server.

[0200] After creating an AI model, users typically need to use it for inference. The process for using an AI model for inference can be illustrated in Figure 18, and includes the following steps:

[0201] Step 1801: The database server receives the attributes fed back by the AI ​​server.

[0202] The attributes provided in this feedback can be the attributes ultimately selected by the AI ​​server for model training. This set of attributes can be called the inference attribute set. The AI ​​server can provide this feedback to the database server after determining the attributes to be used for model training before training begins, or it can provide it along with model parameters and other information after the AI ​​model has been created.

[0203] For example, the attributes used to train the xxx model are {y1, y4}.

[0204] Step 1802: The database server stores the received attributes.

[0205] In the database server, a mapping table can be established between model names and sets of inference attributes. The attributes recorded in this table are the attributes needed during the model inference process. When the database server receives the set of inference attributes from the AI ​​server, it can add the set of inference attributes and the corresponding model name to this mapping table.

[0206] Step 1803: The database server receives the model inference instruction corresponding to the AI ​​model mentioned above.

[0207] Step 1804: Input the data stored in the database server that corresponds to both the model inference instructions and the attributes fed back by the AI ​​server (i.e., the attributes in the inference attribute set) into the AI ​​model to perform model inference and obtain the result data.

[0208] The database server can retrieve the user-specified model name and target row from the model inference command. Then, it can search the inference attribute set corresponding to that model name in the aforementioned correspondence table, retrieve the data of the target row within that attribute set, and determine the AI ​​model corresponding to that model name. Each retrieved row of data is then input into the AI ​​model for inference, yielding the corresponding result data.

[0209] Figures 19 and 20 provide a visual example of the processing flow shown in Figure 18.

[0210] As shown in Figure 19, the database server receives the attributes (which may be the attributes actually used when training the AI ​​model) from the AI ​​server. The attributes {y1, y4} actually used when training the AI ​​model are used. {y1, y4} are used as the inference attribute set, and the inference attribute set and the corresponding model name are recorded.

[0211] As shown in Figure 20, the database server receives the model inference instruction, executes the model inference instruction, obtains the inference attribute set {y1, y4} corresponding to the AI ​​model "xxx", then obtains the data corresponding to both the model inference instruction and the inference attribute set, inputs it into the AI ​​model to perform model inference, and obtains the result data.

[0212] Based on the same technical concept, this disclosure provides an apparatus for creating an AI model. The apparatus is deployed on a database server, as shown in Figure 21. The apparatus includes:

[0213] The receiving module 2110 is used to receive a first model creation instruction carrying a first set of attributes and an AI model fed back from the AI ​​server. The first set of attributes includes multiple attributes, and the first model creation instruction instructs the creation of an AI model based on data corresponding to the first set of attributes stored in the database server. Specifically, it can implement the processing functions of steps 101 and 105 mentioned above, as well as other implicit steps.

[0214] The filtering module 2120 is used to filter out multiple attributes included in the first attribute set based on the data corresponding to the first attribute set, to obtain a second attribute set. Specifically, it can implement the processing function of step 102 above, as well as other implicit steps.

[0215] Execution module 2130 is used to send the data corresponding to the second attribute set stored in the database server to the AI ​​server, and the AI ​​server is used to create the AI ​​model based on the data corresponding to the second attribute set. Specifically, it can implement the processing functions of steps 103 and 104 above, as well as other implicit steps.

[0216] In one possible implementation, the screening module 2120 is used for:

[0217] The target attribute is filtered out from the first attribute set to obtain the second attribute set, wherein the distribution characteristics of the data of the target attribute satisfy the distribution filtering condition, and / or the association characteristics of the data of the target attribute with the data of other attributes satisfy the association filtering condition.

[0218] In one possible implementation, the distribution screening conditions include at least one of the following screening conditions:

[0219] The proportion of null values ​​in the target attribute data reaches a first proportion threshold.

[0220] The number of different values ​​in the target attribute data is less than the number threshold.

[0221] In one possible implementation, the association screening condition includes at least one of the following screening conditions:

[0222] The correlation between the data of the target attribute and the data of other attributes is greater than a first correlation threshold.

[0223] The correlation between the data of the target attribute and the data of other attributes is less than the second correlation threshold.

[0224] The other attributes belong to the first attribute set.

[0225] In one possible implementation, the apparatus further includes a generation module for:

[0226] Generate a second model creation instruction carrying a second set of attributes. The second model creation instruction is used to instruct the data corresponding to the second set of attributes to be sent to the AI ​​server.

[0227] The execution module 2130 is used to: in response to the second model creation instruction, send the data corresponding to the second attribute set stored in the database server to the AI ​​server.

[0228] In one possible implementation, the device further includes a retraining module for:

[0229] The system receives instruction information from the AI ​​server after creating the AI ​​model, wherein the instruction information is used to directly or indirectly indicate the importance of at least one attribute in the second attribute set to model training. Specifically, this can implement the processing function of step 801 above, as well as other implicit steps.

[0230] Based on the indicated information, a third attribute set is generated and recorded. Specifically, this implements the processing function of step 802 above, as well as other implicit steps.

[0231] Receive the model retraining instruction corresponding to the AI ​​model. Specifically, this can implement the processing function of step 1501 above, as well as other implicit steps.

[0232] The data corresponding to the third attribute set stored in the database server is sent to the AI ​​server, which then retrains the AI ​​model based on the data corresponding to the third attribute set. Specifically, this implements the processing functions of steps 1502 and 1503 above, as well as other implicit steps.

[0233] The system receives the retrained AI model from the AI ​​server. Specifically, it can implement the processing function of step 1504 above, as well as other implicit steps.

[0234] In one possible implementation, the device further includes a retraining module for:

[0235] A specified number of data points corresponding to the first attribute set are sent to the AI ​​server, which then determines the indication information based on these data points. Specifically, this implements the processing functions of steps 1101 and 1102 above, as well as other implicit steps.

[0236] The instruction information fed back by the AI ​​server is received. Specifically, this can implement the processing function of step 1103 above, as well as other implicit steps.

[0237] Based on the indicated information, a third attribute set is generated and recorded. Specifically, this implements the processing function of step 1104 above, as well as other implicit steps.

[0238] Receive the model retraining instruction corresponding to the AI ​​model. Specifically, this can implement the processing function of step 1501 above, as well as other implicit steps.

[0239] The data corresponding to the third attribute set stored in the database server is sent to the AI ​​server, which then retrains the AI ​​model based on the data corresponding to the third attribute set. Specifically, this implements the processing functions of steps 1502 and 1503 above, as well as other implicit steps.

[0240] The system receives the retrained AI model from the AI ​​server. Specifically, it can implement the processing function of step 1504 above, as well as other implicit steps.

[0241] In one possible implementation, the retraining module is further configured to:

[0242] Before sending a specified number of data points corresponding to the first attribute set to the AI ​​server, it is determined that the data corresponding to the first attribute set has changed compared to the last time the third attribute set was recorded, and the change ratio reaches a second ratio threshold. Specifically, this can implement the processing function of step 1101 above, as well as other implicit steps.

[0243] In one possible implementation, the indication information includes attributes selected by the AI ​​server for model training. The third set of attributes consists of the attributes selected by the AI ​​server for model training.

[0244] In one possible implementation, the indication information includes importance values ​​corresponding to each of the at least one attribute. The third attribute set consists of attributes whose importance values ​​are greater than an importance threshold.

[0245] In one possible implementation, the device further includes a reasoning module for:

[0246] Receive the attributes fed back by the AI ​​server. Specifically, this can implement the processing functions of steps 1801 and 1802 above, as well as other implicit steps.

[0247] Receive model inference instructions corresponding to the AI ​​model. Specifically, this can implement the processing function of step 1803 above, as well as other implicit steps.

[0248] The data corresponding to the model inference instructions and the attributes fed back by the AI ​​server are input into the AI ​​model for model inference to obtain the result data. Specifically, this can implement the processing function of step 1804 above, as well as other implicit steps.

[0249] In this embodiment, the first set of attributes carried in the model creation instruction is filtered out. Because the attributes used to create the AI ​​model are reduced through this filtering process, the amount of data that the database server needs to transmit to the AI ​​server during the AI ​​model creation process is reduced, improving data transmission efficiency and thus increasing the efficiency of AI model creation.

[0250] The receiving module 2110, the screening module 2120, and the execution module 2130 can all be implemented in software or in hardware. For example, the implementation of the receiving module 2110 will be described below. Similarly, the implementation of the screening module 2120 and the execution module 2130 can refer to the implementation of the receiving module 2110.

[0251] As an example of a software functional unit, the receiving module 2110 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the receiving module 2110 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0252] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0253] As an example of a hardware functional unit, the receiving module 2110 may include at least one computing device, such as a server. Alternatively, the receiving module 2110 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0254] The multiple computing devices included in the receiving module 2110 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the receiving module 2110 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the receiving module 2110 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0255] It should be noted that, in other embodiments, the receiving module 2110, the screening module 2120, and the execution module 2130 can be used for any step in the method of creating an AI model. The steps implemented by the receiving module 2110, the screening module 2120, and the execution module 2130 can be specified as needed. The receiving module 2110, the screening module 2120, and the execution module 2130 respectively implement different steps in the method of creating an AI model to realize all the functions of the device for creating an AI model.

[0256] This disclosure also provides a computing device 100. As shown in FIG22, the computing device 100 includes: a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other via the bus 102. The computing device 100 may be a server or a terminal device. It should be understood that this disclosure does not limit the number of processors and memories in the computing device 100.

[0257] Bus 102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 23, but this does not imply that there is only one bus or one type of bus. Bus 102 can include pathways for transmitting information between various components of computing device 100 (e.g., memory 106, processor 104, communication interface 108).

[0258] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0259] The memory 106 may include volatile memory, such as random access memory (RAM). The memory 106 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0260] The memory 106 stores executable program code, which the processor 104 executes to implement the functions of the aforementioned receiving module 2110, filtering module 2120, and execution module 2130, thereby realizing the method for creating an AI model. That is, the memory 106 stores instructions for creating an AI model.

[0261] The communication interface 108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.

[0262] Based on the same technical concept, this disclosure also provides a database system, which includes a database server on which the above-described apparatus for creating AI models is deployed. In one possible embodiment, the database system further includes an AI server, wherein the database server and the AI ​​server are used to execute the method steps corresponding to any possible embodiment in the above method embodiments. The implementation methods of the database server and the AI ​​server can both refer to the above description of the implementation methods of the apparatus for creating AI models, and will not be repeated here.

[0263] This disclosure also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0264] As shown in Figure 23, the computing device cluster includes at least one computing device 100. The memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for creating AI models.

[0265] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for creating AI models. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for creating AI models.

[0266] It should be noted that the memory 106 in different computing devices 100 within the computing device cluster can store different instructions, each used for a portion of the functions of the device for creating the AI ​​model. That is, the instructions stored in the memory 106 of different computing devices 100 can implement the functions of one or more modules among the aforementioned receiving module 2110, filtering module 2120, and execution module 2130.

[0267] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), and can be a Transmission Control Protocol (TCP) network or a Remote Direct Memory Access (RDMA) network. Figure 24 illustrates one possible implementation. As shown in Figure 24, two computing devices 100A and 100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 106 in computing device 100A stores instructions for performing the functions of the receiving module 2110 and the filtering module 2120. Simultaneously, the memory 106 in computing device 100B stores instructions for performing the functions of the execution module 2130.

[0268] The connection method between the computing device clusters shown in Figure 24 can be such that, considering the method for creating AI models provided in this disclosure requires a large amount of data storage, the functions implemented by the execution module 2130 are delegated to the computing device 100B for execution.

[0269] It should be understood that the functions of computing device 100A shown in Figure 24 can also be performed by multiple computing devices 100. Similarly, the functions of computing device 100B can also be performed by multiple computing devices 100.

[0270] This disclosure also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device clusters shown in Figures 23 and 24. The difference is that the memory 106 of one or more computing devices 100 in this computing device cluster can store the same instructions for creating AI models.

[0271] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for creating AI models. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for creating AI models.

[0272] This disclosure also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored on any available medium. When the computer program product is run on at least one cluster of computing devices, it enables at least one computing device to create an AI model.

[0273] This disclosure also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a cluster of computing devices, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disk, DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to create an AI model, or instruct the computing device to perform a method for creating an AI model.

[0274] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A method for creating an artificial intelligence (AI) model, characterized in that, The method is applied to a database server, and the method includes: The system receives a first model creation instruction carrying a first set of attributes, wherein the first set of attributes includes multiple attributes, and the first model creation instruction is used to instruct the creation of an AI model based on the data corresponding to the first set of attributes stored in the database server. Based on the data corresponding to the first attribute set, the multiple attributes included in the first attribute set are filtered out to obtain the second attribute set. The data corresponding to the second attribute set stored in the database server is sent to the AI ​​server, and the AI ​​server is used to create the AI ​​model based on the data corresponding to the second attribute set. Receive the AI ​​model fed back from the AI ​​server.

2. The method according to claim 1, characterized in that, The second attribute set is obtained by filtering out multiple attributes included in the first attribute set based on the data corresponding to the first attribute set, including: The target attribute is filtered out from the first attribute set to obtain the second attribute set, wherein the distribution characteristics of the data of the target attribute satisfy the distribution filtering condition, and / or the association characteristics of the data of the target attribute with the data of other attributes satisfy the association filtering condition.

3. The method according to claim 2, characterized in that, The distribution screening conditions include at least one of the following screening conditions: The proportion of null values ​​in the target attribute data reaches a first proportion threshold. The number of different values ​​in the target attribute data is less than the number threshold.

4. The method according to claim 2 or 3, characterized in that, The associated screening conditions include at least one of the following screening conditions: The correlation between the data of the target attribute and the data of other attributes is greater than a first correlation threshold; The correlation between the data of the target attribute and the data of other attributes is less than a second correlation threshold; The other attributes belong to the first attribute set.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Receive the instruction information fed back by the AI ​​server after the AI ​​model is created; Based on the indicated information, a third attribute set is generated and the third attribute set is recorded; Receive the model retraining instruction corresponding to the AI ​​model; The data corresponding to the third attribute set stored in the database server is sent to the AI ​​server, and the AI ​​server is used to retrain the AI ​​model based on the data corresponding to the third attribute set. Receive the retrained AI model from the AI ​​server.

6. The method according to any one of claims 1-4, characterized in that, The method further includes: A specified number of data corresponding to the first attribute set are sent to the AI ​​server, and the AI ​​server is used to determine indication information based on the specified number of data corresponding to the first attribute set. Receive the instruction information fed back by the AI ​​server; Based on the indicated information, a third attribute set is generated and the third attribute set is recorded; Receive the model retraining instruction corresponding to the AI ​​model; The data corresponding to the third attribute set stored in the database server is sent to the AI ​​server, and the AI ​​server is used to retrain the AI ​​model based on the data corresponding to the third attribute set. Receive the retrained AI model from the AI ​​server.

7. The method according to claim 6, characterized in that, Before sending the specified number of data corresponding to the first attribute set to the AI ​​server, the method further includes: It is determined that the data corresponding to the first attribute set has changed relative to the last time the third attribute set was recorded, and the change ratio reaches the second ratio threshold.

8. The method according to any one of claims 5-7, characterized in that, The indication information includes the attributes selected by the AI ​​server for model training; The third set of attributes consists of attributes selected by the AI ​​server for model training.

9. The method according to any one of claims 5-7, characterized in that, The indication information includes the importance value corresponding to each of the at least one attribute; The third set of attributes consists of attributes whose importance value is greater than the importance threshold.

10. The method according to any one of claims 1-9, characterized in that, After filtering out multiple attributes included in the first attribute set based on the data corresponding to the first attribute set to obtain the second attribute set, the method further includes: Generate a second model creation instruction carrying a second set of attributes, the second model creation instruction being used to instruct the data corresponding to the second set of attributes to be sent to the AI ​​server; Sending the data corresponding to the second attribute set stored in the database server to the AI ​​server includes: in response to the second model creation instruction, sending the data corresponding to the second attribute set stored in the database server to the AI ​​server.

11. A database system, characterized in that, The database system includes a database server, which is used for: The system receives a first model creation instruction carrying a first set of attributes, wherein the first set of attributes includes multiple attributes, and the first model creation instruction is used to instruct the creation of an AI model based on the data corresponding to the first set of attributes stored in the database server. Based on the data corresponding to the first attribute set, the multiple attributes included in the first attribute set are filtered out to obtain the second attribute set. The data corresponding to the second attribute set stored in the database server is sent to the AI ​​server, and the AI ​​server is used to create the AI ​​model based on the data corresponding to the second attribute set. Receive the AI ​​model fed back from the AI ​​server.

12. The database system according to claim 11, characterized in that, The database server is used for: The target attribute is filtered out from the first attribute set to obtain the second attribute set, wherein the distribution characteristics of the data of the target attribute satisfy the distribution filtering condition, and / or the association characteristics of the data of the target attribute with the data of other attributes satisfy the association filtering condition.

13. The database system according to claim 12, characterized in that, The distribution screening conditions include at least one of the following screening conditions: The proportion of null values ​​in the target attribute data reaches a first proportion threshold. The number of different values ​​in the target attribute data is less than the number threshold.

14. The database system according to claim 12 or 13, characterized in that, The associated screening conditions include at least one of the following screening conditions: The correlation between the data of the target attribute and the data of other attributes is greater than a first correlation threshold; The correlation between the data of the target attribute and the data of other attributes is less than a second correlation threshold; The other attributes belong to the first attribute set.

15. The database system according to any one of claims 11-14, characterized in that, The database server is also used for: Receive the instruction information fed back by the AI ​​server after the AI ​​model is created; Based on the indicated information, a third attribute set is generated and the third attribute set is recorded; Receive the model retraining instruction corresponding to the AI ​​model; The data corresponding to the third attribute set stored in the database server is sent to the AI ​​server, and the AI ​​server is used to retrain the AI ​​model based on the data corresponding to the third attribute set. Receive the retrained AI model from the AI ​​server.

16. The database system according to any one of claims 11-14, characterized in that, The database server is also used for: A specified number of data corresponding to the first attribute set are sent to the AI ​​server, and the AI ​​server is used to determine indication information based on the specified number of data corresponding to the first attribute set. Receive the instruction information fed back by the AI ​​server; Based on the indicated information, a third attribute set is generated and the third attribute set is recorded; Receive the model retraining instruction corresponding to the AI ​​model; The data corresponding to the third attribute set stored in the database server is sent to the AI ​​server, and the AI ​​server is used to retrain the AI ​​model based on the data corresponding to the third attribute set. Receive the retrained AI model from the AI ​​server.

17. The database system according to claim 16, characterized in that, The database server is also used for: It is determined that the data corresponding to the first attribute set has changed relative to the last time the third attribute set was recorded, and the change ratio reaches the second ratio threshold.

18. The database system according to any one of claims 15-17, characterized in that, The indication information includes the attributes selected by the AI ​​server for model training; The third set of attributes consists of attributes selected by the AI ​​server for model training.

19. The database system according to any one of claims 15-17, characterized in that, The indication information includes the importance value corresponding to each of the at least one attribute; The third set of attributes consists of attributes whose importance value is greater than the importance threshold.

20. The database system according to any one of claims 11-19, characterized in that, The database server is also used for: Generate a second model creation instruction carrying a second set of attributes, the second model creation instruction being used to instruct the data corresponding to the second set of attributes to be sent to the AI ​​server; The database server is used for: In response to the second model creation instruction, the data corresponding to the second attribute set stored in the database server is sent to the AI ​​server.

21. The database system according to any one of claims 11-20, characterized in that, The database system also includes the AI ​​server.

22. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-10.

23. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1-10.

24. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1-10.

Images