Effective model replacement technique for resolving objective mismatch in composition of pre-trained machine learning models

Abstract

Disclosed is an effective model replacement technique for resolving an objective mismatch in a composition of pre-trained machine learning models. A pre-trained model (PTM) replacement method performed by a model replacement system according to an embodiment may comprise: a sampling step of sampling user input data via a density and diversity sampling algorithm; an inference step of collecting performance data by replacing a specific model in a composition with some of a plurality of candidate pre-trained models by using the sampled user input data; and a prediction step of predicting expected performance of the remaining candidate pre-trained models by training a meta-prediction model on the basis of the collected performance data.

Description

Effective model replacement techniques for resolving goal mismatches in combinations of pre-trained machine learning models

[0001] The following description concerns model replacement techniques for resolving goal mismatches that may occur when using combinations of pre-trained machine learning models (PTMs).

[0002] Pre-trained models (PTMs) have driven innovation in machine learning by significantly reducing the resources required for model training and greatly improving reusability. Despite these advantages, selecting a pre-trained model suitable for specific system requirements remains challenging due to application heterogeneity and performance variability. To address this, combinations of pre-trained models have been proposed to extend functionality beyond individual models, and on-the-fly approaches are being applied to meet dynamic requirements. However, on-the-fly combinations of pre-trained models can cause objective mismatches, leading to inefficiencies and errors in the combined models. Therefore, underperforming pre-trained models must be replaced efficiently and in a timely manner. This, however, is a very difficult task due to the vast pool of candidates and the enormous amount of time required for evaluation.

[0003] Combinations of existing pre-trained models (PTMs) can lead to performance degradation and errors due to inconsistencies between the inputs and outputs of different models; to resolve this, there is a problem of needing to quickly and efficiently find an optimal alternative pre-trained model.

[0004] To address this, the embodiment introduces a Sample-Infer-Predict framework to quickly and accurately identify the optimal pre-trained model among a large pool of model candidates.

[0005] A method for replacing pre-trained models (PTMs) performed by a model replacement system may include: a sampling step of sampling user input data through a density and diversity sampling algorithm; an inference step of collecting performance data by replacing a specific model within a combination with some of a plurality of candidate pre-trained models using the sampled user input data; and a prediction step of predicting the expected performance of the remaining candidate pre-trained models by training a meta-prediction model based on the collected performance data.

[0006] The above sampling step includes the step of extracting feature information from the user input data through a feature extraction model, generating an embedding vector using the extracted feature information, and calculating a density score of the generated embedding vector, wherein the feature extraction model includes Visual Transformer (ViT) or Word2Vec depending on whether the user input data is image or text data, and the density score can be calculated as the average distance between the generated embedding vector and the embedding vector of the nearest neighbor using a nearest neighbor approach.

[0007] The sampling step may include grouping the generated embedding vectors based on the density score of the calculated embedding vectors, and selecting user input data by considering areas with a density greater than or equal to a preset standard and areas with a density less than or equal to a preset standard for the clusters generated through the grouping.

[0008] The above sampling step may include the step of labeling the selected user input data by the user to generate sample-label pairs.

[0009] The above inference step may include the step of obtaining an output result by performing inference on combinations replaced with some candidate pre-trained models using the sampled user input data and the labels of the sampled user input data, and generating performance data by comparing the obtained output result with the labels.

[0010] The above inference step may include a step of configuring a pair containing descriptive information of a pre-trained model and the generated performance data into a meta-prediction dataset used for meta-prediction learning.

[0011] The prediction step described above includes the step of preprocessing a meta-prediction dataset and training a meta-prediction model using the preprocessed meta-prediction dataset. The preprocessing involves normalizing the meta-prediction dataset if it is numerical data and converting it into a binary form through one-hot encoding if it is categorical data. The meta-prediction model may be selected from a regression model or a classification model depending on the type of performance indicator.

[0012] The prediction step may include a step of ranking a plurality of candidate pre-trained models according to the expected performance of the remaining candidate pre-trained models predicted by the learned meta-prediction model, and determining the pre-trained model with the best prediction performance as a replacement model for the existing model in the combination.

[0013] A computer program stored in a computer-readable storage medium for executing a pre-trained model (PTM) replacement method performed by a model replacement system, wherein the pre-trained model replacement method may include: a sampling step of sampling user input data through a density and diversity sampling algorithm; an inference step of collecting performance data by replacing a specific model within a combination with some of a plurality of candidate pre-trained models using the sampled user input data; and a prediction step of predicting the expected performance of the remaining candidate pre-trained models by training a meta-prediction model based on the collected performance data.

[0014] A model replacement system may include: a sampling unit that samples user input data through a density and diversity sampling algorithm; an inference unit that collects performance data by replacing a specific model within a combination with some of a plurality of candidate pre-trained models using the sampled user input data; and a prediction unit that predicts the expected performance of the remaining candidate pre-trained models by training a meta-prediction model based on the collected performance data.

[0015] Compared to the existing pre-trained model replacement method, processing time is significantly reduced, enabling rapid response in real-time applications (replacement time: 155 seconds vs. existing 28,293 seconds).

[0016] The performance prediction accuracy of the alternative pre-trained model is high, allowing for the reliable selection of the optimal pre-trained model (Precision@k value: maintained at a maximum of 1.0).

[0017] It can reduce user labeling and computation costs by maintaining high accuracy even with a small number of input samples (low Mean Absolute Error (MAE) with fewer than 20 samples).

[0018] In actual applications, efficiency and accuracy can be demonstrated by verifying the effects of the sampling, inference, and prediction steps in cases involving combinations of pre-trained vehicle detection models.

[0019] Figure 1 is a diagram illustrating the life cycle of a PTM combination.

[0020] Figure 2 is an example of an abstract process.

[0021] Figure 3 is an example showing GPU profiler data illustrating input processing and inference times.

[0022] FIG. 4 is a diagram illustrating a pre-trained model substitution operation in one embodiment.

[0023] FIG. 5 is a diagram illustrating a sampling operation in one embodiment.

[0024] FIG. 6 is a diagram showing an algorithm for explaining density and diversity sampling operations in one embodiment.

[0025] FIG. 7 is a diagram illustrating an inference operation in one embodiment.

[0026] FIG. 8 is a block diagram illustrating a model replacement system in one embodiment.

[0027] FIG. 9 is a flowchart illustrating a pre-trained model substitution method in one embodiment.

[0028] Hereinafter, embodiments will be described in detail with reference to the attached drawings.

[0029]

[0030] Figure 1 is a diagram illustrating the lifecycle of a pre-trained model combination.

[0031] To facilitate understanding of the explanation of the model replacement behavior proposed in the embodiments, we will first describe the pre-trained model combinations. To this end, we will define the pre-trained model combination framework, the instant pre-trained model combination lifecycle, and the pre-trained model replacement process.

[0032] We will now describe the pre-trained model combination framework. To effectively utilize pre-trained models, it is assumed that comprehensive model descriptions, typically provided by Model Hubs, are available. These descriptions may include details such as the target task (e.g., image classification), training information (e.g., hyperparameters, optimizer), performance metrics (e.g., accuracy, IoU), training datasets, and base models. This information is essential for understanding the performance and limitations of pre-trained models and for evaluating expected performance.

[0033] However, public model hubs may have incomplete model descriptions and may not include essential information such as training details or performance metrics. This lack of detail can make it difficult to fully understand the performance and limitations of the model. Additionally, the provided metrics may be unreliable or vary significantly in specialized contexts. In the embodiments, it is assumed that essential information (training details, performance metrics) is available to ensure a thorough and reliable analysis.

[0034] We will now describe the lifecycle of pre-trained model compositions. The lifecycle of an instant pre-trained model composition can be defined by applying a framework in web service composition. As shown in Figure 1, the lifecycle of a pre-trained model composition can be divided into four main stages: Composition Definition, Pre-trained Model Selection (PTM Selection), Deployment, and Execution. It can be initiated when user u requests an instant pre-trained model composition. At this time, the composition request may include all requirements (functional and non-functional requirements) for user u's pre-trained model composition, including both functional and non-functional requirements. For example, the composition request may include detecting a specific vehicle in a collection of very low-resolution images.

[0035] In the first step, the combination definition, the requirements of user u can be classified into an abstract process represented as a series of tasks including control and data flow. Through the model description of the hub, a list of potential tasks achievable with existing pre-trained models can be created. Referring to Figure 2, this is an example of an abstract process for vehicle detection.

[0036] In the second step, pre-trained model selection, a pre-trained model to perform the tasks of the abstract process can be selected based on available model descriptions. Consequently, an instantiated combinatorial process can be generated in which all necessary combinatorial elements are selected, even though it is not yet executable.

[0037] In the third step, deployment, pre-trained models selected according to user u's requirements (e.g., monolithic, microservices, serverless) can be packaged. Consequently, the combination generated for user u to execute can be deployed.

[0038] In the final execution stage, pre-trained model combinations are run and can be continuously monitored by the combination runtime. Here, changes in distribution caused by target mismatch can also be detected through monitoring. Additionally, inputs I from all unlabeled users u are collected and recorded by the runtime, which can be utilized in the instant replacement process. Pre-trained models whose performance has degraded due to distribution changes are also detected, and the lifecycle returns to the pre-trained model selection stage to find suitable candidates through replacement.

[0039] Next, we will describe the On-the-fly PTM Replacement operation. When a warning is issued regarding performance degradation due to goal mismatch, it is necessary to replace a problematic pre-trained model experiencing a distribution shift. However, the replacement process is complex due to the large number of available candidate pre-trained models. We assume that multiple syntactically replaceable pre-trained model alternatives exist within the Model Hub. These alternatives are designed to perform tasks similar to the problematic pre-trained model being replaced (e.g., image classification), ensuring compatibility and minimizing integration issues. Additionally, we assume that essential information, such as model descriptions, metadata, and reported performance metrics, is identified and accessible within the Model Hub.

[0040] There are several challenges that must be addressed to ensure a seamless and efficient replacement process in dynamic and complex on-the-spot pre-trained model replacement environments. One significant challenge is the incompleteness of the model descriptions provided by Model Hubs. For example, the descriptions may lack detailed specifications of the training data, performance metrics under various conditions, and information regarding model limitations. This incomplete information makes it difficult to accurately assess how well a model will perform in a specific context, thereby hindering the guarantee of reliable and effective replacement.

[0041] The approach of inserting replacement pre-trained models and evaluating their performance through user input is promising, but it is time-consuming due to the large number of candidate sets and input samples. Extensive testing and performance evaluation for each candidate pre-trained model delay the replacement process. Furthermore, the speed of inference execution on all user inputs is very slow. Often, the time required to process inputs and load them onto the GPU exceeds the actual inference time.

[0042] Referring to Figure 3, this is an example showing GPU profiler data illustrating input processing and inference times. The x-axis represents time, and the y-axis represents various threads and streams. The top section represents the main Python threads that primarily handle input processing, and the bottom section represents GPU streams that take relatively little time for actual inference.

[0043] FIG. 4 is a diagram illustrating a pre-trained model substitution operation in one embodiment.

[0044] To solve the problem presented above, the embodiment proposes a new on-the-fly pre-trained model replacement framework for optimizing the replacement process. As shown in Figure 4, a three-stage methodology (Sample-Infer-Predict) consisting of a sampling stage, an inference stage, and a prediction stage will be described.

[0045] The sampling step can select a representative small sample (a subset of data) from the user input data through a Density and Diversity sampling algorithm that considers the density and diversity of the data. The Density and Diversity sampling algorithm can improve the accuracy of evaluating the pre-trained model by sampling that includes rare cases that the pre-trained model cannot handle well, while still representing the input distribution.

[0046] The inference step can collect performance data by performing inference on sampled user input data using pre-trained models of surrogate candidates. This is an essential process to bridge the gap between model descriptions and actual performance.

[0047] The prediction step can predict the expected performance of the remaining candidate pre-trained models by training a meta-prediction model based on the collected performance data. At this time, the meta-prediction model can efficiently estimate performance by utilizing information listed in the description of each candidate pre-trained model (e.g., accuracy, model based on the training data set, number of downloads and likes, etc.).

[0048] More specifically, the sampling step focuses on reducing the number of inputs required to evaluate the pre-trained model through sampling. Fewer inputs lead to faster computation speeds, which can improve the efficiency of the replacement process. Additionally, reducing the number of inputs decreases the user's labeling workload, thereby enhancing the practicality and usability of the methodology.

[0049] In the embodiments, a new Density and Diversity sampling algorithm may be introduced. The Density and Diversity algorithm can be designed to carefully select a subset of inputs that accurately represents the entire input distribution and considers various cases that may lead to target mismatch between pre-trained models. These diverse cases are important because they are likely to cause performance deviations that might not be detected in general inputs. In the embodiments, by including both general and diverse cases, it is ensured that the selected samples comprehensively represent the input distribution. This ensures that pre-trained model replacement candidates perform robustly in both diverse and general scenarios, which is essential for maintaining consistent performance even with target mismatch.

[0050] First, embedding vectors can be generated by extracting features from the collected user input. Then, a sampling algorithm can select a subset of valid samples based on density and diversity thresholds. Finally, the user can label these samples to generate sample-label pairs to be used in the next inference step.

[0051] 1) Feature Extraction and Density Scoring: Feature extraction constitutes a fundamental step in the analysis and interpretation of user input and may involve the process of transforming raw data into a set of quantifiable features. Embedding vectors, which represent the transformed input as points in a high-dimensional space, facilitate the application of distance-based techniques such as cosine similarity. These techniques are essential for sampling because they allow for the selection of both representative and diverse samples by measuring similarity and diversity between data points. The specific method of feature extraction depends on the type of input data. For example, image features can be extracted using Visual Transformers (ViT), whereas text data can be extracted using Word2Vec.

[0052] After generating embedding vectors, the next step is to calculate a density score for each embedding. Calculating the density score is essential for determining the commonality of inputs and is a critical element of the sampling algorithm. The density score can be calculated using the nearest neighbor approach. Here, the density of an embedding can be calculated as the average distance between that embedding vector and the embedding vector of its nearest neighbor. A higher density score indicates that the input is closely related to other inputs, suggesting commonality within the user input. This average distance can be recorded as the density score for the corresponding embedding.

[0053] 2) Density and Diversity Sampling: To enhance the representativeness and diversity of selected samples from large-scale inputs, a density and diversity sampling algorithm can be designed that incorporates three key criteria: similarity, density, and diversity. Referring to Figure 5, the concept of the user input space, the familiar space of the pre-trained model, and the sampling process considering these spaces is illustrated. The shaded area represents the familiar space where the pre-trained model to be replaced is well-trained and capable of making accurate predictions. Not all user inputs exist within the shaded area, indicating that some user inputs may cause confusion in the pre-trained model's decision-making. Similarly, different circles within this space represent clusters of input data points, and each cluster can group inputs that share similar characteristics.

[0054] The density and diversity sampling algorithm proposed in the embodiments aims to select a diverse set of inputs by considering both high-density and low-density regions within the familiar space of the pre-trained model. High-density regions, where circle clusters are dense, represent common inputs that are closely related to one another. Low-density regions, where circles are small or isolated, represent rare inputs that are uncommon in the familiar space of the pre-trained model. This density and diversity sampling algorithm can prioritize low-density regions so that more unique samples are selected. In this way, the resulting samples can include both common inputs familiar to the pre-trained model and rare inputs that are difficult for the pre-trained model to process.

[0055] Referring to Figure 6, this figure illustrates an algorithm for explaining density and diversity sampling operations. The density and diversity sampling algorithm can use embedding vectors, density scores, and user-specified similarity, density, and diversity thresholds as inputs. The sampling process begins by establishing a set of representative embeddings that is initialized with an empty tensor (line 1). These representative embeddings are constructed incrementally by iterating through the input embeddings, and the input embeddings can be normalized and evaluated based on predefined thresholds for similarity, density, and diversity. To maintain diversity, the cosine similarity between new candidate embeddings and the existing set of representative embeddings can be calculated (line 12). Only candidates with a similarity lower than a specific threshold are selected, thereby ensuring that new samples are not too similar to already selected samples, thus maintaining diversity within the dataset.

[0056] At the same time, the density score of each embedding can be evaluated by comparing it to a threshold derived from the percentile of the total density score (3rd line) (13th line). This criterion ensures that selected samples are not extracted from excessively dense regions of the dataset, thereby providing a more balanced and comprehensive representation.

[0057] In addition, a diversity threshold is applied, and the diversity score (1 minus similarity) can be used to determine whether the new embedding contributes significantly to the diversity of the representative set (line 14). Candidates that meet this criterion and have a density score below the threshold can be added to the representative set.

[0058] This process continues until a maximum number of samples are selected or all embeddings are evaluated, and can return the sampled embeddings and selected indices. Consequently, a subset of embeddings possessing both representativeness and diversity is generated, which can provide a strong selection criterion for subsequent evaluation in the next inference step.

[0059] 3) Sample Labeling: Sample labeling is the process of generating sample-label pairs by annotating input samples, which can increase the robustness of evaluating the performance of pre-trained models and identify optimal replacement samples. While user labeling is essential for evaluating and replacing pre-trained models, the embodiments focus on this process assuming high-quality labeled data. The embodiments focus on evaluating and replacing pre-trained models while avoiding the complexity of the labeling process.

[0060] The inference step involves inferring from previously sampled inputs. The goal of the inference step is to evaluate a portion of the candidate pre-trained models and record their performance within a synthetic context. For example, if there are 150 replacement candidates in the Model Hub, only 40 of them can be executed in synthesis. This partial selection reduces overall computation time, making the on-the-fly replacement process more efficient and rapid for the user. The selection of candidates to execute in the inference step can be performed randomly.

[0061] Referring to Figure 7, this is a diagram illustrating the inference operation. Initially, samples and their corresponding labels obtained in the previous step can be used. These samples can be used as input to execute inference on the current synthesis. In the inference step, the output generated by the synthesis can be compared with user-provided labels. This comparison between the actual labels and the predicted outputs is crucial for identifying performance metrics of the synthesis. Through this iterative inference, a set of model description-performance pairs encapsulating the performance data of each alternative model can be collected. These results form a meta-prediction dataset, which can be used to train the meta-prediction model in the next step.

[0062] While various types of information are provided in model descriptions, reported performance (e.g., accuracy), the dataset used for fine-tuning and training the base model, the base model, and likes and downloads can be utilized in particular. Reported performance, the dataset, and the base model are objective attributes of the pre-trained model and can provide insights into the model's background and performance. However, this information is self-reporting and is not always accurate or reliable for direct use. Therefore, the number of likes and downloads can be included as additional indicators of the model's validity and reliability.

[0063] As a result of the inference step, a predictive dataset containing actual performance within the configuration can be collected by comparing the model descriptions of various replacement candidates with user-provided labels. This meta-predictive dataset can be utilized in the next step of an on-the-fly replacement method to predict the performance of models that were not directly inferred.

[0064] The goal of the prediction phase is to accurately predict the performance of candidates not evaluated in the inference phase, thereby generating a more complete set of candidate information more efficiently. The prediction phase can begin with the preprocessing of the meta-prediction dataset compiled in the inference phase. Preprocessing may include normalizing and encoding model explanatory data to make it suitable for training the meta-prediction model. Specifically, numerical data can be normalized to a common scale, and text data can be converted into a binary format using one-hot encoding for categorical information. Here, the dataset, underlying model information, and categorical information can all be considered.

[0065] The preprocessed dataset is used to train a meta-predictive model, where model descriptions can be used as training features and inference performance as training labels. The meta-predictive model may vary depending on the type of performance metric recorded during the inference phase. If the performance metric is continuous, such as accuracy, regression models such as linear regression, random forest regression, or neural networks can be used. If the performance metric is categorical, classification models such as logistic regression, decision trees, random forests, support vector machines, or neural networks can be used.

[0066] The primary function of the meta-prediction model is to predict the performance of each potential alternative model within the context of a pre-trained model composition. Pre-trained models evaluated during the inference phase are not included here, as their actual performance is already known. Once the meta-prediction model is trained and performance predictions are derived, these predictions can be used to rank the best alternative models. These rankings are based on predicted performance metrics and can ensure the priority of models most likely to demonstrate superior performance within the combination. Consequently, the highest-ranked model in this list can be selected for integration into the pre-trained model combination.

[0067] According to the embodiments, the problem of goal mismatch occurring during the combination process of pre-trained machine learning models can be resolved by utilizing this method. When performance degradation or goal mismatch occurs in component models during model combination or pipelines, this method can be used to quickly identify and replace the optimal alternative model among multiple candidate models. Through this, the reliability and efficiency of AI systems can be enhanced in fields requiring real-time response, such as autonomous driving, medical image analysis, and industrial quality inspection.

[0068] FIG. 8 is a block diagram illustrating a model replacement system in one embodiment, and FIG. 9 is a flowchart illustrating a pre-trained model replacement method in one embodiment.

[0069] The processor of the model replacement system (100) may include a sampling unit (810), an inference unit (820), and a prediction unit (830). These components of the processor may be representations of different functions performed by the processor according to control instructions provided by program code stored in the model replacement system. The processor and the components of the processor may control the model replacement system to perform steps (910 to 930) included in the pre-learned model replacement method of FIG. 9. In this case, the processor and the components of the processor may be implemented to execute instructions according to the code of the operating system included in memory and the code of at least one program.

[0070] The processor can load program code stored in a file of a program for a pre-trained model replacement method into memory. For example, when a program is executed in a model replacement system, the processor can control the model replacement system to load program code from a file of a program into memory under the control of the operating system. At this time, the sampling unit (810), the inference unit (820), and the prediction unit (830) may each be different functional representations of the processor for executing instructions of corresponding parts of the program code loaded into memory to execute subsequent steps (910 to 930).

[0071] In step (910), the sampling unit (810) can sample user input data through a density and diversity sampling algorithm. The sampling unit (810) can extract feature information from user input data through a feature extraction model, generate an embedding vector using the extracted feature information, and calculate a density score of the generated embedding vector. At this time, the feature extraction model may include Visual Transformer (ViT) or Word2Vec depending on whether the user input data is image or text data, and the density score may be calculated as the average distance between the embedding vector generated using the nearest neighbor approach and the embedding vector of the nearest neighbor. The sampling unit (810) can group the generated embedding vectors based on the density score of the calculated embedding vectors, and select user input data by considering areas with a density greater than a preset standard and areas with a density less than a preset standard for the clusters generated through grouping. The sampling unit (810) can generate sample-label pairs by labeling the selected user input data by the user.

[0072] In step (920), the inference unit (820) can collect performance data by replacing a specific model within a combination with some of the candidate pre-trained models using the sampled user input data. The inference unit (820) can obtain an output result by performing inference on the combination replaced with some of the candidate pre-trained models using the sampled user input data and the label of the sampled user input data, and can generate performance data by comparing the obtained output result with the label. The inference unit (820) can configure a pair containing the descriptive information of the pre-trained model and the generated performance data into a meta-prediction dataset used for meta-prediction learning.

[0073] In step (930), the prediction unit (830) can predict the expected performance of the remaining candidate pre-trained models by training a meta-prediction model based on the collected performance data. The prediction unit (830) can preprocess the meta-prediction data set and train the meta-prediction model using the preprocessed meta-prediction data set. At this time, the preprocessing involves normalizing the meta-prediction data set if it is numerical data, and converting it into a binary form through one-hot encoding if it is categorical data. The meta-prediction model can be selected from a regression model or a classification model depending on the type of performance indicator. The prediction unit (830) can rank multiple candidate pre-trained models based on the expected performance of the remaining candidate pre-trained models predicted through the trained meta-prediction model, and determine the pre-trained model with the best prediction performance as a replacement model for the existing model in the combination.

[0074] According to the embodiment, model replacement time can be reduced by predicting the performance of the entire candidate pool through minimal sampling and partial inference from a large pool of pre-trained models. Furthermore, by considering the representativeness and diversity of user input data, it is possible to ensure that the pre-trained model effectively processes various inputs in actual operating environments. Additionally, user labeling tasks and computational costs can be minimized while maintaining the accuracy of performance predictions.

[0075] The model replacement method proposed in the example can be applied to a combination of pre-trained vehicle detection models, and its effectiveness can be verified through the following experiments. A dataset of 36 pre-trained image classification models provided by Hugging Face and 5,849 vehicle images can be used. Replacement time, meta-prediction accuracy (Precision@k), and sampling efficiency (MAE) can be evaluated. The proposed model replacement method reduced replacement time by 181 times compared to the existing full verification method while maintaining the performance prediction accuracy of the replacement models. In particular, by applying density and diversity algorithms, prediction accuracy was improved even with small-scale sampling.

[0076] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include a plurality of processing elements and / or a plurality of types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0077] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0078] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0079] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0080] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. In a pre-trained model (PTM) replacement method performed by a model replacement system, A sampling step that samples user input data through a density and diversity sampling algorithm; An inference step of collecting performance data by replacing a specific model within a combination with some of a plurality of candidate pre-trained models using the sampled user input data above; and A prediction step that predicts the expected performance of the remaining candidate pre-trained models by training a meta-prediction model based on the collected performance data above. A pre-trained model replacement method including 2. In Paragraph 1, The above sampling step is, A step of extracting feature information from the user input data through a feature extraction model, generating an embedding vector using the extracted feature information, and calculating a density score of the generated embedding vector. Includes, The above feature extraction model includes Visual Transformer (ViT) or Word2Vec depending on whether the user's input data is image or text data, and A pre-trained model replacement method characterized by the above density score being calculated as the average distance between the generated embedding vector and the embedding vector of the nearest neighbor using the nearest neighbor approach.

3. In Paragraph 2, The above sampling step is, A step of grouping the generated embedding vectors based on the density scores of the calculated embedding vectors, and selecting user input data by considering areas with a density greater than a preset standard and areas with a density less than a preset standard for the clusters generated through the grouping. A pre-trained model replacement method including 4. In Paragraph 3, The above sampling step is, The step of labeling the selected user input data by the user to generate sample-label pairs A pre-trained model replacement method including 5. In Paragraph 1, The above inference step is, A step of obtaining output results by performing inference on combinations replaced with some candidate pre-trained models using the sampled user input data and the labels of the sampled user input data, and generating performance data by comparing the obtained output results with the labels. A pre-trained model replacement method including 6. In Paragraph 5, The above inference step is, Step of configuring a pair containing explanatory information of a pre-trained model and the generated performance data into a meta-prediction dataset used for meta-prediction learning A pre-trained model replacement method including 7. In Paragraph 1, The above prediction step is, A step of preprocessing a meta-prediction dataset and training a meta-prediction model using the preprocessed meta-prediction dataset. Includes, The above preprocessing normalizes the meta-prediction dataset if it is numerical data, and converts it into a binary form through one-hot encoding if it is categorical data, and A pre-trained model replacement method characterized in that the above meta-prediction model is selected from a regression model or a classification model depending on the type of performance indicator.

8. In Paragraph 7, The above prediction step is, A step of ranking multiple candidate pre-trained models based on the expected performance of the remaining candidate pre-trained models predicted by the above-mentioned meta-prediction model, and determining the pre-trained model with the best prediction performance as a replacement model for the existing model within the combination. A pre-trained model replacement method including 9. A computer program stored on a computer-readable storage medium for executing a pre-trained model (PTM) replacement method performed by a model replacement system, The above-mentioned pre-trained model replacement method is, A sampling step that samples user input data through a density and diversity sampling algorithm; An inference step of collecting performance data by replacing a specific model within a combination with some of a plurality of candidate pre-trained models using the sampled user input data above; and A prediction step that predicts the expected performance of the remaining candidate pre-trained models by training a meta-prediction model based on the collected performance data above. A computer program stored on a computer-readable storage medium that executes.

10. In a model replacement system, A sampling unit that samples user input data through a density and diversity sampling algorithm; An inference unit that collects performance data by replacing a specific model within a combination with some of a plurality of candidate pre-trained models using the sampled user input data above; and A prediction unit that predicts the expected performance of the remaining candidate pre-trained models by training a meta-prediction model based on the collected performance data above. A model replacement system including

Images