Active learning system and method
The active learning system addresses annotation cost and accuracy challenges by selecting high-typicality and low-confidence data points and correcting overconfidence, ensuring high model performance across varying annotation volumes.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- HITACHI HIGH TECH CORP
- Filing Date
- 2024-10-28
- Publication Date
- 2026-05-07
AI Technical Summary
Existing active learning methods face challenges in reducing annotation costs and improving model accuracy, particularly when the number of annotable data points is small or large, as they often lead to data bias or do not effectively utilize data with low confidence or typicality.
An active learning system that calculates confidence and typicality levels of unlabeled data, selecting high-typicality and low-confidence data points for annotation, and corrects overconfidence in the model's confidence scores using integrated confidence calculation, thereby optimizing model training with reduced annotation costs.
The system achieves high model accuracy and reduces annotation costs by effectively utilizing unlabeled data, correcting overconfidence, and improving the performance of machine learning models regardless of the number of annotations possible.
Smart Images
Figure JP2024038333_07052026_PF_FP_ABST
Abstract
Description
Active learning systems and methods
[0001] This disclosure relates to machine learning techniques, particularly active learning.
[0002] In supervised machine learning, reducing annotation costs is a particularly important challenge. Improving the performance of machine learning models (i.e., inference accuracy) requires large amounts of data, but annotating (or labeling) all of this data is extremely costly. This challenge is even more serious in specialized fields such as manufacturing and healthcare, where accurate labeling requires enormous costs and time.
[0003] One way to reduce annotation costs is through active learning. Active learning is a technique that prioritizes and generates data useful for learning by interactively querying the user, and then performs annotation. In active learning, the system first selects multiple data points from unlabeled data that are useful for improving the performance of the machine learning model. The number of data points that can be selected at this time is referred to as the number of annotable data points. The data selected at this time (i.e., the data used to query the user) is referred to as the query. Next, the system interactively queries the user with this query. The user then makes a decision and performs annotation based on the query. The system adds the annotated data as labeled data. The system then trains the machine learning model using the updated labeled data. By repeating this process, the model can be optimized while keeping annotation costs down.
[0004] A common method of active learning is to prioritize selecting data with low confidence levels in the machine learning model. However, simply prioritizing low-confidence data can lead to a bias in queries towards similar data, and may not reduce annotation costs.
[0005] Therefore, Patent Document 1 describes that, in order to correct data bias while prioritizing data with low confidence, "the analysis unit calculates the confidence level of each estimation result of a machine learning model that estimates which of a predetermined set of classification destinations the input unsupervised training data belong to. The selection unit clusters the training data into multiple clusters, and after clustering, selects one or more training data starting with those with the lowest confidence levels."
[0006] JP 2021-47751 Hacohen, G., Dekel, A., and Weinshall, D. (2022). Active learning on a budget: Opposite strategies suit high and low budgets. In Proceedings of the 39th International Conference on Machine Learning.
[0007] As described in Patent Document 1, methods that prioritize data with low confidence levels or diversity when selecting data may not improve the accuracy of machine learning models and may not reduce annotation costs when the number of annotable data points is small.
[0008] On the other hand, when the number of annotable data points is small, selecting typical data, as described in Non-Patent Document 1, is an effective method. Typical data refers to data that has common characteristics among all the data. However, this method does not improve accuracy or reduce annotation costs when the number of annotable data points is large.
[0009] The purpose of this disclosure is to provide a technology that enables high performance (in other words, high model accuracy) regardless of the number of annotations possible, and reduces annotation costs, for the above-mentioned active learning technology.
[0010] A typical embodiment of this disclosure has the following configuration. One embodiment is an active learning system comprising a computer for actively training a machine learning model, the computer comprising: a model inference unit that obtains inference results by the machine learning model based on unlabeled data which is input data; a confidence calculation unit that calculates the confidence level of the unlabeled data based on the inference results; a typicality calculation unit that calculates the typicality of the unlabeled data based on the inference results; a low confidence data selection unit that selects low confidence data based on the confidence level of the unlabeled data; a high typicality data selection unit that selects high typicality data based on the typicality of the unlabeled data; and a data determination unit that selects annotation target data from the selected low confidence data and the low confidence data.
[0011] According to a representative embodiment of this disclosure, the active learning technology described above can achieve high performance (in other words, high model accuracy) regardless of the number of annotations, and can reduce annotation costs. Other issues, configurations, and effects are shown in the embodiments for carrying out the invention.
[0012] A diagram showing the configuration of the active learning system of this embodiment (Embodiment 1). A diagram showing an example configuration of the active learning system (computer system) of this embodiment. A diagram showing the flow of the active learning method using the active learning system of this embodiment. A diagram showing an example configuration of the data selection unit 109 in this embodiment. A diagram showing the processing flow of the data selection unit 109 in this embodiment. A diagram showing an example configuration of the confidence calculation unit 304 in this embodiment. An explanatory diagram explaining the confidence calculated by the cluster confidence calculation module 502 in this embodiment. An explanatory diagram explaining the effect of the confidence integration unit 503 in this embodiment. A schematic diagram showing an example of the initial setup screen in this embodiment. A schematic diagram showing an example of the annotation screen in this embodiment. A diagram showing an example of the hardware configuration of the active learning system (computer system) in this embodiment.
[0013] The embodiments of this disclosure will be described in detail below with reference to the drawings. In the drawings, the same parts will be denoted by the same reference numerals in principle, and repeated descriptions will be omitted.
[0014] In explanations, when describing program-based processing, the focus may sometimes be on the program, functions, or processing units. However, the main hardware component is the processor, or a controller, device, computer, or system composed of such a processor. The computer, using its processor, executes processing according to the program read into memory, utilizing resources such as memory and communication interfaces as appropriate. This realizes the specified functions and processing units. The processor is composed of semiconductor devices such as a CPU / MPU or GPU. Processing is not limited to software program processing; it can also be implemented using dedicated circuits. Applicable dedicated circuits include FPGAs, ASICs, CPLDs, etc.
[0015] The program may be pre-installed as data on the target computer, or it may be distributed as data to the target computer from the program source. The program source may be a program distribution server on a communication network, or a non-transient computer-readable storage medium, such as a memory card or disk. The program may consist of multiple modules. The computer system may consist of multiple devices. The computer system may consist of a client-server system, a cloud computing system, an IoT system, etc. Various types of data and information are composed of structures such as tables and lists, but are not limited to these. Representations such as identification information, identifiers, IDs, names, and numbers are interchangeable.
[0016] [Solution, etc.] The active learning system and method of this embodiment comprises: a confidence calculation unit that calculates confidence based on the inference results of a machine learning model; a typicality calculation unit that calculates typicality based on the inference results of a machine learning model; a high typicality data selection unit that selects data with high typicality (referred to as "high typicality data") with respect to unlabeled data; and a low confidence data selection unit that selects data with low confidence (referred to as "low confidence data") with respect to unlabeled data. The system further comprises a data determination unit that determines whether to select and adopt low confidence data or high typicality data as a query (annotation target data) based on the confidence calculated by the confidence calculation unit.
[0017] Furthermore, in active learning, there is a problem in that when a machine learning model is undertrained, the confidence score is calculated to be higher than the accuracy. This problem is generally called overconfidence. Therefore, the active learning system and method of this embodiment have a function to correct the confidence score in the confidence score calculation unit as a measure to prevent overconfidence in the machine learning model. Specifically, the confidence score calculation unit calculates multiple confidence scores using different calculation methods, for example, two types of confidence scores (classification confidence score and cluster confidence score described later), and calculates one integrated confidence score from these two types of confidence scores.
[0018] <Embodiment 1> The active learning system of this embodiment (referred to as Embodiment 1) will be described using Figures 1 and subsequent figures.
[0019] [Active Learning System] Figure 1 shows the configuration of the active learning system of this embodiment. The active learning system of this embodiment shown in Figure 1 is implemented by a computer system and has functional parts realized by software processing, such as a data selection unit 109, annotation 122, and model learning 124. The main feature of this embodiment is the data selection unit 109.
[0020] The data selection unit 109 is a processing unit that implements the function of selecting unlabeled data (in other words, data to be annotated) that is useful for active learning. The data selection unit 109 includes a model inference unit 301, a typicality calculation unit 302, a clustering unit 303, a confidence calculation unit 304, a two-type selection unit 310, and a data determination unit 307. The two-type selection unit 310 includes a high-typicality data selection unit 305 and a low-confidence data selection unit 306. The data selection unit 109 receives input such as a machine learning model 112, unlabeled data 113, and active learning setting data 114 as data. The data selection unit 109 obtains annotated data 121 (in other words, multiple queries) as output data.
[0021] The model inference unit 301 inputs the unlabeled data 113, which is the input data, into the machine learning model 112 and obtains features and classification probabilities as the inference results of the model. The typicality calculation unit 302 calculates the typicality of the unlabeled data based on the inference results. The confidence calculation unit 304 calculates the confidence of the unlabeled data based on the inference results. The clustering unit 303 performs clustering of the unlabeled data based on the inference results.
[0022] In the two-type selection unit 310, the high-typicality data selection unit 305 selects high-typicality data for each cluster based on the typicality of the unlabeled data. The low-confidence data selection unit 306 selects low-confidence data for each cluster based on the confidence level of the unlabeled data.
[0023] The data determination unit 307 receives the high-typicality data and low-confidence data selected by the two-type selection unit 310, and determines, for each cluster, which unlabeled data (high-typicality data and low-confidence data) to adopt as the query (annotation target data) based on the confidence level of the high-typicality data (obtained from the confidence level calculation unit 304). The data determination unit 307 outputs and stores the annotation target data (query), which is the unlabeled data of the determination result, and information such as the confidence level associated with that data, as annotation target data 121.
[0024] Regarding the management and control of the annotation target data and the associated confidence level information, the data may be managed by associating confidence level information with unlabeled data to form a dataset, or they may be configured to be input and output to each part as separate data / information.
[0025] The data selection unit 109 (particularly the data determination unit 307) obtains high-typicality data and low-confidence data for each cluster of unlabeled data, as summarized in the callouts in the figure. The data selection unit 109 (particularly the data determination unit 307) determines the annotation target data, which is the active learning data, by selecting low-confidence data if the confidence level of the high-typicality data is high, and selecting high-typicality data if the confidence level is low. This selection and determination is performed as the determination of multiple queries that match the set range of possible annotations.
[0026] After data selection, the system, in annotation 122, interactively prompts the user for queries on the screen based on the data to be annotated (multiple queries) 121, and the user performs the annotation (labeling) work on the queries. The annotated data becomes labeled data 122. Subsequently, the system performs model training 124 using the labeled data 122 and the machine learning model 112. This improves the accuracy of the machine learning model 112.
[0027] [Active Learning System (Computer System)] Figure 2 shows an example configuration of the active learning system 100 of this embodiment. The active learning system 100 shown in Figure 2 is a computer system and has an interface 101, a control unit 102, and a storage unit 103.
[0028] Interface 101 has an input unit 104 and an output unit 105. Control unit 102 has a setting unit 106, a data update unit 107, a model learning unit 108, a data selection unit 109, and a data visualization unit 110. Storage unit 103 stores labeled data 111, a machine learning model 112, unlabeled data 113, and active learning setting data 114, etc.
[0029] The input unit 104 receives the input operations of the user and transmits instruction information and the like to the setting unit 106 and the data update unit 107 based on the input operations. Specifically, information such as a machine learning model, unlabeled data, and active learning setting data is sent to the setting unit 106. Also, annotation information of a query (data to be annotated) and the like is sent to the data update unit 107. The annotation information is information on labeling by the user.
[0030] The unlabeled data and the query may be in various forms such as text data, image data, audio data, numerical data, sensor data, etc. Details of the initial setting screen in this embodiment will be described later. In this embodiment, the case of learning an image is illustrated, but it is not limited thereto.
[0031] The output unit 105 realizes a function of presenting a query or the like selected by the data selection unit 109 to the user and requesting annotation (labeling) for the query. The output unit 105 has a screen display with a graphical user interface (GUI). Also, the output unit 105 presents information on the learning status and accuracy of the machine learning model 112 to the user. Thereby, the user performs the annotation work and checks the progress of the learning. Further, the output unit 105 visualizes the feature amounts of the labeled data 111 and the unlabeled data 113 using the information obtained by the data visualization unit 110. Thereby, the relationship between the query data and other data can be visualized, and materials for the user to judge whether to annotate the query or change the query are provided. Details of the annotation screen will be described later.
[0032] The labeled data 111 is a data set with labels attached and is used for the learning and evaluation of the machine learning model 112. The label indicates the class or category to which each data (or data point) belongs. At the start of active learning, this labeled data 111 is often empty. Through annotation, this labeled data 111 gradually increases.
[0033] The machine learning model 112 is not limited to a specific type as long as it is a model for solving classification problems, and any model can be used. For example, it may be a neural network, SVM (support vector machine), or decision tree. Also, the machine learning model 112 includes information on the parameters and hyperparameters of the model.
[0034] The unlabeled data 113 is a data set that has not yet been labeled. In the process of active learning, the unlabeled data 113 is selected by the data selection unit 109 and becomes a query (data to be annotated), which is data annotated by the user. Each data of the unlabeled data 113 may be in various forms such as text, image, audio, numerical data, sensor data, etc.
[0035] The active learning setting data 114 is setting information required during the active learning of the active learning system 100, and various types of data and parameters are stored. This setting information includes, for example, the number of annotatable items, the query set, the data selection method, the threshold value, etc. The number of annotatable items is used for the end condition of the flow in FIG. 5. The query set is a set that holds the queries selected by the data selection unit 109. The data selection method is the method (corresponding algorithm, program, etc.) used in each data selection unit of the two-type selection unit 310 (FIG. 1). The threshold value is the threshold value used for the confidence determination in the data determination unit 307 (FIG. 1).
[0036] The setting unit 106 receives the machine learning model, unlabeled data, and active learning setting data from the input unit 104 and registers them in the machine learning model 112, unlabeled data 113, and active learning data 114 of the storage unit 103, respectively.
[0037] The data update unit 107 performs an operation of adding the data with the annotation information obtained by the input unit 104 attached to the query (data to be annotated) to the labeled data 111 as labeled data, or an operation of deleting the data with the annotation information from the unlabeled data 113.
[0038] The model learning unit 108 trains the machine learning model 112. Specifically, the model learning unit 108 uses unlabeled data 113 to perform self-supervised and unsupervised learning on the machine learning model 112. In addition, the model learning unit 108 uses labeled data 111 to perform supervised learning on the machine learning model 112. Furthermore, the model learning unit 108 also has the function to perform semi-supervised learning using both labeled data 111 and unlabeled data 113.
[0039] The data selection unit 109 implements the function of selecting suitable data from unlabeled data 113 to be used as queries (annotation target data). The data selection unit 109 obtains setting information such as data selection method and threshold from active learning setting data 114. Using information such as inference results obtained from the machine learning model 112, the data selection unit 109 selects data to be adopted as queries from the unlabeled data 113 and stores it as a query set as multiple queries. The query set containing the selected multiple queries is sent to the output unit 105. Details of the data selection unit 109 will be described later.
[0040] The data visualization unit 110 inputs the labeled data 111 and the unlabeled data 113 into the machine learning model 112 and compresses the multidimensional feature data obtained as output to two or three dimensions using a dimensionality reduction method for visualization (display) on the screen. Known dimensionality reduction methods such as t-SNE (t-Distributed Stochastic Neighbor Embedding), PCA (Principal Component Analysis), and autoencoders can be applied. The compressed data is sent to the output unit 105.
[0041] [Active Learning Method] Figure 3 shows the flow of the active learning method in this embodiment. The active learning system shown in Figures 1 and 2 performs processing according to this active learning method. The flow of active learning using the active learning system 100 will be explained according to Figure 3.
[0042] In step S201, in the active learning system 100, the unlabeled data, machine learning model, and active learning setting data entered by the user performing the active learning task through the input unit 104 are registered in the memory unit 103 as unlabeled data 113, machine learning model 112, and active learning setting data 114, respectively.
[0043] Next, in step S202, the model learning unit 108 pre-trains the machine learning model 112 using unlabeled data 113. As for the pre-training method, if there is no labeled data 111 at all, self-supervised learning or unsupervised learning can be used. If there is even a small amount of labeled data 111, supervised learning or semi-supervised learning may be used.
[0044] Next, in step S203, the data selection unit 109 obtains a query set (annotation target data) by selecting data from the unlabeled data 113 using information from the machine learning model 112. Details of query selection in step S203 will be described later with reference to Figure 5.
[0045] Next, in step S204, the user performs annotation (annotation 122 in Figure 1) on the set of queries obtained in step S203 using the annotation screen (described later). The active learning system 100 displays the queries on the annotation screen and processes the user input. The data update unit 107 adds the annotated data to the labeled data 111 and also deletes the annotated queries from the unlabeled data 113.
[0046] Next, in step S205, the model learning unit 108 uses the updated labeled data 111 to perform supervised learning or semi-supervised learning (model learning 124 in Figure 1) on the machine learning model 112.
[0047] Next, in step S206, the active learning system 100 determines, based on predetermined termination conditions, whether to return to step S203 to continue active learning or to terminate active learning. The termination conditions may be, for example, when the desired accuracy of the machine learning model 112 is obtained, when the amount of change in the parameters of the machine learning model 112 is less than a predetermined amount, or when the unlabeled data 113 becomes empty.
[0048] [Data Selection Unit] Figure 4 shows an example of the configuration of the data selection unit 109. The data selection unit 109 includes a machine learning model inference unit 301, a typicality calculation unit 302, a clustering unit 303, a confidence calculation unit 304, a high-typicality data selection unit 305, a low-confidence data selection unit 306, and a data determination unit 307. The setting information of the active learning setting data 114 is input to the data selection unit 109 and used appropriately in each element of the data selection unit 109.
[0049] The machine learning model inference unit 301 first receives the machine learning model 112 and unlabeled data 113. The machine learning model inference unit 301 inputs the unlabeled data 113 (data points) into the machine learning model 112 and outputs the classification probability, which is the probability distribution of belonging to each class, for each unlabeled data point 113 (data point). The machine learning model inference unit 301 also outputs the features of each unlabeled data point 113 (data point).
[0050] Examples of these features include the output of the hidden layers of a neural network. For image data, feature extraction methods such as SIFT (Scale-Invariant Feature Transform) and HOG (Histogram of Oriented Gradients) can be used. For text data, methods such as TF-IDF (Term Frequency-Inverse Document Frequency) and Word2Vec can be used. Furthermore, for numerical data and sensor data, principal component analysis (PCA) and statistical features (mean, standard deviation, etc.) can also be applied.
[0051] The clustering unit 303 clusters the features of the unlabeled data 113 (data points) obtained by the machine learning model inference unit 301. Known clustering methods such as K-means, K-means++, or DBSCAN may be used. The calculated cluster information is sent to the high-typicality data selection unit 305 and the low-confidence data selection unit 306.
[0052] The typicality calculation unit 302 receives feature quantities of unlabeled data 113 (data points) as inference results from the machine learning model inference unit 301 and calculates the typicality of each unlabeled data 113 (data point). For example, the typicality calculation method can use a known technique using density in the feature space described in Non-Patent Literature 1. The calculated typicality information is sent to the high-typicality data selection unit 305.
[0053] The confidence calculation unit 304 receives the classification probability and features of the unlabeled data 113 (data points) as inference results from the machine learning model inference unit 301, and calculates the confidence level for each unlabeled data 113. The calculated confidence level information is sent to the low confidence level data selection unit 306 and the data determination unit 307. Details of the confidence calculation unit 304 will be described later.
[0054] The high-typicality data selection unit 305 selects the data (data point) with the highest typicality calculated by the typicality calculation unit 302 from each cluster indicated by the cluster information, as "high-typicality data" 311, and sends it to the data determination unit 307.
[0055] The low-confidence data selection unit 306 selects the data (data point) with the lowest confidence level calculated by the confidence level calculation unit 304 from each cluster indicated by the cluster information, as "low-confidence data" 312, and sends it to the data determination unit 307.
[0056] The data determination unit 307 first receives high-typicality data 311 for each cluster from the high-typicality data selection unit 305, low-confidence data 312 for each cluster from the low-confidence data selection unit 306, and the confidence level of each data point from the confidence level calculation unit 304. Next, the data determination unit 307 decides for each cluster whether to adopt the high-typicality data 311 or the low-confidence data 312 as a query and add it to the query set, based on the confidence level of the high-typicality data 311. If the confidence level of the high-typicality data 311 is high, the data determination unit 307 determines that the learning efficiency is low and selects the low-confidence data 312 to add to the query set. On the other hand, if the confidence level of the high-typicality data 311 is low, the data determination unit 307 selects the high-typicality data 311 to add to the query set. The determination of high or low confidence levels in the data determination unit 307 is performed, for example, using a dedicated threshold (one of the setting data 114). Furthermore, the data determination unit 307 has the function of sending a query set containing the selected multiple queries to the output unit 105.
[0057] In this embodiment (Figure 4), the high-typicality data selection unit 305 selects the single data with the highest typicality for each cluster as high-typicality data 311, and the low-confidence data selection unit 306 selects the single data with the lowest confidence level for each cluster as low-confidence data 312. However, the system is not limited to this; for example, dedicated thresholds may be set, and one or more data with a typicality equal to or greater than the threshold may be selected as high-typicality data 311, or one or more data with a confidence level below the threshold may be selected as low-confidence data 312.
[0058] [Query Selection Flow of the Data Selection Unit] Figure 5 shows the processing flow of query (annotation target data) selection by the data selection unit 109 in Figure 4. The process of selecting and determining the query by the data selection unit 109 will be explained according to Figure 5.
[0059] First, the active learning system 100 initializes the query set when query selection begins.
[0060] Next, in step S401, the machine learning model inference unit 301 inputs the unlabeled data 113 into the machine learning model 112, and obtains the classification probability and features of the unlabeled data 113 as an inference result from the machine learning model 112.
[0061] Next, in step S402, the clustering unit 303 clusters the features of the unlabeled data 113 obtained in step S401.
[0062] Next, in step S403, the typicality calculation unit 302 calculates the typicality of each unlabeled data 113.
[0063] Next, in step S404, the confidence calculation unit 304 calculates the confidence level for each unlabeled data point 113.
[0064] Next, in step S405, the high-typicality data selection unit 305 selects data with high typicality from each cluster as "high-typicality data" 311.
[0065] Next, in step S406, the low-confidence data selection unit 306 selects data with a low confidence level from each cluster as "low-confidence data" 312.
[0066] Steps S403 to S406 may be performed in parallel.
[0067] Next, in step S407, the data determination unit 307 determines whether the confidence level of the high-typicality data 311 is below a threshold. If it is below the threshold (YES), in step S408, the high-typicality data 311 is selected and added to the query set. On the other hand, if the confidence level of the high-typicality data 311 is higher than the threshold (NO), the data determination unit 307 selects the low-confidence data 312 in step S409 and adds it to the query set.
[0068] Next, in step S410, the data selection unit 109 checks whether the number of elements in the query set (in other words, the number of queries) is equal to or greater than the set value [number of annotations]. If it is less than [number of annotations] (NO), the process returns to after step S402 and is repeated in the same manner. If it is equal to or greater than [number of annotations] (YES), the query selection flow ends. That is, the flow ends when the number of queries in the query set reaches the number of annotations.
[0069] [Confidence Calculation Unit] Figure 6 shows an example of the configuration of the confidence calculation unit 304. The simple confidence level (such as the softmax output) of commonly used machine learning models is prone to "overconfidence" when there is little training data for the model (in other words, when the model is undertrained). Overconfidence refers to a state where the confidence level is high, but the actual accuracy of the machine learning model is low. Therefore, in this embodiment, the confidence calculation unit 304 corrects the confidence level. The confidence calculation unit 304 calculates two types of confidence levels, a first confidence level and a second confidence level, which are calculated using different methods, and calculates a single integrated confidence level (a new confidence level) from the two types of confidence levels.
[0070] The confidence calculation unit 304 includes a classification confidence calculation module 501, a cluster confidence calculation module 502, and a confidence integration unit 503.
[0071] The classification confidence calculation module 501 is, in other words, the first confidence calculation unit. The classification confidence calculation module 501 obtains the class classification probability of the unlabeled data 113 as an inference result from the machine learning model inference unit 301, and calculates the first confidence (classification confidence) and the first provisional label based on the class classification probability. The first confidence is the value of the highest probability among the class classification probabilities. The first provisional label is the class corresponding to the highest probability. The calculated first confidence 1 and first provisional label are sent to the confidence integration unit 503. The first provisional label is also sent to the cluster confidence calculation module 502.
[0072] The cluster confidence calculation module 502 is, in other words, a second confidence calculation unit. The cluster confidence calculation module 502 calculates a second confidence level (cluster confidence) and a second provisional label. First, the cluster confidence calculation module 502 obtains the features of the unlabeled data 113 as an inference result from the machine learning model inference unit 301, and the first provisional label from the classification confidence calculation module 501. Next, the cluster confidence calculation module 502 clusters the features of the unlabeled data 113. This clustering method may be the well-known K-means method or the K-means++ method. The number of clusters is set to match the number of classifications (number of classes) in the classification problem that the machine learning model 112 learns.
[0073] The second provisional label is the label obtained through clustering. The cluster confidence calculation module 502 associates the first provisional label with the second provisional label obtained through clustering. One example of an association method is to assign labels based on the frequency of occurrence of the first provisional label within each cluster. Specifically, one method is to assign the label with the highest number of occurrences of the first provisional label within each cluster to the second provisional label.
[0074] The above provisional labels (first provisional label, second provisional label) are used within the confidence calculation unit 304. For explanatory purposes, the provisional label (inference label) determined by the classification confidence calculation module 501 is referred to as the first provisional label, and the provisional label (inference label) determined by the cluster confidence calculation module 502 is referred to as the second provisional label. Note that there is a correspondence between the class and the label.
[0075] The first confidence level is c １ Let the second confidence level be c. ２ Let's assume the second confidence level (c ２ The confidence level (x) is calculated using the following equation 1, based on the relative distance between the data subject to confidence measurement (let's call it x) and the center point of each cluster.
[0076] Formula 1:
[0077] D in Formula 1 represents an arbitrary distance function (such as Euclidean distance, cosine similarity, etc.), f(x) represents the feature of data x, and μ ｉ represents the center (average) of cluster i. From Formula 1, the second confidence level (c ２ ) becomes high when the feature f(x) is close to the center point of a certain cluster and far from the center points of other clusters. On the other hand, the second confidence level (c ２ ) becomes small when the distances from the centers of multiple clusters are almost equal. From Formula 1, the second confidence level (c ２ ) takes a value in the range from 0 to 1.
[0078] [Second Confidence Level (Cluster Confidence Level)] Figure 7 is an image diagram of the feature space of the unlabeled data 113. Using Figure 7, the second confidence level (c ２ ) in Formula 1 will be explained. For the sake of explanation, the feature space is considered to be two-dimensional, but it can also be of higher dimensions. In the example of Figure 7, there are cluster A (data with label A) and cluster B (data with label B). Labels A and B in Figure 7 and the like are abstractions of examples of specific label values. The feature space diagram 601 in (A) of Figure 7 is when the confidence level is low, and the feature space diagram 602 in (B) of Figure 7 is when the confidence level is high. As in the feature space diagram 601 in (A), when the distances from the center points of each cluster to the data to be calculated for confidence level (indicated by black circles) are almost equal, the second confidence level (c ２ ) becomes low. On the other hand, as in the feature space diagram 602 in (B), when the data to be calculated for confidence level is close to the center point of cluster A and far from the center point of cluster B, for example, the second confidence level (c ２ ) becomes high.
[0079] In the confidence level integration unit 503 of Figure 6, using the first temporary label (denoted as y １ ), the first confidence level (c １ ), the second temporary label (denoted as y ２ ), and the second confidence level (c ２ ), a new confidence level (c), which is the integrated confidence level obtained by integrating the two confidence levels into one, is calculated by the following Formula 2. This new confidence level is the confidence level calculated and output by the confidence level calculation unit 304.
[0080] Formula 2:
[0081] δ(y) in Equation 2 １ , y ２ ) is y １ and y ２ This is an indicator function that returns 1 if they match and 0 otherwise. From this equation 2, the first provisional label (y １ ) and the second temporary label (y ２ If the two tentative labels do not match, the confidence level (c) is 0. Also, if the two tentative labels match, and the first confidence level (c) １ ) and the second confidence level (c ２ The confidence level (c) can only be expected to increase when the ) increases. With this calculation method, the confidence level (c) can be expected to be low when the machine learning model is prone to overconfidence during insufficient training, and to increase as training progresses.
[0082] [Integrated Confidence] Figure 8 shows the effect of the confidence integration unit 503 on integrated confidence. Using Figure 8, we will explain why the confidence (c) in Equation 2 can mitigate overconfidence. Figure 8 shows an example of the feature space 701 when the machine learning model 112 is undertrained and the feature space 702 when the machine learning model 112 is sufficiently trained. For the sake of explanation, the feature space is shown as two-dimensional, but it may be higher-dimensional.
[0083] When training is insufficient, the features of each data point are not sufficiently separated for each class, as shown in the feature space 701. Therefore, the first provisional label (y １ ) and the second temporary label (y ２ Many of the given labels do not match, and the confidence level (c) in Equation 2 becomes 0. As a result, the confidence level (c) tends to be 0 when there is insufficient training, which is prone to overconfidence. Also, even if the two given labels match, the two confidence levels will take values in the range of 0 to 1, so one of them will always be smaller.
[0084] On the other hand, as learning progresses, overconfidence improves, and features are separated into classes, as in the feature space 702. At this time, the first provisional label (y １ ) and the second temporary label (y２ Many of the results in equation 2, δ(y １ , y ２ The second confidence level (c) is 1. Therefore, the confidence level (c) can be expected to be higher compared to when the provisional labels do not match. ２ The second confidence level (c) is calculated based on Equation 1 using the relative distance between the feature of the target data and the center of each cluster, so as each cluster is separated, the second confidence level (c) ２ ) can be expected to increase.
[0085] Therefore, the confidence score (c) calculated and output by the confidence score integration unit 503 tends to be low when the machine learning model is overconfident due to insufficient training, and is expected to increase as training progresses.
[0086] [Initial Setup Screen] Figure 9 shows an example of an initial setup screen 800 for performing the initial setup of the active learning system 100. This initial setup screen 800 has a field 801 for entering the path to the active learning dataset Dir (directory), a field 802 for entering the number of classification classes, and a field 803 for setting the machine learning model. The initial setup screen 800 also includes a select box 804 for setting "data selection method 1" used for selecting high-typicality data, a select box 805 for setting "data selection method 2" used for selecting low-confidence data, a field 806 for specifying the number of annotations possible, a field 807 for entering a threshold, a start button 808 for starting active learning, and so on.
[0087] The user selects the data selection method to be used in the high-typicality data selection unit 305 using the select box 804. The options in select box 804 include, for example, a method for sampling typical data, such as Non-Patent Document 1. The user also sets the data selection method to be used in the low-confidence data selection unit 306 using the select box 805. The options in select box 805 include, for example, Margin Sampling and Uncertainty Sampling. In the threshold input field 807, the user sets the threshold (confidence threshold) used by the data determination unit 307 in step S407 to determine the confidence level of the high-typicality data. After the user has entered and set all items on the initial setup screen 800, when the user presses the start button 808, the active learning process of the active learning system 100 starts according to Figure 3, etc.
[0088] [Annotation Screen] Figure 10 shows an example of an annotation screen 900 provided to the user by the active learning system 100. The annotation screen 900 includes a basic query information field 901, a query display field 902, a feature space display field 903, a label selection button 904, a query change button 905, a next query button 906, and so on.
[0089] In the query's basic information section 901, the basic information of the selected query (annotation target data / unlabeled data) is displayed for the user to review, including the data name (file name), confidence level and / or typicality, and data selection result. The confidence level is the confidence level of the machine learning model 112 regarding the unlabeled data of the query, and is the confidence level used for the decision by the data selection unit 109. The typicality is the typicality of the unlabeled data of the query, and is the typicality of the high-typicality data used for the decision by the data selection unit 109. The data selection result is the result of whether the data selection unit 109 selected (in other words sampled) high-typicality data or low-confidence data for the query, and information on the data selection method used for the selection.
[0090] The query display area 902 displays the content of the query (annotation target data) (for example, an image). The feature space display area 903 displays a graph (for example, a two-dimensional feature space of feature 1 and feature 2) that visualizes the features of the labeled data 111 and unlabeled data 113 obtained by the output unit 105 (Figure 2). In other words, the feature space display area 903 displays a feature distribution diagram that has been dimensionally compressed by the data visualization unit 110 (Figure 2). In the feature space display area 903, the user can check the relationship between the features of the query drawn in the query display area 902 and the features of other data.
[0091] The user can annotate (label) the data displayed in the query display area 902 by operating the label selection button 904. The label selection button 904 is a button with a label value for labeling. For example, when selecting from labels A and B based on classification, the user presses the "Label A" button if they want to assign "Label A" to the query. The user can also check the feature space display area 903, and if they determine that the query does not contribute to improving model accuracy, they can change the query displayed for annotation by pressing the query change button 905. After annotating a query, the user can then press the next query button 906 to display the next query on the screen and annotate the next query.
[0092] The screen may also display an information field 907 related to learning progress, etc. The information field 907 may display the number of annotations, the number of queries in the query set, the number of annotated data, the number of remaining data, etc.
[0093] As described above, the annotation screen 900 displays at least the query automatically selected by the data selection unit 109, as well as data, information, and GUIs to support the annotation work. In the example in Figure 10, information such as confidence level, typicality, data selection method, and number of annotable items generated by the data selection unit 109 is also displayed. This allows the user to perform the annotation work more efficiently.
[0094] [Computer System] Figure 11 shows an example of the hardware and software configuration of the active learning system 100 as a computer system in Figure 2. The computer 1000 that constitutes this computer system has a processor 1001, memory 1002, communication interface device 1003, input / output interface device 1004, storage device 1007, etc., and these are interconnected via an architecture such as a bus.
[0095] The processor 1001 executes processing according to the control program 1002A for the memory 1002. This realizes a predetermined function 1001A (Figures 1 and 2) as an execution module. The processor 1001 may also have an NPU (Neural Processing Unit) suitable for machine learning / AI calculations.
[0096] The memory 1002 and storage device 1007 are composed of non-volatile storage devices, etc. The memory 1002 stores, for example, a control program 1002A, configuration information 1002B, learning data 1002C, etc. Various data and information may be stored in the storage device 1007 (for example, an auxiliary storage device, an external storage device) and read into the memory 1002 as needed. An external server or database may also be used as a storage device.
[0097] The control program 1002A is a computer program for realizing functions 1001A, etc., and is a learning program for realizing active learning. The setting information 1002B is system setting information and user setting information related to the control program 1002A, and includes the active learning setting data 114 in Figure 2. The learning data 1002C is various types of data used in the learning phase, such as image data, machine learning models, and training data, and includes labeled data 111, machine learning models 112, and unlabeled data 113 in Figure 2.
[0098] The communication interface device 1003 is a device that implements a communication interface with a communication network such as a LAN. Input / output interface device 1004 has an input device 1005 and an output device 1006 externally connected to it. The computer 1000 may also have built-in input / output devices. Examples of input devices 1005 include a mouse, keyboard, and microphone. Examples of output devices 1006 include a display, printer, and speakers.
[0099] [Effects, etc.] As explained above, this embodiment enables high performance (in other words, high model accuracy) in active learning regardless of the number of annotations, and reduces annotation costs.
[0100] In prior art examples such as Patent Document 1, the accuracy of the machine learning model does not improve sufficiently when the number of annotable elements is small. Therefore, more annotations are required to improve the accuracy of the machine learning model, increasing the annotation cost. On the other hand, by using this embodiment, the accuracy of the machine learning model can be rapidly improved even when the number of annotable elements is small, and the annotation cost can be reduced compared to prior art examples such as Patent Document 1.
[0101] [Modification] This embodiment uses clustering by the clustering unit 303 (Figure 1), but is not limited to this, and a modification can be made in which clustering is not used. In this configuration, data is selected not by cluster, but by data point of unlabeled data, based on confidence level, etc.
[0102] In this embodiment, the confidence calculation unit 304 (Figure 1) is configured to calculate the combined confidence level from two types of confidence levels (classification confidence level and cluster confidence level). However, the system is not limited to this configuration, and it is equally possible to calculate multiple confidence levels using different calculation methods and then calculate the combined confidence level from these multiple confidence levels.
[0103] [Note] This embodiment includes the following configuration.
[0104] An active learning program or recording medium storing the program for causing a computer to perform processing for actively learning a machine learning model, the program comprising the steps to be performed by the computer: a model inference step for obtaining inference results by the machine learning model based on unlabeled data which is input data; a confidence calculation step for calculating the degree of confidence regarding the unlabeled data based on the inference results; a typicality calculation step for calculating the typicality of the unlabeled data based on the inference results; a low confidence data selection step for selecting low confidence data based on the degree of confidence regarding the unlabeled data; a high typicality data selection step for selecting high typicality data based on the typicality of the unlabeled data; and a data determination step for selecting annotation target data from the selected low confidence data and the low confidence data.
[0105] Although embodiments of this disclosure have been described in detail above, the invention is not limited to the embodiments described above and can be modified in various ways without departing from the gist of the invention. Each embodiment can be modified by adding, deleting, or replacing components, except for essential components. Unless otherwise specified, each component may be singular or plural. Combinations of each embodiment and its variations are also possible.
[0106] 109...Data selection unit, 112...Machine learning model, 113...Unlabeled data, 114...Setting data for active learning, 301...Model inference unit, 302...Generality calculation unit, 303...Clustering unit, 304...Confidence calculation unit, 310...Two-type selection unit, 305...High-typicality data selection unit, 306...Low-confidence data selection unit, 307...Data judgment unit, 121...Annotation target data, 122...Annotation, 123...Labeled data, 124...Model training.
Claims
1. An active learning system comprising a computer for actively training a machine learning model, wherein the computer includes: a model inference unit that obtains inference results by the machine learning model based on unlabeled data which is input data; a confidence calculation unit that calculates the confidence level of the unlabeled data based on the inference results; a typicality calculation unit that calculates the typicality of the unlabeled data based on the inference results; a low confidence data selection unit that selects low confidence data based on the confidence level of the unlabeled data; a high typicality data selection unit that selects high typicality data based on the typicality of the unlabeled data; and a data determination unit that selects data to be annotated from the selected low confidence data and the low confidence data.
2. An active learning system according to claim 1, wherein the data determination unit selects the low confidence data if the confidence level for the high typicality data is higher than or equal to a threshold, and selects the high typicality data if the confidence level for the high typicality data is lower than or equal to a threshold.
3. An active learning system according to claim 1, wherein the confidence calculation unit comprises a plurality of confidence calculation modules with different methods for calculating the confidence, and a confidence integration unit that calculates a single integrated confidence by combining the plurality of confidences calculated by the plurality of confidence calculation modules.
4. An active learning system according to claim 3, wherein the confidence calculation unit has a classification confidence calculation module as one of the plurality of confidence calculation modules, and the classification confidence calculation module calculates a classification confidence as a first provisional label and a first confidence using the class classification probability based on the inference result.
5. An active learning system according to claim 4, wherein the confidence calculation unit has a cluster confidence calculation module as one of the plurality of confidence calculation modules, and the cluster confidence calculation module clusters the feature quantities obtained as the inference result and calculates a cluster confidence as a second provisional label and a second confidence based on the relative distance between the clustered feature quantities and the center of each cluster.
6. In the active learning system according to claim 5, the confidence integration unit integrates the first confidence and the second confidence into one confidence (c) based on the following formula when the first pseudo-label is y １ , the first confidence is c １ , the second pseudo-label is y ２ , and the second confidence is c １ : c = δ(y １ , y ２ ) · c １ · c ２ , where δ is an indicator function that returns 1 if y １ and y ２ are the same, and 0 if they are different; active learning system.
7. An active learning system according to claim 1, comprising: a clustering unit that clusters the unlabeled data based on the inference results; a low-confidence data selection unit that selects low-confidence data for each cluster of the clustered results based on the confidence level of the unlabeled data; a high-typicality data selection unit that selects high-typicality data for each cluster of the clustered results based on the typicality of the unlabeled data; and a data determination unit that selects the data to be annotated for each cluster.
8. An active learning system according to claim 1, wherein, based on the annotation target data selected by the data determination unit, the annotation target data is displayed on the annotation screen along with the data selection result from the low confidence data and the high typicality data, the confidence level, or the typicality level.
9. An active learning method to be performed in an active learning system comprising a computer for actively training a machine learning model, the method comprising the steps performed by the computer: a model inference step to obtain an inference result by the machine learning model based on unlabeled data which is input data; a confidence calculation step to calculate a degree of confidence regarding the unlabeled data based on the inference result; a typicality calculation step to calculate a degree of typicality regarding the unlabeled data based on the inference result; a low confidence data selection step to select low confidence data based on the degree of confidence regarding the unlabeled data; a high typicality data selection step to select high typicality data based on the typicality regarding the unlabeled data; and a data determination step to select data to be annotated from the selected low confidence data and the low confidence data.
10. An active learning method according to claim 9, wherein the data determination step includes selecting the low confidence data if the confidence level for the high typicality data is higher than or equal to a threshold, and selecting the high typicality data if the confidence level for the high typicality data is lower than or equal to a threshold.