Model training system, model training method, and model training program

Abstract

A model training system (2) uses a machine learning model (56) to perform inference on the basis of input data and outputs inference results. The model training system identifies a correct answer of the inferences. The model training system generates annotated data and identifies weaknesses in inferencing on the basis of a plurality of annotated data sets and inference results. The model training system generates collection conditions for collecting input data that addresses weaknesses. The model training system divides a plurality of pieces of collected data into training data and evaluation data. The model training system trains and evaluates machine learning models, identifies weaknesses in the machine learning models if evaluation results are unsatisfactory, and generates collection conditions that address the weaknesses.

Description

Model learning system, model learning method and model learning program Cross-Reference to Related Applications

[0001] This international application claims the benefit of Japanese Patent Application No. 2025-055450 filed with the Japan Patent Office on March 28, 2025, the entire disclosure of which is incorporated into this international application by reference.

[0002] The present disclosure relates to a model learning system, a model learning method, and a model learning program for training a machine learning model.

[0003] Patent Document 1 describes a system that collects vehicle data from a plurality of in-vehicle devices mounted on each of a plurality of vehicles.

[0004] In the development of machine learning models mounted on vehicles, generation of training data for training the machine learning model, training of the machine learning model, and evaluation of the machine learning model are performed on a center side that collects vehicle data from a plurality of in-vehicle devices.

[0005] Japanese Unexamined Patent Publication No. 2023-84379

[0006] As a result of detailed studies by the inventor, it has been found that, for example, an engineer may have to perform annotation by themselves to generate training data, which increases the engineer's workload in the development of machine learning models.

[0007] The present disclosure reduces the workload of engineers developing machine learning models.

[0008] One aspect of the present disclosure is a model learning system including an inference result output unit, a correct answer identification unit, an annotation execution unit, a weak point identification unit, a collection condition generation unit, a data collection unit, a learning data generation unit, a model learning unit, a model evaluation unit, and a re-execution unit.

[0009] The inference result output unit is configured to perform inference based on input data using a machine learning model and output an inference result.

[0010] The correct answer identification unit is configured to identify a correct answer for inference based on input data.

[0011] The annotation execution unit is configured to perform annotation based on the input data input to the inference result output unit and the correct answer data indicating the correct answer identified by the correct answer identification unit, thereby generating annotated data that includes the input data.

[0012] The weakness identification unit is configured to identify weaknesses in the machine learning model's inference based on multiple annotated data sets and multiple inference results output by the inference result output unit for each of the multiple annotated data sets.

[0013] The collection condition generation unit is configured to generate collection conditions for collecting input data that addresses weaknesses.

[0014] The data collection unit is configured to collect data based on the generated collection conditions.

[0015] The learning data generation unit is configured to generate multiple learning data and multiple evaluation data by splitting multiple collected data, which are multiple data collected by the data collection unit, into learning data and evaluation data.

[0016] The model learning unit is configured to train a machine learning model using multiple training datasets.

[0017] The model evaluation unit is configured to evaluate the machine learning model after it has been trained by the model learning unit, using multiple evaluation datasets.

[0018] The re-execution unit is configured to repeat a cycle in which, if the evaluation result by the model evaluation unit is unsatisfactory, it activates the weakness identification unit and the collection condition generation unit to identify weaknesses in the machine learning model, generates collection conditions corresponding to the identified weaknesses, and sequentially executes data collection by the data collection unit using the newly generated collection conditions, generation of training data and evaluation data by the training data generation unit, training by the model training unit, and evaluation by the model evaluation unit.

[0019] The model learning system of this disclosure, configured in this manner, generates collection conditions using an inference result output unit, a correct answer identification unit, an annotation execution unit, a weakness identification unit, and a collection condition generation unit. Subsequently, it sequentially operates a data collection unit, a training data generation unit, a model learning unit, a model evaluation unit, a weakness identification unit, and a collection condition generation unit, and repeats this cycle to train a machine learning model. Therefore, the model learning system of this disclosure can reduce the intervention of engineers required to train machine learning models and reduce the workload of engineers developing machine learning models.

[0020] Another aspect of this disclosure is a model learning method performed by a model learning system.

[0021] In the model learning method disclosed herein, the model learning system uses a machine learning model to perform inference based on input data and outputs the inference result.

[0022] The model learning system identifies the correct inference based on the input data.

[0023] The model learning system performs annotation based on the input data and the ground truth data that identifies the correct answer, generating annotated data that includes the input data.

[0024] The model learning system identifies weaknesses in the machine learning model's inference based on multiple annotated datasets and multiple inference results output for each of these datasets.

[0025] The model learning system generates collection conditions for collecting input data that addresses weaknesses.

[0026] The model learning system collects data based on the generated collection conditions.

[0027] The model learning system generates multiple training data and multiple evaluation data by dividing the collected data into training data and evaluation data.

[0028] The model learning system uses multiple training datasets to train a machine learning model.

[0029] The model learning system evaluates the machine learning model, which has been trained by the model learning unit, using multiple evaluation datasets.

[0030] The model learning system, if the evaluation result is unsatisfactory, identifies the weaknesses of the machine learning model, generates collection conditions corresponding to the identified weaknesses, and then sequentially performs data collection using the newly generated collection conditions, generates training data and evaluation data, trains the machine learning model, and evaluates the machine learning model, repeating this cycle.

[0031] The model learning method described herein is a method that is executed within the model learning system described herein, and by executing this method, the same effects as those of the model learning system described herein can be obtained.

[0032] A further aspect of this disclosure is a model learning program for causing a computer to function as an inference result output unit, a correct answer identification unit, an annotation execution unit, a weakness identification unit, a collection condition generation unit, a data collection unit, a training data generation unit, a model learning unit, a model evaluation unit, and a re-execution unit.

[0033] A computer controlled by the model learning program of this disclosure can constitute part of the model learning system of this disclosure and can achieve the same effects as the model learning system of this disclosure.

[0034] This is a block diagram showing the configuration of the data collection system. This is a block diagram showing the configuration of the data collection device. This is a block diagram showing the configuration of the center. This is a block diagram showing the procedure for the inference process. This is a block diagram showing the procedure for the weakness identification process. This is a block diagram showing the procedure for the learning process. This is a block diagram showing the configuration of the data collection device and the center. This is a flowchart showing the inference process. This is a flowchart showing the weakness identification process. This is a flowchart showing the learning process.

[0035] Embodiments of the present disclosure will be described below with reference to the drawings.

[0036] As shown in Figure 1, the data acquisition system 1 of this embodiment comprises a plurality of data acquisition devices 2 and a center 3.

[0037] The data collection device 2 is mounted on the vehicle and has the function of communicating data with the center 3 via a wide-area wireless communication network NW. The data collection device 2 transmits the data collected in the vehicle to the center 3.

[0038] Center 3 is a device that manages the data acquisition system 1. Center 3 has the function of communicating data with multiple data acquisition devices 2 via a wide-area wireless communication network NW. Center 3 stores the data transmitted from the multiple data acquisition devices 2.

[0039] As shown in Figure 2, the data acquisition device 2 comprises a control unit 11, a CAN communication unit 12, a storage unit 13, and a communication unit 14. CAN stands for Controller Area Network.

[0040] The control unit 11 is an electronic control device centered around a microcomputer equipped with a CPU 21, ROM 22, RAM 23, etc. The various functions of the microcomputer are realized by the CPU 21 executing a program stored in a non-transitional physical recording medium. In this example, the ROM 22 corresponds to the non-transitional physical recording medium that stores the program. Furthermore, the execution of this program executes a method corresponding to the program. Note that some or all of the functions executed by the CPU 21 may be configured hardware-wise by one or more ICs, etc. Also, the number of microcomputers constituting the control unit 11 may be one or more.

[0041] The CAN communication unit 12 is connected to be capable of data communication with a plurality of ECUs via a communication line, and transmits and receives data in accordance with the CAN communication protocol. Specifically, the plurality of ECUs connected to the CAN communication unit 12 are an engine ECU that performs engine control, a brake ECU that performs brake control, a steering ECU that performs steering control, a suspension ECU that performs suspension control, an ECU that controls turning on / off of lights, and the like. In FIG. 2, only ECU 16, ECU 17, and ECU 18 are shown as ECUs connected to the CAN communication unit 12. ECU is an abbreviation for Electronic Control Unit.

[0042] The storage unit 13 is a storage device for storing various types of data. The storage unit 13 is provided with a collected data database (hereinafter, collected data DB) 13a, an evaluation history data database (hereinafter, evaluation history data DB) 13b, a learning dataset database (hereinafter, learning dataset DB) 13c, and an evaluation dataset database (hereinafter, evaluation dataset DB) 13d.

[0043] The communication unit 14 performs data communication with the center 3 via the wide-area wireless communication network NW.

[0044] As shown in FIG. 3, the center 3 includes a control unit 31, a communication unit 32, and a storage unit 33.

[0045] The control unit 31 is an electronic control device mainly configured by a microcomputer including a CPU 41, a ROM 42, a RAM 43, and the like. Various functions of the microcomputer are realized by the CPU 41 executing a program stored in a non-transitional tangible recording medium. In this example, the ROM 42 corresponds to the non-transitional tangible recording medium storing the program. Furthermore, execution of this program implements a method corresponding to the program. Note that part or all of the functions executed by the CPU 41 may be configured in hardware by one or a plurality of ICs or the like. Furthermore, the number of microcomputers configuring the control unit 31 may be one or plural.

[0046] The communication unit 32 performs data communication with a plurality of data collection devices 2 via the wide-area wireless communication network NW.

[0047] The storage unit 33 is a storage device for storing various data.

[0048] FIG. 4 is a block diagram showing the procedure of inference processing executed by the data collection device 2.

[0049] As shown in FIG. 4, the data collection device 2 executes a user-oriented inference task 51. The user-oriented inference task 51 infers the driver's intention by using, as input data, dashboard image data representing an image displayed on a dashboard of a vehicle and driver operation data representing an operation performed by the driver of the vehicle. In the present embodiment, the user-oriented inference task 51 is performed by an AI model 56. AI is an abbreviation for Artificial Intelligence. The AI model 56 is a machine learning model generated by executing machine learning using a collected dataset.

[0050] Examples of the operation performed by the driver include an accelerator operation, a brake operation, and a steering operation. Further, examples of the operation performed by the driver also include an operation specified based on image data obtained by an in-vehicle camera capturing the driver.

[0051] The user-oriented inference task 51, for example, outputs an inference result that satisfies a preset high likelihood condition. In the present embodiment, the high likelihood condition is, for example, that the likelihood is 0.5 or higher.

[0052] The user-oriented inference task 51, for example, outputs a first inference result, a second inference result, and a third inference result in descending order of likelihood. The likelihoods of the first inference result, the second inference result, and the third inference result are, for example, 0.9, 0.7, and 0.5, respectively. For the first inference result, for example, the driver's intention is "open the driver's seat window". For the second inference result, for example, the driver's intention is "open all windows".

[0053] The user-facing inference task 51 may provide the vehicle occupant with a highly likely inference result (i.e., a first inference result), or it may provide the result to an in-vehicle application (not shown) that provides a service to the vehicle occupant according to the inference result. Examples of such in-vehicle applications include those that automatically open and close the vehicle windows, automatically control the air conditioning inside the vehicle, and assist with vehicle driving, in accordance with the estimated user's intentions.

[0054] The data collection device 2 executes the correct answer identification task 52. The correct answer identification task 52 is a process that identifies the correct answer of the inference result based on the inference result of the user-directed inference task 51 and reaction data (e.g., driver reaction data) that shows the user's (e.g., occupant such as the driver) reaction. For example, the correct answer identification task 52 identifies the inference result of the user-directed inference task 51 (e.g., open the driver's side window) that matches the user's reaction (e.g., the user opened the driver's side window, and the user did not close the driver's side window that the in-car app opened) as the correct answer, and identifies the inference result of the user-directed inference task 51 that does not match the user's reaction as an incorrect answer.

[0055] The correct answer identification task 52 may identify the correct driver's intention based on the input data of the user-facing reasoning task 51 (i.e., dashboard image data and driver operation data), the reasoning results of the user-facing reasoning task 51 (i.e., the first, second, and third reasoning results), and driver reaction data indicating the driver's response. For example, the user-facing reasoning task 51 outputs information identified from the input and output of the user-facing reasoning task 51 (e.g., the reasoning basis described later) along with the reasoning result, and when the user responds indicating that both the reasoning basis and the reasoning result are correct, the correct answer identification task 52 may identify the reasoning result as correct. When the user responds indicating that at least one of the reasoning basis and the reasoning result is incorrect, the correct answer identification task 52 may identify the reasoning result as incorrect.

[0056] The data collection device 2 links dashboard image data, driver operation data, and driver reaction data with correct answer data indicating the correct answer identified by the correct answer identification task 52, and stores them as collected data in the collected data DB 13a. The data collection device 2 may also link dashboard image data, driver operation data, and driver reaction data with incorrect answer data indicating the incorrect answer identified by the correct answer identification task 52, and store them as collected data in the collected data DB 13a.

[0057] The data acquisition device 2 performs annotation on the collected data stored in the collected data DB 13a, adding correct labels. Incorrect labels may also be added during the annotation process.

[0058] The data collection device 2 stores the dashboard image data and driver operation data, to which correct and incorrect labels have been added through annotation, as annotated data in the evaluation history data DB 13b.

[0059] Figure 5 is a block diagram showing the procedure for weakness identification processing performed by the data acquisition device 2.

[0060] As shown in Figure 5, the data acquisition device 2 performs an annotated data count check to determine whether the number of annotated data is equal to or greater than a preset processing start determination number.

[0061] If the number of annotated data points is equal to or greater than the number used to determine the start of processing, the data acquisition device 2 executes the user-facing inference task 51. The user-facing inference task 51 outputs, for example, the first inference result, the second inference result, and the third inference result in descending order of likelihood.

[0062] Furthermore, by saving the inference results of the user-facing inference task 51 in the above inference process, it is possible to omit executing the user-facing inference task 51 in the weakness identification process. However, there is a risk that the AI ​​model 56 of the user-facing inference task 51 may be updated by another process before the number of annotated data exceeds the number of processing start determinations. For this reason, in this embodiment, the data acquisition device 2 executes the user-facing inference task 51 in the weakness identification process.

[0063] Next, the data acquisition device 2 performs the weakness identification task 53. The weakness identification task 53 is performed by the AI ​​model 57.

[0064] The data collection device 2 links the first, second, and third inference results of the user-facing inference task 51 with the input data for the weakness identification task 53 (i.e., dashboard image data and driver operation data) and stores them in the evaluation history data DB 13b.

[0065] The weakness identification task 53 includes an inference basis transformation process, a classification process, and a weakness detection process.

[0066] The inference basis conversion process is a process that outputs the first, second, and third recognition results, which are obtained by converting the inference basis for each of the first, second, and third inference results output by the user-facing inference task 51 into text, based on the first, second, and third inference results and the annotated data corresponding to the first, second, and third inference results. The inference basis conversion process includes, for example, inputting input data (i.e., dashboard image data and driver operation data) and inference results into a large-scale language model and causing the large-scale language model to output text representing the inference basis.

[0067] For example, the text supporting the first inference result (i.e., the first recognition result) could be "Voice recognition result + Because there is no one in the driver's seat." For example, the text supporting the second inference result (i.e., the second recognition result) could be "Voice recognition result + It was determined that ventilation is needed due to rising temperature inside the vehicle."

[0068] The classification process involves mapping one or more features corresponding to the basis for the inference results of the annotated data corresponding to the first, second, and third inference results onto a feature space composed of a predetermined set of features. Examples of features include brightness and the number of pedestrians.

[0069] The classification process outputs a classification result showing the result of the above mapping, and features corresponding to the basis for the inference result. The classification result allows for visualization of the entire feature space. Therefore, engineers can determine what features the AI ​​model 57 has identified as being included in the scene, thereby improving explainability.

[0070] The weakness detection process combines qualitative weakness detection using qualitative data and quantitative weakness detection using quantitative data to identify weaknesses in the AI ​​model 56 for the user-facing inference task 51.

[0071] Both qualitative and quantitative weakness detection focus on cases where the most likely inference result (i.e., the first inference result) differs from the correct answer.

[0072] Qualitative weakness detection involves analyzing the inference basis text for the first inference result using a grammatical analysis model, for example, with grammatical rules such as 5W1H, to determine which features the AI ​​model 57 is focusing on.

[0073] Quantitative weakness detection calculates the difference in features between the first inference result and the ground truth data, based on the classification results in the feature space. The difference includes information such as distance in the feature space, where the magnitude of the distance corresponds to the magnitude of the difference.

[0074] The weakness detection process then identifies the features with the largest differences among the features that the AI ​​model 57 focuses on as features that represent weaknesses (hereinafter referred to as weakness features).

[0075] Next, the data acquisition device 2 executes the data acquisition condition generation task 54. The data acquisition condition generation task 54 is performed by the AI ​​model 58.

[0076] The collection condition generation task 54 generates a collection condition file based on one or more weak features identified by the weak detection process. The collection condition file contains one or more collection conditions for the data acquisition device 2 to perform data acquisition. For example, the collection condition generation task 54 generates a collection condition file that sets collection conditions for collecting data from scenes containing weak features.

[0077] The data collection device 2 stores the one or more weakness features output by the weakness identification task 53 and the collection condition file output by the collection condition generation task 54 in the evaluation history data DB 13b, linking them to the input data (i.e., dashboard image data, driver operation data, and the first, second, and third inference results) input to the weakness identification task 53.

[0078] Figure 6 is a block diagram showing the steps of the learning process performed by the data acquisition device 2.

[0079] As shown in Figure 6, the data acquisition device 2 acquires data based on the acquisition condition file generated by the acquisition condition generation task 54 and stores the acquired data in the acquisition data DB 13a.

[0080] The collection conditions set in the collection conditions file generated by the collection conditions generation task 54 are conditions for collecting data from scenes corresponding to weaknesses. Therefore, the collected data collected based on the collection conditions file generated by the collection conditions generation task 54 can be considered as data collected from scenes corresponding to weaknesses.

[0081] The data acquisition device 2 performs annotation to add correct labels to the collected data stored in the collected data DB 13a that has not yet been annotated with correct labels. As a result, the collected data collected based on the collection condition file generated by the collection condition generation task 54 is assigned the correct labels corresponding to the collection condition file.

[0082] The data collection device 2 divides multiple annotated data sets into training data and evaluation data based on predefined splitting criteria. The splitting criteria are, for example, the proportion of data points per class. Classes are classified by, for example, time of day (e.g., morning, noon, night, etc.) or weather (e.g., sunny, cloudy, rainy, etc.).

[0083] The data collection device 2 stores multiple annotated data sets, which have been divided into training data, as training datasets in the training dataset DB 13c. The data collection device 2 also stores multiple annotated data sets, which have been divided into evaluation data, as evaluation datasets in the evaluation dataset DB 13d.

[0084] The data collection device 2 performs a data count check to determine whether the number of training data and the number of evaluation data are equal to or greater than a predetermined threshold.

[0085] If the number of training data and the number of evaluation data are both equal to or greater than the standard number, the data acquisition device 2 uses the training data stored in the training data set DB 13c and the evaluation data stored in the evaluation data set DB 13d to train and evaluate the AI ​​model 56 that performs the user-oriented inference task 51.

[0086] Specifically, the data acquisition device 2 first performs training on the user-facing inference task 51 using the training dataset. As a result, the parameters in the AI ​​model 56 corresponding to the user-facing inference task 51 are changed.

[0087] Next, after the user-facing inference task 51 has finished training, the data acquisition device 2 evaluates the inference results of the trained user-facing inference task 51 using the evaluation dataset and outputs the evaluation results of the user-facing inference task 51.

[0088] Next, the data acquisition device 2 uses the evaluation results for the user-facing inference task 51, along with input data (i.e., dashboard image data, driver operation data, and ground truth data) and output data (i.e., weak feature data) from multiple past cycles of the weakness identification task 53, to perform learning and evaluation of the weakness identification task 53.

[0089] For example, if the evaluation result indicates that the inference of the user-facing inference task 51 does not meet the pre-set evaluation criteria, the data acquisition device 2 modifies the parameters related to the weak feature (i.e., the output data of the weakness identification task 53 in the previous cycle) used to generate the collection conditions set for collecting the training data and evaluation data used in the training of the user-facing inference task 51.

[0090] Furthermore, the data collection device 2 can learn, based on the input and output data and evaluation results of the weakness identification task 53 in multiple past cycles, what kind of output data the weakness identification task 53 outputs when given certain input data, and whether or not the inference of the user-facing inference task 51 has improved as a result. Based on these learning results and the current evaluation results for the user-facing inference task 51, the data collection device 2 modifies the parameters in the AI ​​model 57 that correspond to the weakness identification task 53.

[0091] Furthermore, the data acquisition device 2 uses the evaluation results for the user-facing inference task 51 and the input and output data for the weakness identification task 53 from the previous cycle to evaluate how the weakness identification task 53 has improved as a result of the changes made in the previous cycle.

[0092] Next, the data acquisition device 2 uses the evaluation results for the user-facing inference task 51, along with the input data (i.e., weak features) and output data (i.e., collection condition files) from multiple past cycles of the collection condition generation task 54, to perform training and evaluation of the collection condition generation task 54.

[0093] For example, if the evaluation result indicates that the inference of the user-facing inference task 51 does not meet the pre-set evaluation criteria, the data acquisition device 2 modifies the parameters related to the collection condition file used to generate the collection conditions set for collecting the training data and evaluation data used in the training of the user-facing inference task 51.

[0094] Furthermore, the data acquisition device 2 can learn, based on the input and output data and evaluation results of the data acquisition condition generation task 54 in multiple past cycles, what kind of output data the data acquisition condition generation task 54 outputs when certain input data is received, and whether or not the inference of the user-facing inference task 51 has improved as a result. Based on these learning results and the current evaluation results for the user-facing inference task 51, the data acquisition device 2 modifies the parameters in the AI ​​model 58 that correspond to the data acquisition condition generation task 54.

[0095] Furthermore, the data acquisition device 2 uses the evaluation results for the user-facing inference task 51 and the input and output data for the collection condition generation task 54 from the previous cycle to evaluate how the collection condition generation task 54 has improved as a result of the changes made in the previous cycle.

[0096] The data acquisition device 2 generates evaluation results for the learning of the user-oriented inference task 51 by performing learning and evaluation of the user-oriented inference task 51, and stores the generated evaluation results in the evaluation history data DB 13b.

[0097] The data collection device 2 generates evaluation results for the data collection condition generation task 54 by performing learning and evaluation of the data collection condition generation task 54, and stores the generated evaluation results in the evaluation history data DB 13b.

[0098] The data collection device 2 generates evaluation results for the data collection condition generation task 54 by performing learning and evaluation of the data collection condition generation task 54, and stores the generated evaluation results in the evaluation history data DB 13b.

[0099] If the evaluation result of the user-facing inference task 51 is satisfactory, the data collection device 2 replaces the AI ​​model 56 running the user-facing inference task 51 with a newly trained AI model 56. On the other hand, if the evaluation result of the user-facing inference task 51 is unsatisfactory, the data collection device 2 retrieves the data used when the evaluation result was unsatisfactory from the evaluation history data DB 13b and executes the weakness identification task 53 and the collection condition generation task 54 again.

[0100] Similarly, if the evaluation result of the weakness identification task 53 is satisfactory, the data acquisition device 2 replaces the AI ​​model 57 performing the weakness identification task 53 with a newly trained AI model 57. On the other hand, if the evaluation result of the weakness identification task 53 is unsatisfactory, the data acquisition device 2 retrieves the data used when the evaluation result was unsatisfactory from the evaluation history data DB 13b and executes the weakness identification task 53 and the collection condition generation task 54 again.

[0101] Furthermore, if the evaluation result of the collection condition generation task 54 is satisfactory, the data acquisition device 2 replaces the AI ​​model 58 executing the collection condition generation task 54 with a newly trained AI model 58. On the other hand, if the evaluation result of the collection condition generation task 54 is unsatisfactory, the data acquisition device 2 retrieves the data used when the evaluation result was unsatisfactory from the evaluation history data DB 13b and executes the weakness identification task 53 and the collection condition generation task 54 again.

[0102] As shown in Figure 7, the storage unit 13 of the data acquisition device 2 stores the learning and evaluation script 131, AI model application data 132, dataset construction profile 133, model evaluation profile 134, inference flow log 135, weakness identification and collection condition generation flow log 136, and learning and evaluation flow log 137.

[0103] The learning and evaluation script 131 is a file containing programs for performing the learning and evaluation of AI models 56, 57, and 58.

[0104] The AI ​​model application data 132 includes data for setting up AI models 56, 57, and 58, and executable files for applications that perform the inference process, weakness identification process, and learning process described above.

[0105] The dataset construction profile 133 includes data indicating the splitting criteria for dividing multiple annotated data into training data and evaluation data.

[0106] The model evaluation profile 134 includes data indicating the number of processing start decisions set to initiate the weakness identification task 53 and the collection condition generation task 54.

[0107] The inference flow log 135 includes data indicating the operational status of the inference process described above. The weakness identification / collection condition generation flow log 136 includes data indicating the operational status of the weakness identification process described above. The learning / evaluation flow log 137 includes data indicating the operational status of the learning process described above.

[0108] Center 3 comprises a log monitoring unit 61 and an OTA unit 62 as functional blocks realized by the CPU 41 executing a program stored in ROM 42. OTA stands for Over The Air.

[0109] The data acquisition device 2 uses the communication unit 14 to transmit the inference flow log 135, the weakness identification / collection condition generation flow log 136, and the learning / evaluation flow log 137 to the log monitoring unit 61 of the center 3.

[0110] The log monitoring unit 61 provides engineers with information contained in the inference flow log 135, the weakness identification / collection condition generation flow log 136, and the learning / evaluation flow log 137, in response to requests from engineers developing AI models.

[0111] The log monitoring unit 61 determines whether or not there are any problems with the operation status of the above-mentioned inference process, weakness identification process, and learning process based on the inference flow log 135, the weakness identification / collection condition generation flow log 136, and the learning / evaluation flow log 137.

[0112] If a new AI model is generated to solve the problem that has occurred, the newly generated AI model is distributed to the data acquisition device 2 by the OTA unit 62.

[0113] Next, the procedure for the inference process performed by the data acquisition device 2 will be described. The inference process is a process that is repeatedly executed while the control unit 11 of the data acquisition device 2 is operating.

[0114] When the inference process is executed, the CPU 21 of the control unit 11 acquires the dashboard image data, driver operation data, and driver reaction data that were most recently input to the data acquisition device 2 in S10, as shown in Figure 8.

[0115] In S20, the CPU 21 executes a user-facing inference task 51 using the dashboard image data and driver operation data acquired in S10.

[0116] In S30, the CPU 21 displays the first inference result generated by the user-facing inference task 51, for example, on a dashboard.

[0117] In S40, the CPU 21 obtains the dashboard image data, driver operation data, and driver reaction data acquired in S10, as well as the inference results of the user-facing inference task 51 (i.e., the first, second, and third inference results).

[0118] In S50, the CPU 21 executes the correct answer identification task 52 using the data and inference results acquired in S40.

[0119] In S60, the CPU 21 links the dashboard image data and driver operation data with the correct answer data indicating the correct answer identified by the correct answer identification task 52 and saves them to the collected data DB 13a.

[0120] In step S70, the CPU 21 performs annotation, which involves adding correct labels to the collected data stored in the collected data DB 13a.

[0121] In S80, the CPU 21 saves the dashboard image data and driver operation data, to which correct labels have been added by annotation, as annotated data to the evaluation history data DB 13b, and then terminates the inference process.

[0122] Next, the procedure for the vulnerability identification process performed by the data acquisition device 2 will be explained. The vulnerability identification process is a process that is repeatedly performed while the control unit 11 of the data acquisition device 2 is operating.

[0123] When the weakness identification process is executed, the CPU 21 of the control unit 11 determines in S110 whether or not annotated data has been accumulated, as shown in Figure 9. Specifically, the CPU 21 determines whether or not the number of annotated data is equal to or greater than a preset number for determining when to start processing.

[0124] If no annotated data has been accumulated, the CPU 21 terminates the weakness identification process. On the other hand, if annotated data has been accumulated, the CPU 21 retrieves the accumulated annotated data in S120.

[0125] In S130, the CPU 21 executes a user-facing inference task 51 using the annotated data acquired in S120.

[0126] In S140, the CPU 21 links the inference result of the user-facing inference task 51 in S130 with the input data to the user-facing inference task 51 in S130 (i.e., dashboard image data and driver operation data) and saves it to the evaluation history data DB 13b.

[0127] In S150, the CPU 21 executes the above-mentioned inference basis transformation process.

[0128] CPU 21 executes the above classification process in S160.

[0129] CPU 21 performs the above qualitative weakness detection in S170.

[0130] CPU 21 performs the above quantitative weakness detection in S180.

[0131] In S190, the CPU 21 determines whether or not there are any features with a large difference. If there are no features with a large difference, the CPU 21 terminates the weakness identification process. On the other hand, if there are features with a large difference, the CPU 21 identifies the features with a large difference as weakness features in S200.

[0132] In S210, the CPU 21 stores one or more weakness features identified in S200 in the evaluation history data DB 13b, linking them to the input data entered into the weakness identification task 53.

[0133] CPU 21 executes the collection condition generation task 54 in S220.

[0134] In S230, the CPU 21 links the collection condition file output by the collection condition generation task 54 to the input data entered into the weakness identification task 53, saves it to the evaluation history data DB 13b, and terminates the weakness identification process.

[0135] Next, the procedure for the learning process performed by the data acquisition device 2 will be described. The learning process is a process that is repeatedly executed while the control unit 11 of the data acquisition device 2 is operating.

[0136] When the learning process is executed, the CPU 21 of the control unit 11 collects data in S300 based on the generated collection condition file, as shown in Figure 10, and saves the collected data (hereinafter referred to as collected data) to the collected data DB 13a.

[0137] In S310, the CPU 21 determines whether or not a predetermined number of data points have been collected. If a predetermined number of data points has not been collected, the CPU 21 terminates the learning process. On the other hand, if a predetermined number of data points have been collected, the CPU 21 performs annotation in S320, adding correct labels to the collected data.

[0138] In S330, the CPU 21 divides the multiple annotated collected data into training data and evaluation data based on predefined splitting criteria.

[0139] In S340, the CPU 21 determines whether the number of training data and the number of evaluation data are equal to or greater than a predetermined threshold. If the number of training data and the number of evaluation data are not equal to or greater than the threshold, the CPU 21 terminates the training process.

[0140] On the other hand, if the number of training data and the number of evaluation data are both above the standard number, the CPU 21 retrieves the previous evaluation results of the weakness identification task 53 and the collection condition generation task 54 from the evaluation history data DB 13b in S350.

[0141] CPU 21 performs training for user-facing inference task 51 in S360.

[0142] In S370, the CPU 21 performs an evaluation of the user-facing inference task 51 after the training was performed in S360.

[0143] CPU 21 performs the learning of the weakness identification task 53 in S380.

[0144] CPU 21 performs the evaluation of the weakness identification task 53 in S390.

[0145] CPU 21 performs training for the data collection condition generation task 54 in S400.

[0146] In S410, the CPU 21 performs an evaluation of the data collection condition generation task 54.

[0147] In S420, the CPU 21 saves the evaluation results of the processes in S370, S390, and S410 (i.e., the evaluation results of the user-facing inference task 51, the weakness identification task 53, and the collection condition generation task 54) to the evaluation history data DB 13b. The CPU 21 also saves the evaluation results of the processes in S370, S390, and S410 in association with the input data entered into the weakness identification task 53.

[0148] CPU 21 determines in S430 whether the evaluation result is a pass or fail. The processing in S430 is performed for each of the evaluation results of the processes in S370, S390, and S410.

[0149] If the evaluation result is a pass, the CPU 21 replaces the AI ​​model performing the task corresponding to the pass evaluation result with the newly trained AI model in S440, and terminates the training process.

[0150] On the other hand, if the evaluation result is unsatisfactory, the CPU 21 retrieves the evaluation history data from the evaluation history data DB 13b for the time when the evaluation result was unsatisfactory (i.e., the input data entered into the weakness identification task 53 and the collection condition generation task 54 in the current cycle) at S450, and then executes the weakness identification task 53 and the collection condition generation task 54 again to terminate the learning process.

[0151] The data collection device 2 configured in this way uses the AI ​​model 56 to perform inference based on dashboard image data and driver operation data (hereinafter referred to as input data), and outputs the inference results.

[0152] The data collection device 2 identifies the correct answer for inference based on the input data.

[0153] The data collection device 2 performs annotation based on the input data and the correct answer data that identifies the correct answer, and generates annotated data that includes the input data.

[0154] The data collection device 2 identifies weaknesses in the inference performed by the AI ​​model 56 based on multiple annotated data and the first, second, and third inference results output for each of the multiple annotated data.

[0155] The data collection device 2 generates collection conditions for collecting input data that addresses weaknesses.

[0156] The data acquisition device 2 collects data based on the generated collection conditions.

[0157] The data collection device 2 generates multiple training data and multiple evaluation data by dividing the collected data into training data and evaluation data.

[0158] The data collection device 2 trains the AI ​​model 56 using multiple training data sets.

[0159] The data collection device 2 evaluates the AI ​​model 56 after training using multiple evaluation data sets.

[0160] If the evaluation result is unsatisfactory, the data collection device 2 executes a weakness identification task 53 and a data collection condition generation task 54 to identify weaknesses in the AI ​​model 56, generates data collection conditions corresponding to the identified weaknesses, and then sequentially performs data collection using the newly generated data collection conditions, generates training data and evaluation data, trains the AI ​​model 56, and evaluates the AI ​​model 56, repeating this cycle as one cycle.

[0161] Such a data acquisition device 2 identifies the correct inference based on input data, generates annotated data, and further identifies weaknesses to generate collection conditions. Subsequently, it sequentially performs data acquisition, generation of training data and evaluation data, training of the AI ​​model 56, evaluation of the AI ​​model 56, and generation of collection conditions, and by repeatedly executing this cycle, the AI ​​model 56 can be trained. Therefore, the data acquisition device 2 can reduce the intervention of engineers required to train the AI ​​model 56 and reduce the workload of engineers developing the AI ​​model 56.

[0162] Furthermore, the data collection device 2 infers the driver's intentions based on the input data. The data collection device 2 identifies the correct answer to the inference based on driver reaction data that shows the driver's response. As a result, the data collection device 2 can generate annotated data for training the AI ​​model 56 that infers the driver's intentions.

[0163] The data collection device 2 also outputs one or more inference results (e.g., first, second, and third inference results) that satisfy a pre-set high-likelihood condition indicating a high likelihood. The data collection device 2 identifies weaknesses based on the inference results that do not match the correct answer among the one or more inference results that satisfy the high-likelihood condition.

[0164] Specifically, the data collection device 2 generates the reasoning basis for inference results that do not match the correct answer based on the input and output of the AI ​​model 56, and identifies weaknesses by identifying the features corresponding to the reasoning basis. The data collection device 2 also identifies features corresponding to weaknesses by mapping the input data when the AI ​​model 56 outputs an inference result that does not match the correct answer in a feature space composed of a set of features.

[0165] Furthermore, based on the evaluation results of the AI ​​model 56, the data collection device 2 replaces the currently operating AI model 56 with the trained AI model 56 if the AI ​​model 56 has been improved. This allows the data collection device 2 to provide the driver with an AI model 56 with improved inference accuracy.

[0166] Furthermore, in each of the multiple cycles, the data acquisition device 2 stores evaluation data linked to the evaluation results of the AI ​​model 56 in the evaluation history data DB 13b. The data acquisition device 2 then uses the evaluation data from the multiple cycles stored in the evaluation history data DB 13b to train the AI ​​models 57 and 58. This allows the data acquisition device 2 to improve the inference accuracy of the AI ​​models 57 and 58.

[0167] Furthermore, the data acquisition device 2 evaluates the learning of AI models 57 and 58 using evaluation data from multiple cycles. The data acquisition device 2 then stores the evaluation results of AI models 57 and 58 in the evaluation history data DB 13b, linked to the evaluation data for the current cycle. This allows the data acquisition device 2 to evaluate the learning of AI models 57 and 58 and to verify the evaluation based on the evaluation results.

[0168] The multiple data sets collected based on the collection conditions generated by the data collection device 2 are collected from a single vehicle equipped with the data collection device 2. This allows the data collection device 2 to repeat the above cycle for training the AI ​​model 56 within a single vehicle.

[0169] Furthermore, the data acquisition device 2 is mounted on the vehicle. This allows the data acquisition device 2 to repeat the above cycle for learning the AI ​​model 56 within a single vehicle.

[0170] In the embodiments described above, the data acquisition device 2 corresponds to a model learning system, the AI ​​model 56 corresponds to a machine learning model, S20 and S120 correspond to processing as an inference result output unit, S50 corresponds to processing as a correct answer identification unit, and S70 corresponds to processing as an annotation execution unit.

[0171] Furthermore, S110 to S200 correspond to processing as a weakness identification unit, S220 corresponds to processing as a collection condition generation unit, S300 corresponds to processing as a data collection unit, and S310 to S330 correspond to processing as a learning data generation unit.

[0172] Furthermore, S360 corresponds to processing as a model learning unit, S370 corresponds to processing as a model evaluation unit, and S450 corresponds to processing as a re-execution unit.

[0173] Furthermore, the driver corresponds to the user, the driver reaction data corresponds to reaction data, S440 corresponds to processing as a model replacement unit, AI model 56 corresponds to the first machine learning model, and AI models 57 and 58 correspond to the second machine learning models.

[0174] Furthermore, S210, S230, and S420 correspond to processing as a data storage unit, S380 and S400 correspond to processing as a second machine learning model learning unit, S390 and S410 correspond to processing as a second machine learning model evaluation unit, and S420 corresponds to processing as an evaluation storage unit.

[0175] Although one embodiment of the present disclosure has been described above, the present disclosure is not limited to the above embodiment and can be implemented in various modified forms.

[0176] [Modification 1] In the above embodiment, a form was shown in which the correct answer is identified by the driver's reaction. However, in cases where it is necessary to identify the correct answer by the driver's reaction, there is a concern that there will be insufficient training data linked to the driver's reaction. Taking this into consideration, a condition may be added as a collection condition to acquire data with similar features to the scene in which the driver's reaction occurred, without the condition of the driver's reaction, so that training data similar to the data in which the driver's reaction occurred can be collected.

[0177] For example, if you want to train the system to prevent automatic wiper operation from being overwritten by the driver by training it to recognize scenes where the automatic wiper operation was overwritten by the driver, you can set up data collection conditions that collect only data where the image features of the scene are similar.

[0178] Specifically, the data acquisition device 2 identifies the feature quantities of the input data for the user-oriented inference task 51, which identifies the correct answer based on the driver's response, links the identified feature quantities to the correct answer, and stores them as correct answer-linked feature quantities.

[0179] The data collection device 2 then generates annotated data for the input data of the user-oriented inference task 51, which could not obtain driver reaction data and could not identify the correct answer based on the driver's response. This data is considered correct if it has a similarity of a predetermined level or higher to the correct answer-associated features.

[0180] As a result, the data acquisition device 2 can generate training data and evaluation data even when it is unable to acquire driver reaction data. Therefore, when the data acquisition device 2 repeats the above cycle with a single vehicle, it can suppress situations in which there is a shortage of training data and evaluation data.

[0181] The control unit 11 and its method described in this disclosure may be implemented by a dedicated computer provided by configuring a processor and memory programmed to perform one or more functions embodied by a computer program. Alternatively, the control unit 11 and its method described in this disclosure may be implemented by a dedicated computer provided by configuring a processor by one or more dedicated hardware logic circuits. Alternatively, the control unit 11 and its method described in this disclosure may be implemented by one or more dedicated computers configured by a combination of a processor and memory programmed to perform one or more functions and a processor configured by one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by the computer on a computer-readable non-transitional tangible recording medium. The method for realizing the functions of each part included in the control unit 11 does not necessarily need to include software, and all of its functions may be realized using one or more hardware components.

[0182] Multiple functions of one component in the above embodiment may be realized by multiple components, or one function of one component may be realized by multiple components. Furthermore, multiple functions of multiple components may be realized by one component, or one function realized by multiple components may be realized by one component. Also, some parts of the configuration of the above embodiment may be omitted. Furthermore, at least some parts of the configuration of the above embodiment may be added to or replaced with the configuration of other above embodiments.

[0183] In addition to the data acquisition device 2 described above, this disclosure can also be realized in various forms, such as a system that uses the data acquisition device 2 as a component, a program for making the computer function as the data acquisition device 2, a non-transitional tangible recording medium such as a semiconductor memory on which this program is recorded, and a model learning method. [Technical concept disclosed in this specification] [Item 1] An inference result output unit (S20, 56) configured to perform inference based on input data using a machine learning model (56) and output the inference result.S120) A correct answer identification unit (S50) configured to identify the correct answer of the inference based on the input data; an annotation execution unit (S70) configured to perform annotation based on the input data input to the inference result output unit and the correct answer data indicating the correct answer identified by the correct answer identification unit, and to generate annotated data including the input data; a weakness identification unit (S110 to S200) configured to identify weaknesses in the inference by the machine learning model based on a plurality of annotated data and a plurality of inference results output by the inference result output unit for each of the plurality of annotated data; a collection condition generation unit (S220) configured to generate collection conditions for collecting the input data corresponding to the weaknesses; and a data collection unit (S300) configured to collect data based on the generated collection conditions. A learning data generation unit (S310 to S330) configured to generate multiple learning data and multiple evaluation data by dividing multiple collected data, which are multiple data collected by the data collection unit, into learning data and evaluation data; a model learning unit (S360) configured to train the machine learning model using the multiple learning data; and a model evaluation unit (S370) configured to evaluate the machine learning model after it has been trained by the model learning unit using the multiple evaluation data. Model learning system (2) includes a re-execution unit (S450) configured to repeat the cycle, in which, if the evaluation result by the model evaluation unit is unsatisfactory, the weakness identification unit and the collection condition generation unit are activated to identify the weakness of the machine learning model, the collection conditions corresponding to the identified weakness are generated, and the data collection unit collects data using the newly generated collection conditions, the learning data generation unit generates the learning data and evaluation data, the model learning unit performs learning, and the model evaluation unit performs evaluation, as one cycle.

[0184] [Item 2] A model learning system according to Item 1, wherein the inference result output unit is configured to infer the user's intent based on the input data, and the correct answer identification unit is configured to identify the correct answer of the inference based on reaction data indicating the user's response.

[0185] [Item 3] A model learning system according to Item 1 or Item 2, wherein the inference result output unit is configured to output one or more inference results that satisfy a preset high-likelihood condition indicating a high likelihood, and the weakness identification unit is configured to identify weaknesses based on the inference results that do not match the correct answer among the one or more inference results that satisfy the high-likelihood condition.

[0186] [Item 4] A model learning system as described in Item 3, wherein the weakness identification unit is configured to identify the weakness by generating inference basis inferred by the machine learning model for the inference result that does not match the correct answer, based on the input and output of the machine learning model, and by identifying feature quantities corresponding to the inference basis.

[0187] [Item 5] A model learning system according to Item 3 or Item 4, wherein the weakness identification unit is configured to identify the feature corresponding to the weakness by mapping the input data when the machine learning model outputs an inference result that does not match the correct answer in a feature space composed of a plurality of pre-set features.

[0188] [Item 6] A model learning system according to any one of Items 1 to 5, further comprising a model replacement unit (S440) configured to replace the machine learning model in operation with the machine learning model after it has been trained by the model learning unit when the machine learning model has been improved based on the evaluation results of the model evaluation unit.

[0189] [Item 7] A model learning system according to any one of Items 1 to 6, wherein the machine learning model of the inference result output unit is set as a first machine learning model (56), the weakness identification unit and the collection condition generation unit are configured to identify weaknesses and generate collection conditions using a second machine learning model (57, 58) different from the first machine learning model, and the model learning system further comprises: a data storage unit (S210, S230, S420) configured to store evaluation data linking the input / output data of the weakness identification unit and the collection condition generation unit with the evaluation results of the first machine learning model in each of the multiple cycles, and a second machine learning model learning unit (S380, S400) configured to learn the second machine learning model using the evaluation data of the multiple cycles stored by the data storage unit.

[0190] [Item 8] A model learning system as described in Item 7, further comprising: a second machine learning model evaluation unit (S390, S410) configured to evaluate the learning of the second machine learning model by the second machine learning model learning unit using the evaluation data of a plurality of cycles; and an evaluation storage unit (S420) configured to save the evaluation result of the second machine learning model by the second machine learning model evaluation unit in association with the evaluation data for the current cycle.

[0191] [Item 9] A model learning system according to any one of Items 1 to 7, wherein the plurality of collected data collected based on the collection conditions generated by the collection condition generation unit are data collected from a single vehicle.

[0192] [Item 10] A model learning system as described in Item 9, wherein the learning data generation unit identifies the features of the input data of the machine learning model that have identified the correct answer based on the user's response as correct answer-associated features, and for the input data of the machine learning model that has not been able to obtain reaction data indicating the user's response and therefore could not identify the correct answer based on the user's response, it is treated as the correct answer and annotation is performed on the condition that it has a similarity of a predetermined or higher to the correct answer-associated features, thereby generating the learning data and the evaluation data.

[0193] [Item 11] A model learning system as described in Item 9 or Item 10, wherein the model learning system is mounted on the vehicle.

[0194] [Item 12] A model learning method executed by a model learning system (2), comprising: using a machine learning model (56), performing inference based on input data and outputting inference results; identifying the correct answer of the inference based on the input data; performing annotation based on the input data and the correct answer data indicating the identified correct answer to generate annotated data including the input data; identifying weaknesses in the inference by the machine learning model based on a plurality of annotated data and a plurality of inference results output for each of the plurality of annotated data; generating collection conditions for collecting the input data corresponding to the weaknesses; collecting data based on the generated collection conditions; generating a plurality of training data and a plurality of evaluation data by dividing the plurality of collected data into training data and evaluation data; training the machine learning model using the plurality of training data; and evaluating the machine learning model after training using the plurality of evaluation data. A model learning method that, if the evaluation result is unsatisfactory, identifies the weakness of the machine learning model, generates collection conditions corresponding to the identified weakness, sequentially performs data collection using the newly generated collection conditions, generates training data and evaluation data, trains the machine learning model, and evaluates the machine learning model, with the cycle being repeated as one cycle.

[0195] [Item 13] An inference result output unit (S20, 56) configured to use a machine learning model (56) to perform inference based on input data and output the inference result.S120), a correct answer identification unit (S50) configured to identify the correct answer of the inference based on the input data, an annotation execution unit (S70) configured to perform annotation based on the input data input to the inference result output unit and the correct answer data indicating the correct answer identified by the correct answer identification unit, and to generate annotated data including the input data, a weakness identification unit (S110 to S200) configured to identify weaknesses in the inference by the machine learning model based on a plurality of annotated data and a plurality of inference results output by the inference result output unit for each of the plurality of annotated data, a collection condition generation unit (S220) configured to generate collection conditions for collecting the input data corresponding to the weaknesses, a data collection unit (S300) configured to collect data based on the generated collection conditions, a learning data generation unit (S310 to S330) configured to generate a plurality of learning data and a plurality of evaluation data by dividing a plurality of collected data, which are a plurality of data collected by the data collection unit, into learning data and evaluation data. A model learning program that functions as a model learning unit (S360) configured to learn the machine learning model using multiple training data, a model evaluation unit (S370) configured to evaluate the machine learning model after it has been learned by the model learning unit using multiple evaluation data, and a re-execution unit (S450) configured to repeat the cycle, in which, if the evaluation result by the model evaluation unit is unsatisfactory, the weakness identification unit and the collection condition generation unit are activated to identify the weaknesses of the machine learning model, the collection conditions corresponding to the identified weaknesses are generated, and the data collection unit collects data using the newly generated collection conditions, the learning data generation unit generates the training data and evaluation data, the model learning unit performs training, and the model evaluation unit performs evaluation, in which this is performed sequentially as one cycle.

Claims

1. An inference result output unit (S20, S120) configured to perform inference based on input data using a machine learning model (56) and output the inference result; a correct answer identification unit (S50) configured to identify the correct answer of the inference based on the input data; an annotation execution unit (S70) configured to perform annotation based on the input data input to the inference result output unit and the correct answer data indicating the correct answer identified by the correct answer identification unit, and to generate annotated data including the input data; a weakness identification unit (S110 to S200) configured to identify weaknesses in the inference by the machine learning model based on a plurality of annotated data and a plurality of inference results output by the inference result output unit for each of the plurality of annotated data; a collection condition generation unit (S220) configured to generate collection conditions for collecting the input data corresponding to the weaknesses; and a data collection unit (S300) configured to collect data based on the generated collection conditions. A learning data generation unit (S310 to S330) configured to generate multiple learning data and multiple evaluation data by dividing multiple collected data, which are multiple data collected by the data collection unit, into learning data and evaluation data; a model learning unit (S360) configured to train the machine learning model using the multiple learning data; and a model evaluation unit (S370) configured to evaluate the machine learning model after it has been trained by the model learning unit using the multiple evaluation data. Model learning system (2) comprising a re-execution unit (S450) configured to repeat a cycle in which, if the evaluation result by the model evaluation unit is unsatisfactory, the weakness identification unit and the collection condition generation unit are activated to identify the weakness of the machine learning model, the collection conditions corresponding to the identified weakness are generated, and the data collection unit collects data using the newly generated collection conditions, the learning data generation unit generates the learning data and evaluation data, the model learning unit performs learning, and the model evaluation unit performs evaluation, in a single cycle.

2. A model learning system according to claim 1, wherein the inference result output unit is configured to infer the user's intent based on the input data, and the correct answer identification unit is configured to identify the correct answer of the inference based on reaction data indicating the user's response.

3. A model learning system according to claim 1 or claim 2, wherein the inference result output unit is configured to output one or more inference results that satisfy a preset high-likelihood condition indicating a high likelihood, and the weakness identification unit is configured to identify weaknesses based on the inference results that do not match the correct answer among the one or more inference results that satisfy the high-likelihood condition.

4. A model learning system according to claim 3, wherein the weakness identification unit is configured to identify weaknesses by generating inference basis inferred by the machine learning model for the inference result that does not match the correct answer, based on the input and output of the machine learning model, and by identifying feature quantities corresponding to the inference basis.

5. A model learning system according to claim 3, wherein the weakness identification unit is configured to identify the feature corresponding to the weakness by mapping the input data when the machine learning model outputs an inference result that does not match the correct answer in a feature space composed of a plurality of pre-set features.

6. A model learning system according to claim 1 or claim 2, further comprising a model replacement unit (S440) configured to replace the machine learning model in operation with the machine learning model after it has been trained by the model learning unit when the machine learning model has been improved based on the evaluation results of the model evaluation unit.

7. A model learning system according to claim 1 or claim 2, wherein the machine learning model of the inference result output unit is set as a first machine learning model (56), the weakness identification unit and the collection condition generation unit are configured to identify weaknesses and generate collection conditions using a second machine learning model (57, 58) different from the first machine learning model, and the model learning system further comprises: a data storage unit (S210, S230, S420) configured to store evaluation data linking the input / output data of the weakness identification unit and the collection condition generation unit with the evaluation results of the first machine learning model in each of the plurality of cycles, and a second machine learning model learning unit (S380, S400) configured to learn the second machine learning model using the evaluation data of the plurality of cycles stored by the data storage unit.

8. A model learning system according to claim 7, further comprising: a second machine learning model evaluation unit (S390, S410) configured to evaluate the learning of the second machine learning model by the second machine learning model learning unit using the evaluation data of a plurality of cycles; and an evaluation storage unit (S420) configured to store the evaluation result of the second machine learning model by the second machine learning model evaluation unit in association with the evaluation data for the current cycle.

9. A model learning system according to claim 1 or claim 2, wherein the plurality of collected data collected based on the collection conditions generated by the collection condition generation unit are data collected from a single vehicle.

10. A model learning system according to claim 9, wherein the learning data generation unit identifies the features of the input data of the machine learning model that has identified the correct answer based on the user's response as correct answer-associated features, and generates the learning data and evaluation data by considering the input data of the machine learning model that could not obtain reaction data indicating the user's response and could not identify the correct answer based on the user's response as correct answers and performing annotation on the condition that it has a similarity of a predetermined or higher to the correct answer-associated features.

11. A model learning system according to claim 9, wherein the model learning system is mounted on the vehicle.

12. A model learning method executed by a model learning system (2), comprising: using a machine learning model (56), performing inference based on input data and outputting an inference result; identifying the correct answer of the inference based on the input data; performing annotation based on the input data and the correct answer data indicating the identified correct answer to generate annotated data including the input data; identifying weaknesses in the inference by the machine learning model based on a plurality of annotated data and a plurality of inference results output for each of the plurality of annotated data; generating collection conditions for collecting the input data corresponding to the weaknesses; collecting data based on the generated collection conditions; generating a plurality of training data and a plurality of evaluation data by dividing the plurality of collected data into training data and evaluation data; training the machine learning model using the plurality of training data; and evaluating the machine learning model after training using the plurality of evaluation data. A model learning method that, if the evaluation result is unsatisfactory, identifies the weakness of the machine learning model, generates collection conditions corresponding to the identified weakness, sequentially performs data collection using the newly generated collection conditions, generates training data and evaluation data, trains the machine learning model, and evaluates the machine learning model, with the cycle being repeated as one cycle.

13. An inference result output unit (S20, S120) configured to perform inference based on input data using a machine learning model (56) and output the inference result; a correct answer identification unit (S50) configured to identify the correct answer of the inference based on the input data; an annotation execution unit (S70) configured to perform annotation based on the input data input to the inference result output unit and the correct answer data indicating the correct answer identified by the correct answer identification unit, and to generate annotated data including the input data; a weakness identification unit (S110 to S200) configured to identify weaknesses in the inference by the machine learning model based on a plurality of annotated data and a plurality of inference results output by the inference result output unit for each of the plurality of annotated data; a collection condition generation unit (S220) configured to generate collection conditions for collecting the input data corresponding to the weaknesses; and a data collection unit (S300) configured to collect data based on the generated collection conditions. A model learning program comprising: a learning data generation unit (S310 to S330) configured to generate multiple learning data and multiple evaluation data by dividing multiple collected data, which are multiple data collected by the data collection unit, into learning data and evaluation data; a model learning unit (S360) configured to learn the machine learning model using the multiple learning data; a model evaluation unit (S370) configured to evaluate the machine learning model after it has been learned by the model learning unit using the multiple evaluation data; and a re-execution unit (S450) configured to function as a cycle in which, if the evaluation result by the model evaluation unit is unsatisfactory, the weakness identification unit and the collection condition generation unit are activated to identify the weakness of the machine learning model, the collection conditions corresponding to the identified weakness are generated, and the data collection unit uses the newly generated collection conditions to sequentially perform data collection, the learning data generation unit generates the learning data and evaluation data, the model learning unit performs learning, and the model evaluation unit performs evaluation, and repeats the cycle.

Images