Machine learning model, model training method, data collection system, and on-vehicle device

Abstract

A machine learning model (50) causes a computer to function as a first input unit (S10, S120), an inference result generation unit (S20, S130), a first output unit (S30, S140), a second input unit (S120), a weak point identification unit (S150 to S200), a collection condition generation unit (S220), and a second output unit (S230). The first input unit receives input of first input data. The inference result generation unit performs inference on the basis of the first input data and generates an inference result. The first output unit outputs the inference result. The second input unit receives input of second input data including a plurality of pieces of first input data and a plurality of pieces of ground truth data. The weak point identification unit identifies a weak point in the inference. The collection condition generation unit generates a collection condition corresponding to the weak point. The second output unit outputs collection condition data indicating the collection condition.

Description

Machine learning model, model learning method, data collection system and in-vehicle device Cross-reference to Related Applications

[0001] This international application claims the benefit of Japanese Patent Application No. 2025-055451 filed with the Japan Patent Office on March 28, 2025, the entire disclosure of which is incorporated into this international application by reference.

[0002] The present disclosure relates to a machine learning model for collecting data.

[0003] Patent Document 1 describes a system that collects vehicle data from a plurality of in-vehicle devices respectively mounted on a plurality of vehicles.

[0004] In the development of machine learning models mounted on vehicles, generation of training data for training the machine learning model, training of the machine learning model, and evaluation of the machine learning model are performed on a center side that collects vehicle data from a plurality of in-vehicle devices.

[0005] Japanese Unexamined Patent Publication No. 2023-84379

[0006] As a result of detailed studies by the inventors, it has been found that when the configuration of a data collection system using a machine learning model becomes complicated, the work load on engineers in the development of the machine learning model increases.

[0007] The present disclosure reduces the work load of engineers developing machine learning models.

[0008] One aspect of the present disclosure is a machine learning model for causing a computer to function as a first input unit, an inference result generation unit, a first output unit, a second input unit, a weakness identification unit, a collection condition generation unit, and a second output unit.

[0009] The first input unit is configured to input first input data.

[0010] The inference result generation unit is configured to perform inference based on the first input data and generate one or a plurality of inference results.

[0011] The first output unit is configured to output the one or more inference results generated by the inference result generation unit.

[0012] The second input unit is configured to receive a dataset as second input data, which includes a plurality of first input data and a plurality of correct answer data indicating the correct answer to the inference result generated by the inference result generation unit based on each of the plurality of first input data.

[0013] The weakness identification unit is configured to identify weaknesses in the inference performed by the inference result generation unit based on the second input data.

[0014] The collection condition generation unit is configured to generate collection conditions for collecting first input data that corresponds to weaknesses.

[0015] The second output unit is configured to output collection condition data indicating the collection conditions as second output data.

[0016] The machine learning model of this disclosure, configured in this manner, can sequentially operate the inference result generation unit, the weakness identification unit, and the collection condition generation unit by inputting multiple first input data to the inference result generation unit via the second input unit, and output second output data via the second output unit. In other words, the machine learning model of this disclosure can operate the inference result generation unit, the weakness identification unit, and the collection condition generation unit in conjunction within the machine learning model by inputting to the inference result generation unit. Therefore, when generating collection conditions for training the machine learning model of this disclosure, it is not necessary to prepare three types of input data for the inference result generation unit, the weakness identification unit, and the collection condition generation unit, respectively, but only one type of input data to be input to the second input unit is required, thus reducing the workload of engineers developing machine learning models.

[0017] Another aspect of this disclosure is a model learning method in which a model learning device performs training on a machine learning model comprising an inference result generation unit, a weakness identification unit, and a collection condition generation unit.

[0018] In the model learning method of this disclosure, the model learning device collects input data, which is input to the inference result generation unit, as collected data based on the generated collection conditions, and then trains a machine learning model using the multiple collected data, repeating this cycle.

[0019] The data used to train the machine learning model in each of the multiple cycles includes a dataset comprising, as first training data for training the inference result generation unit of the machine learning model, multiple input data collected in the current cycle, and multiple ground truth data representing the correct answers to the inference results generated by the inference result generation unit based on each of the multiple input data.

[0020] The data used to train the machine learning model in each of the multiple cycles includes, as second training data for training the machine learning model's weakness identification unit and collection condition generation unit, the data input to the machine learning model when generating collection conditions in previous cycles, and the data output by the collection condition generation unit when generating collection conditions in previous cycles.

[0021] The model training method described herein is a method performed to train the machine learning model described herein, and by performing this method, the same effects as the machine learning model described herein can be obtained.

[0022] A further aspect of this disclosure is a machine learning model for causing a computer to function as a first input unit, an inference result generation unit, a first output unit, a second input unit, a weakness identification unit, and a weakness output unit.

[0023] The weakness output unit is configured to output weakness data that indicates the weaknesses.

[0024] The machine learning model of this disclosure, configured in this manner, can sequentially operate the inference result generation unit and the weakness identification unit by inputting multiple first input data to the inference result generation unit via the second input unit, and output weakness data via the weakness output unit. In other words, the machine learning model of this disclosure can operate the inference result generation unit and the weakness identification unit in conjunction within the machine learning model by inputting to the inference result generation unit. Therefore, when identifying weaknesses in order to train the machine learning model, the machine learning model of this disclosure does not need to separately prepare two types of input data for the inference result generation unit and the weakness identification unit, but only one type of input data to be input to the second input unit is required, thus reducing the workload of engineers developing machine learning models.

[0025] Another aspect of this disclosure is a model learning method in which a model learning device learns a machine learning model comprising an inference result generation unit and a weakness identification unit.

[0026] In the model learning method of this disclosure, the model learning device collects input data, which is input to the inference result generation unit, as collected data based on the generated collection conditions, and then trains a machine learning model using the multiple collected data, repeating this cycle.

[0027] The data used to train the machine learning model in each of the multiple cycles includes a dataset comprising, as first training data for training the inference result generation unit of the machine learning model, multiple input data collected in the current cycle, and multiple ground truth data representing the correct answers to the inference results generated by the inference result generation unit based on each of the multiple input data.

[0028] The data used to train the machine learning model in each of the multiple cycles includes, as second training data for training the weakness identification unit of the machine learning model, the data input to the machine learning model when identifying weaknesses in previous cycles, and the data output by the weakness identification unit in previous cycles.

[0029] The model training method described herein is a method performed to train the machine learning model described herein, and by performing this method, the same effects as the machine learning model described herein can be obtained.

[0030] A further aspect of the present disclosure is a data acquisition system comprising a plurality of on-board devices, each mounted in a plurality of vehicles and configured to transmit vehicle data, each including at least information about the vehicle on which it is mounted; and a center configured to receive vehicle data from the plurality of on-board devices.

[0031] In the data collection system disclosed herein, the center includes a machine learning model that causes a computer to function as a first input unit, an inference result generation unit, a first output unit, a second input unit, a weakness identification unit, a collection condition generation unit, and a second output unit.

[0032] The center is configured to distribute the collection condition data to multiple in-vehicle devices.

[0033] Multiple in-vehicle devices are configured to collect vehicle data based on collection condition data distributed from the center.

[0034] The machine learning model uses multiple vehicle data points transmitted from the in-vehicle device as the first input data, which is included in the second input data.

[0035] The data collection system disclosed herein is a system equipped with the machine learning model disclosed herein and can achieve the same effects as the machine learning model disclosed herein.

[0036] A further aspect of this disclosure is an in-vehicle device mounted on a vehicle, the in-vehicle device comprising a machine learning model for causing a computer to function as a first input unit, an inference result generation unit, a first output unit, a second input unit, a weakness identification unit, a collection condition generation unit, and a second output unit.

[0037] The in-vehicle device of this disclosure is configured to collect vehicle data based on collection condition data output by a machine learning model.

[0038] The machine learning model uses multiple vehicle data points collected by the in-vehicle device as the first input data, which is included in the second input data.

[0039] The in-vehicle device of this disclosure is a device equipped with the machine learning model of this disclosure, and can achieve the same effects as the machine learning model of this disclosure.

[0040] This is a block diagram showing the configuration of the data collection system of the first embodiment. This is a block diagram showing the configuration of the data collection device of the first embodiment. This is a block diagram showing the configuration of the center of the first embodiment. This is a block diagram showing the configuration of the integrated AI model of the first embodiment. This is a block diagram showing the procedure for the inference process of the first embodiment. This is a block diagram showing the procedure for the weakness identification process of the first embodiment. This is a block diagram showing the procedure for the learning process of the first embodiment. This is a flowchart showing the inference process of the first embodiment. This is a flowchart showing the weakness identification process of the first embodiment. This is a flowchart showing the learning process of the first embodiment. This is a block diagram showing the configuration of the data collection device of the second embodiment. This is a block diagram showing the configuration of the center of the second embodiment. This is a block diagram showing the configuration of the integrated AI model of the second embodiment. This is a block diagram showing the procedure for the weakness identification process of the second embodiment. This is a block diagram showing the procedure for the learning process of the second embodiment. This is a flowchart showing the weakness identification process of the second embodiment. This is a block diagram showing the configuration of the integrated AI model of the third embodiment.

[0041] [First Embodiment] A first embodiment of the present disclosure will be described below with reference to the drawings.

[0042] As shown in Figure 1, the data acquisition system 1 of this embodiment comprises a plurality of data acquisition devices 2 and a center 3.

[0043] The data collection device 2 is mounted on the vehicle and has the function of communicating data with the center 3 via a wide-area wireless communication network NW. The data collection device 2 transmits the data collected in the vehicle to the center 3.

[0044] The center 3 is a device that manages the data collection system 1. The center 3 has a function of performing data communication with a plurality of data collection devices 2 via a wide area wireless communication network NW. The center 3 accumulates data transmitted from the plurality of data collection devices 2.

[0045] As shown in FIG. 2, the data collection device 2 includes a control unit 11, a CAN communication unit 12, a storage unit 13, and a communication unit 14. CAN is an abbreviation for Controller Area Network.

[0046] The control unit 11 is an electronic control device configured mainly with a microcomputer including a CPU 21, a ROM 22, a RAM 23, and the like. Various functions of the microcomputer are realized by the CPU 21 executing a program stored in a non-transitional tangible recording medium. In this example, the ROM 22 corresponds to the non-transitional tangible recording medium storing the program. Furthermore, execution of this program implements a method corresponding to the program. Note that part or all of the functions executed by the CPU 21 may be configured in hardware by one or more ICs or the like. Furthermore, the number of microcomputers configuring the control unit 11 may be one or plural.

[0047] The CAN communication unit 12 is connected to be capable of data communication with a plurality of ECUs via a communication line, and transmits and receives data in accordance with a CAN communication protocol. Specifically, the plurality of ECUs connected to the CAN communication unit 12 include an engine ECU that performs engine control, a brake ECU that performs brake control, a steering ECU that performs steering control, a suspension ECU that performs suspension control, and an ECU that controls turning on / off of lights, among others. In FIG. 2, only ECU 16, ECU 17, and ECU 18 are shown as ECUs connected to the CAN communication unit 12. ECU is an abbreviation for Electronic Control Unit.

[0048] The storage unit 13 is a storage device for storing various types of data. The storage unit 13 is equipped with a collection data database (hereinafter referred to as the collection data DB) 13a, an evaluation history data database (hereinafter referred to as the evaluation history data DB) 13b, a learning dataset database (hereinafter referred to as the learning dataset DB) 13c, and an evaluation dataset database (hereinafter referred to as the evaluation dataset DB) 13d.

[0049] The communications unit 14 performs data communication with the center 3 via the wide-area wireless communication network NW.

[0050] As shown in Figure 3, the center 3 comprises a control unit 31, a communication unit 32, and a storage unit 33.

[0051] The control unit 31 is an electronic control device centered around a microcomputer equipped with a CPU 41, ROM 42, RAM 43, etc. The various functions of the microcomputer are realized by the CPU 41 executing a program stored in a non-transitional physical recording medium. In this example, the ROM 42 corresponds to the non-transitional physical recording medium that stores the program. Furthermore, the execution of this program executes a method corresponding to the program. Note that some or all of the functions executed by the CPU 41 may be configured hardware-wise by one or more ICs, etc. Also, the number of microcomputers constituting the control unit 31 may be one or more.

[0052] The communication unit 32 performs data communication with multiple data acquisition devices 2 via a wide-area wireless communication network NW.

[0053] The memory unit 33 is a memory device for storing various types of data.

[0054] As shown in Figure 4, the data acquisition device 2 includes an integrated AI model 50 as a functional block realized by the CPU 21 executing a program stored in the ROM 22. AI stands for Artificial Intelligence. The integrated AI model 50 is a machine learning model generated by performing machine learning using the collected dataset.

[0055] The integrated AI model 50 performs the user-facing inference task 51, the correct answer identification task 52, the weakness identification task 53, and the collection condition generation task 54, which will be described later.

[0056] Figure 5 is a block diagram showing the steps of the inference process performed by the integrated AI model 50.

[0057] As shown in Figure 5, the integrated AI model 50 performs a user-facing inference task 51. The user-facing inference task 51 takes dashboard image data, which shows images displayed on the vehicle's dashboard, and driver operation data, which shows operations performed by the vehicle's driver, as input data to infer the driver's intentions.

[0058] Driver actions include, for example, acceleration, braking, and steering. Additionally, driver actions identified based on image data captured by an on-board camera are also included.

[0059] The user-facing inference task 51 outputs, for example, an inference result that satisfies a pre-set high-likelihood condition. In this embodiment, the high-likelihood condition is, for example, that the likelihood is 0.5 or higher.

[0060] The user-facing inference task 51 outputs, for example, a first inference result, a second inference result, and a third inference result in descending order of likelihood. The likelihoods of the first, second, and third inference results are, for example, 0.9, 0.7, and 0.5, respectively. The first inference result is, for example, that the driver's intention is "to open the driver's side window." The second inference result is, for example, that the driver's intention is "to open all windows."

[0061] The user-facing inference task 51 provides the vehicle occupants with the most likely inference result (i.e., the first inference result) by displaying it, for example, on the dashboard. Alternatively, the user-facing inference task 51 may provide the most likely inference result (i.e., the first inference result) to an in-vehicle application (not shown) that provides services to the vehicle occupants according to the inference result. Examples of such in-vehicle applications include those that automatically open and close the vehicle windows, automatically control the in-vehicle air conditioning, and assist with vehicle driving, in accordance with the estimated user's intentions.

[0062] The integrated AI model 50 performs a correct answer identification task 52. The correct answer identification task 52 is a process that identifies the correct answer to the inference result based on the inference result of the user-directed inference task 51 and reaction data (e.g., driver reaction data) that shows the user's (e.g., occupant such as the driver) reaction. For example, the correct answer identification task 52 identifies the inference result of the user-directed inference task 51 (e.g., open the driver's side window) as the correct answer if it matches the user's reaction (e.g., the user opened the driver's side window, and the user did not close the driver's side window that the in-car app opened), and identifies the inference result of the user-directed inference task 51 that does not match the user's reaction as the incorrect answer.

[0063] The correct answer identification task 52 may identify the correct driver's intention based on the input data of the user-facing reasoning task 51 (i.e., dashboard image data and driver operation data), the reasoning results of the user-facing reasoning task 51 (i.e., the first, second, and third reasoning results), and driver reaction data indicating the driver's response. For example, the user-facing reasoning task 51 outputs information identified from the input and output of the user-facing reasoning task 51 (e.g., the reasoning basis described later) along with the reasoning result, and when the user responds indicating that both the reasoning basis and the reasoning result are correct, the correct answer identification task 52 may identify the reasoning result as correct. When the user responds indicating that at least one of the reasoning basis and the reasoning result is incorrect, the correct answer identification task 52 may identify the reasoning result as incorrect.

[0064] The integrated AI model 50 links dashboard image data, driver operation data, and driver reaction data with correct answer data indicating the correct answer identified by the correct answer identification task 52, and stores them as collected data in the collected data DB 13a. The integrated AI model 50 may also link dashboard image data, driver operation data, and driver reaction data with incorrect answer data indicating the incorrect answer identified by the correct answer identification task 52, and store them as collected data in the collected data DB 13a.

[0065] The integrated AI model 50 performs annotation on the collected data stored in the collected data DB 13a, adding correct labels. Incorrect labels may also be added during the annotation process.

[0066] The integrated AI model 50 stores the dashboard image data and driver operation data, to which correct and incorrect labels have been added through annotation, as annotated data in the evaluation history data DB 13b.

[0067] Figure 6 is a block diagram showing the procedure for weakness identification processing performed by the integrated AI model 50.

[0068] As shown in Figure 6, the integrated AI model 50 performs an annotated data count check to determine whether the number of annotated data is equal to or greater than a pre-set processing start determination number.

[0069] If the number of annotated data points is greater than or equal to the number of data points required to initiate processing, the integrated AI model 50 executes a user-facing inference task 51. The user-facing inference task 51 outputs, for example, a first inference result, a second inference result, and a third inference result in descending order of likelihood.

[0070] Furthermore, by saving the inference results of the user-facing inference task 51 in the above inference process, it is possible to omit executing the user-facing inference task 51 in the weakness identification process. However, there is a risk that the user-facing inference task 51 may be updated by another process before the number of annotated data exceeds the number of processing start determinations. For this reason, in this embodiment, the integrated AI model 50 executes the user-facing inference task 51 in the weakness identification process.

[0071] Next, the integrated AI model 50 performs the weakness identification task 53.

[0072] The integrated AI model 50 links the first, second, and third inference results of the user-facing inference task 51 with the input data for the weakness identification task 53 (i.e., dashboard image data and driver operation data) and stores them in the evaluation history data DB 13b.

[0073] The weakness identification task 53 includes an inference basis transformation process, a classification process, and a weakness detection process.

[0074] The inference basis conversion process is a process that outputs the first, second, and third recognition results, which are obtained by converting the inference basis for each of the first, second, and third inference results output by the user-facing inference task 51 into text, based on the first, second, and third inference results and the annotated data corresponding to the first, second, and third inference results. The inference basis conversion process includes, for example, inputting input data (i.e., dashboard image data and driver operation data) and inference results into a large-scale language model and causing the large-scale language model to output text representing the inference basis.

[0075] For example, the text supporting the first inference result (i.e., the first recognition result) could be "Voice recognition result + Because there is no one in the driver's seat." For example, the text supporting the second inference result (i.e., the second recognition result) could be "Voice recognition result + It was determined that ventilation is needed due to rising temperature inside the vehicle."

[0076] The classification process involves mapping one or more features corresponding to the basis for the inference results of the annotated data corresponding to the first, second, and third inference results onto a feature space composed of a predetermined set of features. Examples of features include brightness and the number of pedestrians.

[0077] The classification process outputs a classification result showing the result of the above mapping, and features corresponding to the basis for the inference result. The classification result allows for visualization of the entire feature space. Therefore, engineers can determine what features the integrated AI model 50 has identified as being included in the scene, thereby improving explainability.

[0078] The weakness detection process combines qualitative weakness detection using qualitative data and quantitative weakness detection using quantitative data to identify weaknesses in the user-facing inference task 51.

[0079] Both qualitative and quantitative weakness detection focus on cases where the most likely inference result (i.e., the first inference result) differs from the correct answer.

[0080] Qualitative weakness detection involves analyzing the inference basis text for the first inference result using a grammatical analysis model, for example, with grammatical rules such as 5W1H, to determine which features the integrated AI model 50 is focusing on.

[0081] Quantitative weakness detection calculates the difference in features between the first inference result and the ground truth data, based on the classification results in the feature space. The difference includes information such as distance in the feature space, where the magnitude of the distance corresponds to the magnitude of the difference.

[0082] The weakness detection process then identifies the features with the largest differences among the features that the integrated AI model 50 focuses on as features that represent weaknesses (hereinafter referred to as weakness features).

[0083] Next, the integrated AI model 50 executes the collection condition generation task 54.

[0084] The collection condition generation task 54 generates a collection condition file based on one or more weak features identified by the weak detection process. The collection condition file contains one or more collection conditions for the data acquisition device 2 to perform data acquisition. For example, the collection condition generation task 54 generates a collection condition file that sets collection conditions for collecting data from scenes containing weak features.

[0085] The data collection device 2 stores the one or more weakness features output by the weakness identification task 53 and the collection condition file output by the collection condition generation task 54 in the evaluation history data DB 13b, linking them to the input data (i.e., dashboard image data, driver operation data, and the first, second, and third inference results) input to the weakness identification task 53.

[0086] Figure 7 is a block diagram showing the steps of the learning process performed by the data acquisition device 2.

[0087] As shown in Figure 7, the data acquisition device 2 acquires data based on the acquisition condition file generated by the acquisition condition generation task 54 and stores the acquired data in the acquisition data DB 13a.

[0088] The collection conditions set in the collection conditions file generated by the collection conditions generation task 54 are conditions for collecting data from scenes corresponding to weaknesses. Therefore, the collected data collected based on the collection conditions file generated by the collection conditions generation task 54 can be considered as data collected from scenes corresponding to weaknesses.

[0089] The data acquisition device 2 performs annotation to add correct labels to the collected data stored in the collected data DB 13a that has not yet been annotated with correct labels. As a result, the collected data collected based on the collection condition file generated by the collection condition generation task 54 is assigned the correct labels corresponding to the collection condition file.

[0090] The data collection device 2 divides multiple annotated data sets into training data and evaluation data based on predefined splitting criteria. The splitting criteria are, for example, the proportion of data points per class. Classes are classified by, for example, time of day (e.g., morning, noon, night, etc.) or weather (e.g., sunny, cloudy, rainy, etc.).

[0091] The data collection device 2 stores multiple annotated data sets, which have been divided into training data, as training datasets in the training dataset DB 13c. The data collection device 2 also stores multiple annotated data sets, which have been divided into evaluation data, as evaluation datasets in the evaluation dataset DB 13d.

[0092] The data collection device 2 performs a data count check to determine whether the number of training data and the number of evaluation data are equal to or greater than a predetermined threshold.

[0093] If the number of training data and the number of evaluation data are both equal to or greater than the standard number, the data acquisition device 2 uses the training data stored in the training data set DB 13c, the evaluation data stored in the evaluation data set DB 13d, and the various data stored in the evaluation history data DB 13b to perform training and evaluation of the integrated AI model 50.

[0094] Specifically, the data acquisition device 2 first performs training on the user-facing inference task 51 using the training dataset. This changes the parameters in the integrated AI model 50 that correspond to the user-facing inference task 51.

[0095] Next, after the user-facing inference task 51 has finished training, the data acquisition device 2 evaluates the inference results of the trained user-facing inference task 51 using the evaluation dataset and outputs the evaluation results of the user-facing inference task 51.

[0096] Next, the data acquisition device 2 uses the evaluation results for the user-facing inference task 51, along with input data (i.e., dashboard image data, driver operation data, and ground truth data) and output data (i.e., weak feature data) from multiple past cycles of the weakness identification task 53, to perform learning and evaluation of the weakness identification task 53.

[0097] For example, if the evaluation result indicates that the inference of the user-facing inference task 51 does not meet the pre-set evaluation criteria, the data acquisition device 2 modifies the parameters related to the weak feature (i.e., the output data of the weakness identification task 53 in the previous cycle) used to generate the collection conditions set for collecting the training data and evaluation data used in the training of the user-facing inference task 51.

[0098] Furthermore, the data acquisition device 2 can learn, based on the input and output data and evaluation results of the weakness identification task 53 in multiple past cycles, what kind of output data the weakness identification task 53 outputs when given certain input data, and whether or not the inference of the user-facing inference task 51 has improved as a result. Based on these learning results and the current evaluation results for the user-facing inference task 51, the data acquisition device 2 modifies the parameters corresponding to the weakness identification task 53.

[0099] Furthermore, the data acquisition device 2 uses the evaluation results for the user-facing inference task 51 and the input and output data for the weakness identification task 53 from the previous cycle to evaluate how the weakness identification task 53 has improved as a result of the changes made in the previous cycle.

[0100] Next, the data acquisition device 2 uses the evaluation results for the user-facing inference task 51, along with the input data (i.e., weak features) and output data (i.e., collection condition files) from multiple past cycles of the collection condition generation task 54, to perform training on the collection condition generation task 54.

[0101] For example, if the evaluation result indicates that the inference of the user-facing inference task 51 does not meet the pre-set evaluation criteria, the data acquisition device 2 modifies the parameters related to the collection condition file used to generate the collection conditions set for collecting the training data and evaluation data used in the training of the user-facing inference task 51.

[0102] Furthermore, the data acquisition device 2 can learn, based on the input and output data and evaluation results of the data acquisition condition generation task 54 in multiple past cycles, what kind of output data the data acquisition condition generation task 54 outputs when certain input data is received, and whether or not the inference of the user-facing inference task 51 has improved as a result. Based on these learned results and the current evaluation results for the user-facing inference task 51, the data acquisition device 2 modifies the parameters corresponding to the data acquisition condition generation task 54.

[0103] Furthermore, the data acquisition device 2 uses the evaluation results for the user-facing inference task 51 and the input and output data for the collection condition generation task 54 from the previous cycle to evaluate how the collection condition generation task 54 has improved as a result of the changes made in the previous cycle.

[0104] Therefore, the weakness identification task 53 and the data collection condition generation task 54 are updated in conjunction with parameter changes associated with the learning of the user-facing inference task 51.

[0105] Next, if the data collection device 2 determines that the evaluation result of the integrated AI model 50 (i.e., the evaluation result for the user-facing inference task 51) is satisfactory (i.e., the inference for the user-facing inference task 51 meets the evaluation criteria), it replaces the integrated AI model 50 currently in operation with the newly trained integrated AI model 50.

[0106] On the other hand, if the evaluation result of the integrated AI model 50 is unsatisfactory (i.e., the inference of the user-facing inference task 51 does not meet the evaluation criteria), the data acquisition device 2 retrieves the evaluation history data from the evaluation history data DB 13b for the time when the evaluation result was unsatisfactory (i.e., the input data entered into the weakness identification task 53 in the current cycle), and then executes the weakness identification task 53 and the collection condition generation task 54 again.

[0107] Next, the procedure for the inference process performed by the integrated AI model 50 will be described. The inference process is a process that is repeatedly executed while the control unit 11 of the data acquisition device 2 is operating.

[0108] When the inference process is executed, the integrated AI model 50 acquires the dashboard image data, driver operation data, and driver reaction data that were most recently input to the data acquisition device 2 in S10, as shown in Figure 8.

[0109] In S20, the integrated AI model 50 performs a user-facing inference task 51 using the dashboard image data and driver operation data acquired in S10.

[0110] In S30, the integrated AI model 50 displays the first inference result generated by the user-facing inference task 51, for example, on a dashboard.

[0111] In S40, the integrated AI model 50 acquires the dashboard image data, driver operation data, and driver reaction data acquired in S10, as well as the inference results of the user-facing inference task 51 (i.e., the first, second, and third inference results).

[0112] In S50, the integrated AI model 50 performs the correct answer identification task 52 using the data and inference results acquired in S40.

[0113] In S60, the integrated AI model 50 links the dashboard image data and driver operation data with the correct answer data identified by the correct answer identification task 52 and stores them in the collected data DB 13a.

[0114] In step S70, the integrated AI model 50 performs annotation on the collected data stored in the collected data DB 13a, adding correct labels.

[0115] In step S80, the integrated AI model 50 saves the dashboard image data and driver operation data, to which correct labels have been added by annotation, as annotated data in the evaluation history data DB 13b, and terminates the inference process.

[0116] Next, the procedure for the weakness identification process performed by the integrated AI model 50 will be explained. The weakness identification process is a process that is repeatedly performed while the control unit 11 of the data acquisition device 2 is operating.

[0117] When the weakness identification process is executed, the integrated AI model 50 determines in S110 whether or not annotated data has been accumulated, as shown in Figure 9. Specifically, the integrated AI model 50 determines whether or not the number of annotated data is equal to or greater than a preset number of processing start determinations.

[0118] If no annotated data has been accumulated, the integrated AI model 50 terminates the weakness identification process. On the other hand, if annotated data has been accumulated, the integrated AI model 50 retrieves the accumulated annotated data in S120.

[0119] In S130, the integrated AI model 50 executes a user-facing inference task 51 using the annotated data acquired in S120.

[0120] In S140, the integrated AI model 50 links the inference result of the user-facing inference task 51 in S130 with the input data for the user-facing inference task 51 in S130 (i.e., dashboard image data and driver operation data) and saves it to the evaluation history data DB 13b.

[0121] In S150, the integrated AI model 50 performs the above-mentioned inference basis transformation process.

[0122] The integrated AI model 50 performs the above classification process in S160.

[0123] The integrated AI model 50 performs the qualitative weakness detection described above in S170.

[0124] The integrated AI model 50 performs the quantitative weakness detection described above in S180.

[0125] In S190, the integrated AI model 50 determines whether or not there are any features with large differences. If there are no features with large differences, the integrated AI model 50 terminates the weakness identification process. On the other hand, if there are features with large differences, the integrated AI model 50 identifies these features with large differences as weakness features in S200.

[0126] In S210, the integrated AI model 50 stores one or more weak feature quantities identified in S200 in the evaluation history data DB 13b, linking them to the input data entered into the weak identification task 53.

[0127] In S220, the integrated AI model 50 executes the data collection condition generation task 54.

[0128] In step S230, the integrated AI model 50 links the collection condition file output by the collection condition generation task 54 to the input data entered into the weakness identification task 53, saves it to the evaluation history data DB 13b, and terminates the weakness identification process.

[0129] Next, the procedure for the learning process performed by the data acquisition device 2 will be described. The learning process is a process that is repeatedly executed while the control unit 11 of the data acquisition device 2 is operating.

[0130] When the learning process is executed, the CPU 21 of the control unit 11 collects data in S300 based on the generated collection condition file, as shown in Figure 10, and saves the collected data (hereinafter referred to as collected data) to the collected data DB 13a.

[0131] In S310, the CPU 21 determines whether or not a predetermined number of data points have been collected. If a predetermined number of data points has not been collected, the CPU 21 terminates the learning process. On the other hand, if a predetermined number of data points have been collected, the CPU 21 performs annotation in S320, adding correct labels to the collected data.

[0132] In S330, the CPU 21 divides the multiple annotated collected data into training data and evaluation data based on predefined splitting criteria.

[0133] In S340, the CPU 21 determines whether the number of training data and the number of evaluation data are equal to or greater than a predetermined threshold. If the number of training data and the number of evaluation data are not equal to or greater than the threshold, the CPU 21 terminates the training process.

[0134] On the other hand, if the number of training data and the number of evaluation data are both above the standard number, the CPU 21 retrieves the previous evaluation results of the weakness identification task 53 and the collection condition generation task 54 from the evaluation history data DB 13b in S350.

[0135] CPU 21 performs training and evaluation of the integrated AI model 50 in S360.

[0136] In S370, the CPU 21 saves the evaluation result of the processing in S360 to the evaluation history data DB 13b. The CPU 21 also saves the evaluation result of the processing in S360 in association with the input data entered into the integrated AI model 50.

[0137] The CPU 21 determines in S380 whether the evaluation result of the integrated AI model 50 is satisfactory or not.

[0138] If the evaluation result is satisfactory, the CPU 21 replaces the operational integrated AI model 50 with the newly trained integrated AI model 50 in S390 and terminates the training process.

[0139] On the other hand, if the evaluation result is unsatisfactory, the CPU 21 retrieves the evaluation history data from the evaluation history data DB 13b for the time when the evaluation result was unsatisfactory (i.e., the input data entered into the weakness identification task 53 in the current cycle) at S400, and then executes the weakness identification task 53 and the collection condition generation task 54 again, terminating the learning process.

[0140] The integrated AI model 50 configured in this way is a machine learning model that enables the computer to function as a first input unit, an inference result generation unit, a first output unit, a second input unit, a weakness identification unit, a collection condition generation unit, and a second output unit.

[0141] The first input unit is configured to receive dashboard image data and driver operation data (hereinafter referred to as the first input data).

[0142] The inference result generation unit is configured to perform inference based on the first input data and generate one or more inference results.

[0143] The first output unit is configured to output one or more inference results generated by the inference result generation unit.

[0144] The second input unit is configured to receive multiple annotated data (hereinafter referred to as "second input data") which includes multiple first input data and multiple correct answer data indicating the correct answer to the inference result generated by the inference result generation unit based on each of the multiple first input data.

[0145] The weakness identification unit is configured to identify weaknesses in the inference performed by the inference result generation unit based on the second input data.

[0146] The collection condition generation unit is configured to generate collection conditions for collecting first input data that corresponds to weaknesses.

[0147] The second output unit is configured to output a collection condition file (hereinafter referred to as the second output data) that indicates the collection conditions.

[0148] Such an integrated AI model 50 can sequentially operate the inference result generation unit, the weakness identification unit, and the collection condition generation unit by inputting multiple first input data to the inference result generation unit via the second input unit, and output second output data via the second output unit. In other words, the integrated AI model 50 can operate the inference result generation unit, the weakness identification unit, and the collection condition generation unit in conjunction within the machine learning model by inputting to the inference result generation unit. Therefore, when generating collection conditions for training the integrated AI model 50, it is not necessary to prepare three types of input data for the inference result generation unit, the weakness identification unit, and the collection condition generation unit, but only one type of input data to be input to the second input unit is required, thus reducing the workload of the engineers developing the integrated AI model 50.

[0149] Furthermore, the integrated AI model 50 is a machine learning model that causes the computer to function as a third output unit, which is configured to output data indicating the basis for each of the one or more inference results generated by the inference result generation unit, based on the second input data.

[0150] Such an integrated AI model 50 can provide engineers with information to consider for what reasons the integrated AI model 50 identified weaknesses.

[0151] The integrated AI model 50 is a machine learning model configured to function as a fourth output unit that further maps the second input data into a feature space composed of a set of predefined features, thereby outputting mapping data that shows the result of mapping the second input data into the feature space.

[0152] Such an integrated AI model 50 can provide engineers with information to consider when determining which features the integrated AI model 50 focused on to identify weaknesses.

[0153] The integrated AI model 50 is a machine learning model that enables the computer to function as a fifth output unit, which is configured to output weakness features that indicate the weaknesses identified by the weakness identification unit.

[0154] Such an integrated AI model 50 can provide engineers with information to consider when determining what features the integrated AI model 50 used to generate the data collection conditions.

[0155] Furthermore, the inference result output unit is configured to infer the driver's intentions based on the first input data. The integrated AI model 50 is a machine learning model that enables the computer to function as a third input unit, a correct answer identification unit, and a sixth output unit.

[0156] The third input unit is configured to receive data (hereinafter referred to as "third input data") that includes a plurality of first input data, a plurality of inference results output by the inference result output unit for each of the plurality of first input data, and driver reaction data indicating the driver's response for each of the plurality of first input data.

[0157] The correct answer identification unit is configured to identify the correct answer for each of the multiple first input data based on the third input data.

[0158] The sixth output unit is configured to output multiple correct answer data, each of which indicates the correct answer identified for each of the multiple first input data.

[0159] Such an integrated AI model 50 can generate multiple annotated data sets, each containing multiple first input data sets and multiple correct answer data sets for each of the multiple first input data sets.

[0160] The data acquisition device 2 also trains the integrated AI model 50.

[0161] The data collection device 2 collects the first input data, which is input to the inference result generation unit, as collected data based on the generated collection conditions. Furthermore, it repeats the cycle of training the integrated AI model 50 using the multiple collected data, which constitutes one cycle.

[0162] In each of the multiple cycles, the data used to train the machine learning model includes the first training data and the second training data.

[0163] The first training data is a dataset comprising multiple input data collected in the current cycle, and multiple ground truth data representing the correct answers for the inference results generated by the inference result generation unit based on each of the input data.

[0164] The second set of training data consists of data input to the integrated AI model 50 when generating collection conditions in multiple past cycles, and data output by the collection condition generation unit when generating collection conditions in multiple past cycles.

[0165] The model learning method performed by the data collection device 2 is a method performed to train the integrated AI model 50, and by performing this method, the same effects as the integrated AI model 50 can be obtained.

[0166] The data acquisition device 2 also includes an integrated AI model 50. The data acquisition device 2 is configured to collect vehicle data based on a collection condition file output by the integrated AI model 50. The integrated AI model 50 uses the multiple vehicle data collected by the data acquisition device 2 as the first input data included in the second input data.

[0167] Such a data acquisition device 2 is a device equipped with an integrated AI model 50, and can achieve the same effects as the integrated AI model 50.

[0168] In the embodiments described above, the integrated AI model 50 corresponds to a machine learning model, and the data acquisition device 2 corresponds to an in-vehicle device and a model learning device.

[0169] Furthermore, S10 and S120 correspond to processing as the first input unit, S20 and S130 correspond to processing as the inference result generation unit, and S30 and S140 correspond to processing as the first output unit.

[0170] Furthermore, multiple annotated data corresponds to a dataset, S120 corresponds to processing as a second input unit, S150 to S200 corresponds to processing as a weakness identification unit, S220 corresponds to processing as a collection condition generation unit, the collection condition file corresponds to collection condition data, and S230 corresponds to processing as a second output unit.

[0171] Furthermore, the first, second, and third recognition results correspond to data indicating the basis for inference and the third output data, and S150 corresponds to processing as the third output unit. The classification results correspond to mapping data and the fourth output data, and S160 corresponds to processing as the fourth output unit.

[0172] Furthermore, the weak feature quantity corresponds to the data indicating the weakness and the fifth output data, and S210 corresponds to processing as the fifth output unit.

[0173] Furthermore, the driver corresponds to the user, the driver reaction data corresponds to the reaction data, S40 corresponds to processing as the third input unit, S50 corresponds to processing as the correct answer identification unit, the multiple correct answer data corresponds to the sixth output data, and S60 corresponds to processing as the sixth output unit.

[0174] [Second Embodiment] A second embodiment of the present disclosure will be described below with reference to the drawings. In the second embodiment, the parts that differ from the first embodiment will be described. Common components will be denoted by the same reference numerals.

[0175] The data acquisition system 1 of the second embodiment differs from the first embodiment in that the configuration of the data acquisition device 2 and the center 3 has been changed.

[0176] The data collection device 2 of the second embodiment differs from the first embodiment in that, as shown in Figure 11, the evaluation history data DB 13b, the learning data set DB 13c, and the evaluation data set DB 13d are omitted in the storage unit 13.

[0177] The center 3 of the second embodiment differs from the first embodiment in that, as shown in Figure 12, the storage unit 33 is provided with a collection data DB 33a, an evaluation history data DB 33b, a learning dataset DB 33c, and an evaluation dataset DB 33d.

[0178] The data acquisition device 2 of the second embodiment acquires data based on the acquisition condition file distributed from the center 3 and transmits the acquired data to the center 3.

[0179] As shown in Figure 13, the center 3 of the second embodiment includes an integrated AI model 100 as a functional block realized by the CPU 41 executing a program stored in the ROM 42.

[0180] The integrated AI model 100 is an AI model that performs user-facing inference tasks 51, weakness identification tasks 53, and data collection condition generation tasks 54.

[0181] Figure 14 is a block diagram showing the procedure for weakness identification processing performed by the integrated AI model 100.

[0182] As shown in Figure 14, the integrated AI model 100 retrieves annotated data from the evaluation history data DB 33b and executes the user-facing inference task 51. The annotated data is prepared in advance and stored in the evaluation history data DB 33b.

[0183] Next, the integrated AI model 100 performs the weakness identification task 53.

[0184] The integrated AI model 100 links the first, second, and third inference results of the user-facing inference task 51 with the input data for the weakness identification task 53 and stores them in the evaluation history data DB 33b.

[0185] The weakness identification task 53 includes an inference basis transformation process, a classification process, and a weakness detection process.

[0186] The weakness detection process combines qualitative and quantitative weakness detection to identify weaknesses in the user-facing inference task 51.

[0187] Next, the integrated AI model 100 executes the collection condition generation task 54.

[0188] The collection condition generation task 54 generates a collection condition file based on one or more weak features identified by the weak feature identification process.

[0189] The integrated AI model 100 stores the one or more weakness features output by the weakness identification task 53 and the collection condition file output by the collection condition generation task 54 in the evaluation history data DB 33b, linking them to the input data entered into the weakness identification task 53.

[0190] Center 3 distributes the collection condition file generated by the integrated AI model 100 to multiple data collection devices 2 that constitute the data collection system 1.

[0191] Figure 15 is a block diagram showing the steps of the learning process performed by Center 3.

[0192] As shown in Figure 15, the data acquisition device 2 acquires data based on the acquisition condition file generated by the acquisition condition generation task 54 and transmits the acquired data to the center 3.

[0193] Center 3 stores the data transmitted from data acquisition device 2 in the collected data DB 33a.

[0194] Center 3 performs annotation on collected data stored in the collected data DB 33a that does not have a correct label attached, adding the correct label. As a result, the collected data collected based on the collection condition file generated by the collection condition generation task 54 is assigned the correct label corresponding to the collection condition file.

[0195] Center 3 divides the multiple annotated data sets into training data and evaluation data based on predefined splitting criteria.

[0196] Center 3 stores the multiple annotated data sets, which have been divided into training data, as training datasets in DB33c. Center 3 also stores the multiple annotated data sets, which have been divided into evaluation data, as evaluation datasets in DB33d.

[0197] Center 3 performs a data count check to determine whether the number of training data and the number of evaluation data are equal to or greater than the predetermined thresholds.

[0198] If the number of training data and the number of evaluation data are both equal to or greater than the standard number, Center 3 uses the training data stored in training data DB 33c, the evaluation data stored in evaluation data DB 33d, and the various data stored in evaluation history data DB 33b to train and evaluate the integrated AI model 100.

[0199] Next, if Center 3 determines that the evaluation result of the integrated AI model 100 is satisfactory, it replaces the operational integrated AI model 100 with the newly trained integrated AI model 100.

[0200] On the other hand, if the evaluation result of the integrated AI model 100 is unsatisfactory, Center 3 retrieves the evaluation history data from the evaluation history data DB 33b for the time when the evaluation result was unsatisfactory, and then executes the weakness identification task 53 and the collection condition generation task 54 again.

[0201] Next, the procedure for the weakness identification process performed by the integrated AI model 100 will be explained. The weakness identification process is initiated when a pre-set start condition is met during the operation of the control unit 31 of the center 3.

[0202] When the weakness identification process is executed, the integrated AI model 100 retrieves the annotated data from the evaluation history data DB 33b in S510, as shown in Figure 16.

[0203] The integrated AI model 100 executes the user-facing inference task 51 in S520.

[0204] In step S530, the integrated AI model 100 links the inference results of the user-facing inference task 51 in step S520 with the input data for the user-facing inference task 51 in step S520 and saves them in the evaluation history data DB 33b.

[0205] The integrated AI model 100 performs the above-mentioned inference basis transformation process in S540.

[0206] The integrated AI model 100 performs the above classification process in S550.

[0207] The integrated AI model 100 performs the qualitative weakness detection described above in S560.

[0208] The integrated AI model 100 performs the above quantitative weakness detection in S570.

[0209] In S580, the integrated AI model 100 determines whether or not there are any features with large differences. If there are no features with large differences, the integrated AI model 100 terminates the weakness identification process. On the other hand, if there are features with large differences, the integrated AI model 100 identifies these features with large differences as weakness features in S590.

[0210] In step S600, the integrated AI model 100 stores one or more weakness features identified in step S590 in the evaluation history data DB 33b, linking them to the input data entered into the weakness identification task 53.

[0211] In S610, the integrated AI model 100 executes the data collection condition generation task 54.

[0212] In step S620, the integrated AI model 100 links the collection condition file output by the collection condition generation task 54 to the input data entered into the weakness identification task 53, saves it to the evaluation history data DB 33b, and terminates the weakness identification process.

[0213] The data acquisition system 1 configured in this way includes a plurality of data acquisition devices 2, each of which is mounted on a plurality of vehicles and configured to transmit vehicle data that includes at least information about the vehicle on which it is mounted, and a center 3 configured to receive vehicle data from the plurality of data acquisition devices 2.

[0214] Center 3 is equipped with an integrated AI model 100.

[0215] The integrated AI model 100 is a machine learning model that enables a computer to function as a first input unit, an inference result generation unit, a first output unit, a second input unit, a weakness identification unit, a collection condition generation unit, and a second output unit.

[0216] Center 3 is configured to distribute the collection condition file to multiple data collection devices 2.

[0217] Multiple data acquisition devices 2 are configured to collect vehicle data based on a collection condition file distributed from the center 3.

[0218] The integrated AI model 100 uses multiple vehicle data transmitted from the data acquisition device 2 as the first input data included in the second input data.

[0219] Such a data collection system 1 is a device equipped with an integrated AI model 100, and can achieve the same effects as the integrated AI model 100.

[0220] In the embodiments described above, the integrated AI model 100 corresponds to a machine learning model, S510 corresponds to processing as a first input unit, S520 corresponds to processing as an inference result generation unit, and S530 corresponds to processing as a first output unit.

[0221] Furthermore, S510 corresponds to processing as a second input unit, S540 to S590 correspond to processing as a weakness identification unit, S610 corresponds to processing as a collection condition generation unit, and S620 corresponds to a second output unit.

[0222] Furthermore, S540 corresponds to processing as the third output unit, S550 corresponds to processing as the fourth output unit, and S600 corresponds to processing as the fifth output unit.

[0223] [Third Embodiment] A third embodiment of the present disclosure will be described below with reference to the drawings. In the third embodiment, the parts that differ from the first embodiment will be described. Common components will be denoted by the same reference numerals.

[0224] The data acquisition system 1 of the third embodiment differs from the first embodiment in that the configuration of the data acquisition device 2 has been changed.

[0225] As shown in Figure 17, the data acquisition device 2 is equipped with an integrated AI model 150 instead of an integrated AI model 50.

[0226] The integrated AI model 150 is an AI model that performs user-facing inference tasks 51, correct answer identification tasks 52, and weakness identification tasks 53.

[0227] The control unit 11 of the data collection device 2 generates a data collection condition file by executing a data collection condition generation task 54 using the weak feature quantities output from the integrated AI model 150.

[0228] The integrated AI model 150 configured in this way is a machine learning model that enables the computer to function as a first input unit, an inference result generation unit, a first output unit, a second input unit, a weakness identification unit, and a weakness output unit.

[0229] The weakness output unit is configured to output weakness features that indicate weaknesses.

[0230] Such an integrated AI model 150 can sequentially operate the inference result generation unit and the weakness identification unit by inputting multiple first input data to the inference result generation unit via the second input unit, and output weakness features via the weakness output unit. In other words, the integrated AI model 150 can operate the inference result generation unit and the weakness identification unit in conjunction within the integrated AI model 150 by inputting to the inference result generation unit. Therefore, when identifying weaknesses in order to train the integrated AI model 150, it is not necessary to separately prepare two types of input data for the inference result generation unit and the weakness identification unit, and only one type of input data to be input to the second input unit is required, thus reducing the workload of the engineers developing the integrated AI model 150.

[0231] The data acquisition device 2 also trains the integrated AI model 150.

[0232] The data collection device 2 collects input data, which is input to the inference result generation unit, as collected data based on the generated collection condition file. Furthermore, it repeats the cycle of training the integrated AI model 150 using the multiple collected data, which constitutes one cycle.

[0233] In each of the multiple cycles, the data used to train the machine learning model includes the first training data and the second training data.

[0234] The first training data is a dataset comprising multiple input data collected in the current cycle, and multiple ground truth data representing the correct answers for the inference results generated by the inference result generation unit based on each of the input data.

[0235] The second set of training data consists of data input to the integrated AI model 150 when identifying weaknesses in multiple past cycles, and data output by the weakness identification unit in multiple past cycles.

[0236] The model learning method executed by the data acquisition device 2 is a method performed to train the integrated AI model 150, and by executing this method, the same effects as the integrated AI model 150 can be obtained.

[0237] In the embodiments described above, the integrated AI model 150 corresponds to a machine learning model, S10 corresponds to processing as a first input unit, S20 corresponds to processing as an inference result generation unit, and S30 corresponds to processing as a first output unit.

[0238] Furthermore, S120 corresponds to processing as a second input unit, S150 to S200 correspond to processing as a weakness identification unit, and S210 corresponds to processing as a weakness output unit.

[0239] Although one embodiment of the present disclosure has been described above, the present disclosure is not limited to the above embodiment and can be implemented in various modified forms.

[0240] [Modification 1] In the above embodiment, a form was shown in which the correct answer is identified by the driver's reaction. However, in cases where it is necessary to identify the correct answer by the driver's reaction, there is a concern that there will be insufficient training data linked to the driver's reaction. Taking this into consideration, a condition may be added as a collection condition to acquire data with similar features to the scene in which the driver's reaction occurred, without the condition of the driver's reaction, so that training data similar to the data in which the driver's reaction occurred can be collected.

[0241] For example, if you want to train the system to prevent automatic wiper operation from being overwritten by the driver by training it to recognize scenes where the automatic wiper operation was overwritten by the driver, you can set up data collection conditions that collect only data where the image features of the scene are similar.

[0242] Specifically, the integrated AI model 50 identifies the feature quantities of the input data for the user-facing inference task 51, which identifies the correct answer based on the driver's response, and stores the identified feature quantities as correct answer-associated feature quantities, linking them to the correct answer.

[0243] The integrated AI model 50 then generates annotated data for the input data of the user-facing inference task 51, which could not obtain driver reaction data and could not identify the correct answer based on the driver's response. This data is considered correct if it has a similarity of a predetermined level or higher to the correct answer-associated features.

[0244] As a result, the integrated AI model 50 can generate training data and evaluation data even when driver reaction data cannot be acquired. Therefore, when the integrated AI model 50 repeats the above cycle with a single vehicle, it can suppress situations in which there is a shortage of training data and evaluation data.

[0245] The control units 11, 31 and their methods described in this disclosure may be implemented by a dedicated computer provided by configuring a processor and memory programmed to execute one or more functions embodied by a computer program. Alternatively, the control units 11, 31 and their methods described in this disclosure may be implemented by a dedicated computer provided by configuring a processor by one or more dedicated hardware logic circuits. Alternatively, the control units 11, 31 and their methods described in this disclosure may be implemented by one or more dedicated computers configured by a combination of a processor and memory programmed to execute one or more functions and a processor configured by one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by the computer on a computer-readable non-transitional tangible recording medium. The methods for realizing the functions of each part included in the control units 11, 31 do not necessarily need to include software, and all of its functions may be realized using one or more hardware components.

[0246] Multiple functions of one component in the above embodiment may be realized by multiple components, or one function of one component may be realized by multiple components. Furthermore, multiple functions of multiple components may be realized by one component, or one function realized by multiple components may be realized by one component. Also, some parts of the configuration of the above embodiment may be omitted. Furthermore, at least some parts of the configuration of the above embodiment may be added to or replaced with the configuration of other above embodiments.

[0247] In addition to the integrated AI models 50, 100, and 150 described above, this disclosure can also be realized in various forms, such as a system that uses the integrated AI models 50, 100, and 150 as components, a program for making a computer function as the integrated AI models 50, 100, and 150, a non-transitional tangible recording medium such as semiconductor memory that records this program, and a model learning method. [Technical Concept Disclosed in This Specification] [Item 1] A computer comprising: a first input unit (S10, S120, S510) configured to input first input data; an inference result generation unit (S20, S130, S520) configured to perform inference based on the first input data and generate one or more inference results; a first output unit (S30, S140, S530) configured to output one or more of the inference results generated by the inference result generation unit; a second input unit (S120, S510) configured to input a dataset including a plurality of the first input data and a plurality of correct answer data indicating the correct answer to the inference result generated by the inference result generation unit based on each of the plurality of the first input data as second input data; and a weakness identification unit (S150 to S200, S540 to S590) configured to identify weaknesses in the inference performed by the inference result generation unit based on the second input data. A machine learning model (50, 100) is configured to function as a collection condition generation unit (S220, S610) that generates collection conditions for collecting the first input data corresponding to the aforementioned weakness, and a second output unit (S230, S620) that outputs collection condition data indicating the collection conditions as second output data.

[0248] [Item 2] A machine learning model according to Item 1, wherein the computer is configured to function as a third output unit (S150, S540) that outputs data indicating the basis for inference for each of the one or more inference results generated by the inference result generation unit as third output data, based on the second input data.

[0249] [Item 3] A machine learning model according to Item 1 or Item 2, wherein the computer is configured to function as a fourth output unit (S160, S550) that further maps the second input data into a feature space composed of a plurality of pre-set features, thereby outputting mapping data as fourth output data that shows the result of mapping the second input data into the feature space.

[0250] [Item 4] A machine learning model described in any one of Items 1 to 3, wherein the computer is configured to function as a fifth output unit (S210, S600) that outputs data indicating the weakness identified by the weakness identification unit as fifth output data.

[0251] [Item 5] A machine learning model (50) described in any one of Items 1 to 4, wherein the inference result generation unit is configured to infer the user's intent based on the first input data, and the computer is further configured to function as: a third input unit (S40) configured to input data including a plurality of the first input data, a plurality of the inference results output by the first output unit for each of the plurality of the first input data, and reaction data indicating the user's response for each of the plurality of the first input data as third input data; a correct answer identification unit (S50) configured to identify the correct answer of the inference based on the first input data for each of the plurality of the first input data based on the third input data; and a sixth output unit (S60) configured to output a plurality of correct answer data indicating the correct answer identified for each of the plurality of the first input data as sixth output data.

[0252] [Item 6] A model learning method in which a model learning device (2) learns a machine learning model (50) comprising: an inference result generation unit (S20) configured to perform inference based on input data and generate one or more inference results; a weakness identification unit (S150 to S200) configured to identify weaknesses in the inference performed by the inference result generation unit; and a collection condition generation unit (S220) configured to generate collection conditions for collecting the input data corresponding to the weaknesses, wherein the model learning device (2) learns the machine learning model, and the input data input to the inference result generation unit is collected as collection data based on the generated collection conditions, and the machine learning model is learned using the multiple collected data, with each of the multiple cycles having a data set as first learning data for learning the inference result generation unit of the machine learning model, comprising a multiple input data set collected in the current cycle and a multiple correct answer data set indicating the correct answer to the inference result generated by the inference result generation unit based on each of the multiple input data sets, A model learning method for learning the weakness identification unit and the collection condition generation unit of the machine learning model, comprising, as second learning data, data input to the machine learning model when generating collection conditions in multiple past cycles, and data output by the collection condition generation unit when generating collection conditions in multiple past cycles.

[0253] [Item 7] A machine learning model (150) for a computer to function as: a first input unit (S10) configured to input first input data; an inference result generation unit (S20) configured to perform inference based on the first input data and generate one or more inference results; a first output unit (S30) configured to output one or more of the inference results; a second input unit (S120) configured to input a dataset as second input data, which includes a plurality of the first input data and a plurality of correct answer data indicating the correct answer to the inference result generated by the inference result generation unit based on each of the plurality of the first input data; a weakness identification unit (S150 to S200) configured to identify weaknesses in the inference performed by the inference result generation unit based on the second input data; and a weakness output unit (S210) configured to output weakness data indicating the weaknesses.

[0254] [Item 8] A model learning method in which a model learning device (2) learns a machine learning model (150) comprising an inference result generation unit (S20) configured to perform inference based on input data and generate one or more inference results, and a weakness identification unit (S150 to S200) configured to identify weaknesses in the inference performed by the inference result generation unit, wherein the input data input to the inference result generation unit is collected as collected data based on generated collection conditions, and the machine learning model is learned using the multiple collected data, with the cycle being repeated as one cycle, and the data used to learn the machine learning model in each of the multiple cycles includes a dataset as first learning data for learning the inference result generation unit of the machine learning model, comprising a multiple input data set collected in the current cycle and a multiple correct answer data set indicating the correct answer to the inference result generated by the inference result generation unit based on each of the multiple input data sets, A model learning method for learning the weakness identification unit of the machine learning model, comprising, as second training data for learning the weakness identification unit of the machine learning model, data input to the machine learning model when identifying weaknesses in multiple past cycles, and data output by the weakness identification unit in multiple past cycles.

[0255] [Item 9] A plurality of in-vehicle devices (2) are installed in each of a plurality of vehicles and are configured to transmit vehicle data which includes at least information about the vehicle on which they are installed, and a center (3) is configured to receive the vehicle data from the plurality of in-vehicle devices, wherein the center comprises a computer, a first input unit (S510) configured to input first input data, an inference result generation unit (S520) configured to perform inference based on the first input data and generate one or more inference results, a first output unit (S530) configured to output one or more of the inference results generated by the inference result generation unit, a second input unit (S510) configured to input a dataset as second input data which includes a plurality of first input data and a plurality of correct answer data which indicate the correct answer to the inference result generated by the inference result generation unit based on each of the plurality of first input data, and a weakness identification unit (S540 to S590) configured to identify weaknesses in the inference made by the inference result generation unit based on the second input data. A data collection system (1) comprising: a collection condition generation unit (S610) configured to generate collection conditions for collecting the first input data corresponding to the aforementioned weaknesses; and a machine learning model (100) configured to function as a second output unit (S620) configured to output the collection condition data indicating the collection conditions as second output data; wherein the center is configured to distribute the collection condition data to a plurality of the in-vehicle devices; the plurality of the in-vehicle devices are configured to collect the vehicle data based on the collection condition data distributed from the center; and the machine learning model uses the plurality of vehicle data transmitted from the in-vehicle devices as the first input data included in the second input data.

[0256] [Item 10] An in-vehicle device (2) mounted on a vehicle, the in-vehicle device comprising: a computer, a first input unit (S10, S120) configured to input first input data; an inference result generation unit (S20, S130) configured to perform inference based on the first input data and generate one or more inference results; a first output unit (S30, S140) configured to output one or more of the inference results generated by the inference result generation unit; a second input unit (S120) configured to input a dataset including a plurality of the first input data and a plurality of correct answer data indicating the correct answer to the inference result generated by the inference result generation unit based on each of the plurality of the first input data as second input data; a weakness identification unit (S150 to S200) configured to identify weaknesses in the inference performed by the inference result generation unit based on the second input data; a collection condition generation unit (S220) configured to generate collection conditions for collecting the first input data corresponding to the weaknesses; and An in-vehicle device comprising a machine learning model (50) for functioning as a second output unit (S230) configured to output collection condition data indicating the collection conditions as second output data, wherein the device is configured to collect vehicle data based on the collection condition data output by the machine learning model, and the machine learning model uses a plurality of the vehicle data collected by the in-vehicle device as the first input data included in the second input data.

Claims

1. The computer comprises: a first input unit (S10, S120, S510) configured to input first input data; an inference result generation unit (S20, S130, S520) configured to perform inference based on the first input data and generate one or more inference results; a first output unit (S30, S140, S530) configured to output one or more of the inference results generated by the inference result generation unit; a second input unit (S120, S510) configured to input a dataset including a plurality of the first input data and a plurality of correct answer data indicating the correct answer to the inference result generated by the inference result generation unit based on each of the plurality of the first input data as second input data; a weakness identification unit (S150 to S200, S540 to S590) configured to identify weaknesses in the inference performed by the inference result generation unit based on the second input data; a collection condition generation unit (S220, S610) configured to generate collection conditions for collecting the first input data corresponding to the weaknesses; and A machine learning model (50, 100) is provided to function as a second output unit (S230, S620) configured to output collection condition data indicating the aforementioned collection conditions as second output data.

2. A machine learning model according to claim 1, wherein the computer is configured to function as a third output unit (S150, S540) that outputs data indicating the basis for inference for each of the one or more inference results generated by the inference result generation unit as third output data, based on the second input data.

3. A machine learning model according to claim 1 or claim 2, wherein the computer is configured to function as a fourth output unit (S160, S550) that further maps the second input data into a feature space composed of a plurality of pre-set features, thereby outputting mapping data as fourth output data that shows the result of mapping the second input data into the feature space.

4. A machine learning model according to claim 1 or claim 2, wherein the computer is configured to function as a fifth output unit (S210, S600) that outputs data indicating the weakness identified by the weakness identification unit as fifth output data.

5. A machine learning model according to claim 1 or claim 2, wherein the inference result generation unit is configured to infer the user's intent based on the first input data, and the machine learning model (50) for causing the computer to function as: a third input unit (S40) configured to input data including a plurality of the first input data, a plurality of the inference results output by the first output unit for each of the plurality of the first input data, and reaction data indicating the user's response for each of the plurality of the first input data as third input data; a correct answer identification unit (S50) configured to identify the correct answer of the inference based on the first input data for each of the plurality of the first input data based on the third input data; and a sixth output unit (S60) configured to output a plurality of correct answer data indicating the correct answer identified for each of the plurality of the first input data as sixth output data.

6. A model learning method in which a model learning device (2) learns a machine learning model (50) comprising: an inference result generation unit (S20) configured to perform inference based on input data and generate one or more inference results; a weakness identification unit (S150 to S200) configured to identify weaknesses in the inference performed by the inference result generation unit; and a collection condition generation unit (S220) configured to generate collection conditions for collecting the input data corresponding to the weaknesses, wherein the model learning device (2) learns the machine learning model, and the input data input to the inference result generation unit is collected as collection data based on the generated collection conditions, and the machine learning model is learned using the multiple collected data, with each of the multiple cycles having a dataset, as first learning data for learning the inference result generation unit of the machine learning model, comprising multiple input data collected in the current cycle and multiple correct answer data indicating the correct answer to the inference result generated by the inference result generation unit based on each of the multiple input data, A model learning method for learning the weakness identification unit and the collection condition generation unit of the machine learning model, comprising, as second learning data, data input to the machine learning model when generating collection conditions in multiple past cycles, and data output by the collection condition generation unit when generating collection conditions in multiple past cycles.

7. A machine learning model (150) for a computer to function as: a first input unit (S10) configured to input first input data; an inference result generation unit (S20) configured to perform inference based on the first input data and generate one or more inference results; a first output unit (S30) configured to output one or more of the inference results; a second input unit (S120) configured to input a dataset as second input data, which includes a plurality of the first input data and a plurality of correct answer data indicating the correct answer to the inference result generated by the inference result generation unit based on each of the plurality of the first input data; a weakness identification unit (S150 to S200) configured to identify weaknesses in the inference performed by the inference result generation unit based on the second input data; and a weakness output unit (S210) configured to output weakness data indicating the weaknesses.

8. A model learning method in which a model learning device (2) learns a machine learning model (150) comprising an inference result generation unit (S20) configured to perform inference based on input data and generate one or more inference results, and a weakness identification unit (S150 to S200) configured to identify weaknesses in the inference performed by the inference result generation unit, wherein the input data input to the inference result generation unit is collected as collected data based on generated collection conditions, and the machine learning model is learned using the multiple collected data, with the cycle being repeated as one cycle, and the data used to learn the machine learning model in each of the multiple cycles includes, as first learning data for learning the inference result generation unit of the machine learning model, a dataset comprising multiple input data collected in the current cycle and multiple correct answer data indicating the correct answer to the inference result generated by the inference result generation unit based on each of the multiple input data, A model learning method for learning the weakness identification unit of the machine learning model, comprising, as second training data for learning the weakness identification unit of the machine learning model, data input to the machine learning model when identifying weaknesses in multiple past cycles, and data output by the weakness identification unit in multiple past cycles.

9. The system comprises a plurality of in-vehicle devices (2) mounted on each of a plurality of vehicles and configured to transmit vehicle data including at least information about the vehicle on which it is mounted, and a center (3) configured to receive the vehicle data from the plurality of in-vehicle devices, wherein the center includes a computer configured to: a first input unit (S510) configured to input first input data; an inference result generation unit (S520) configured to perform inference based on the first input data and generate one or more inference results; a first output unit (S530) configured to output one or more of the inference results generated by the inference result generation unit; a second input unit (S510) configured to input a dataset as second input data, which includes a plurality of first input data and a plurality of correct answer data indicating the correct answer to the inference result generated by the inference result generation unit based on each of the plurality of first input data; and a weakness identification unit (S540 to S590) configured to identify weaknesses in the inference performed by the inference result generation unit based on the second input data. A data collection system (1) comprising: a collection condition generation unit (S610) configured to generate collection conditions for collecting the first input data corresponding to the aforementioned weaknesses; and a machine learning model (100) configured to function as a second output unit (S620) configured to output the collection condition data indicating the collection conditions as second output data; wherein the center is configured to distribute the collection condition data to a plurality of the in-vehicle devices; the plurality of the in-vehicle devices are configured to collect the vehicle data based on the collection condition data distributed from the center; and the machine learning model uses the plurality of vehicle data transmitted from the in-vehicle devices as the first input data included in the second input data.

10. An in-vehicle device (2) mounted on a vehicle, the in-vehicle device comprising: a computer, a first input unit (S10, S120) configured to input first input data; an inference result generation unit (S20, S130) configured to perform inference based on the first input data and generate one or more inference results; a first output unit (S30, S140) configured to output one or more of the inference results generated by the inference result generation unit; a second input unit (S120) configured to input a dataset including a plurality of the first input data and a plurality of correct answer data indicating the correct answer to the inference result generated by the inference result generation unit based on each of the plurality of the first input data as second input data; a weakness identification unit (S150 to S200) configured to identify weaknesses in the inference performed by the inference result generation unit based on the second input data; a collection condition generation unit (S220) configured to generate collection conditions for collecting the first input data corresponding to the weaknesses; and An in-vehicle device comprising a machine learning model (50) for functioning as a second output unit (S230) configured to output collection condition data indicating the collection conditions as second output data, wherein the device is configured to collect vehicle data based on the collection condition data output by the machine learning model, and the machine learning model uses a plurality of the vehicle data collected by the in-vehicle device as the first input data included in the second input data.

Images