Learning device, identification device, learning method, and learning program

Abstract

A missing data determination unit (7) determines whether or not there are sample data of a first modality and sample data of a second modality that are synchronized with each other. A loss calculation unit (identification loss calculation unit (10) for modality 1, identification loss calculation unit (13) for modality 2, and inter-feature-amount distance loss calculation unit (14)) performs different loss calculations depending on whether there are sample data of the first modality and sample data of the second modality that are synchronized with each other, or there is no sample data of the first modality and sample data of the second modality that are synchronized with each other and there is only one of the sample data. A model update unit (15) performs learning using a loss value obtained by the loss calculation by the loss calculation unit.

Description

Learning device, identification device, learning method and learning program

[0001] The present disclosure relates to machine learning using data of one or more modalities.

[0002] In recent years, multimodal learning has become widespread. Multimodal learning is machine learning using multimodal data. Multimodal data is data of two or more types of modalities. When specific unimodal data is input to a machine learning model during operation, multimodal learning can achieve higher accuracy than unimodal learning. Unimodal data is data of one type of modality. Furthermore, unimodal learning is machine learning using unimodal data.

[0003] For example, Patent Document 1 proposes a multimodal learning method that performs learning by reducing the distance between feature amounts of modality data that can be used during learning and operation and feature amounts of modality data that can be used only during learning. Furthermore, when using multimodal data during both learning and operation, techniques robust to modality missing have been proposed. For example, Patent Document 2 discloses a technique that can handle cases where one modality is missing during operation due to sensor failure or the like.

[0004] Japanese Unexamined Patent Application Publication No. 2020-064568, Japanese Unexamined Patent Application Publication No. 2022-066879

[0005] In the technique of Patent Document 1, in the case of classification tasks such as image recognition, learning is performed by bringing feature amounts of data of different modalities assigned the same class label closer together. With the technique of Patent Document 1, when multiple labels are assigned to one piece of data, such as in object detection and semantic occupancy prediction, feature amounts of synchronized multimodal data can be brought closer together, and learning can be performed. However, in practice, the sampling frequency of each sensor often differs. Therefore, data with one missing modality cannot be used for learning according to Patent Document 1.

[0006] The technology described in Patent Document 2 can handle modality loss during operation. However, it does not consider how to handle data where modality is missing during training.

[0007] In light of these considerations, the primary objective of this disclosure is to enable effective learning whether or not there is sample data for two mutually synchronized modalities, or whether or not there is sample data for only one of them.

[0008] The learning device according to this disclosure includes: a sample data determination unit that determines whether there are sample data of a first modality that are synchronized with each other and sample data of a second modality that is a modality different from the first modality; a loss calculation unit that performs different loss calculations depending on whether there are sample data of a first modality that are synchronized with each other and sample data of a second modality, or whether there are no sample data of a first modality that are synchronized with each other and only one of the sample data exists; and a learning unit that performs learning using the loss value obtained by the loss calculation performed by the loss calculation unit.

[0009] This disclosure performs different loss calculations depending on whether there are sample data for two mutually synchronized modalities or only one sample data. Furthermore, this disclosure uses the loss values ​​obtained from the loss calculation for training. Therefore, according to this disclosure, effective training can be performed whether there are sample data for two mutually synchronized modalities or only one sample data.

[0010] A diagram showing an example of the functional configuration of a multimodal learning device according to Embodiment 1. A flowchart showing an example of the operation of a multimodal learning device according to Embodiment 1. A diagram showing an example of the functional configuration of a unimodal data identification device according to Embodiment 1. A flowchart showing an example of the operation of a unimodal data identification device according to Embodiment 1. A diagram showing an example of the functional configuration of a multimodal learning device for semantic occupancy prediction according to Embodiment 2. A diagram showing an example of a method for collecting image data and point cloud data according to Embodiment 2. A diagram explaining the calculation of distance loss between features according to Embodiment 2. A flowchart showing an example of the operation of a multimodal learning device for semantic occupancy prediction according to Embodiment 2. A diagram showing an example of the functional configuration of an image input semantic occupancy prediction device according to Embodiment 2. A flowchart showing an example of the operation of an image input semantic occupancy prediction device according to Embodiment 2. A diagram showing an example of the hardware configuration of a multimodal learning device according to Embodiment 1. A diagram showing an example of the hardware configuration of a unimodal data identification device according to Embodiment 1.

[0011] The embodiments will be described below with reference to the drawings. In the following description of the embodiments and in the drawings, the same reference numerals indicate the same part or a corresponding part.

[0012] Embodiment 1. <Learning> ***Description of Configuration*** Figure 1 shows an example of the functional configuration of the multimodal learning device 1 according to this embodiment. The multimodal learning device 1 is a learning device. The operation procedure of the multimodal learning device 1 corresponds to the learning method. The program that realizes the operation of the multimodal learning device 1 corresponds to the learning program. The multimodal learning device 1 consists of a multimodal data collection unit 2, a data processing unit 3, a model unit 4, and a storage unit 5.

[0013] The multimodal data acquisition unit 2 collects two types of modal data from multimodal data that includes images, point clouds, audio, text, etc. The multimodal data acquisition unit 2 collects image data and point cloud data, for example, using one or more cameras and one or more LiDAR or millimeter-wave radars. LiDAR stands for Light Detection and Ranging. The multimodal data acquisition unit 2 writes the collected data to the data storage unit 16. The data of the two modalities of the data collected by the multimodal data acquisition unit 2 are called Modality 1 and Modality 2. Modality 2 is data of a different modality than Modality 1.

[0014] The data processing unit 3 processes modality 1 and modality 2 collected by the multimodal data acquisition unit 2. The data processing unit 3 consists of an input processing unit 6 and a missing data determination unit 7.

[0015] The input processing unit 6 reads sample data included in the collected data from the data storage unit 16. Then, the input processing unit 6 outputs the read sample data to the missing data determination unit 7. The sample data for modality 1 corresponds to the sample data for the first modality. The sample data for modality 2 corresponds to the sample data for the second modality. Hereafter, the sample data for modality 1 will also be simply referred to as the sample for modality 1. Similarly, the sample data for modality 2 will also be simply referred to as the sample for modality 2.

[0016] The missing data detection unit 7 determines whether one of the modalities in the input sample data is missing. The missing data detection unit 7 then outputs the determination result to the input processing unit 6. The missing data detection unit 7 also determines whether there is a sample of modality 2 that is synchronized with the sample of modality 1. The missing data detection unit 7 corresponds to the sample data detection unit. Furthermore, the processing performed by the missing data detection unit 7 corresponds to the sample data detection processing.

[0017] The input processing unit 6 outputs sample data to the model unit 4 according to the determination result of the missing data determination unit 7. For example, if the missing data determination unit 7 determines that the samples for modality 2 are missing, the input processing unit 6 outputs the samples for modality 1 to the feature extraction unit 8 for modality 1. For example, if the missing data determination unit 7 determines that the sample data for both modalities is not missing, the input processing unit 6 outputs the samples for modality 1 to the feature extraction unit 8 for modality 1 and outputs the samples for modality 2 to the feature extraction unit 11 for modality 2.

[0018] Model Unit 4 updates the parameters of the machine learning model using multimodal data. Model Unit 4 consists of a feature extraction unit 8 for modality 1, an identification unit 9 for modality 1, an identification loss calculation unit 10 for modality 1, a feature extraction unit 11 for modality 2, an identification unit 12 for modality 2, an identification loss calculation unit 13 for modality 2, an inter-feature distance loss calculation unit 14, and a model update unit 15.

[0019] The Modality 1 Feature Extraction Unit 8 acquires a sample of Modality 1 and extracts the feature quantities of the Modality 1 sample as the first feature quantities. The Modality 1 Feature Extraction Unit 8 then outputs the extracted feature quantities of the Modality 1 sample (the first feature quantities) to the Modality 1 Identification Unit 9. The Modality 1 Feature Extraction Unit 8 corresponds to the first feature quantity extraction unit.

[0020] The Modality 1 Identification Unit 9 acquires the feature quantities (first feature quantities) of the Modality 1 sample and performs identification of the Modality 1 sample. The Modality 1 Identification Unit 9 outputs the identification result to the Modality 1 Identification Loss Calculation Unit 10. For example, in the case of object detection, the Modality 1 Identification Unit 9 identifies the class label and bounding box of the object included in the feature quantities of the Modality 1 sample. The Modality 1 Identification Unit 9 corresponds to the first identification unit.

[0021] The Modality 1 identification loss calculation unit 10 calculates the loss value using the identification results for the Modality 1 sample. In other words, the Modality 1 identification loss calculation unit 10 performs identification loss calculation using the Modality 1 sample.

[0022] The Modality 2 Feature Extraction Unit 11 acquires a sample of Modality 2 and extracts the feature quantities of the Modality 2 sample as second feature quantities. The Modality 2 Feature Extraction Unit 11 then outputs the extracted feature quantities of the Modality 2 sample (second feature quantities) to the Modality 2 Identification Unit 12. The Modality 2 Feature Extraction Unit 11 corresponds to the second feature quantity extraction unit.

[0023] The Modality 2 Identification Unit 12 acquires the feature quantities (second feature quantities) of the Modality 2 samples and identifies the Modality 2 samples. The Modality 2 Identification Unit 12 outputs the identification result to the Modality 2 Identification Loss Calculation Unit 13. The Modality 2 Identification Unit 12 corresponds to the second identification unit.

[0024] The Modality 2 identification loss calculation unit 13 calculates the loss value using the identification results for the Modality 2 sample. In other words, the Modality 2 identification loss calculation unit 13 performs identification loss calculation using the Modality 2 sample.

[0025] The loss values ​​calculated by the Modality 1 identification loss calculation unit 10 and the Modality 2 identification loss calculation unit 13 represent the magnitude of the discrepancy between the predicted and actual values. The smaller the loss value, the more accurate the identification result.

[0026] The feature extraction unit 8 for modality 1, the identification unit 9 for modality 1, the feature extraction unit 11 for modality 2, and the identification unit 12 for modality 2 are composed of machine learning models such as Transformer or multilayer perceptron (MLP).

[0027] The feature-inter-feature distance loss calculation unit 14 acquires the feature quantities of the sample from modality 1 and the feature quantities of the sample from modality 2. The feature-inter-feature distance loss calculation unit 14 then calculates a loss value related to the distance between the two feature quantities. The smaller the loss value calculated by the feature-inter-feature distance loss calculation unit 14, the higher the similarity of the sample feature quantities between the two modalities. For example, the feature-inter-feature distance loss calculation unit 14 calculates a value obtained by inverting the sign of the cosine similarity between the two feature quantities.

[0028] The Modality 1 discrimination loss calculation unit 10, the Modality 2 discrimination loss calculation unit 13, and the feature distance loss calculation unit 14 are collectively referred to as the loss calculation unit. As will be described later, the loss calculation unit performs different loss calculations depending on whether there are mutually synchronized samples of Modality 1 and Modality 2, or whether there are no mutually synchronized samples of Modality 1 and Modality 2, and only one sample data exists. When there are mutually synchronized samples of Modality 1 and Modality 2, the Modality 1 discrimination loss calculation unit 10 calculates the discrimination loss of the Modality 1 samples. The Modality 2 discrimination loss calculation unit 13 calculates the discrimination loss of the Modality 2 samples. Furthermore, the feature distance loss calculation unit 14 calculates the loss for the distance between the first feature and the second feature. These loss calculations are collectively referred to as composite loss calculations. On the other hand, if only samples from Modality 1 exist, the Modality 1 identification loss calculation unit 10 performs the identification loss calculation for the Modality 1 samples. Also, if only samples from Modality 2 exist, the Modality 2 identification loss calculation unit 13 performs the identification loss calculation for the Modality 2 samples. Either of these identification loss calculations is called a single-loss calculation.

[0029] The processing performed by the loss calculation unit (identification loss calculation unit 10 for modality 1, identification loss calculation unit 13 for modality 2, and feature distance loss calculation unit 14) corresponds to loss calculation processing.

[0030] The model update unit 15 performs learning using the loss values ​​obtained from the loss calculation performed by the loss calculation unit. In other words, if a composite loss calculation is performed, the model update unit 15 performs learning using the loss values ​​obtained from the composite loss calculation. On the other hand, if a single loss calculation is performed, the model update unit 15 performs learning using the loss values ​​obtained from the single loss calculation. If a composite loss calculation is performed, the model update unit 15 learns the parameters of the modality 1 feature extraction unit 8, the modality 1 identification unit 9, the modality 2 feature extraction unit 11, and the modality 2 identification unit 12 that reduce the sum of the loss values ​​of the composite loss calculation. On the other hand, if a single loss calculation is performed, the model update unit 15 learns the parameters of the modality 1 feature extraction unit 8 or the modality 2 feature extraction unit 11, and the parameters of the modality 1 identification unit 9 or the modality 2 identification unit 12 that reduce the loss value of the single loss calculation. The model update unit 15 learns parameters, for example, using the stochastic gradient descent method. The model update unit 15 writes information about the updated parameters obtained through learning to the model storage unit 17. The model update unit 15 corresponds to the learning unit. Furthermore, the processing performed by the model update unit 15 corresponds to the learning process.

[0031] The memory unit 5 stores various types of information. The memory unit 5 consists of a data storage unit 16 and a model storage unit 17. The data storage unit 16 stores data collected by the multimodal data acquisition unit 2. The model storage unit 17 stores the parameters of the model unit 4.

[0032] The multimodal learning device 1 is, for example, a computer having the hardware configuration shown in Figure 11.

[0033] The multimodal learning device 1 comprises a processor 901, a main memory 902, an auxiliary memory 903, and a communication device 904 as hardware. Functional components, excluding the storage unit 5 in Figure 1, are implemented, for example, by programs. The auxiliary memory 903 stores programs that implement these functions. These programs are loaded from the auxiliary memory 903 to the main memory 902. The processor 901 then executes these programs to perform the operation of the functional components in Figure 1. The storage unit 5 in Figure 1 is implemented, for example, by the auxiliary memory 903.

[0034] ***Explanation of Operation*** Next, the operation of the multimodal learning device 1 will be explained. Figure 2 is a flowchart showing the learning operation by the multimodal learning device 1 according to Embodiment 1.

[0035] In step S101, the multimodal data acquisition unit 2 acquires data from modality 1 and modality 2. In step S101, it may be possible to acquire samples from modality 1 and modality 2 that are synchronized with each other, or it may be possible to acquire samples from only one modality because they are not synchronized with each other. The multimodal data acquisition unit 2 writes the acquired data to the data storage unit 16.

[0036] In step S102, the loop is initiated.

[0037] In step S103, the input processing unit 6 reads sample data from the data storage unit 16.

[0038] In step S104, the missing data determination unit 7 determines whether the sample data read by the input processing unit 6 is missing data for one of the modalities. Based on the determination result output by the missing data determination unit 7, the input processing unit 6 selects a feature extraction unit and outputs the sample data to the selected feature extraction unit. Specifically, if there are synchronized samples for modality 1 and modality 2, the input processing unit 6 outputs the modality 1 sample to the modality 1 feature extraction unit 8. The input processing unit 6 also outputs the modality 2 sample to the modality 2 feature extraction unit 11 (the process proceeds to step S108). On the other hand, if there are no synchronized samples for modality 1 and modality 2, and only one sample data exists, the input processing unit 6 outputs the existing sample data to the corresponding feature extraction unit (the process proceeds to step S105). In other words, if only samples for modality 1 are present, the input processing unit 6 outputs the samples for modality 1 to the feature extraction unit 8 for modality 1. If only samples for modality 2 are present, the input processing unit 6 outputs the samples for modality 2 to the feature extraction unit 11 for modality 2.

[0039] In step S105, one of the feature extraction units extracts the features of the sample data. That is, if only samples of modality 1 exist, the feature extraction unit 8 for modality 1 extracts the features of the samples of modality 1. On the other hand, if only samples of modality 2 exist, the feature extraction unit 11 for modality 2 extracts the features of the samples of modality 2.

[0040] In step S106, any of the discriminating units calculates a discrimination result using the feature quantity calculated in step S105. That is, when the feature quantity of the modality 1 sample is extracted in step S105, the modality 1 discriminating unit 9 calculates a discrimination result for the modality 1 sample using the feature quantity of the modality 1 sample. On the other hand, when the feature quantity of the modality 2 sample is extracted in step S105, the modality 2 discriminating unit 12 calculates a discrimination result for the modality 2 sample using the feature quantity of the modality 2 sample.

[0041] In step S107, any of the discrimination loss calculation units calculates a loss value related to the discrimination result using the discrimination result calculated in step S106. That is, when a discrimination result for the modality 1 sample is calculated in step S106, the modality 1 discrimination loss calculation unit 10 performs discrimination loss calculation on the discrimination result for the modality 1 sample using the discrimination result for the modality 1 sample, and calculates a loss value. On the other hand, when a discrimination result for the modality 2 sample is calculated in step S106, the modality 2 discrimination loss calculation unit 13 performs discrimination loss calculation on the discrimination result for the modality 2 sample using the discrimination result for the modality 2 sample, and calculates a loss value.

[0042] In step S108, the modality 1 feature extraction unit 8 extracts the feature quantity of the modality 1 sample. Also, the modality 2 feature extraction unit 11 extracts the feature quantity of the modality 2 sample.

[0043] In step S109, the inter-feature distance loss calculation unit 14 calculates a loss value related to the distance between the feature quantity of the modality 1 sample and the feature quantity of the modality 2 sample.

[0044] In step S110, the modality 1 discriminating unit 9 calculates a discrimination result for the modality 1 sample using the feature quantity of the modality 1 sample. Also, the modality 2 discriminating unit 12 calculates a discrimination result for the modality 2 sample using the feature quantity of the modality 2 sample.

[0045] In step S111, the identification loss calculation unit for modality 1 10 performs identification loss calculation on the identification result of the modality 1 sample using the identification result for the modality 1 sample, and calculates a loss value. Also, the identification loss calculation unit for modality 2 13 performs identification loss calculation on the identification result of the modality 2 sample using the identification result for the modality 2 sample, and calculates a loss value.

[0046] In step S112, the model update unit 15 learns parameters using the calculated loss and updates the parameters. More specifically, when composite loss calculation has been performed (steps S108 to S111), the model update unit 15 reduces the sum of loss values from the composite loss calculation, and learns the parameters of the feature extraction unit for modality 1 8, the parameters of the identification unit for modality 1 9, the parameters of the feature extraction unit for modality 2 11, and the parameters of the identification unit for modality 2 12. On the other hand, when single loss calculation has been performed (steps S105 to S107), the model update unit 15 reduces the loss value from the single loss calculation, and learns the parameters of the feature extraction unit for modality 1 8 or the parameters of the feature extraction unit for modality 2 11, and the parameters of the identification unit for modality 1 9 or the parameters of the identification unit for modality 2 12.

[0047] The processing from step S102 to step S113 is repeated.

[0048] Finally, in step S114, the model update unit 15 writes information relating to the updated trained parameters into the model storage unit 17.

[0049] <Inference> ***Description of Configuration*** Figure 3 shows an example of a functional configuration of the unimodal data identification device 18 according to the present embodiment. The unimodal data identification device 18 is an identification device. The unimodal data identification device 18 is only capable of identifying data of modality 1. The unimodal data identification device 18 identifies data of modality 1 using a machine learning model trained by the aforementioned multimodal learning device 1.

[0050] The unimodal data identification device 18 consists of a data acquisition unit 19 for modality 1, an input processing unit 20, a model unit 21, an output processing unit 22, and a storage unit 23.

[0051] The Modality 1 data acquisition unit 19 collects data from Modality 1. The Modality 1 data acquisition unit 19 collects, for example, image data from one or more cameras as Modality 1 data. The Modality 1 data acquisition unit 19 writes the collected image data to the data storage unit 24.

[0052] The input processing unit 20 processes the image data collected by the modality 1 data acquisition unit 19. The input processing unit 20 reads sample data included in the collected image data from the data storage unit 24. Then, the input processing unit 20 outputs the read sample data to the model unit 21.

[0053] The model unit 21 calculates the identification result for the sample data of modality 1. The model unit 21 consists of a feature extraction unit 218 for modality 1 and an identification unit 219 for modality 1. The feature extraction unit 218 for modality 1 reads the learned parameters of the feature extraction unit 8 for modality 1 obtained by the multimodal learning device 1 from the model storage unit 17. Then, the feature extraction unit 218 for modality 1 extracts the features of the sample data of modality 1 using the read parameters. The identification unit 219 for modality 1 reads the learned parameters of the identification unit 9 for modality 1 obtained by the multimodal learning device 1 from the model storage unit 17. Then, the identification unit 219 for modality 1 calculates the identification result for the sample data of modality 1 using the read parameters.

[0054] The output processing unit 22 outputs the identification result of the Modality 1 identification unit 219. The output processing unit 22 displays the identification result on a display, for example.

[0055] The memory unit 23 stores various types of information. The memory unit 23 consists of a data storage unit 24 and a model storage unit 17. The data storage unit 24 stores data collected by the modality 1 data acquisition unit 19. The model storage unit 17 stores information regarding the parameters of the model unit 4 learned by the multimodal learning device 1.

[0056] The unimodal data identification device 18 is, for example, a computer having the hardware configuration shown in Figure 12.

[0057] The unimodal data identification device 18 includes, as hardware, a processor 911, a main memory 912, an auxiliary memory 913, and a communication device 914. Functional components, excluding the storage unit 23 in Figure 3, are implemented, for example, by programs. The auxiliary memory 913 stores programs that implement these functions. These programs are loaded from the auxiliary memory 913 to the main memory 912. The processor 911 then executes these programs to perform the operation of the functional components in Figure 3. The storage unit 23 in Figure 3 is implemented, for example, by the auxiliary memory 913.

[0058] ***Explanation of Operation*** Next, we will explain an example of the operation of the unimodal data identification device 18. In other words, we will explain the operation of the machine learning model trained by the multimodal learning device 1 during operation. Figure 4 is a flowchart showing an example of the operation of the unimodal data identification device 18. As mentioned above, the unimodal data identification device 18 is only capable of identifying data of modality 1.

[0059] In step S201, the Modality 1 data acquisition unit 19 collects data for Modality 1. The Modality 1 data acquisition unit 19 writes the collected data to the data storage unit 24.

[0060] In step S202, the model unit 21 reads the learned parameters of the feature extraction unit 8 for modality 1 and the learned parameters of the identification unit 9 for modality 1 from the model storage unit 17.

[0061] In step S203, the input processing unit 20 reads sample data included in the data collected from the data storage unit 24. The input processing unit 20 outputs the read sample data to the feature extraction unit 218 for modality 1.

[0062] In step S204, the feature extraction unit 218 for modality 1 extracts features from the sample data of modality 1 using the parameters learned by the feature extraction unit 8 for modality 1. The feature extraction unit 218 for modality 1 outputs the extracted features to the identification unit 219 for modality 1.

[0063] In step S205, the Modality 1 identification unit 219 calculates an identification result for the Modality 1 sample data based on the feature quantities calculated by the Modality 1 feature extraction unit 218, using the learned parameters of the Modality 1 identification unit 9.

[0064] In step S206, the output processing unit 22 outputs the identification result for the sample data of modality 1.

[0065] ***Explanation of the Effects of the Embodiment*** In this embodiment, different loss calculations are performed depending on whether there are sample data from two mutually synchronized modalities or only one sample data. Then, in this embodiment, learning is performed using the loss value obtained from the loss calculation. Therefore, according to this embodiment, effective learning can be performed whether there are sample data from two mutually synchronized modalities or only one sample data.

[0066] Furthermore, in this embodiment, if one of the two modalities is missing, the sample data of the modality that is not missing is used to train the parameters of the corresponding feature extraction and discrimination units. This increases the amount of data that can be used as training data. As a result, the diversity of the training data increases, and a unimodal machine learning model with high generalization performance can be generated.

[0067] Furthermore, by making the features between modalities similar through the feature distance loss calculation unit, high-quality features that consider both modalities can be generated. Therefore, it is possible to generate models with higher accuracy than unimodal learning.

[0068] Furthermore, consider a scenario where one modality excels in the identification task, but only data from the other modality is available during operation. The multimodal learning device according to this embodiment can bring the feature vectors of the modality used during operation closer to the high-quality feature vectors of the modality that excels in the identification task. As a result, it becomes possible to generate high-quality feature vectors during operation, thereby improving the accuracy of the identification model during operation.

[0069] Embodiment 2. Embodiment 1 described a multimodal learning method and an operation method in a unimodal environment. This embodiment describes a multimodal learning method and an operation method in a unimodal environment for semantic occupancy prediction. This embodiment mainly describes the differences from Embodiment 1. Matters not described below are the same as in Embodiment 1.

[0070] <Learning> ***Explanation of Configuration*** Figure 5 shows an example of the functional configuration of the multimodal learning device 25 for semantic occupancy prediction. The multimodal learning device 25 for semantic occupancy prediction is also a learning device. The operation procedure of the multimodal learning device 25 for semantic occupancy prediction corresponds to the learning method. The program that realizes the operation of the multimodal learning device 25 for semantic occupancy prediction corresponds to the learning program. The multimodal learning device 25 for semantic occupancy prediction consists of a multimodal data collection unit 26, a data processing unit 27, a model unit 28, and a storage unit 29.

[0071] The multimodal data acquisition unit 26 collects image data and point cloud data. For example, as shown in Figure 6, suppose that four RGB cameras 83, 84, 85, and 86 are installed on the vehicle 81. The multimodal data acquisition unit 26 collects image data from four viewpoints of the four RGB cameras 83, 84, 85, and 86. Also, suppose that one LiDAR 82 is installed on the vehicle 81. The multimodal data acquisition unit 26 collects point cloud data from the LiDAR 82. Note that the multimodal data acquisition unit 26 may collect image data and point cloud data from a robot, drone, etc., instead of the vehicle 81 in Figure 6. Note that the number and position of cameras for collecting image data are not specified. Similarly, the number and position of LiDARs for collecting point cloud data are not specified. Also, millimeter-wave radar may be used instead of LiDAR. Note that the sample data of the collected image data and the sample data of the point cloud data are not necessarily synchronized. The multimodal data acquisition unit 26 writes the acquired data to the data storage unit 40.

[0072] The data processing unit 27 processes the data collected by the multimodal data acquisition unit 26. The data processing unit 27 consists of an input processing unit 30 and a missing data determination unit 31. The input processing unit 30 reads sample data included in the collected data from the data storage unit 40. The input processing unit 30 then outputs the read sample data to the missing data determination unit 31.

[0073] The missing data determination unit 31 determines whether one modality of the input sample is missing. The missing data determination unit 31 then outputs the determination result to the input processing unit 30. The missing data determination unit 31 also determines whether there is a point cloud sample data (also called a point cloud sample) that is synchronized with the image sample data (also called an image sample).

[0074] The input processing unit 30 outputs sample data to the model unit 28 according to the determination result of the missing data determination unit 31. For example, if the missing data determination unit 31 determines that a point cloud sample is missing, the input processing unit 30 outputs the image sample to the image feature extraction unit 32. For example, if the missing data determination unit 31 determines that sample data for both modalities is not missing, the input processing unit 30 outputs the image sample to the image feature extraction unit 32 and outputs the point cloud sample to the point cloud feature extraction unit 35.

[0075] The model unit 28 updates the parameters of the machine learning model using multimodal data. The model unit 28 consists of an image feature extraction unit 32, an image identification unit 33, an image identification loss calculation unit 34, a point cloud feature extraction unit 35, a point cloud identification unit 36, a point cloud identification loss calculation unit 37, a three-dimensional spatial feature representation distance loss calculation unit 38, and a model update unit 39.

[0076] The image feature extraction unit 32 acquires an image sample and calculates a three-dimensional spatial feature representation of the image sample. The three-dimensional spatial feature representation holds information about the three-dimensional space to be reconstructed by semantic occupancy prediction. The three-dimensional spatial feature representation can be a voxel representation, a TPV (Tri-Perspective View) representation, a BEV (Bird's-Eye View) representation, etc. The image feature extraction unit 32 is not limited to these and can calculate feature quantities for any position in the three-dimensional space to be reconstructed as a three-dimensional spatial feature representation. Feature representations are included in the three-dimensional spatial feature representation. The image feature extraction unit 32 then extracts feature quantities of the image sample from the calculated three-dimensional spatial feature representation. The image feature extraction unit 32 corresponds to the first feature quantity extraction unit.

[0077] The image identification unit 33 acquires feature quantities obtained from the three-dimensional spatial feature representation of the image sample and identifies the image sample. The image identification unit 33 outputs the identification result to the image identification loss calculation unit 34. The image identification unit 33 corresponds to the first identification unit.

[0078] The image recognition loss calculation unit 34 calculates a loss value using the recognition result for the image sample.

[0079] The point cloud feature extraction unit 35 acquires point cloud samples and calculates a three-dimensional spatial feature representation of the point cloud samples. The point cloud feature extraction unit 35 then extracts feature quantities from the calculated three-dimensional spatial feature representation of the point cloud samples. The point cloud feature extraction unit 35 corresponds to a second feature quantity extraction unit.

[0080] The point cloud identification unit 36 ​​acquires feature quantities obtained from the three-dimensional spatial feature representation of the point cloud sample and identifies the point cloud sample. The point cloud identification unit 36 ​​outputs the identification result to the point cloud identification loss calculation unit 37. The point cloud identification unit 36 ​​corresponds to the second identification unit.

[0081] The point cloud identification loss calculation unit 37 calculates the loss value using the identification result for the point cloud sample.

[0082] The image recognition loss calculation unit 34 and the point cloud recognition loss calculation unit 37 calculate loss values ​​such that the smaller the value, the more accurate the recognition result.

[0083] The image feature extraction unit 32, the image identification unit 33, the point cloud feature extraction unit 35, and the point cloud identification unit 36 ​​are composed of machine learning models such as Transformer or multilayer perceptron (MLP).

[0084] As shown in Figure 7, the inter-three-dimensional spatial feature representation distance loss calculation unit 38 calculates a loss value for the distance between two feature quantities for the same three-dimensional position feature quantities between the two modalities, based on the inter-three-dimensional spatial feature representations of both modalities. Figure 7 illustrates the inter-feature distance loss calculation. In Figure 7, the inter-three-dimensional spatial feature representations of both modalities are voxel representations, but the same applies when other three-dimensional spatial feature representations are used. Different three-dimensional spatial feature representations may be used between the modalities. The smaller the loss value calculated by the inter-three-dimensional spatial feature representation distance loss calculation unit 38, the higher the similarity of the sample feature quantities between the two modalities.

[0085] The image recognition loss calculation unit 34, the point cloud recognition loss calculation unit 37, and the three-dimensional spatial feature representation distance loss calculation unit 38 are collectively called the loss calculation unit. As will be described later, the loss calculation unit performs different loss calculations depending on whether there are mutually synchronized image samples and point cloud samples, or whether there are no mutually synchronized image samples and point cloud samples and only one sample data exists. When mutually synchronized image samples and point cloud samples exist, the image recognition loss calculation unit 34 performs the recognition loss calculation for the image samples. The point cloud recognition loss calculation unit 37 also performs the recognition loss calculation for the point cloud samples. Furthermore, the three-dimensional spatial feature representation distance loss calculation unit 38 performs the loss calculation for the distance between the feature quantities of the image samples and the feature quantities of the point cloud samples. These loss calculations are collectively called the composite loss calculation. On the other hand, when only image samples exist, the image recognition loss calculation unit 34 performs the recognition loss calculation for the image samples. Furthermore, if only point cloud samples exist, the point cloud identification loss calculation unit 37 performs the identification loss calculation for the point cloud samples. Either of these identification loss calculations is called a single-loss calculation.

[0086] The processing performed by the loss calculation unit (image identification loss calculation unit 34, point cloud identification loss calculation unit 37, and three-dimensional spatial feature representation distance loss calculation unit 38) corresponds to loss calculation processing.

[0087] The model update unit 39 performs learning using the loss values ​​obtained by the loss calculation unit. If a composite loss calculation is performed, the model update unit 39 performs learning using the loss values ​​obtained by the composite loss calculation. On the other hand, if a single loss calculation is performed, the model update unit 39 performs learning using the loss values ​​obtained by the single loss calculation. If a composite loss calculation is performed, the model update unit 39 learns the parameters of the image feature extraction unit 32, the image identification unit 33, the point cloud feature extraction unit 35, and the point cloud identification unit 36 ​​that reduce the sum of the loss values ​​of the composite loss calculation. On the other hand, if a single loss calculation is performed, the model update unit 39 learns the parameters of the image feature extraction unit 32 or the point cloud feature extraction unit 35, and the parameters of the image identification unit 33 or the point cloud identification unit 36 ​​that reduce the loss value of the single loss calculation. The model update unit 39 writes information about the updated parameters obtained by learning to the model storage unit 41. The model update unit 39 corresponds to the learning unit. Furthermore, the processing performed by the model update unit 39 corresponds to the learning process.

[0088] The memory unit 29 stores various types of information. The memory unit 29 consists of a data storage unit 40 and a model storage unit 41. The data storage unit 40 stores data collected by the multimodal data acquisition unit 26. The model storage unit 41 stores the parameters of the model unit 28.

[0089] The multimodal learning device 25 for semantic occupancy prediction is, for example, a computer having the hardware configuration shown in Figure 11.

[0090] The semantic occupancy prediction multimodal learning device 25 comprises, as hardware, a processor 901, a main memory 902, an auxiliary memory 903, and a communication device 904. Functional components, excluding the storage unit 29 in Figure 5, are implemented, for example, by programs. The auxiliary memory 903 stores programs that implement these functions. These programs are loaded from the auxiliary memory 903 to the main memory 902. The processor 901 then executes these programs to perform the operation of the functional components in Figure 5. The storage unit 29 in Figure 5 is implemented, for example, by the auxiliary memory 903.

[0091] ***Explanation of Operation*** Next, the operation of the semantic occupancy prediction multimodal learning device 25 will be explained. Figure 8 is a flowchart showing the learning operation by the semantic occupancy prediction multimodal learning device 25 according to Embodiment 2.

[0092] In step S301, the multimodal data acquisition unit 262 acquires image data and point cloud data. In step S301, it may be possible to acquire mutually synchronized image samples and point cloud samples, or it may be possible to acquire only samples from one modality because they are not mutually synchronized. The multimodal data acquisition unit 26 writes the acquired data to the data storage unit 40.

[0093] In step S302, the loop is initiated.

[0094] In step S303, the input processing unit 30 reads sample data from the data storage unit 40.

[0095] In step S304, the missing data determination unit 31 determines whether the sample read by the input processing unit 30 is a sample with missing data in one of the modalities. Based on the determination result output by the missing data determination unit 31, the input processing unit 30 selects a feature extraction unit and outputs the sample data to the selected feature extraction unit. Specifically, if there are mutually synchronized image samples and point cloud samples, the input processing unit 30 outputs the image samples to the image feature extraction unit 32. The input processing unit 30 also outputs the point cloud samples to the point cloud feature extraction unit 35 (processing proceeds to step S308). On the other hand, if there are no mutually synchronized image samples and point cloud samples, and only one of the sample data exists, the input processing unit 30 outputs the existing sample data to the corresponding feature extraction unit (processing proceeds to step S305). In other words, if only image samples exist, the input processing unit 30 outputs the image samples to the image feature extraction unit 32. Furthermore, if only point cloud samples exist, the input processing unit 30 outputs the point cloud samples to the point cloud feature extraction unit 35.

[0096] In step S305, one of the feature extraction units calculates the three-dimensional spatial feature representation of the sample data. Specifically, if only image samples exist, the image feature extraction unit 32 calculates the three-dimensional spatial feature representation of the image samples. On the other hand, if only point cloud samples exist, the point cloud feature extraction unit 35 calculates the three-dimensional spatial feature representation of the point cloud samples.

[0097] In step S306, one of the identification units calculates an identification result using the features obtained from the three-dimensional spatial feature representation calculated in step S305. That is, if the feature quantities of the three-dimensional spatial feature representation of the image sample are extracted in step S305, the image identification unit 33 calculates an identification result for the image sample using the feature quantities of the image sample. On the other hand, if the feature quantities of the three-dimensional spatial feature representation of the point cloud sample are extracted in step S305, the point cloud identification unit 36 ​​calculates an identification result for the point cloud sample using the feature quantities of the point cloud sample.

[0098] In step S307, one of the identification loss calculation units calculates a loss value related to the identification result using the identification result calculated in step S306. That is, if an identification result for an image sample is calculated in step S306, the image identification loss calculation unit 34 calculates a loss value related to the identification result for the image sample. On the other hand, if an identification result for a point cloud sample is calculated in step S306, the point cloud identification loss calculation unit 37 calculates a loss value related to the identification result for the point cloud sample.

[0099] In step S308, the image feature extraction unit 32 calculates the three-dimensional spatial feature representation of the image sample. The point cloud feature extraction unit 35 also calculates the three-dimensional spatial feature representation of the point cloud sample.

[0100] In step S309, the three-dimensional spatial feature representation distance loss calculation unit 38 calculates a loss value related to the distance between the three-dimensional spatial feature representation of the image sample and the three-dimensional spatial feature representation of the point cloud sample for the same three-dimensional position feature quantity.

[0101] In step S310, the image identification unit 33 calculates an identification result for the image sample using the feature quantities of the image sample. The point cloud identification unit 36 ​​also calculates an identification result for the point cloud sample using the feature quantities of the point cloud sample.

[0102] In step S311, the image recognition loss calculation unit 34 calculates a loss value related to the recognition result for the image sample using the recognition result for the image sample. The image recognition loss calculation unit 34 also calculates a loss value related to the recognition result for the point cloud sample using the recognition result for the point cloud sample.

[0103] In step S312, the model update unit 39 learns and updates the parameters using the calculated loss. More specifically, if a composite loss calculation is performed (steps S308 to S311), the model update unit 39 learns the parameters of the image feature extraction unit 328, the image identification unit 33, the point cloud feature extraction unit 35, and the point cloud identification unit 36 ​​that reduce the sum of the loss values ​​in the composite loss calculation. On the other hand, if a single loss calculation is performed (steps S305 to S307), the model update unit 15 learns the parameters of the image feature extraction unit 32 or the point cloud feature extraction unit 35, and the parameters of the image identification unit 33 or the point cloud identification unit 36 ​​that reduce the loss value in the single loss calculation.

[0104] The process from step S302 to step S113 is repeated.

[0105] Finally, in step S314, the model update unit 39 writes information about the updated learned parameters to the model storage unit 41.

[0106] <Inference> ***Description of Configuration*** Figure 9 shows an example of the functional configuration of the image input semantic occupancy prediction device 42 according to this embodiment. The image input semantic occupancy prediction device 42 is an identification device. The image input semantic occupancy prediction device 42 is only capable of identifying image data. The image input semantic occupancy prediction device 42 identifies image data using a machine learning model learned by the multimodal learning device 25 for semantic occupancy prediction described above.

[0107] The image input semantic occupancy prediction device 42 consists of an image data acquisition unit 43, an input processing unit 44, a model unit 45, an output processing unit 46, and a storage unit 47.

[0108] The image data collection unit 43 collects image data. The image data collection unit 43 collects image data, for example, from one or more cameras. The image data collection unit 43 writes the collected image data to the data storage unit 24.

[0109] The input processing unit 44 processes the image data collected by the image data acquisition unit 43. The input processing unit 44 reads sample data included in the collected image data from the data storage unit 24. Then, the input processing unit 44 outputs the read sample data to the model unit 45.

[0110] The model unit 45 calculates the identification result for the sample image data. The model unit 45 consists of an image feature extraction unit 452 and an image identification unit 453. The image feature extraction unit 452 reads the trained parameters of the image feature extraction unit 32 obtained by the semantic occupancy prediction multimodal learning device 25 from the model storage unit 41. Then, the image feature extraction unit 452 extracts the features of the sample image data using the read parameters. The image identification unit 453 reads the trained parameters of the image identification unit 33 obtained by the semantic occupancy prediction multimodal learning device 25 from the model storage unit 41. Then, the image identification unit 453 calculates the identification result for the sample image data using the read parameters.

[0111] The output processing unit 46 outputs the identification result of the image identification unit 453. The output processing unit 46 displays the identification result on a display, for example.

[0112] The memory unit 47 stores various types of information. The memory unit 47 consists of a data storage unit 48 and a model storage unit 41. The data storage unit 48 stores data collected by the image data collection unit 43. The model storage unit 41 stores information about the parameters of the model unit 4 learned by the semantic occupancy prediction multimodal learning device 25.

[0113] The image input semantic occupancy prediction device 42 is, for example, a computer having the hardware configuration shown in Figure 12.

[0114] The image input semantic occupancy prediction device 42 includes, as hardware, a processor 911, a main memory 912, an auxiliary memory 913, and a communication device 914. Functional components, excluding the storage unit 47 in Figure 9, are implemented, for example, by programs. The auxiliary memory 913 stores programs that implement these functions. These programs are loaded from the auxiliary memory 913 to the main memory 912. The processor 911 then executes these programs to perform the operation of the functional components in Figure 9. The storage unit 47 in Figure 9 is implemented, for example, by the auxiliary memory 913.

[0115] ***Explanation of Operation*** Next, an example of the operation of the image input semantic occupancy prediction device 42 will be explained. In other words, the operation of the machine learning model trained by the semantic occupancy prediction multimodal learning device 25 will be explained. Figure 10 is a flowchart of an example of the operation of the image input semantic occupancy prediction device 42. As mentioned above, the image input semantic occupancy prediction device 42 is only capable of identifying image data.

[0116] In step S401, the image data acquisition unit 43 acquires image data. The image data acquisition unit 43 writes the acquired data to the data storage unit 48.

[0117] In step S402, the model unit 45 reads the learned parameters of the image feature extraction unit 32 and the learned parameters of the image recognition unit 33 from the model storage unit 41.

[0118] In step S403, the input processing unit 44 reads sample data included in the data collected from the data storage unit 48. The input processing unit 44 outputs the read sample data to the image feature extraction unit 32.

[0119] In step S404, the image feature extraction unit 452 calculates a three-dimensional spatial feature representation of the image sample using the parameters learned by the image feature extraction unit 32. The image feature extraction unit 32 then extracts feature quantities from the three-dimensional spatial feature representation. The image feature extraction unit 32 outputs the extracted feature quantities to the image identification unit 453.

[0120] In step S405, the image identification unit 453 calculates an identification result for an image sample based on the feature quantities calculated by the image feature extraction unit 452, using the learned parameters of the image identification unit 33.

[0121] In step S406, the output processing unit 46 outputs the identification result for the image sample.

[0122] ***Effects of the Embodiment*** In Embodiment 2, in addition to the effects of Embodiment 1, it is possible to sample a certain number of three-dimensional positions used to calculate the distance between three-dimensional spatial feature representations. This makes it possible to realize multimodal learning with improved computational efficiency.

[0123] Although Embodiments 1 and 2 have been described above, these two embodiments may be implemented in combination. Alternatively, one of these two embodiments may be implemented in part. Alternatively, these two embodiments may be implemented in part in combination. Furthermore, the configurations and procedures described in these two embodiments may be modified as necessary.

[0124] ***Supplementary Explanation of Hardware Configuration*** Here, we will provide supplementary explanations of the hardware configurations shown in Figures 11 and 12. Below, we will provide a representative explanation of the hardware configuration in Figure 11, but the following explanation also applies to the hardware configuration in Figure 12.

[0125] The processor 901 is a CPU, DSP, etc. CPU stands for Central Processing Unit. DSP stands for Digital Signal Processor. The main memory 902 is RAM. RAM stands for Random Access Memory. The auxiliary memory 903 is a ROM, flash memory, HDD, etc. ROM stands for Read Only Memory. HDD stands for Hard Disk Drive. The communication device 904 is an electronic circuit that performs data communication processing. The communication device 904 is, for example, a communication chip or NIC. NIC stands for Network Interface Card.

[0126] Furthermore, the auxiliary storage device 903 also stores the OS. OS stands for Operating System. At least a portion of the OS is executed by the processor 901. While executing at least a portion of the OS, the processor 901 executes a program that realizes the functions of the functional components shown in Figure 1 or Figure 5. By executing the OS, the processor 901 performs task management, memory management, file management, communication control, etc. Also, at least one of the information, data, signal values, and variable values ​​indicating the processing results of the functional components shown in Figure 1 or Figure 5 is stored in at least one of the main memory 902, auxiliary storage device 903, registers in the processor 901, and cache memory. Furthermore, the program that realizes the functions of the functional components shown in Figure 1 or Figure 5 may be stored on a portable recording medium such as a magnetic disk, flexible disk, optical disk, compact disk, Blu-ray® disk, or DVD. A portable recording medium containing the program that realizes the functions of the functional components shown in Figure 1 or Figure 5 may be distributed.

[0127] Furthermore, at least one of the "parts" of the functional components shown in Figure 1 or Figure 5 may be read as "circuit," "process," "procedure," "process," or "circuitry." Also, the multimodal learning device 1 and the semantic occupancy prediction multimodal learning device 25 may each be implemented by a processing circuit. The processing circuit is, for example, a logic IC, GA, ASIC, or FPGA. IC stands for Integrated Circuit. GA stands for Gate Array. ASIC stands for Application Specific Integrated Circuit. FPGA stands for Field-Programmable Gate Array. In this case, the functional components shown in Figure 1 or Figure 5 are each implemented as part of a processing circuit. In this specification, the higher-level concept encompassing both the processor and the processing circuit is referred to as "processing circuitry." In other words, the processor and the processing circuit are specific examples of "processing circuitry."

[0128] Finally, the various aspects of this disclosure are summarized below as an appendix. (Appendix 1) A learning device comprising: a sample data determination unit that determines whether or not there are sample data of a first modality that are synchronized with each other and sample data of a second modality that is a modality different from the first modality; a loss calculation unit that performs different loss calculations depending on whether there are sample data of a first modality that are synchronized with each other and sample data of a second modality, or whether there are no sample data of a first modality that are synchronized with each other and only one of the sample data exists; and a learning unit that performs learning using the loss value obtained by the loss calculation performed by the loss calculation unit. (Note 2) The learning device according to Note 1, wherein the loss calculation unit performs a composite loss calculation, which is a loss calculation using the sample data of the first modality and the sample data of the second modality that are synchronized with each other, when there is a sample data of the first modality and a sample data of the second modality that are synchronized with each other, and when there is no sample data of the first modality and a sample data of the second modality that are synchronized with each other, and when there is a sample data of only one of them, it performs a single loss calculation, which is a loss calculation using the sample data that is available, and the learning unit performs learning using the loss value obtained by the composite loss calculation when the loss calculation unit has performed the composite loss calculation, and performs learning using the loss value obtained by the single loss calculation when the loss calculation unit has performed the single loss calculation. (Note 3) The learning device according to Note 2, wherein the loss calculation unit performs, as the composite loss calculation, the identification loss calculation of the sample data of the first modality, the identification loss calculation of the sample data of the second modality, and the loss calculation of the distance between the feature quantities of the sample data of the first modality and the feature quantities of the sample data of the second modality, and as the single loss calculation, the identification loss calculation of the sample data of the first modality or the identification loss calculation of the sample data of the second modality.(Note 4) The learning device further comprises: a first feature extraction unit that extracts the feature quantities of the sample data of the first modality as first feature quantities; a second feature extraction unit that extracts the feature quantities of the sample data of the second modality as second feature quantities; a first identification unit that uses the first feature quantities to identify the sample data of the first modality; and a second identification unit that uses the second feature quantities to identify the sample data of the second modality, wherein the loss calculation unit performs, as the composite loss calculation, an identification loss calculation for the identification result by the first identification unit, an identification loss calculation for the identification result by the second identification unit, and a loss calculation for the distance between the first feature quantity and the second feature quantity, and as the single loss calculation, an identification loss calculation for the identification result by the first identification unit or an identification loss calculation for the identification result by the second identification unit, as described in Note 3. (Note 5) The learning device according to Note 4, wherein the learning unit learns the parameters of the first feature extraction unit, the parameters of the second feature extraction unit, the parameters of the first identification unit, and the parameters of the second identification unit to reduce the loss value obtained by the composite loss calculation when the composite loss calculation is performed, and learns the parameters of the first feature extraction unit or the parameters of the second feature extraction unit, and the parameters of the first identification unit or the parameters of the second identification unit to reduce the loss value obtained by the single loss calculation when the single loss calculation is performed. (Note 6) The learning device according to any one of Notes 1 to 5, wherein the sample data determination unit determines whether there are mutually synchronized image sample data of the first modality and point cloud sample data of the second modality, and the loss calculation unit performs different loss calculations depending on whether there are mutually synchronized image sample data and point cloud sample data or whether there are mutually synchronized image sample data and point cloud sample data and only one of the sample data exists.(Note 7) An identification device that identifies data of the first modality or data of the second modality using the learning results of the learning unit described in Note 1. (Note 8) A learning method in which a computer determines whether there are mutually synchronized sample data of the first modality and sample data of the second modality which is a modality different from the first modality; the computer performs different loss calculations depending on whether there are mutually synchronized sample data of the first modality and sample data of the second modality, or whether there are mutually synchronized sample data of the first modality and sample data of the second modality and only one of the sample data exists; and the computer performs learning using the loss value obtained by the loss calculation. (Note 9) A learning program comprising: a sample data determination process that determines whether there are sample data of a first modality that are synchronized with each other and sample data of a second modality that is a modality different from the first modality; a loss calculation process that performs different loss calculations depending on whether there are sample data of a first modality that are synchronized with each other and sample data of a second modality, or whether there are no sample data of a first modality that are synchronized with each other and only one of the sample data exists; and a learning process that performs learning using the loss value obtained by the loss calculation performed by the loss calculation process.

[0129] 1 Multimodal learning device, 2 Multimodal data acquisition unit, 3 Data processing unit, 4 Model unit, 5 Storage unit, 6 Input processing unit, 7 Missing data detection unit, 8 Feature extraction unit for Modality 1, 9 Identification unit for Modality 1, 10 Identification loss calculation unit for Modality 1, 11 Feature extraction unit for Modality 2, 12 Identification unit for Modality 2, 13 Identification loss calculation unit for Modality 2, 14 Inter-feature distance loss calculation unit, 15 Model update unit, 16 Data storage unit, 17 Model storage unit, 18 Unimodal data identification device, 19 Data acquisition unit for Modality 1, 20 Input processing unit, 21 Model unit, 218 Feature extraction unit for Modality 1, 219 Identification unit for Modality 1, 22 Output processing unit, 23 Storage unit, 24 Data storage unit, 25 Multimodal learning device for semantic occupancy prediction, 26 Multimodal data acquisition unit, 27 1. Data processing unit, 28. Model unit, 29. Storage unit, 30. Input processing unit, 31. Missing data determination unit, 32. Image feature extraction unit, 33. Image identification unit, 34. Image identification loss calculation unit, 35. Point cloud feature extraction unit, 36. Point cloud identification unit, 37. Point cloud identification loss calculation unit, 38. Three-dimensional spatial feature representation distance loss calculation unit, 39. Model update unit, 40. Data storage unit, 41. Model storage unit, 42. Image input semantic occupancy prediction device, 43. Image data collection unit, 44. Input processing unit, 45. Model unit, 45. Image feature extraction unit, 453. Image identification unit, 46. Output processing unit, 47. Storage unit, 48. Data storage unit, 901. Processor, 902. Main memory, 903. Auxiliary storage device, 904. Communication device, 911. Processor, 912. Main memory, 913. Auxiliary storage device, 914. Communication device.

Claims

1. A learning device comprising: a sample data determination unit that determines whether there are mutually synchronized sample data for a first modality and sample data for a second modality that is a modality different from the first modality; a loss calculation unit that performs different loss calculations depending on whether mutually synchronized sample data for the first modality and sample data for the second modality exist, or whether mutually synchronized sample data for the first modality and sample data for the second modality do not exist and only one of the sample data exists; and a learning unit that performs learning using the loss value obtained by the loss calculation performed by the loss calculation unit.

2. The learning device according to claim 1, wherein the loss calculation unit performs a composite loss calculation, which is a loss calculation using the sample data of the first modality and the sample data of the second modality that are synchronized with each other, when there is a sample data of the first modality and a sample data of the second modality that are synchronized with each other, and when there is no sample data of the first modality and a sample data of the second modality that are synchronized with each other, and when there is a sample data of only one of them, it performs a single loss calculation, which is a loss calculation using the sample data that is available, and the learning unit performs learning using the loss value obtained by the composite loss calculation when the loss calculation unit has performed the composite loss calculation, and performs learning using the loss value obtained by the single loss calculation when the loss calculation unit has performed the single loss calculation.

3. The learning device according to claim 2, wherein the loss calculation unit performs, as the composite loss calculation, the identification loss calculation of the sample data of the first modality, the identification loss calculation of the sample data of the second modality, and the loss calculation of the distance between the feature quantities of the sample data of the first modality and the feature quantities of the sample data of the second modality, and as the single loss calculation, the identification loss calculation of the sample data of the first modality or the identification loss calculation of the sample data of the second modality.

4. The learning device further comprises: a first feature extraction unit that extracts feature quantities of sample data of the first modality as first feature quantities; a second feature extraction unit that extracts feature quantities of sample data of the second modality as second feature quantities; a first identification unit that identifies the sample data of the first modality using the first feature quantities; and a second identification unit that identifies the sample data of the second modality using the second feature quantities, wherein the loss calculation unit performs, as the composite loss calculation, an identification loss calculation for the identification result by the first identification unit, an identification loss calculation for the identification result by the second identification unit, and a loss calculation for the distance between the first feature quantity and the second feature quantity; and as the single loss calculation, an identification loss calculation for the identification result by the first identification unit or an identification loss calculation for the identification result by the second identification unit.

5. The learning device according to claim 4, wherein the learning unit learns, when the composite loss calculation is performed, the parameters of the first feature extraction unit, the parameters of the second feature extraction unit, the parameters of the first identification unit, and the parameters of the second identification unit to reduce the loss value obtained by the composite loss calculation, and when the single loss calculation is performed, the learning unit learns the parameters of the first feature extraction unit or the parameters of the second feature extraction unit, the parameters of the first identification unit or the parameters of the second identification unit to reduce the loss value obtained by the single loss calculation.

6. The learning device according to any one of claims 1 to 5, wherein the sample data determination unit determines whether there are mutually synchronized image sample data of the first modality and point cloud sample data of the second modality, and the loss calculation unit performs different loss calculations depending on whether there are mutually synchronized image sample data and point cloud sample data or whether there are mutually synchronized image sample data and point cloud sample data and only one of the sample data exists.

7. An identification device that identifies data of the first modality or data of the second modality using the learning results from the learning unit described in any one of claims 1 to 6.

8. A learning method comprising: a computer determining whether there are mutually synchronized sample data for a first modality and sample data for a second modality that is a different modality from the first modality; the computer performing different loss calculations depending on whether there are mutually synchronized sample data for the first modality and sample data for the second modality, or whether there are no mutually synchronized sample data for the first modality and sample data for the second modality and only one of them exists; and the computer performing learning using the loss values ​​obtained from the loss calculations.

9. A learning program comprising: a sample data determination process that determines whether there are sample data of a first modality that are synchronized with each other and sample data of a second modality that is a modality different from the first modality; a loss calculation process that performs different loss calculations depending on whether there are sample data of a first modality that are synchronized with each other and sample data of a second modality, or whether there are no sample data of a first modality that are synchronized with each other and only one of them is present; and a learning process that performs learning using the loss value obtained by the loss calculation performed by the loss calculation process.

Images