Information processing device, information processing method, and program

Abstract

Provided is an information processing device comprising: an annotation data acquisition unit that acquires annotation data including a weak word identified by the difference between text outputted for an input video by a learning model that outputs, on the basis of a video captured by a camera mounted on the vehicle, text representing the situation of a vehicle in the video, and correct answer data of the text for the input video; a parameter acquisition unit that acquires a video parameter including the situation of the vehicle and the situation in the surroundings of the vehicle in the input video; a road identification unit that, on the basis of the video parameter, identifies, from three-dimensional map data for simulation including a plurality of types of roads, a similar road similar to the road on which the vehicle is traveling in the input video; and a video generation unit that generates a video for causing the learning model to learn the weak word on the basis of the annotation data by using a portion of the similar road identified by the road identification unit in the three-dimensional map data.

Description

Information processing device, information processing method, and program

[0001] This invention relates to an information processing device, an information processing method, and a program.

[0002] Patent Document 1 describes a system comprising control means, the control means having a function to acquire target location information and target attribute information from a holding means that holds in advance the location and attributes of a predetermined object in association with each other; a function to acquire an image taken on a vehicle; and a function to associate the acquired image with the target attribute information based on the position of the image and the target location information. Patent Document 2 describes a driving support device comprising: surrounding environment imaging means for acquiring an image of the surrounding environment of a moving object; risk factor determination means for inputting the image into a learning model generated by machine learning using a multilayer neural network to recognize the surrounding conditions occurring in the surrounding environment of the moving object and to determine risk factors in the surrounding environment of the moving object based on the recognized surrounding conditions; determination result output means for outputting the determination result of the risk factor determination means; and determination criterion changing means for changing the determination criteria for risk factors by updating the part of the learning model that determines risk factors in the surrounding environment of the moving object based on the surrounding conditions. Patent Document 3 describes an explanatory text generation device that includes an image explanatory text generation unit that generates an explanatory text for an image based on the features of the image; an operating behavior explanatory text generation unit that generates an explanatory text for operating behavior based on the features of the image and the explanatory text generated by the image explanatory text generation unit; and an explanatory text generation unit that generates an explanatory text for an image or an explanatory text for operating behavior that occurs after the generation of the explanatory text for the image and the explanatory text for operating behavior based on the explanatory text generated by the image explanatory text generation unit and the explanatory text generated by the operating behavior explanatory text generation unit. Patent Document 4 describes an explanatory text generation device that includes an acquisition unit that acquires material features representing each of the materials used in a work and video features extracted from each of the videos for each process in which the work is filmed, an update unit that identifies the actions on the materials included in the video for each process based on the video features of each of the videos for each process and updates the material features of the identified materials according to the identified actions, and a generation unit that generates text explaining the procedure of the work for each process based on the updated material features, the identified actions, and the video features.[Prior Art Documents] [Patent Documents] [Patent Document 1] Japanese Patent Application Laid-Open No. 2023-117185 [Patent Document 2] Japanese Patent Application Laid-Open No. 2018-173860 [Patent Document 3] Japanese Patent Application Laid-Open No. 2021-174172 [Patent Document 4] Japanese Patent Application Laid-Open No. 2023-053742.

[0003] Conventionally, there has been known a technique of analyzing a video captured by a camera mounted on a vehicle using an AI model to generate text representing the situation of the vehicle. In this technique, a cycle of executing inference of a video, evaluating and analyzing the inference result, generating a new video based on the evaluation / analysis result, adding the generated video to a dataset, and repeating the cycle is used to generate a dataset. Then, the AI model is updated by repeating the process of updating the AI model using the generated dataset. However, even when such an automatic cycle is repeatedly executed, the inference evaluation value may not improve in some videos. Therefore, there is a need to construct a new method for generating a dataset that does not depend on the automatic cycle.

[0004] According to an embodiment of the present invention, an information processing device is provided. The information processing device may include an annotation data acquisition unit that acquires annotation data including weak words identified by the difference between the text output for an input video by a learning model that outputs text representing the situation of the vehicle in the video based on a video captured by a camera mounted on the vehicle, and the correct answer data of the text for the input video. The information processing device may include a parameter acquisition unit that acquires a plurality of parameters including the situation of the vehicle and the surrounding situation of the vehicle in the input video. The information processing device may include a road identification unit that identifies a similar road similar to the road on which the vehicle travels in the input video from three-dimensional map data for simulation including a plurality of types of roads based on the plurality of parameters. The information processing device may include a video generation unit that generates a video for causing the learning model to learn the weak words based on the annotation data using the portion of the road identified by the road identification unit in the three-dimensional map data.

[0005] In the information processing device, the road identification unit may identify roads from the three-dimensional map data whose similarity to the roads in the input video is higher than a predetermined threshold, based on the plurality of parameters.

[0006] In any of the above-mentioned information processing devices, the video generation unit may generate video that reproduces the status of the vehicle included in the plurality of parameters on the road in the three-dimensional map data identified by the road identification unit.

[0007] In any of the above-mentioned information processing devices, the plurality of parameters may include the number of lanes on the road on which the vehicle is traveling in the input video, and the road identification unit may identify roads from the three-dimensional map data that have the same number of lanes as the number of lanes included in the plurality of parameters as similar roads.

[0008] In any of the above-mentioned information processing devices, the plurality of parameters may include the size of the intersection when the vehicle in the input video is driving through an intersection, and the road identification unit may identify intersections from the three-dimensional map data whose size is the same as the size of the intersection included in the plurality of parameters as similar roads.

[0009] In any of the above-mentioned information processing devices, the plurality of parameters may include the type of curve when the vehicle in the input video is traveling on a curve, and the road identification unit may identify curves from the three-dimensional map data that are the same type of curve as the type of curve included in the plurality of parameters as similar roads.

[0010] In any of the above-mentioned information processing devices, the plurality of parameters may include the type of area in which the road on which the vehicle is traveling in the input video is located, and the road identification unit may identify roads in the same area as the type of area included in the plurality of parameters from the three-dimensional map data as similar roads.

[0011] Any of the above-mentioned information processing devices may include a model update unit that updates the learning model using a plurality of learning data sets, which include the video generated by the video generation unit and the annotation data.

[0012] Any of the above-mentioned information processing devices may include a RAN control unit for controlling the RAN and an AI processing unit for performing AI (Artificial Intelligence) processing, and the AI ​​processing unit may include the annotation data acquisition unit, the parameter acquisition unit, and the video generation unit. Examples of AI processing include AI processing related to RAN control (sometimes referred to as RAN control AI processing) and AI processing not related to RAN control (sometimes referred to as non-RAN control AI processing). An example of RAN control AI processing is RIC (RAN Intelligent Controller). RIC is a technology that uses AI to optimize the wireless resources of the RAN and automate RAN operation. RICs include Non-RT RICs (Non-Real Time RICs) and Near-RT RICs (Near-Real Time RICs). Non-RT RICs are sometimes called Centralized RICs. Non-RT RICs are located within SMOs (Service Management and Orchestrations) that manage and orchestrate RANs. Non-RT RICs generate and notify policies related to RAN control and send information to Near-RT RICs. For example, a Non-RT RIC generates a learning model for RAN control by performing machine learning using data collected from the RAN and sends it to a Near-RT RIC. Near-RT RICs are sometimes called Distributed RICs. Compared to Non-RT RICs, Near-RT RICs are located closer to RAN nodes (RUs (Radio Units), DUs (Distributed Units), CUs (Central Units)) and perform tasks such as controlling RAN nodes and resources. Near-RT RICs perform processing that is more real-time than Non-RT RICs. For example, Near-RT RICs perform inference processing related to RAN control using trained models acquired from Non-RT RICs. RAN control AI processing is not limited to RICs. Non-RAN control AI processing can be so-called MEC AI applications.Non-RAN controlled AI processing includes learning and inference processing of arbitrary AIs that are not related to RAN control.

[0013] According to one embodiment of the present invention, an information processing method performed by a computer is provided. The information processing method may include an annotation data acquisition step in which a learning model that outputs text representing the situation of the vehicle in a video based on a video captured by a camera mounted on the vehicle acquires annotation data that includes weak words identified by the difference between the text output for the input video and the correct text data for the input video. The information processing method may include a parameter acquisition step in which a plurality of parameters, including the situation of the vehicle and the surrounding situation of the vehicle in the input video, are acquired. The information processing method may include a road identification step in which, based on the plurality of parameters, similar roads similar to the roads on which the vehicle travels in the input video are identified from three-dimensional map data for simulation, which includes a plurality of types of roads. The information processing method may include a video generation step in which a video is generated using the portion of the road identified by the road identification unit from the three-dimensional map data to allow the learning model to learn the weak words based on the annotation data.

[0014] According to one embodiment of the present invention, a program is provided to cause a computer to execute the following steps: an annotation data acquisition step of acquiring annotation data that includes weak words identified by the difference between the text output for an input video and the correct text data for the input video; a parameter acquisition step of acquiring a plurality of parameters including the vehicle's condition and the surrounding conditions of the vehicle in the input video; a road identification step of identifying similar roads similar to the roads the vehicle travels on in the input video from a 3D map data for simulation including a plurality of types of roads based on the plurality of parameters; and a video generation step of generating a video for the learning model to learn the weak words based on the annotation data using the portion of the road identified by the road identification unit from the 3D map data.

[0015] It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. Furthermore, subcombinations of these features may also constitute an invention.

[0016] This diagram schematically shows an example of the environment of the information processing device 100. This is an explanatory diagram for explaining the processing performed by the information processing device 100. This is an explanatory diagram for explaining the processing performed by the information processing device 100. This is an explanatory diagram for explaining the processing performed by the information processing device 100. This diagram schematically shows an example of the functional configuration of the information processing device 100. This diagram schematically shows an example of the environment of the information processing device 100. This diagram schematically shows an example of the hardware configuration of the computer 1200 that functions as the information processing device 100.

[0017] The present invention will be described below through embodiments, but these embodiments are not intended to limit the scope of the claims. Furthermore, not all combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0018] Figure 1 schematically shows an example of the environment of the information processing device 100. In the example shown in Figure 1, the information processing device 100 is connected to a network 50. The network 50 may include a cloud network. The network 50 may include the internet. The network 50 may include a mobile communication network. The mobile communication network may conform to any of the following mobile communication systems: LTE (Long Term Evolution) communication system, 5G (5th Generation) communication system, 3G (3rd Generation) communication system, and 6G (6th Generation) communication system or later.

[0019] For example, the information processing device 100 may be located in a cloud network. The information processing device 100 may also be located in a mobile communication network. For example, the information processing device 100 may be located in a MEC (Multi-access Edge Computing) system. The information processing device 100 may also be located in a core network.

[0020] The information processing device 100 receives video 210 captured by a camera 202 mounted on the vehicle 200 from the vehicle 200. The information processing device 100 may receive the video 210 via a mobile communication network. The video 210 may be transmitted by the camera 202 or by a communication device mounted on the vehicle 200.

[0021] Vehicle 200 may be a so-called autonomous vehicle. Camera 202 may be a drive recorder. Camera 202 may be a camera that captures images for use in controlling autonomous driving. Camera 202 may be any other type of camera. Vehicle 200 does not have to be an autonomous vehicle.

[0022] The information processing device 100 may generate training data for updating a learning model that takes a video 210 as input and outputs text describing the status of the vehicle 200 in the video 210. The learning model can be anything as long as it can output text describing the status of the vehicle 200 in the video 210 in response to the input video 210. For example, the learning model may be generated by machine learning using multiple training data sets that include a video 210 in which the status of the vehicle 200 is known and text describing the status of the known vehicle. For example, the learning model may be a so-called video generation AI. If the learning model is a so-called video generation AI, the learning model may be initially tuned to output text describing the status of the vehicle 200 in response to the video 210. Usually, initial tuning alone is not sufficient, and appropriate updates to the learning model are necessary to improve accuracy.

[0023] Figure 2 is an explanatory diagram illustrating the processing performed by the information processing device 100. The information processing device 100 stores a learning model 101 that takes a video 210 as input and outputs text describing the status of the vehicle 200 in the video 210.

[0024] The information processing device 100 uses a plurality of video parameters 109, including the status of the vehicle 200 and the surrounding conditions of the vehicle 200, to simulate a video 210 captured by the camera 202, and performs learning using the plurality of video parameters 109. The information processing device 100 may generate a video 210 that simulates a video 210 captured by the camera 202 using three-dimensional map data for simulation that includes multiple types of roads. The three-dimensional map data may be a vast, virtually generated three-dimensional map that includes roads, structures such as buildings, natural objects such as trees, etc. The three-dimensional map data may be a virtual reproduction of an existing space, or it may be a virtual generation of a non-existent space.

[0025] The information processing device 100 may perform learning using a video 210 captured by the camera 202 to which multiple video parameters 109, including the status of the vehicle 200 and the surrounding environment of the vehicle 200 in the video 210, are associated. In this case, for example, a person views the video 210 captured by the camera 202, generates multiple video parameters 109 representing the status of the vehicle 200 and the surrounding environment of the vehicle 200, and associates them with the video 210.

[0026] The video parameter 109 may include "Common," "Own Vehicle," and "Other Vehicles" as major categories. The major categories may also include other categories.

[0027] The "Common" category may include "Roads," "Weather," and "Time" as subcategories. The "Common" category may further include at least one of the following as subcategories: "Intersection Size," "Traffic Signals," "Curve Type," and "Area Type." The "Common" category may also include other subcategories. The "Road" subcategory may include options such as 1 lane, 2 lanes, 3 lanes, 4 lanes, and 5 lanes. The "Road" subcategory may also include other options. The "Weather" subcategory may include options such as sunny, cloudy, and rainy. The "Weather" subcategory may also include other options. The "Time" subcategory may include options such as 6:00, 12:00, and 18:00. The "Time" subcategory may also include other options. The "Intersection Size" subcategory may include large, medium, and small. The "Intersection Size" subcategory may also include other options. The "Traffic Signals" subcategory may include red, blue, and yellow. The subcategory "Type of Curve" may include gentle, sharp, and hairpin curves. The subcategory "Type of Curve" may also include options other than these. The subcategory "Type of Area" may refer to the type of area where the road the vehicle travels is located, and may include suburbs, urban areas, and commercial districts. The subcategory "Type of Area" may also include options other than these.

[0028] The "My Vehicle" category may include "Speed" and "Lane" as subcategories. The "My Vehicle" category may also include other subcategories. The "Speed" subcategory may include 30 km / h, 60 km / h, and 100 km / h as options. The "Speed" subcategory may also include other options. The "Lane" subcategory may include right, middle, and left as options. The "Lane" subcategory may also include other options.

[0029] The "Other Vehicles" category may include subcategories such as "Speed," "Lane," and "Direction." The "Other Vehicles" category may also include subcategories other than those listed above. The "Speed" subcategory may include options such as 0 km / h, 30 km / h, and 60 km / h. The "Speed" subcategory may also include options other than those listed above. The "Lane" subcategory may include options such as right, middle, and left. The "Lane" subcategory may also include options other than those listed above.

[0030] The information processing device 100 performs inference 102 on the video 210 using the learning model 101. The information processing device 100 inputs the video 210 to the learning model 101 and generates text that describes the situation of the vehicle 200 in the video 210. The generated text may be referred to as the generated statement 104.

[0031] The information processing device 100 performs an evaluation / analysis 106 on the generated sentence 104. The information processing device 100 stores in advance the correct data 108 and video parameters 109 for the video 210. The information processing device 100 may perform the evaluation / analysis 106 by comparing the generated sentence 104 with the correct data 108.

[0032] The correct answer data 108 is pre-registered for the video 210. The correct answer data 108 may be generated and registered by a person who views the video 210. The correct answer data 108 may also be registered by a person who views the video 210, for example, by inputting the video 210 into a learning model 101 and modifying the generated sentence 104 output from the learning model. The correct answer data 108 may be registered for the video 210 by any other method, as long as it accurately represents the state of the vehicle 200 in the video 210.

[0033] The information processing device 100 identifies weak words 110 by evaluation / analysis 106 based on the difference between the generated sentence 104 output by the learning model 101 for the video 210 and the correct answer data 108 for the video 210. For example, the information processing device 100 identifies weak words 110 as words that are included in the correct answer data 108 but not in the generated sentence 104.

[0034] The information processing device 100 performs annotation data generation 112 for the weak word 110. The information processing device 100 stores annotation categories 114 in advance and generates multiple annotation data 116 using the weak word 110 and the annotation categories 114.

[0035] Annotation category 114 includes multiple categories. For each of the multiple categories, annotation category 114 includes multiple options.

[0036] Multiple categories in annotation category 114 may include "action." "Action" represents the actions of the vehicle. Examples of "action" options include stopping, going straight (forward), reversing, turning right, turning left, accelerating, decelerating, changing lanes to the right, changing lanes to the left, etc. The "action" options may include only some of these, or may include all of them.

[0037] Multiple categories in annotation category 114 may include "causative objects." A "causative object" represents the object that caused the reason for the action. Examples of "causative object" options include pedestrians, riders (motorcycles), passenger cars, trucks, buses, trains, motorcycles, bicycles, signs, red lights, yellow lights, green lights, balls, multiple pedestrians, fire trucks, ambulances, police cars, arrow signals, obstacles, etc. The "causative object" options may include only some of these, or may include all of them.

[0038] Multiple categories in annotation category 114 may include "object motion." "Object motion" represents the action of the causative object. Examples of "object motion" options include walking, standing still, falling, running, making contact, giving a stop signal, in front of a store, running out, etc. The "object motion" options may include only some of these, or may include all of them.

[0039] Multiple categories in annotation category 114 may include "absolute object location." "Absolute object location" represents the absolute location of an object. Examples of options for "absolute object location" include roadways, shoulders (roadsides), crosswalks, sidewalks, bicycle paths, fire stations, hospitals, police stations, etc. Options for "absolute object location" may include only some of these, or may include all of them.

[0040] Multiple categories in annotation category 114 may include "Object Position (Relative)". "Object Position (Relative)" represents the relative position of an object as seen from the vehicle. Examples of "Object Position (Relative)" options include forward (in the path of travel), right front, left front, etc. The "Object Position (Relative)" options may include only some of these, or may include all of them.

[0041] Multiple categories in annotation category 114 may include "driving intention." "Driving intention" represents the planned route of the vehicle. Examples of options for "driving intention" include going straight (forward), reversing, turning right, turning left, etc. The options for "driving intention" may include only some of these, or may include all of them.

[0042] A plurality of categories of the annotation category 114 may include "own vehicle position". "Own vehicle position" represents the traveling position of the own vehicle (including near facilities). Examples of options for "own vehicle position" include intersections, straight roads, curves, etc. The options for "own vehicle position" may include only some of these, or may include others.

[0043] The information processing device 100 identifies a category among the plurality of categories of the annotation category 114 in which one of the options matches the weakness word 110, fixes the option of the category to the weakness word 110, and changes the options of the other categories, thereby generating a plurality of variations of combinations of the plurality of options of the plurality of categories, and for each of the plurality of variations, generates annotation data 116 using the options of the plurality of categories.

[0044] As a specific example, as illustrated in FIG. 3, when the weakness word 110 is a straight road, the information processing device 100 fixes the option of the "own vehicle position" category to the straight road, and changes the options of the categories of "action", "causal object", "object action", "object position (absolute)", "object position (relative)", and "travel intention", thereby generating a plurality of variations of combinations of the plurality of options of the plurality of categories. In the example shown in FIG. 3, the information processing device 100 generates one variation where "action" is straight ahead, "causal object" is an obstacle, "object action" is none, "object position (absolute)" is the lane, "object position (relative)" is ahead, "travel intention" is straight ahead, and "own vehicle position" is a straight road. The information processing device 100 generates annotation data 116 of "The vehicle is on a straight road and is trying to go straight. Since there is no obstacle ahead, the vehicle is going straight" using the plurality of options of the variation. The generation of the annotation data 116 may be executed by a rule-based method using the annotation category 114. The generation of the annotation data 116 may also be executed using AI. For example, the generation of the annotation data 116 may be executed using text generation AI. The generation of the annotation data 116 may also be performed manually.

[0045] The information processing device 100 may perform video design 118 for each of the multiple annotation data 116. The annotation data 116 is data that represents the status of the vehicle 200, and is not data for generating a video. Therefore, the annotation data 116 alone does not have enough parameters to generate a video. Thus, conventionally, people have supplemented the parameters necessary for video generation based on their experience. However, this method places a heavy burden on people and is time-consuming, which is one of the reasons why the overall processing efficiency has decreased.

[0046] In response, the information processing device 100 according to this embodiment performs video design 118 using video parameters 109. Based on the video parameters 109, the information processing device 100 identifies similar roads from three-dimensional map data 140 for simulation, which include multiple types of roads, that are similar to the roads on which the vehicle 200 travels in the video 210. As described above, the three-dimensional map data 140 may be a vast, virtually generated three-dimensional map, and may include roads, structures such as buildings, natural objects such as trees, etc. The three-dimensional map data 140 may be a virtual reproduction of an existing space, or it may be a virtual generation of a non-existent space.

[0047] The information processing device 100 may identify similar roads from the 3D map data 140 based on the surrounding conditions of the vehicle 200 included in the video parameters 109. As a specific example, as illustrated in Figure 4, the information processing device 100 uses the size of the road and intersection among the video parameters 109 to identify a medium-sized intersection on a two-lane road from the 3D map data 140. The information processing device 100 identifies at least one part of the 3D map data 140 that matches the conditions. If there are multiple parts of the 3D map data 140 that match the conditions, the information processing device 100 may identify the part with a higher degree of matching the conditions. If there are multiple parts that match the conditions, the information processing device 100 may identify multiple parts in order of decreasing degree of matching the conditions.

[0048] The information processing device 100 executes video generation 124 for generating a video 126 for training the learning model 101 with weak points words based on the annotation data 116 by using a specified part of the 3D map data 140. The information processing device 100 may generate the video 126 by using the annotation category 114, the annotation data 116, the video parameters 109, and a specified part of the 3D map data 140.

[0049] The information processing device 100 may generate the video 126 by a rule-based method. For example, the information processing device 100 generates a video 126 that reflects all elements included in the annotation category 114, the annotation data 116, and the video parameters 109 for a specified part of the 3D map data 140. The information processing device 100 may also generate a video 126 that reflects a part of the elements included in the annotation category 114, the annotation data 116, and the video parameters 109 for a specified part of the 3D map data 140.

[0050] The information processing device 100 may generate the video 126 by using AI. For example, the information processing device 100 generates a prompt for instructing to generate the video 126 by using a specified part of the 3D map data 140 and all elements included in the annotation category 114, the annotation data 116, and the video parameters 109, and inputs it to the video generation AI to generate the video 126. The information processing device 100 may also generate a prompt for instructing to generate the video 126 by using a specified part of the 3D map data 140 and a part of all elements included in the annotation category 114, the annotation data 116, and the video parameters 109, and inputs it to the video generation AI to generate the video 126.

[0051] For example, in the example shown in Figure 4, the information processing device 100 generates a video 126 for a specified portion of the 3D map data 140, with the weather set to cloudy, the time to 12 o'clock, the vehicle's speed to 30 km / h, the vehicle's lane to the left, the traffic light in the vehicle's lane to green, no obstacles in front of the vehicle and proceeding straight, the other vehicle's speed to 30 km / h, the other vehicle's lane to the right, and the other vehicle's direction to the right.

[0052] The information processing device 100 may generate a video 126 for each of the multiple annotation data 116, and for each of the multiple annotation data 116, it may generate training data 128 that includes the generated video 126 and the annotation data 116.

[0053] The information processing device 100 may perform fine-tuning 130 of the learning model 101 using the generated learning data 128. The information processing device 100 may update the learning model 101 for each of the learning data 128 so that when a video 126 is input, a generated sentence 104 based on the annotation data 116 is generated.

[0054] As described above, annotation data 116 is data that represents the situation of the vehicle 200, and not data for generating a video. Therefore, annotation data 116 alone is insufficient for generating a video. Conventionally, people have supplemented the parameters necessary for video generation based on their experience. However, this method places a heavy burden on people and is time-consuming, contributing to a decrease in overall processing efficiency. In contrast, the information processing device 100 according to this embodiment uses video parameters 109 to identify similar roads that are similar to the surrounding situation of the vehicle 200 in the video 210 input from the 3D map data 140. By using the identified similar roads, it is possible to eliminate the need for parameters to generate road images. Furthermore, the information contained in video parameters 109 can be used to supplement the parameters necessary for video generation. This significantly reduces the burden on people and allows for the generation of video parameters at a faster speed than when supplemented by people, thereby increasing the update speed of the learning model 101. By increasing the update speed of the learning model 101, the accuracy of generating a generated statement 104 that accurately describes the situation of the vehicle 200 from video 210 taken by a camera 202 mounted on the vehicle 200 can be improved more quickly. This can improve the accuracy of monitoring the vehicle 200 using the generated statement 104, reduce the monitoring load on the vehicle 200, and potentially make a significant contribution to the development of the automotive industry.

[0055] In this example, the explanation uses the case where all processing is performed by the information processing device 100, but this is not the only case. The information processing device 100 may perform only video design 118 and video generation 124. The information processing device 100 may perform only video design 118, video generation 124, and fine tuning 130.

[0056] Figure 5 schematically shows an example of the functional configuration of the information processing device 100. The information processing device 100 includes a storage unit 150, a registration unit 152, a video acquisition unit 154, an inference unit 156, an evaluation and analysis unit 158, an annotation data generation unit 160, an annotation data acquisition unit 162, a parameter acquisition unit 164, a road identification unit 166, a video generation unit 168, and a model update unit 170. It is not necessarily required that the information processing device 100 include all of these.

[0057] The registration unit 152 registers various data. The registration unit 152 stores the registered data in the storage unit 150. The registration unit 152 may register the learning model 101. The registration unit 152 may register the video 210, the video parameters 109, and the correct answer data 108. The registration unit 152 may register the annotation category 114.

[0058] The video acquisition unit 154 acquires the video 210. During learning execution, the video acquisition unit 154 may acquire the video 210 generated by the simulation. During learning execution, the video acquisition unit 154 may acquire the video 210 captured by the camera 202 of the vehicle 200, for which the correct answer data 108 and video parameters 109 have been generated. After learning is complete, when generating a generated statement 104 that actually represents the situation of the vehicle 200, the video acquisition unit 154 may receive the video 210 from the vehicle 200 via a mobile communication network. The video acquisition unit 154 may receive multiple videos 210 from a server that collects videos 210 from multiple vehicles 200.

[0059] The inference unit 156 performs inference 102 on the video 210 acquired by the video acquisition unit 154. The inference unit 156 generates a generated sentence 104 by inputting the video 210 into the learning model 101.

[0060] The evaluation and analysis unit 158 ​​performs an evaluation / analysis 106 on the generated sentence 104 generated by the inference unit 156. The evaluation and analysis unit 158 ​​compares the generated sentence 104 generated from the video 210 with the correct answer data 108 corresponding to the video 210. The evaluation and analysis unit 158 ​​identifies weak words 110 based on the difference between the generated sentence 104 generated from the video 210 and the correct answer data 108 corresponding to the video 210. For example, the evaluation and analysis unit 158 ​​identifies words that are included in the correct answer data 108 but not in the generated sentence 104 as weak words 110.

[0061] The annotation data generation unit 160 performs annotation data generation 112 for the weak word 110 identified by the evaluation analysis unit 158. The annotation data generation unit 160 generates multiple annotation data 116 using the weak word 110 and the annotation category 114. The annotation data generation unit 160 may identify a category among the multiple categories of the annotation category 114 in which one of the options matches the weak word 110, fix the option in that category to the weak word 110, and change the options in the other categories to generate multiple variations of combinations of multiple options in multiple categories, and generate annotation data 116 for each of the multiple variations using the options in multiple categories.

[0062] The annotation data acquisition unit 162 acquires a plurality of annotation data 116 generated by the annotation data generation unit 160.

[0063] The parameter acquisition unit 164 acquires video parameters 109 corresponding to the input video 210. The parameter acquisition unit 164 may acquire the video parameters 109 from the storage unit 150.

[0064] The road identification unit 166 identifies similar roads from the 3D map data 140 that are similar to the roads on which the vehicle 200 is traveling in the input video 210, based on the video parameters 109 acquired by the parameter acquisition unit 164.

[0065] The road identification unit 166 may identify roads as similar roads from the 3D map data 140 if the number of lanes is the same as the number of lanes indicated by "Road" in the video parameters 109. The road identification unit 166 may identify intersections as similar roads from the 3D map data 140 if the vehicle 200 in the input video 210 is traveling through an intersection if the vehicle 200 in the input video 210 is traveling through a curve if the road identification unit 166 identifies curves as similar roads from the 3D map data 140 if the type is the same as the type indicated by "Type of curve" in the video parameters 109. The road identification unit 166 may identify roads in areas as similar roads from the 3D map data 140 if the type is the same as the type indicated by "Type of area" in the video parameters 109. The road identification unit 166 may identify portions of the 3D map data 140 that match multiple of the video parameters 109, including "road," "intersection size," "curve type," and "area type."

[0066] The road identification unit 166 may identify roads in the 3D map data 140 whose similarity to the roads in the input video 210 is higher than a predetermined threshold, based on the video parameters 109. The road identification unit 166 may determine the similarity between the roads in the input video 210 and the roads in the 3D map data 140 based on, for example, the number of matches with the video parameters 109, namely "road," "intersection size," "curve type," and "area type." For example, the road identification unit 166 determines that the greater the number of matches, the higher the similarity.

[0067] The road identification unit 166 may determine the similarity of each of the video parameters 109, namely "road," "intersection size," "curve type," and "area type," and then determine the similarity between the roads in the input video 210 and the roads in the 3D map data 140 based on these four similarities. For example, the road identification unit 166 may determine that the closer the number of lanes, the higher the similarity for "road." For example, the road identification unit 166 may determine that the closer the size of the "intersection," the higher the similarity for "intersection size." For example, the road identification unit 166 may determine that the more similar the types of "curves," the higher the similarity for "area type." For example, the road identification unit 166 may determine that the more similar the types, the higher the similarity for "area type." The road identification unit 166 may determine the similarity between the roads in the input video 210 and the roads in the 3D map data 140 by adding, multiplying, weighted addition, or weighted multiplication of the four similarities.

[0068] The video generation unit 168 uses the portions of similar roads identified by the road identification unit 166 from the 3D map data 140 to generate a video 126 based on the annotation data acquisition unit 162, which is used to train the learning model 101 on weak words 110.

[0069] The video generation unit 168 may generate a video 126 based on the portion of the 3D map data 140 that is similar to the road identified by the road identification unit 166 and the video parameters 109.

[0070] For example, the video generation unit 168 generates a video 126 that reproduces the surrounding conditions of a vehicle 200 on a road similar to the 3D map data 140 identified by the road identification unit 166. For example, the video generation unit 168 generates a video 126 that reproduces at least one of the weather, time of day, and traffic signals on a road similar to the 3D map data 140 identified by the road identification unit 166.

[0071] For example, the video generation unit 168 generates a video 126 that reproduces the situation of a vehicle 200 included in the video parameters 109 on a road similar to the 3D map data 140 identified by the road identification unit 166. For example, the video generation unit 168 generates a video 126 that reproduces at least one of the vehicle's speed and lane position included in the video parameters 109 on a road similar to the 3D map data 140 identified by the road identification unit 166. For example, the video generation unit 168 generates a video 126 that reproduces at least one of the other vehicle's speed, lane position, and direction included in the video parameters 109 on a road similar to the 3D map data 140 identified by the road identification unit 166.

[0072] The video generation unit 168 may generate a video 126 based on the portion of the three-dimensional map data 140 that is similar to the road identified by the road identification unit 166, the video parameters 109, and the annotation data 116. The video generation unit 168 may further generate a video 126 that reproduces at least one of the annotation data 116, namely "action," "causative object," "object action," "object position (absolute)," "object position (relative)," "driving intention," and "vehicle position."

[0073] The video generation unit 168 may generate the video 126 using a rule-based method, as described above. The video generation unit 168 may also generate the video 126 using AI, as described above.

[0074] The video generation unit 168 may generate a video 126 generated from the annotation data 116, and training data that includes the annotation data 116.

[0075] The model update unit 170 updates the learning model 101 using multiple training data sets generated by the video generation unit 168. The model update unit 170 may update the learning model 101 for each of the multiple training data sets 128 so that when a video 126 is input, a generated sentence 104 based on the annotation data 116 is generated.

[0076] As described above, the information processing device 100 does not have to include all of the following: storage unit 150, registration unit 152, video acquisition unit 154, inference unit 156, evaluation and analysis unit 158, annotation data generation unit 160, annotation data acquisition unit 162, parameter acquisition unit 164, road identification unit 166, evaluation and analysis unit 158, and model update unit 170. For example, the information processing device 100 does not have to include the inference unit 156, evaluation and analysis unit 158, and annotation data generation unit 160. In this case, the inference unit 156, evaluation and analysis unit 158, and annotation data generation unit 160 are provided by other devices. For example, the information processing device 100 does not have to include the inference unit 156 and evaluation and analysis unit 158. In this case, the inference unit 156 and evaluation and analysis unit 158 ​​are provided by other devices. For example, the information processing device 100 does not have to include the inference unit 156. In this case, the inference unit 156 is provided by other devices. For example, the information processing device 100 does not need to include an evaluation and analysis unit 158. In this case, the evaluation and analysis unit 158 ​​is provided by another device. For example, the information processing device 100 does not need to include an annotation data generation unit 160. In this case, the annotation data generation unit 160 is provided by another device.

[0077] Figure 6 schematically shows an example of the environment of the information processing device 100. The information processing device 100 may be deployed in a mobile communication system consisting of a management infrastructure 300, a plurality of distributed infrastructures 400, and a plurality of wireless base stations 500. In this mobile communication system, the management infrastructure 300 and the plurality of distributed infrastructures 400 cooperate to control the RAN 510 and perform AI processing. Mobile communication services are provided to the vehicle 200 by the RAN 510. The RAN 510 may be a virtualized vRAN (Virtual RAN), and the mobile communication system may perform control of the vRAN. The RAN 510 may also be a physical RAN, and the mobile communication system may perform control of the physical RAN.

[0078] The distributed infrastructure 400 may be data centers located in various locations. The distributed infrastructure 400 may be composed of multiple devices. The information processing device 100 may be located on the distributed infrastructure 400. The information processing device 100 may function as a BBU (BaseBand Unit), and the wireless base station 500 may function as an RRU (Remote Radio Unit). The information processing device 100 may implement a CU. The distributed infrastructure 400 may implement a DU. The information processing device 100 may implement a UPF (User Plane Function).

[0079] The management infrastructure 300 may be a data center that manages multiple distributed infrastructures 400. The management infrastructure 300 may be composed of multiple devices. The management infrastructure 300 may be implemented on a virtualization infrastructure consisting of multiple devices. The management infrastructure 300 may be implemented by a single device. In other words, the management infrastructure 300 may be a management device.

[0080] The management infrastructure 300 may be called the Core Brain, and the distributed infrastructure 400 may be called the Regional Brain. Note that Figure 7 illustrates a case where a single-layer distributed infrastructure 400 is located below the management infrastructure 300, but this is not the only case. The distributed infrastructure 400 may have multiple layers. For example, if two layers of distributed infrastructure 400 are located below the management infrastructure 300, the management infrastructure 300 may be called the Core Brain, the lower-layer distributed infrastructure 400 may be called the Regional Brain, and the further lower-layer distributed infrastructure 400 may be called the Sub-Regional Brain.

[0081] The information processing device 100, located on the distributed infrastructure 400, may have one or more CPUs (Central Processing Units) and one or more GPUs (Graphics Processing Units). The information processing device 100 may have multiple superchips, each connected by an interconnect. This interconnect may be memory-consistent and capable of achieving high bandwidth and low latency. Thus, the information processing device 100 may have both CPU resources and GPU resources as computing resources.

[0082] The information processing device 100 may include a RAN control unit 180 for controlling the RAN 510 and an AI processing unit 190 for performing AI processing. In this case, the AI ​​processing unit 190 may include an annotation data acquisition unit 162, a parameter acquisition unit 164, a road identification unit 166, and a video generation unit 168. The AI ​​processing unit 190 may further include a model update unit 170. The AI ​​processing unit 190 may further include a registration unit 152, a video acquisition unit 154, an inference unit 156, an evaluation and analysis unit 158, and an annotation data generation unit 160.

[0083] Figure 7 schematically shows an example of the hardware configuration of a computer 1200 that functions as an information processing device 100. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of the device according to this embodiment, or to cause the computer 1200 to execute operations associated with the device according to this embodiment or such one or more "parts", and / or to cause the computer 1200 to execute a process or a stage of such process according to this embodiment. Such a program may be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0084] The computer 1200 according to this embodiment includes a CPU 1212, a GPU 1213, a RAM 1214, and a graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive 1226 may be a DVD-ROM drive and a DVD-RAM drive, etc. The storage device 1224 may be a hard disk drive and a solid-state drive, etc. The computer 1200 also includes legacy input / output units such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0085] The CPU 1212 operates according to the programs stored in the ROM 1230 and RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires the image data generated by the CPU 1212 and stores it in the frame buffer provided in the RAM 1214 or within itself, so that the image data is displayed on the display device 1218.

[0086] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive 1226 reads programs or data from the DVD-ROM 1227, etc., and provides them to the storage device 1224. The IC card drive reads programs and data from the IC card and / or writes programs and data to the IC card.

[0087] The ROM 1230 stores boot programs and / or hardware-dependent programs of the computer 1200, which are executed by the computer 1200 when activated. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via USB ports, parallel ports, serial ports, keyboard ports, mouse ports, etc.

[0088] The program is provided on a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The program is read from the computer-readable storage medium and installed on a storage device 1224, RAM 1214, or ROM 1230, which are examples of computer-readable storage media, and executed by the CPU 1212. The information processing described within these programs is read by the computer 1200, resulting in coordination between the program and the various types of hardware resources described above. The apparatus or method may be configured to realize the operation or processing of information in accordance with the use of the computer 1200.

[0089] For example, when communication is performed between a computer 1200 and an external device, the CPU 1212 may execute a communication program loaded into the RAM 1214 and, based on the processing described in the communication program, instruct the communication interface 1222 to perform communication processing. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in a recording medium such as the RAM 1214, storage device 1224, DVD-ROM 1227, or IC card, transmits the read transmission data to the network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

[0090] Furthermore, the CPU 1212 may read all or necessary parts of a file or database stored on an external recording medium such as a storage device 1224, a DVD drive 1226 (DVD-ROM 1227), or an IC card into the RAM 1214, and perform various types of processing on the data in the RAM 1214. The CPU 1212 may then write the processed data back to the external recording medium.

[0091] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and subjected to information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgments, conditional branching, unconditional branching, information retrieval / replacement, etc., as described throughout this disclosure and specified by the program instruction sequence, and write the results back to the RAM 1214. The CPU 1212 may also retrieve information in files, databases, etc., within the recording medium. For example, if a plurality of entries are stored in the recording medium, each having an attribute value of a first attribute associated with an attribute value of a second attribute, the CPU 1212 may search among the plurality of entries for an entry that matches the specified condition for the attribute value of the first attribute, read the attribute value of the second attribute stored in that entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0092] The program or software module described above may be stored on or near the computer 1200 in a computer-readable storage medium. Alternatively, a recording medium such as a hard disk or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the program to the computer 1200 via the network.

[0093] In this embodiment, blocks in the flowchart and block diagram may represent a stage in a process in which an operation is performed or a "part" of a device that has the role of performing an operation. A particular stage and "part" may be implemented by a dedicated circuit, a programmable circuit supplied with computer-readable instructions stored on a computer-readable storage medium, and / or a processor supplied with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuit may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuit may include reconfigurable hardware circuits, such as field-programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), which include logical AND, logical OR, exclusive OR, negated AND, negated OR, and other logical operations, flip-flops, registers, and memory elements.

[0094] A computer-readable storage medium may include any tangible device capable of storing instructions to be executed by a suitable device, and as a result, a computer-readable storage medium having instructions stored therein will comprise a product that includes instructions that can be executed to create means for performing operations specified in a flowchart or block diagram. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable storage media may include floppy disks (registered trademark), diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital versatile disk (DVD), Blu-ray (registered trademark) disk, memory stick, integrated circuit card, etc.

[0095] Computer-readable instructions may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, Java®, C++, and conventional procedural programming languages ​​such as the C programming language or similar programming languages.

[0096] Computer-readable instructions may be provided locally or via a wide area network (WAN) such as a local area network (LAN) or the internet to a processor or programmable circuit of a general-purpose computer, a special-purpose computer, or another programmable data processing device, so that the processor or programmable circuit of the programmable data processing device, such as a computer, may execute the computer-readable instructions to generate means for performing operations specified in a flowchart or block diagram. Here, the computer may be a PC (personal computer), a tablet computer, a smartphone, a workstation, a server computer, a general-purpose computer, or a special-purpose computer, and may also be a computer system in which multiple computers are connected. Such a computer system in which multiple computers are connected is also called a distributed computing system and is a computer in a broad sense. In a distributed computing system, multiple computers execute a program collectively by having each computer execute a part of the program and passing data during program execution between computers as needed.

[0097] Examples of processors include computer processors, central processing units (CPUs), processing units, microprocessors, digital signal processors, controllers, and microcontrollers. A computer may have one or more processors. In a multiprocessor system with multiple processors, each processor executes a portion of the program, and the processors collectively execute the program by passing program execution data between them as needed. For example, in the execution of multitasks, each of the multiple processors may execute a portion of each task in small chunks by switching tasks at each time slice. In this case, which part of a program each processor executes changes dynamically. Which part of a program each of the multiple processors executes may also be statically determined by multiprocessor-aware programming.

[0098] By using the invention according to this embodiment, the accuracy of the learning model 101 that can be used to monitor the vehicle 200 can be improved, and the safety level of the vehicle 200 can be improved, thereby contributing to the achievement of at least one of the Sustainable Development Goals (SDGs): Goal 7 "Affordable and Clean Energy", Goal 9 "Industry, Innovation and Infrastructure", and Goal 11 "Sustainable Cities and Communities".

[0099] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above embodiments. It will be clear from the claims that such modified or improved forms may also be included in the technical scope of the present invention.

[0100] It should be noted that the execution order of operations, procedures, steps, and stages in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not explicitly stated as "before" or "prior to," and that these can be performed in any order unless the output of a previous operation is used in a later operation. Even if the operation flow in the claims, specifications, and drawings is described using phrases such as "first," and "next," for convenience, this does not mean that it is mandatory to perform the operations in that order.

[0101] 50 Network, 100 Information Processing Device, 101 Learning Model, 102 Inference, 104 Generated Sentence, 106 Evaluation / Analysis, 108 Ground Response Data, 109 Video Parameters, 110 Weakness Words, 112 Annotation Data Generation, 114 Annotation Category, 116 Annotation Data, 118 Video Design, 124 Video Generation, 126 Video, 128 Training Data, 130 Fine Tuning, 140 3D Map Data, 150 Storage Unit, 152 Registration Unit, 154 Video Acquisition Unit, 156 Inference Unit, 158 Evaluation and Analysis Unit, 160 Annotation Data Generation Unit, 162 Annotation Data Acquisition Unit, 164 Parameter Acquisition Unit, 166 Road Identification Unit, 168 Video Generation Unit, 170 Model Update Unit, 180 RAN Control Unit, 190 AI Processing Unit, 200 Vehicle, 202 Camera, 210 Video, 300 Management infrastructure, 400 Distributed infrastructure, 500 Wireless base station, 510 RAN, 1200 Computer, 1210 Host controller, 1212 CPU, 1213 GPU, 1214 RAM, 1216 Graphics controller, 1218 Display device, 1220 Input / Output controller, 1222 Communication interface, 1224 Storage device, 1226 DVD drive, 1227 DVD-ROM, 1230 ROM, 1240 Input / Output chip

Claims

1. An information processing device comprising: an annotation data acquisition unit that acquires annotation data including weak words identified by the difference between the text output for an input video and the correct text data for the input video, which is a learning model that outputs text representing the situation of the vehicle in a video based on a video captured by a camera mounted on the vehicle; a parameter acquisition unit that acquires video parameters including the situation of the vehicle and the surrounding situation of the vehicle in the input video; a road identification unit that identifies similar roads similar to the roads on which the vehicle travels in the input video from a 3D map data for simulation including multiple types of roads based on the video parameters; and a video generation unit that generates a video for training the learning model on the weak words based on the annotation data, using the portion of the 3D map data of similar roads identified by the road identification unit.

2. The information processing device according to claim 1, wherein the road identification unit identifies roads from the three-dimensional map data whose similarity to the road in the input video is higher than a predetermined threshold, based on the video parameters.

3. The information processing apparatus according to claim 1 or 2, wherein the video generation unit generates a video that reproduces the situation of the vehicle included in the video parameters on a similar road in the three-dimensional map data identified by the road identification unit.

4. The information processing device according to any one of claims 1 to 3, wherein the video parameters include the number of lanes on the road on which the vehicle is traveling in the input video, and the road identification unit identifies roads from the three-dimensional map data that have the same number of lanes as the number of lanes included in the video parameters as similar roads.

5. The information processing device according to any one of claims 1 to 4, wherein the video parameters include the size of the intersection when the vehicle in the input video is driving through an intersection, and the road identification unit identifies, from the three-dimensional map data, an intersection whose size is the same as the size of the intersection included in the video parameters as the similar road.

6. The information processing device according to any one of claims 1 to 5, wherein the video parameters include the type of curve when the vehicle in the input video is traveling on a curve, and the road identification unit identifies a curve whose type is the same as the type of curve included in the video parameters from the three-dimensional map data as a similar road.

7. The information processing apparatus according to any one of claims 1 to 6, wherein the video parameters include the type of area in which the road on which the vehicle is traveling in the input video is located, and the road identification unit identifies roads in the same area as the type of area included in the video parameters from the three-dimensional map data as similar roads.

8. The information processing apparatus according to any one of claims 1 to 7, further comprising a model update unit that updates the learning model using a plurality of learning data sets including the video generated by the video generation unit and the annotation data.

9. An information processing apparatus according to any one of claims 1 to 8, comprising a RAN control unit for controlling RAN and an AI processing unit for performing AI processing, wherein the AI ​​processing unit includes the annotation data acquisition unit, the parameter acquisition unit, and the video generation unit.

10. An information processing method performed by a computer, comprising: an annotation data acquisition step, in which a learning model that outputs text representing the situation of the vehicle in a video based on a video captured by a camera mounted on the vehicle, acquires annotation data including weak words identified by the difference between the text output for an input video and the correct text data for the input video; a parameter acquisition step, in which video parameters including the situation of the vehicle and the surrounding situation of the vehicle in the input video are acquired; a road identification step, in which similar roads similar to the roads on which the vehicle travels in the input video are identified from three-dimensional map data for simulation including multiple types of roads based on the video parameters; and a video generation step, in which a video is generated using the portion of the road identified in the road identification step from the three-dimensional map data to allow the learning model to learn the weak words based on the annotation data.

11. A program for causing a computer to execute the following steps: an annotation data acquisition step, in which a learning model that outputs text representing the situation of a vehicle in a video based on a video captured by a camera mounted on the vehicle acquires annotation data including weak words identified by the difference between the text output for an input video and the correct text data for the input video; a parameter acquisition step, in which video parameters including the situation of the vehicle and the surrounding situation of the vehicle in the input video are acquired; a road identification step, in which similar roads similar to the roads on which the vehicle travels in the input video are identified from three-dimensional map data for simulation including multiple types of roads based on the video parameters; and a video generation step, in which a video is generated using the portion of the road identified in the road identification step from the three-dimensional map data to allow the learning model to learn the weak words based on the annotation data.

Images