Information processing program, information processing method, and information processing device

Abstract

Provided are an information processing program, an information processing method, and an information processing device which can reduce inference time and improve the inference accuracy by multimodal AI. An information processing program according to the present information causes a computer to perform: a data acquisition step for acquiring video data; a token conversion step for performing conversion into tokens that correspond in number to the video data acquired by the data acquisition step, on the basis of the amount of information included in the video data; and a data processing step for inputting, to a machine learning model, the tokens which have been generated by the token conversion step and causing the machine learning model to execute processing of the video data corresponding to the number of tokens.

Description

Information Processing Program, Information Processing Method, and Information Processing Apparatus

[0001] The present invention relates to an information processing program, an information processing method, and an information processing apparatus.

[0002] In recent years, research on multimodal AI (Artificial Intelligence) that handles information combining different types of information such as language and images has been actively conducted. As a technology related to multimodal AI, for example, a technology for generating answers to questions about the situation shown in a video is known.

[0003] Conventionally, in multimodal AI, regardless of the length and resolution of the video, the encoded visual information has been converted into fixed-length tokens and input into an LLM (Large Language Model) (see, for example, Non-Patent Documents 1 and 2).

[0004] Jean-Baptiste Alayrac, et al., “Flamingo: a Visual Language Model for Few-Shot Learning”, [online], 15 November 2022, arXiv:2204.14198, [searched on October 18, 2024], Internet <URL: https: / / arxiv.org / pdf / 2204.14198> Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, Chunyuan Li, “LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models”, [online], 28 July 2024, arXiv:2407.07895, [searched on October 18, 2024], Internet <URL: https: / / arxiv.org / abs / 2407.07895>

[0005] The information processing program according to the present invention causes a computer to perform the following steps: a data acquisition step of acquiring video data; a token conversion step of converting the video data acquired in the data acquisition step into a number of tokens corresponding to the video data based on the amount of information contained in the video data acquired in the data acquisition step; and a data processing step of inputting the tokens generated in the token conversion step into a machine learning model and causing the machine learning model to perform processing of the video data according to the number of tokens.

[0006] Figure 1 is a system configuration diagram showing an overview of the automated driving system. Figure 2 is a block diagram of the automated driving support device according to the first embodiment. Figure 3 is a diagram showing an overview of an example of a token conversion unit and multimodal AI. Figure 4 is a diagram showing an example of driving support according to the number of tokens using a land-based judgment criterion. Figure 5 is a diagram showing an example of driving support according to the number of tokens using a time-series judgment criterion. Figure 6 is a flowchart of the driving support processing by the automated driving support device according to the first embodiment. Figure 7 is a diagram showing an example of driving support processing when the number of tokens is fixed. Figure 8 is a diagram showing an example of driving support processing using the automated driving support device according to the first embodiment. Figure 9 is a diagram for explaining the efficiency of GPU resources by the automated driving support device according to the first embodiment. Figure 10 is a diagram showing the relationship between the automated driving domain and the remote monitoring domain. Figure 11 is a block diagram of the automated driving support device according to the second embodiment. Figure 12 is a flowchart of the driving support processing by the automated driving support device according to the second embodiment. Figure 13 is a diagram showing an example of a computer that executes an information processing program related to driving support.

[0007] Hereinafter, embodiments of the information processing program, information processing method, and information processing apparatus according to the present application will be described in detail with reference to the drawings. However, the technology is not limited by these embodiments. Furthermore, in the drawings, identical parts are denoted by the same reference numerals, and redundant explanations are omitted.

[0008] (First Embodiment) (1. Introduction) When tokens are of fixed length, there is a risk of increased inference time and decreased inference accuracy. For example, because tokens are of fixed length, even with images containing little information, a predetermined number of tokens must be processed, resulting in wasted time for inference. Also, in the case of images containing a large amount of information, the number of tokens may be insufficient to adequately represent the information in the image, in which case data loss occurs and inference accuracy decreases. Therefore, it is difficult to shorten inference time and improve inference accuracy with a technology that uses fixed-length tokens to enable multimodal AI to perform inference. As a result, it may be difficult to provide appropriate and sufficient driving assistance in an automated driving system in which, for example, a multimodal AI is deployed at a base station controlled by a RIC (RAN Intelligent Controller).

[0009] Figure 1 is a system configuration diagram showing an overview of the autonomous driving system. As shown in Figure 1, the autonomous driving system 1 includes a base station 10, a vehicle 20, and a remote monitoring system 30.

[0010] Vehicle 20 is capable of communicating with base station 10. Vehicle 20 transmits various operational information, such as driving video, to base station 10. Vehicle 20 then receives driving support, such as support information related to autonomous driving, from base station 10 and remote monitoring system 30.

[0011] The remote monitoring system 30 is a system for remote monitoring by remote operators such as the automated operation supervisor to remotely monitor the vehicle 20. The remote monitoring system 30 is connected to the base station 10 via a network.

[0012] The remote monitoring system 30 receives requests from the base station 10 for assistance with the operation of the vehicle 20. The remote monitoring system 30 can monitor the vehicle 20 and provide driving assistance in response to these requests. For example, the remote monitoring system 30 can notify remote monitors about the arrangement of personnel to be dispatched to the scene in the event of an accident involving the vehicle 20, or about how to respond if unexpected behavior occurs during autonomous driving.

[0013] The base station 10 transmits and receives data with the vehicle 20 via wireless communication. The base station 10 is also connected to the remote monitoring system 30 via a network, and they exchange data with each other.

[0014] The base station 10 has an automated driving support system 100. The automated driving support system 100 has a large-scale GPU (Graphics Processing Unit).

[0015] The autonomous driving system 1 is required to provide the vehicle 20 with higher performance and comprehensive support in conjunction with the remote monitoring system 30 by utilizing the large-scale GPUs of the base station 10, thereby realizing a fulfilling autonomous driving society. However, while the GPU resources of a single base station 10 are limited, the resources required by the generative AI realized by the GPU are increasing year by year.

[0016] One of the factors that determines the inference cost of a multimodal AI is the length of the input tokens. A token is the basic unit of information that a multimodal AI handles when performing inference. Converting target data into tokens that a multimodal AI can process is called tokenization. A multimodal AI tokenizes video data and performs inference by understanding the information for each token.

[0017] Traditionally, while the length of text tokens is variable, the length of video tokens, i.e., the number of tokens, has often been fixed. In that case, processing video requires a certain amount of GPU resources. However, since the amount of information in video varies depending on the location and circumstances in which it is filmed, if the number of tokens is fixed, a small amount of information leads to an increase in inference time, and a large amount of information leads to the loss of information during LLM input. Therefore, in an autonomous driving system that performs inference with a fixed number of tokens, the inference cost increases linearly as the number of vehicles 20 increases, and there is a risk that resources will not be able to cover more than a certain number of vehicles 20.

[0018] Therefore, in the following embodiment, the amount of information in the video is estimated from the number of tokens used when encoding the video, and the length of the text applied to the video is made variable to reduce inference cost and inference time and improve inference accuracy. Since the situations encountered by each vehicle 20 vary, lightweight inference is performed for vehicles 20 in safe situations, and large-scale inference is performed for vehicles 20 in complex situations to reduce the average load and enable monitoring of more vehicles 20. The determination of whether a situation is safe or complex is made based on the number of tokens applied to the video.

[0019] (2. Automated Driving Support System) Figure 2 is a block diagram of the automated driving support system according to the first embodiment. Next, the details of the automated driving support system 100 according to this embodiment will be described with reference to Figure 2. As shown in Figure 2, the automated driving support system 100 has a data acquisition unit 101, a token conversion unit 102, a data processing unit 103, and a notification unit 104.

[0020] The data acquisition unit 101 receives video data from images captured by an on-board camera 22 mounted on the vehicle 20. The images from the on-board camera 22 are, for example, images of the outside of the vehicle as seen from the vehicle 20. The images of the outside may be images of the front of the vehicle 20, or they may also include images of the left, right, and rear. In addition, the images from the on-board camera 22 may also include images of the inside of the vehicle.

[0021] The data acquisition unit 101 outputs the received video data to the token conversion unit 102. Here, the data acquisition unit 101 may output all frames included in the video data to the token conversion unit 102, or it may output frames that have been thinned out to predetermined intervals.

[0022] The token conversion unit 102 has an image encoder. The token conversion unit 102 receives video data input from the data acquisition unit 101. Next, the token conversion unit 102 extracts each frame from the video data one by one. Then, using the image encoder, the token conversion unit 102 selects and discards information in the images of the extracted frames and converts it into a number of tokens corresponding to the amount of information. Hereinafter, the image shown in the extracted frame will be referred to as the "target image". Subsequently, the token conversion unit 102 outputs the tokens generated from the target image to the multimodal AI 131 of the data processing unit 103.

[0023] For example, the token conversion unit 102 divides the image into numerous image blocks and selects several important image blocks based on dependencies and correlations between them. The token conversion unit 102 then reduces redundancy in the image by having the selected image blocks retain information about other related image blocks, thereby determining the number of tokens. In this case, the token conversion unit 102 can assign a small number of tokens to simple images, such as large areas where image blocks with the same content are consecutive, and assign a larger number of tokens as the image becomes more complex.

[0024] The image encoder held by the token conversion unit 102 may include, for example, LaVIT (Yang Jin et al., “UNIFIED LANGUAGE-VISION PRETRAINING IN LLM WITH DYNAMIC DISCRETE VISUAL TOKENIZATION”, 22 March 2024). The image encoder can use LaVIT to convert the image features of the target image into tokens. In this way, the token conversion unit 102 converts the amount of information contained in the video data acquired by the data acquisition unit 101 into a number of tokens corresponding to the video data.

[0025] The data processing unit 103 uses tokens obtained from the token conversion unit 102 to analyze the video data and make decisions such as answering questions and notifying the remote monitoring system 30 of response requests. The data processing unit 103 includes a multimodal AI 131 and a driver support unit 132.

[0026] The multimodal AI 131 is a machine learning model that generates answers to questions based on video data and text data. The multimodal AI 131 has multiple predefined question templates related to the driving of the vehicle 20, such as questions about road conditions. Here, the questions are assigned levels according to the complexity of the road conditions.

[0027] The multimodal AI 131 may have two levels of questions: basic questions and questions for detailed investigation. Basic questions are, for example, "Please describe the situation in the video." Questions for detailed investigation to analyze the driving situation in more detail are, for example, "Please tell me what action I should take in my vehicle" or "Please tell me the situation of the passengers inside the vehicle."

[0028] In this embodiment, the case in which the multimodal AI 131 performs inference using pre-set questions has been described, but the method of obtaining questions is not limited to this. For example, the multimodal AI 131 may receive question input from the remote monitoring system 30 and answer those questions.

[0029] The multimodal AI 131 has a pre-defined average range of token counts corresponding to the judgment criteria. For example, in the case of land-based judgment criteria, the multimodal AI 131 can pre-store an average range of token counts in a certain area of ​​land through which the vehicle 20 travels. In the case of time-series judgment criteria, the multimodal AI 131 may also store an average range of token counts at each point along the route taken by the vehicle 20. In this case, the multimodal AI 131 can also use different average ranges for each time period. As for how to determine the average range, the multimodal AI 131 may calculate the average range from statistics of the number of tokens previously input, or it may receive information on the average range manually.

[0030] The multimodal AI 131 receives the tokens into which the target image has been converted by the token conversion unit 102. Next, the multimodal AI 131 obtains the number of tokens into which the target image has been converted.

[0031] The multimodal AI 131 determines whether the number of tokens in the target image falls within the average range corresponding to the judgment criteria. If the number of tokens falls within the average range, the amount of information in the target image is considered to be the same as under normal circumstances and normal operation is assumed. Therefore, if the number of tokens falls within the average range, the multimodal AI 131 selects a basic question appropriate to the target image from its stored question templates. Then, the multimodal AI 131 encodes the basic question and converts it into tokens.

[0032] Next, the multimodal AI 131 takes the tokens of the target image and the tokens of the basic question as input and performs answer inference. After that, the multimodal AI 131 outputs the answer to the basic question, which is the inference result.

[0033] Here, we will explain an example where the multimodal AI 131 uses the basic question, "Please describe the situation in the video." In this case, since the number of tokens falls within the average range and is considered normal operation, the multimodal AI 131 may output a response such as, "It is a straight road with nothing in it, so please go straight."

[0034] In contrast, if the number of tokens is not within the average range, the multimodal AI 131 determines whether the number of tokens is greater than the average range. If the number of tokens is greater than the average range, it is considered that special road conditions are occurring at that location. Therefore, if the multimodal AI 131 determines that the number of tokens is greater than the average range, it selects basic questions and questions for detailed investigation from its stored question templates, in addition to questions appropriate to the target image. Then, the multimodal AI 131 encodes the basic questions and questions for detailed investigation and converts them into tokens.

[0035] Next, the multimodal AI 131 takes the tokens of the target image, as well as the tokens of the basic questions and the questions for detailed investigation, as input and performs answer inference. After that, the multimodal AI 131 outputs the answers to the basic questions and the questions for detailed investigation, which are the inference results.

[0036] Here, we will explain an example where the multimodal AI 131 uses the question "Please describe the situation in the video" as a basic question and the question "Please tell me what action I should take with my vehicle" as a question for detailed investigation. In this case, since the number of tokens is higher than the average range and is considered a special situation, the multimodal AI 131 may output a response to the basic question such as "Construction is underway ahead, and traffic controllers are directing traffic." Furthermore, the multimodal AI 131 may output a response to the question for detailed investigation such as "Please drive your vehicle according to the instructions of the traffic controllers."

[0037] In response to this, if the number of tokens is not greater than the average range, the multimodal AI 131 determines that the amount of information in the target image is less than the average range. If the amount of information in the target image is less than the average range, it is considered that the road conditions at that location are safer than usual. Therefore, the multimodal AI 131 terminates processing without selecting a question or inferring an answer.

[0038] In this embodiment, in order to reduce the overall load on the autonomous driving system 1 as much as possible and allocate processing power to other processes, the multimodal AI 131 is not allowed to perform inference when the amount of information in the target image is less than the average range, but this is not limited to this. Even when the amount of information in the target image is less than the average range, the multimodal AI 131 may still answer basic questions.

[0039] Figure 3 shows an overview of an example of a token conversion unit and a multimodal AI. For example, the token conversion unit 102 has an image encoder 121. The multimodal AI 131 has a text encoder 311 and an LLM 312.

[0040] Token conversion unit 102 receives, for example, an input of video data including target image 120. Then, token conversion unit 102 extracts the frame in which target image 120 appears from the video data. Next, token conversion unit 102 inputs target image 120 to image encoder 121. And token conversion unit 102 outputs tokens 300 with the number of tokens corresponding to the information amount of target image 120, which has been converted by image encoder 121, to multimodal AI 131.

[0041] Multimodal AI 131 selects question 301 corresponding to the number of tokens of tokens 300 from the question formats it holds. Then, multimodal AI 131 inputs the selected question 301 to text encoder 311 to convert it into tokens 302.

[0042] Here, text encoder 311 shown in FIG. 3 may be realized by any known technique capable of extracting text feature amounts from text. For example, text encoder 311 may include a recurrent neural network (RNN: Recurrent Neural Network). And text encoder 311 is not particularly limited as long as it is an encoder that extracts text feature amounts from text and converts them into tokens using an RNN.

[0043] Thereafter, multimodal AI 131 inputs tokens 300 generated from target image 120 and tokens 302 generated from question 301 to LLM 312. And multimodal AI 131 obtains an answer to question 301 output from LLM 312.

[0044] Here, in the present embodiment, the automatic driving support device 100 has been described with a configuration in which it has a token conversion unit 102 separately from the multimodal AI 131 and each operates as a different mechanism, but it is not limited to this. For example, multimodal AI 131 may be configured as one AI having the function of token conversion unit 102.

[0045] Returning to FIG. 2, the description will be continued. The driving support unit 132 receives the input of the answer output from the multimodal AI 131. Next, the driving support unit 132 determines the response based on the type and content of the answer. Then, the driving support unit 132 executes driving support processing according to the determined response.

[0046] For example, when the answer is an answer to a basic question, the driving support unit 132 determines the notification of the answer to the driver. Then, the driving support unit 132 transmits the answer to the driving support device 21 of the vehicle 20 to notify the driver of the answer.

[0047] In addition, when the answer includes an answer for a detailed investigation in addition to the answer to a basic question, the driving support unit 132 checks the content of the answer and determines the response. For example, when the content of the answer is to instruct the stop of the vehicle 20, the driving support unit 132 determines the stop of the vehicle 20. Also, when it is determined from the content of the answer that the response of the remote monitor is necessary, the driving support unit 132 determines the notification to the remote monitor. Then, the driving support unit 132 transmits the answer to the driving support device 21 of the vehicle 20 and instructs the notification unit 104 to transmit a message of a response request to the remote monitoring system 30. Further, the driving support unit 132 instructs the driving support device 21 to stop the vehicle 20 to stop the vehicle 20. In addition, the driving support unit 132 may record the answer to the question and the like as a log.

[0048] Here, an example of driving support by the multimodal AI 131 and the driving support unit 132 will be described. The first case is a case where the vehicle is stopped due to traffic control. The multimodal AI 131 selects, for example, the following three questions from the question templates. Question #1 is "Please tell me the current situation." Question #2 is "Please tell me the action to be taken by my vehicle." Question #3 is "Please tell me the condition of the passengers in the vehicle."

[0049] In this case, the multimodal AI 131 will output a response to question #1, for example, "Construction is underway ahead and traffic controllers are directing traffic." The multimodal AI 131 will also output a response to question #2, for example, "Traffic controllers are asking you to stop, so please stop." The multimodal AI 131 will also output a response to question #3, for example, "There is nothing wrong inside the vehicle."

[0050] The driver support unit 132 decides to stop the vehicle 20 and notify the remote monitor based on the responses output from the multimodal AI 131. The driver support unit 132 then instructs the vehicle 20's driver support device 21 to stop the vehicle 20. The driver support unit 132 also transmits the responses output from the multimodal AI 131 to the remote monitoring system 30 to notify the remote monitor.

[0051] The second case involves a passenger being injured due to a sudden stop caused by someone jumping out onto the tracks. In this case as well, the multimodal AI 131 can select questions #1 to #3 from the question template, similar to the first case.

[0052] In this case, the multimodal AI 131 would output a response to question #1, for example, "The car came to a sudden stop because a child ran out in front of it." The multimodal AI 131 would also output a response to question #2, for example, "The child is still standing still, so please stop." The multimodal AI 131 would also output a response to question #3, for example, "The passenger is slumped forward and not moving. Therefore, the passenger may have lost consciousness. Contact the remote monitor and request assistance."

[0053] Based on the responses output from the multimodal AI 131, the driver support unit 132 decides to prompt the remote monitor to input additional questions or make an appeal to passengers. The driver support unit 132 then sends a message to the remote monitoring system 30 prompting the remote monitor to input additional questions and make an appeal to passengers, thereby notifying the remote monitor. In this case, when an additional question from the remote monitor is sent to the driver support unit 132 via the notification unit 104, the driver support unit 132 inputs the additional question into the multimodal AI 131, outputs an answer to that additional question, and sends that answer to the remote monitoring system 30. The remote monitor may also make an appeal to passengers using other means of communication.

[0054] In this way, the data processing unit 103 inputs the tokens generated by the token conversion unit 102 into the multimodal AI 131, which is a machine learning model, and causes the multimodal AI 131 to process the video data according to the number of tokens. A basic question is an example of a first question, and a question for detailed investigation is an example of a second question. That is, if the number of tokens falls within a predetermined average range, the data processing unit 103 has the multimodal AI 131 perform inference for the first question. If the number of tokens is above the predetermined average range, the data processing unit 103 has the multimodal AI 131 perform inference for a second question that includes the first question and is more detailed than the first question. Furthermore, if the number of tokens is below the predetermined average range, the data processing unit 103 does not need to have the multimodal AI 131 perform inference. The data processing unit 103 also supports the driving of the vehicle 20 based on the processing results of the video data. Notifying the vehicle 20 of messages and stopping the vehicle 20 are examples of driving support.

[0055] The notification unit 104 receives instructions from the driver support unit 132 to send a message to the remote monitoring system 30. The notification unit 104 then sends a message to the remote monitoring system 30 via the network, notifying the remote monitor of the necessary actions.

[0056] Sending a message requesting a response from the data processing unit 103 to the remote monitoring system 30 via the notification unit 104 is an example of "alert output." That is, if the number of tokens is above a predetermined average range, the data processing unit 103 outputs an alert based on the processing results of the video data.

[0057] (3. Examples of driving assistance based on the number of tokens) Figure 4 is a diagram showing an example of driving assistance based on the number of tokens using land-specific criteria. Referring to Figure 4, an example of driving assistance based on the number of tokens using land-specific criteria will be explained. Here, the explanation will be given in the case where the multimodal AI 131 sets the average range of the number of tokens on this land to 60-80.

[0058] The token conversion unit 102 extracts a frame containing the target image 201 from the video data of a vehicle 20 traveling at a first point in a specific area. The token conversion unit 102 then uses an image encoder to convert the target image 201 into tokens. In this case, the number of tokens obtained by converting the target image 201 is 70.

[0059] The multimodal AI 131 receives the tokens converted from the target image 201 and determines whether the number of tokens falls within the average range. The multimodal AI 131 then selects a basic question and converts the question into tokens using a text encoder. The multimodal AI 131 then inputs the tokens from the target image 201 and the tokens from the question into the LLM and obtains the response 211 output from the LLM, such as "Please continue to drive safely."

[0060] The driver assistance unit 132 decides whether to notify the driver of the response 211 and transmits the response 211 to the driver assistance device 21 of the vehicle 20.

[0061] Furthermore, the token conversion unit 102 extracts a frame containing the target image 202 from the video data of a vehicle 20 traveling at a different second location on the same land. The token conversion unit 102 then uses an image encoder to convert the target image 202 into tokens. In this case, the number of tokens obtained by converting the target image 202 is also 70.

[0062] In this case as well, since the number of tokens falls within the average range, the multimodal AI 131 selects a basic question and converts the question into tokens using a text encoder. Then, the multimodal AI 131 inputs the tokens from the target image 202 and the tokens from the question into the LLM and obtains the answer 212 output from the LLM, such as "Please continue to drive safely."

[0063] The driver assistance unit 132 decides whether to notify the driver of the response 212 and transmits the response 212 to the driver assistance device 21 of the vehicle 20.

[0064] Furthermore, the token conversion unit 102 extracts a frame containing the target image 203 from the video data of a vehicle 20 traveling at yet another third location on the same land. The token conversion unit 102 then uses an image encoder to convert the target image 203 into tokens. In this case, the number of tokens obtained by converting the target image 203 is 100.

[0065] The multimodal AI 131 receives tokens converted from the target image 201 and determines that the number of tokens is greater than the average range. The multimodal AI 131 then selects basic questions and questions for detailed investigation and converts the questions into tokens using a text encoder. The multimodal AI 131 then inputs the tokens from the target image 201 and the question tokens into the LLM and obtains the response 213 output from the LLM, such as "The driving conditions are different from the surrounding area." and "Slow down and be prepared to react to unexpected actions."

[0066] The driver support unit 132 decides to notify the driver of the answer 213 and to enhance monitoring by the remote monitor. The driver support unit 132 then sends a message to the remote monitoring system 30 via the notification unit 104 to notify the remote monitor of the enhanced monitoring of the vehicle 20 traveling at the third point. The driver support unit 132 also records a log of the answers to the questions. The driver support unit 132 also sends the answer 213 to the driver support device 21 of the vehicle 20.

[0067] Figure 5 shows an example of driver assistance based on the number of tokens using time-series criteria. Referring to Figure 5, an example of driver assistance based on the number of tokens using time-series criteria will be described. Here, the multimodal AI 131 has an average range 230 as information for determining the number of tokens at this point.

[0068] The token conversion unit 102 extracts a frame containing the target image 221 from the video data of the vehicle 20 traveling at this location at time t1. Then, the token conversion unit 102 converts the target image 221 into tokens using an image encoder. In this case, the number of tokens obtained by converting the target image 221 is 231.

[0069] The multimodal AI 131 receives the tokens converted from the target image 221 as input and determines that the number of tokens 231 falls within the average range 230. Then, the multimodal AI 131 selects a basic question and converts the question into tokens using a text encoder. Finally, the multimodal AI 131 inputs the tokens from the target image 201 and the tokens from the question into the LLM and obtains the answer output from the LLM.

[0070] The driver assistance unit 132 decides whether to notify the driver of the response and transmits the response obtained from the multimodal AI 131 to the driver assistance device 21 of the vehicle 20.

[0071] Furthermore, the token conversion unit 102 extracts a frame containing the target image 222 from the video data of the vehicle 20 traveling at this location at time t2. Then, the token conversion unit 102 converts the target image 222 into tokens using an image encoder. In this case, the number of tokens obtained by converting the target image 222 is 232.

[0072] In this case as well, since the number of tokens 232 is included in the average range 230, the multimodal AI 131 selects a basic question and converts the question into tokens using a text encoder. Then, the multimodal AI 131 inputs the tokens of the target image 222 and the tokens of the question into the LLM and obtains the answer output from the LLM.

[0073] The driver assistance unit 132 decides whether to notify the driver of the response and transmits the response obtained from the multimodal AI 131 to the driver assistance device 21 of the vehicle 20.

[0074] Furthermore, the token conversion unit 102 extracts a frame containing the target image 223 from the video data of the vehicle 20 traveling at this location at time t3. Then, the token conversion unit 102 converts the target image 223 into tokens using an image encoder. In this case, the number of tokens obtained by converting the target image 223 is 233.

[0075] The multimodal AI 131 receives tokens converted from the target image 223 and determines that the number of tokens 233 is greater than the average range 230. The multimodal AI 131 then selects basic questions and questions for detailed investigation and converts the questions into tokens using a text encoder. The multimodal AI 131 then inputs the tokens from the target image 223 and the tokens from the questions into the LLM and obtains the detailed answers output from the LLM.

[0076] The driver support unit 132 decides whether to notify the driver of a detailed response and transmits the detailed response to the driver support device 21 of the vehicle 20. Furthermore, the driver support unit 132 decides on a response based on the type and content of the question. Then, according to the decided response, the driver support unit 132 performs driver support processing such as notifying the remote monitoring system 30 of a request for action to be taken by the remote monitor.

[0077] (4. Driving Support Processing) Figure 6 is a flowchart of the driving support processing by the automated driving support device according to the first embodiment. Next, the flow of the driving support processing by the automated driving support device 100 according to this embodiment will be explained with reference to Figure 6. Note that the following steps S1 to S11 can be executed in a different order. Also, some of the following steps S1 to S11 may be omitted.

[0078] The data acquisition unit 101 acquires video data captured by the on-board camera 22 of the vehicle 20 (step S1).

[0079] The token conversion unit 102 extracts frames from the video data. Then, the token conversion unit 102 uses an image encoder to convert the target images contained in the frames into tokens (step S2).

[0080] The multimodal AI 131 receives input of tokens obtained by transforming the target image 201 and determines whether the number of tokens falls within the average range (step S3).

[0081] If the number of tokens falls within the average range (Step S3: Affirmative), the multimodal AI 131 selects a basic question (Step S4).

[0082] Next, the multimodal AI 131 converts the basic question into tokens using a text encoder. Then, the multimodal AI 131 inputs the tokens of the target image and the tokens of the basic question into the LLM, causing the LLM to perform answer inference (step S5). After that, the driving assistance process proceeds to step S11.

[0083] On the other hand, if the number of tokens is not included in the average range (step S3: negative), the multimodal AI 131 determines whether the number of tokens is greater than the average range (step S6). If the number of tokens is less than the average range (step S6: negative), the multimodal AI 131 terminates the driving assistance process.

[0084] On the other hand, if the number of tokens is greater than the average range (step S6: affirmative), the multimodal AI 131 selects basic questions and questions for further investigation (step S7).

[0085] Next, the multimodal AI 131 uses a text encoder to convert the basic questions and questions for detailed investigation into tokens. Then, the multimodal AI 131 inputs the tokens of the target image, as well as the tokens of the basic questions and questions for detailed investigation into the LLM, causing the LLM to perform answer inference (step S8).

[0086] The driver support unit 132 determines whether to notify the remote monitoring system 30 of a message based on the type and content of the response obtained from the multimodal AI 131, and sends the message to the remote monitoring system 30 via the notification unit 104 (step S9).

[0087] Furthermore, the driver support unit 132 records the answers to basic questions and questions for detailed investigation as a log (step S10).

[0088] Subsequently, the driver assistance unit 132 sends answers to basic questions and questions for detailed investigation, and gives instructions for automated driving to the driver assistance device 21 of the vehicle 20, thereby providing driver assistance (step S11).

[0089] (5. Comparison with the case where the number of tokens is fixed) Figure 7 is a diagram showing an example of the driving support process when the number of tokens is fixed. Figure 8 is a diagram showing an example of the driving support process using the automatic driving support device according to the first embodiment. Here, we will compare the case where the number of tokens is fixed with the case where the automatic driving support device 100 according to this embodiment is used, referring to Figures 7 and 8.

[0090] Let's explain the case using an image 111 of an average road condition, as shown in Figure 7. In this case, the image 111 is input to an image encoder 401, which has a fixed number of tokens after conversion, and tokens 412 are generated. Also, since the question does not change depending on the number of tokens, for example, a basic question 411 such as "Please tell me about the situation in the video." is input to a text encoder 402. Then, tokens 412 and the tokens obtained by converting question 411 are input to the LLM 403, and as an inference result for the answer, the answer 414 "It is a straight road with nothing in it, so please go straight." is obtained.

[0091] Next, we will explain the case using image 112 of a complex road situation. In this case, image 112 is input to image encoder 401 and token 413 is generated. Furthermore, since the question does not change depending on the number of tokens, a basic question such as "Please describe the situation in the video," such as question 411, is input to text encoder 402, just as in the case of image 111. Then, token 413 and the token obtained by converting question 411 are input to LLM 403, and as an inference result for the answer, the answer 415, "There is a person standing still, so please stop," is obtained.

[0092] In this case, tokens 412 and 413 are tokens based on images with different amounts of information, but both have the same number of tokens. Therefore, there is a possibility that token 412 is redundant, or that some of the information from image 112 is missing in token 413. Also, because the same question is used, the level of the answer will be about the same even if images 111 and 112, which have different amounts of information, are used. In other words, when using an image with less information, redundant processing may occur, or when using an image with more information, a sufficient answer may not be obtained.

[0093] Next, referring to Figure 8, we will describe the case in which an image 111 of average road conditions is used for the automated driving support device 100 according to this embodiment. In this case, the token conversion unit 102 generates tokens 322 using an image encoder 121 in which the number of tokens changes according to the amount of information in the image. Since the number of tokens in this case is within the average range, the multimodal AI 131 inputs a question 321, such as a basic question, "Please tell me the situation in the image," into the text encoder 311. The multimodal AI 131 then inputs the tokens 322 and the tokens obtained by converting the question 321 into the LLM 312 and obtains the answer 323, "It is a straight road with nothing in it, so please go straight," as the answer inference result.

[0094] Next, we will explain the case where an image 112 of a complex road situation is used. In this case, the token conversion unit 102 inputs the image 112 to the image encoder 121 and generates tokens 332. Furthermore, since the number of tokens is greater than the average range, the multimodal AI 131 selects questions 331, which include, for example, a basic question such as "Please tell me the situation in the video," and a question for detailed investigation such as "Please tell me what action I should take with my vehicle." The multimodal AI 131 then inputs questions 331 to the text encoder 311. The multimodal AI 131 then inputs tokens 322 and the tokens converted from questions 331 to the LLM 403. As an inference result for the answer, the multimodal AI 131 obtains the answer 333, which is "Construction is underway ahead, and traffic controllers are directing traffic." and "Please drive your vehicle according to the instructions of the traffic controllers."

[0095] In this case, the token conversion unit 102 makes the number of tokens 332 for the information-rich image 112 and the number of tokens 322 for the information-poor image different. That is, token 332 can contain a sufficient amount of information about the information-rich image 112. Furthermore, questions can be asked that are appropriate to the amount of information in the image, and the multimodal AI 131 can generate answers to appropriate questions and provide appropriate answers according to the situation.

[0096] (6. Effects) The automated driving support device 100 according to this embodiment changes the number of tokens according to the amount of information contained in the target image and converts it into tokens, and generates questions according to the number of tokens. Next, the automated driving support device 100 uses the tokens of the target image according to the amount of information and the tokens of the question according to the number of tokens to perform inference of the answer to the question. Then, the automated driving support device 100 performs driving support according to the inferred answer.

[0097] As a result, the automated driving support system 100 can appropriately set the number of tokens for the target image and use the information contained in the target image for inference without excess or deficiency. Furthermore, it can use questions corresponding to the number of tokens, and the automated driving support system 100 can perform inference using appropriate questions corresponding to the target image. Therefore, the automated driving support system 100 can contribute to shortening the inference time and improving the inference accuracy of the multimodal AI 131.

[0098] Figure 9 is a diagram illustrating the efficiency of GPU resource utilization by the automated driving support system according to the first embodiment. Here, the automated driving support system 100 of the base station 10 receives video data from vehicles 511 to 514. Vehicles 511 and 512 are in a safe state, vehicle 513 is in normal road conditions, and vehicle 514 is in dangerous road conditions. Here, it is assumed that the number of tokens is less than the average range in a safe state, the number of tokens is within the average range in normal road conditions, and the number of tokens is more than the average range in dangerous road conditions.

[0099] Situation 501 describes communication when the number of tokens is fixed. In this case, since the number of tokens is fixed and the questions do not change according to the number of tokens, the automated driving support system 100 processes roughly the same amount of data for each of the vehicles 511 to 514. As a result, the inference load of the multimodal AI 131 is applied to the MEC (Multi-access Edge Computing)-GPU of the automated driving support system 100, multiplied by the number of vehicles 511 to 1514, resulting in high consumption of GPU resources.

[0100] On the other hand, situation 502 shows communication when using the automated driving support device 100 according to the first embodiment. In this case, for vehicles 511 and 512 that are in a safe state, the multimodal AI 131 does not perform inference, so the amount of data processing is reduced. Also, for vehicle 513 in normal road conditions, the multimodal AI 131 assigns the usual number of tokens to the image and asks basic questions, so the amount of data processing is equivalent to, for example, when the number of tokens is fixed. In contrast, for vehicle 514 in dangerous road conditions, the multimodal AI 131 increases the number of tokens and uses questions to analyze the road conditions in more detail, so the amount of processing increases. In this case, the automated driving support device 100 can allocate the GPU resources saved on the data for vehicles 511 and 512 to data processing for vehicle 514.

[0101] In this way, the multimodal AI 131 can reduce the average processing load by performing lightweight reasoning for vehicles 511-513 that are in a safe and normal state, and large-scale reasoning for vehicle 514 that is in a complex situation. Therefore, the automated driving support system 100 can monitor more vehicles 20.

[0102] Figure 10 shows the relationship between the domain of autonomous driving and the domain of remote monitoring. In Figure 10, the vertical axis shows the frequency of occurrence of a situation, and the horizontal axis shows the complexity of the road conditions. Graph 521 shows the general relationship between the frequency of occurrence and the complexity of the road conditions.

[0103] As shown in Graph 521, the more complex the road conditions, the less frequently such situations occur. Range 522 is the area that can be handled by the autonomous driving of the vehicle 20. Range 522 includes images such as Image 113, which show situations where there is not much information and can be handled by autonomous driving. Range 525 is an area that is difficult for autonomous driving to handle, such as situations where support from the remote monitoring system 30 is required. Range 525 includes images such as Image 114, which show situations where there is a lot of information and autonomous driving is insufficient to handle. In events in Range 525, vehicle costs and complexity increase.

[0104] Here, if the range 522 that can be handled by autonomous driving is expanded, the area 525 that is difficult for autonomous driving to handle will decrease. A smaller area 525 means, for example, that the number of situations requiring support from the remote monitoring system 30 can be reduced, and the support from the remote monitoring system 30 can be more concentrated on each individual situation.

[0105] Furthermore, by using the automated driving support device 100 according to the first embodiment, it is possible to expand the range 525 by improving the accuracy of automated driving. Therefore, by using the automated driving support device 100 according to the first embodiment, it becomes possible to provide more appropriate support and to reduce overall vehicle costs.

[0106] (Second Embodiment) (1. Details of the Automated Driving Support System) Figure 11 is a block diagram of the automated driving support system according to the second embodiment. Next, with reference to Figure 11, the details of the automated driving support system 100 according to this embodiment will be described. In the following description, the operation of each part, which is the same as in the first embodiment, will be omitted. In this embodiment, the vehicle 20 has a CAN (Controller Area Network) 23 and a measuring device 24.

[0107] CAN23 is an in-vehicle device that communicates using a predetermined communication standard. CAN23 outputs driving-related information such as speed and steering angle.

[0108] The measuring device 24 is a device that processes various types of information mounted on the vehicle 20. The measuring device 25 is, for example, a Lidar (Light Detection and Ranging) or radar. Lidar uses laser light to detect objects. Radar uses millimeter-wave wavelength signals to detect objects at long distances or objects passing through fog or clouds. The measuring device 24 outputs measurement information, which is the measurement result.

[0109] The multimodal AI 131 compares the number of tokens in the target image input from the token conversion unit 102 with the average range. If the number of tokens is greater than the average range, the multimodal AI 131 performs the following processing.

[0110] The multimodal AI 131 acquires driving-related information from the CAN 23 installed in the vehicle 20. The multimodal AI 131 also acquires measurement information from the measuring device 24 installed in the vehicle 20. Hereafter, driving-related information and measurement information will be collectively referred to as vehicle information.

[0111] Furthermore, the multimodal AI 131 selects basic questions and detailed investigation questions from its stored question templates that are appropriate for the image. Next, the multimodal AI 131 encodes the basic questions and detailed investigation questions and converts them into tokens. Then, the multimodal AI 131 estimates an answer using the tokens of the target image, the tokens of the basic questions and detailed investigation questions, and the vehicle information as input.

[0112] Here, the driving-related information acquired from CAN23 and the measurement information acquired from measuring device24 are examples of "additional information." In other words, the data processing unit 103 acquires additional information related to the video data and inputs this additional information along with tokens into the multimodal AI 131, which is a machine learning model, to process the video data.

[0113] (2. Driving Support Processing) Figure 12 is a flowchart of the driving support processing by the automated driving support device according to the second embodiment. Next, the flow of the driving support processing by the automated driving support device 100 according to this embodiment will be explained with reference to Figure 12. Note that steps S21 to S32 below can be executed in a different order. Also, some of the steps S21 to S32 below may be omitted.

[0114] The data acquisition unit 101 acquires video data captured by the on-board camera 22 of the vehicle 20 (step S21).

[0115] The token conversion unit 102 extracts frames from the video data. Then, the token conversion unit 102 uses an image encoder to convert the target images contained in the frames into tokens (step S22).

[0116] The multimodal AI 131 receives input of tokens obtained by transforming the target image 201 and determines whether the number of tokens falls within the average range (step S23).

[0117] If the number of tokens falls within the average range (step S23: affirmative), the multimodal AI 131 selects a basic question (step S24).

[0118] Next, the multimodal AI 131 converts the basic question into tokens using a text encoder. Then, the multimodal AI 131 inputs the tokens of the target image and the tokens of the basic question into the LLM, causing the LLM to perform answer inference (step S25). After that, the driving assistance process proceeds to step S32.

[0119] On the other hand, if the number of tokens is not included in the average range (step S23: negative), the multimodal AI 131 determines whether the number of tokens is greater than the average range (step S26). If the number of tokens is less than the average range (step S26: negative), the multimodal AI 131 terminates the driving support process.

[0120] On the other hand, if the number of tokens is greater than the average range (step S26: affirmative), the multimodal AI 131 acquires vehicle information from the CAN 23 and measuring device 24 of the vehicle 20 (step S27).

[0121] Next, the multimodal AI 131 selects basic questions and questions for detailed investigation (step S28).

[0122] Next, the multimodal AI 131 uses a text encoder to convert the basic questions and questions for detailed investigation into tokens. The multimodal AI 131 then inputs the tokens of the target image, the tokens of the basic questions and questions for detailed investigation, and the vehicle information into the LLM, causing the LLM to perform answer inference (step S29).

[0123] The driver support unit 132 determines whether to notify the remote monitoring system 30 of a message based on the type and content of the response obtained from the multimodal AI 131, and sends the message to the remote monitoring system 30 via the notification unit 104 (step S30).

[0124] Furthermore, the driver support unit 132 records the answers to basic questions and questions for detailed investigation as a log (step S31).

[0125] Subsequently, the driver assistance unit 132 sends answers to basic questions and questions for detailed investigation, and gives instructions for automated driving to the driver assistance device 21 of the vehicle 20, thereby providing driver assistance (step S32).

[0126] (3. Effects) The automated driving support device 100 according to this embodiment uses vehicle information in addition to tokens of the target image corresponding to the amount of information and tokens of the question corresponding to the number of tokens to perform inference of the answer to the question. As a result, the automated driving support device 100 can perform inference that takes into account the actual operating conditions more closely, and can contribute to improving the inference accuracy of the multimodal AI 131.

[0127] (Other Use Cases) The above description focused on remote monitoring of autonomous driving, but information processing devices with similar functions to the autonomous driving support device 100 can also be applied to other fields, as long as they perform control using inference by a multimodal AI 131 based on video and text.

[0128] For example, such an information processing device can be applied to robot monitoring. In this case, the information processing device acquires video data monitoring the operation of robots installed in factories, restaurants, etc., and when an error occurs, it uses questions about how to deal with the error to have the multimodal AI 131 infer countermeasures for the error. The information processing device can then perform processes such as requesting action from the support department.

[0129] Furthermore, the information processing device can be applied to monitoring and responding immediately to hospitalized patients. For example, it can acquire video data to monitor the health status of hospitalized patients 24 hours a day, and if an abnormal condition is detected, it can use questions related to written documents to have the multimodal AI 131 infer the name of the disease. The information processing device can then perform processes such as conveying the information to a doctor.

[0130] (Program) It is also possible to create a program in which the processes performed by the automated driving support device 100 described in the above embodiment are written in a language that can be executed by a computer. In this case, the same effects as in the above embodiment can be obtained by having the computer execute the program. Furthermore, the same processes as in the above embodiment may be realized by recording such a program on a computer-readable recording medium and having the computer read and execute the program recorded on this recording medium.

[0131] (Hardware Configuration) Figure 13 shows an example of a computer that executes an information processing program related to driving assistance. The automatic driving assistance device 100 is implemented by a computer system 1000 as shown in Figure 13. The computer system 1000 has a CPU 1100, RAM 1200, ROM 1300, HDD 1400, communication interface (I / F) 1500, input / output interface (I / F) 1600, and media interface (I / F) 1700.

[0132] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, and controls each component. The ROM 1300 stores boot programs executed by the CPU 1100 when the computer system 1000 starts up, as well as programs that depend on the hardware of the computer system 1000.

[0133] The HDD 1400 stores programs executed by the CPU 1100, as well as data used by such programs. The communication interface 1500 receives data from other devices via the communication network NW and sends it to the CPU 1100, and transmits the data generated by the CPU 1100 to other devices via the communication network NW.

[0134] The CPU 1100 controls output devices such as displays and printers, and input devices such as keyboards and mice, via the input / output interface 1600. The CPU 1100 acquires data from input devices via the input / output interface 1600. The CPU 1100 also outputs the generated data to output devices via the input / output interface 1600.

[0135] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical Disc), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0136] For example, when the computer system 1000 functions as the automated driving assistance device 100 of this disclosure, the CPU 1100 of the computer system 1000 realizes the functions of the automated driving assistance device 100 by executing a program loaded on the RAM 1200. The HDD 1400 stores the data in the storage device 130. The CPU 1100 of the computer system 1000 reads and executes these programs from the recording medium 1800, but as another example, these programs may be obtained from another device via a communication network NW.

[0137] Although some embodiments of the present invention have been described in detail above with reference to the drawings, these are merely examples, and the invention can be implemented in various forms with modifications and improvements based on the knowledge of those skilled in the art, starting with the embodiments described in the disclosure section of the invention.

[0138] (Other) Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, and all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0139] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0140] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not contradictory. Also, the terms "section, module, unit" described above can be replaced with "means," "circuit," etc. For example, the acquisition unit can be replaced with acquisition means or acquisition circuit.

[0141] The technologies according to each embodiment described above can improve operational efficiency in an automated society by utilizing AI, and will serve as an innovative technological foundation for an automated society, thereby contributing to the achievement of Sustainable Development Goal (SDG) 9, "Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation."

[0142] 10 Base station 20 Vehicle 21 Driving support device 22 In-vehicle camera 30 Remote monitoring system 101 Data acquisition unit 102 Token conversion unit 103 Data processing unit 104 Notification unit 131 Multimodal AI 132 Driving support unit

Claims

1. An information processing program characterized by causing a computer to execute a data acquisition step of acquiring video data; a token conversion step of converting the video data acquired in the data acquisition step into a number of tokens corresponding to the video data based on the amount of information contained in the video data acquired in the data acquisition step; and a data processing step of inputting the tokens generated in the token conversion step into a machine learning model and causing the machine learning model to perform processing of the video data according to the number of tokens.

2. The information processing program according to claim 1, wherein the data processing step includes the process by which the machine learning model performs inference for the first question if the number of tokens falls within a predetermined average range, and the machine learning model performs inference for the first question and for a second question which is more detailed than the first question if the number of tokens is greater than or equal to a predetermined average range.

3. The information processing program according to claim 1, characterized in that the data processing step includes a process in which the machine learning model does not perform inference if the number of tokens is less than a predetermined average range.

4. The information processing program according to claim 3, characterized in that the data processing step includes a process of acquiring additional information relating to the video data, inputting the additional information together with the token into the machine learning model, and causing the machine learning model to perform processing of the video data.

5. The information processing program according to claim 3, characterized in that the data acquisition step includes a process for acquiring the video data captured by an imaging device mounted on the vehicle, and the data processing step includes a process for supporting the driving of the vehicle based on the processing results of the video data.

6. The information processing program according to claim 3, characterized in that the data processing step includes outputting an alert based on the processing result of the video data if the number of tokens is above a predetermined average range.

7. An information processing method characterized in that an information processing device performs a data acquisition step of acquiring video data; a token conversion step of converting the amount of information contained in the video data acquired by the data acquisition step into a number of tokens corresponding to the video data; and a data processing step of inputting the tokens generated by the token conversion step into a machine learning model and causing the machine learning model to perform processing of the video data according to the number of tokens.

8. An information processing apparatus comprising: a data acquisition unit that acquires video data; a token conversion unit that converts the amount of information contained in the video data acquired by the data acquisition unit into a number of tokens corresponding to the video data; and a data processing unit that inputs the tokens generated by the token conversion unit into a machine learning model and causes the machine learning model to perform processing of the video data according to the number of tokens.

Images