Information processing device and information processing method

Abstract

The present invention is provided with: a frame selection unit (101) which, on the basis of an input moving image and a prompt, outputs data indicating a selected frame, which is a frame used for answering the prompt, from among frames included in the moving image; and an answer unit (103) which, on the basis of the data output by the frame selection unit (101), outputs data indicating an answer to the prompt.

Description

Information Processing Apparatus and Information Processing Method

[0007] ,

[0006] ,

[0001] The present disclosure relates to an information processing apparatus and an information processing method.

[0002] In recent years, research on MLLMs (Multimodal LLMs (Vid-LLMs)) for Video Understanding has been active (see, for example, Non-Patent Document 1). More specifically, various methods have been proposed for MLLMs, such as a method of analyzing a video with an Analyzer, a method of inputting an embedding of a video into an LLM, and a method of combining them.

[0003] [2312.17432v4] Video Understanding with Large Language Models: A Survey (arxiv.org)

[0004] As described above, the development of MLLMs capable of handling videos is active. Generally, in MLLMs, the more frames there are, the more memory is required for inference. Although MLLMs can be used in aspects such as security or safety in surveillance operations, personal information such as face images is often captured, so the demand for inference at the edge rather than in the cloud is considered high. On the other hand, since there are memory limitations in edge inference, memory reduction is necessary.

[0005] The present disclosure has been made to solve the above problems, and an object thereof is to provide an information processing apparatus capable of reducing memory when generating an answer to a prompt compared to the conventional art.

[0006] The information processing apparatus according to the present disclosure includes a frame selection unit that outputs data indicating a selected frame, which is a frame used for an answer to the prompt, among the frames included in the input video based on the input video and the prompt, and an answer unit that outputs data indicating an answer to the prompt based on the data output by the frame selection unit.

[0007] According to the present disclosure, since it is configured as described above, it is possible to reduce memory when generating an answer to a prompt compared to the conventional art.

[0008] This figure shows an example of the configuration of an information processing device according to Embodiment 1. This flowchart shows an example of the operation of an information processing device according to Embodiment 1. This figure shows an example of a video input to the frame selection unit in Embodiment 1. This figure shows an example of video division by the frame selection unit in Embodiment 1. This figure shows an example of processing results by the frame selection unit in Embodiment 1. This figure shows an example of the content of data stored in the answer database in Embodiment 1. This figure shows an example of the configuration of an information processing device according to Embodiment 2. This flowchart shows an example of the operation of an information processing device according to Embodiment 2. This figure shows an example of processing results by the frame selection unit in Embodiment 2. This figure shows an example of processing results by the frame selection unit in Embodiment 2, where the number of previously selected frames is greater than the specified number of frames. This figure shows an example of the content of data stored in the answer database in Embodiment 2. This figure shows an example of the configuration of an information processing device according to Embodiment 3. This flowchart shows an example of the operation of an information processing device according to Embodiment 3. Figures 14A and 14B show examples of input to the frame selection unit in Embodiment 4, where Figure 14A shows an example of a video input to the frame selection unit, and Figure 14B shows an example of constraint information included in the prompt input to the frame selection unit. This figure shows an example of the configuration of an information processing device according to Embodiment 5. This figure shows an example of a video input to the frame selection unit in Embodiment 5. This figure shows an example of the processing result by the frame selection unit in Embodiment 5. Figures 18A and 18B show examples of the hardware configuration of the information processing device according to Embodiments 1-5.

[0009] The embodiments will be described in detail below with reference to the drawings. Embodiment 1. Figure 1 is a diagram showing an example of the configuration of the information processing device 1 according to Embodiment 1. The information processing device 1 includes, for example, a frame selection unit 101, a response database (response DB) 102, and a response unit 103, as shown in Figure 1.

[0010] The frame selection unit 101 outputs data indicating which frames from the video are used to answer the prompt, based on the input video and prompt.

[0011] The prompt input to the frame selection unit 101 is a prompt entered by a user, for example, via an input device (not shown), such as a keyboard, in response to the video input to the frame selection unit 101. This prompt is text, but may also include an image in addition to text. The video input to the frame selection unit 101 is, for example, a video obtained by an imaging device (not shown). This video may be input directly to the information processing device 1, or it may be converted into still images frame by frame at a specified FPS before being input to the information processing device 1. Furthermore, the conversion to still images frame by frame may be performed by the information processing device 1. In addition, a timestamp may be added to the still images when they are converted to still images frame by frame.

[0012] In the following, a prompt input to the frame selection unit 101 may be referred to as an external input prompt to distinguish it from other prompts. Also, in the following, a frame used to respond to a prompt obtained by the frame selection unit 101 may be referred to as a selected frame to distinguish it from other frames.

[0013] Furthermore, the frame selection unit 101 may also output data indicating reason information that shows at least one of the reasons for selecting a frame and the reasons for not selecting any other frames.

[0014] The reason information described above may include information indicating at least one of the following: a summary of the video, and the position of each frame within the video. Examples of the position of a frame within the video include information regarding whether the frame is necessary to respond to an external input prompt, information indicating the differences between the frame and other frames, information indicating the characteristics of the frame, and information indicating the characteristics of the frame within the video as a whole. The frames included in the video may include at least one of the selected frame and other frames.

[0015] The frame selection unit 101 may also output data indicating the timestamp of the video.

[0016] In Embodiment 1, the frame selection unit 101 directly selects frames using MLLM. Furthermore, when the frame selection unit 101 outputs data indicating additional information which is at least one of reason information and a timestamp, the frame selection unit 101 directly selects frames and generates the additional information using MLLM.

[0017] As an example of the processing procedure of the frame selection unit 101, first, the frame selection unit 101 receives video and external input prompts as input. Then, the frame selection unit 101 inputs the received video and external input prompts to the MLLM. The frame selection unit 101 then requests the MLLM to select the frames necessary to answer the external input prompts from the video. Furthermore, if the frame selection unit 101 outputs data indicating additional information, which is at least one of reason information and a timestamp, it requests the MLLM to generate the additional information in addition to the frame selection request. The frame selection unit 101 then receives the frames selected by the MLLM and outputs data indicating the selected frames. Furthermore, if additional information is generated by the MLLM, the frame selection unit 101 receives the additional information in addition to the selected frames and outputs data indicating the selected frames and the additional information.

[0018] The video input to the frame selection unit 101 may be one video or multiple videos. If the video input to the frame selection unit 101 is multiple videos, the frame selection unit 101 may perform the above processing for each video.

[0019] Furthermore, the frame selection unit 101 may divide the input video into specified frame counts (Sn) and perform the above processing for each divided frame, which is a group of divided frames. For example, the specified frame count (Sn) may be specified by the user, or the information processing device 1 may automatically specify the number that can be processed from its own computing resources. Generally, it is thought that there is often a limit to the number that can be processed based on the GPU memory usage when generating answers using MLLM used by the frame selection unit 101.

[0020] Furthermore, the above division process is not limited to application to the information processing device 1 according to Embodiment 1, but can also be applied to the information processing device 1 according to other embodiments.

[0021] The response database 102 stores the data output by the frame selection unit 101. In this case, it is desirable that the response database 102 stores the data output by the frame selection unit 101 in association with data indicating the corresponding external input prompt.

[0022] Examples of the answer database 102 include non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), and EEPROM (Electrically EPROM), as well as magnetic disks, flexible disks, optical disks, compact disks, minidiscs, or DVDs (Digital Versatile Discs).

[0023] Figure 1 shows the case where the response database 102 is located inside the information processing device 1. However, the response database 102 is not limited to this configuration and may be located outside the information processing device 1.

[0024] The response unit 103 outputs data indicating the response to the external input prompt based on the data output by the frame selection unit 101. At this time, the response unit 103 obtains the data output by the frame selection unit 101 by reading data from the response database 102 and performs the above processing. The response unit 103 generates the response to the external input prompt using MLLM. The response obtained by the response unit 103 is text, but this response may also include images in addition to text.

[0025] The output destination for the data from the response unit 103 may be, for example, a display device (not shown) that displays the information indicated by the data, or a storage device (not shown) that stores the data.

[0026] An example of the processing procedure for the answer unit 103 is as follows: First, the answer unit 103 reads data corresponding to the external input prompt from the answer database 102. Then, the answer unit 103 inputs the read data and the external input prompt to the MLLM and requests the MLLM to generate an answer for the external input prompt. Finally, the answer unit 103 receives the answer generated by the MLLM and outputs data indicating that answer.

[0027] The MLLM used in the frame selection unit 101 and the MLLM used in the answer unit 103 may be the same or different. Furthermore, the frame selection unit 101 and the answer unit 103 may use an internally held MLLM or an external MLLM.

[0028] Next, an example of the operation of the information processing device 1 according to Embodiment 1 shown in Figure 1 will be explained with reference to Figures 2 to 6. In the following explanation, we will describe the case in which the frame selection unit 101 uses MLLM to select frames and generate reason information and timestamps. Furthermore, in the following explanation, we will describe the case in which the frame selection unit 101 divides a video.

[0029] In the information processing device 1 according to Embodiment 1 shown in Figure 1, for example, as shown in Figure 2, first the frame selection unit 101 receives video and prompt input (step ST101).

[0030] For example, in Figure 3, a video showing a shopping street is input to the frame selection unit 101. Then, a prompt such as, "It appears a fight has broken out. What were the people involved doing?" is input to the frame selection unit 101 in response to this video.

[0031] Next, the frame selection unit 101 divides the video into segments of a specified number of frames (Sn) (step ST102). In the example in Figure 4, the specified number of frames (Sn) is 6, and the frame selection unit 101 divides the video shown in Figure 3 into segments of 6 frames. In the example in Figure 4, three divided frames are created.

[0032] Next, the frame selection unit 101, based on the external input prompt, uses MLLM to select a frame to be used as a response to the external input prompt for each divided frame, and generates reason information and a timestamp (step ST103). The reason information may include information indicating at least one of the following: a summary of the video and the position of each frame within the video.

[0033] In the example in Figure 5, the frame selection unit 101 uses MLLM to select a frame and generate reason information and a timestamp for the topmost of the three divided frames shown in Figure 4 (the first divided frame). In this case, for example, the frame selection unit 101 inputs a prompt to MLLM such as, "Based on all the information in the video, you need to answer the external input prompt. In the first divided frame, we want to select frames that are likely to be related to the question indicated by the external input prompt. Please provide a summary of the first divided frame and answer the following for each frame within the first divided frame: (1) timestamp, (2) relationship to the question: yes / no, (3) reason for (2) and the position of that frame in the first divided frame." Then, for example, in response to the above prompt, MLLM provides the answer, as a summary of the first divided frame, "A white person appears in the middle, and a gray person also appears at the end." Furthermore, as shown in Figure 5, for example, MLLM responds to the first and second frames of the first divided frame as having no relevance to the question, stating that "the subject is not visible and is not related to the question," and describing its position as "an unnecessary frame in the video." MLLM also responds to the third through fifth frames of the first divided frame as having relevance to the question, stating that "there is a subject, and only a white person is visible," and describing its position as "this is the first appearance of a white person in this video." Finally, MLLM responds to the sixth frame of the first divided frame as having relevance to the question, stating that "there is a subject, and both a white person and a gray person are visible," and describing its position as "this is the first appearance of a gray person in this video." Each frame within the first divided frame is also timestamped. Note that Figure 5 is an illustrative example of MLLM's responses; in reality, MLLM's responses may be composed in natural language.

[0034] Furthermore, in the example in Figure 5, MLLM provides a combined answer for the first and second frames of the first divided frame because they contain the same content; however, individual answers may also be provided. Similarly, in the example in Figure 5, MLLM provides a combined answer for the third through fifth frames of the first divided frame because they contain the same content; however, individual answers may also be provided.

[0035] The data obtained by the frame selection unit 101, including the selected frame, reason information, and timestamp, is stored in the response database 102. Figure 6 shows an example of the data stored in the response database 102. In practice, the above data is stored in the response database 102 in a format such as JSON.

[0036] Next, the response unit 103 generates a response to the external input prompt using MLLM based on the selected frame, reason information, and timestamp (step ST104).

[0037] For example, the response unit 103 inputs a prompt to the MLLM, such as, "Please respond to the external input prompt using the data stored in the response database 102." Then, for example, in response to the above prompt, the MLLM provides a response such as, "The fight occurred at 01:00. The people involved are the white person and the gray person. The white person appeared from the left at 00:20 and disappeared to the left after the fight. They reappeared from the left at 02:10. The gray person appeared from the right at 00:50 and disappeared to the right after the fight. They reappeared from the right at 02:40. The fight scene is shown below," and displays the relevant frames. Note that the MLLM may also display frame numbers or timestamps instead of video or images.

[0038] As described above, in the information processing device 1 according to Embodiment 1, the frame selection unit 101 outputs data indicating the frame among the frames contained in the video that will be used to answer the prompt, based on the input video and the prompt. As a result, in the information processing device 1 according to Embodiment 1, the number of frames used by the answer unit 103 when generating an answer to the prompt using MLLM can be reduced, and the memory used to generate an answer to the prompt can be reduced compared to conventional methods. As a result, the functions of the information processing device 1 can be applied even when they are installed on edge devices with limited memory.

[0039] While systems exist that select frames from an input video and use those frames to generate answers to input questions, conventional systems select frames using a predetermined method regardless of the input question. On the other hand, in a video question-answering system that is not limited to a specific domain, it is expected that user questions will be diverse, and if the frame selection method is predetermined, there is a possibility that the selected frame may not be appropriate for use in answering the question. In contrast, the information processing device 1 according to Embodiment 1 has a frame selection unit 101 that outputs data indicating which frames from the video are to be used to answer the prompt, based on the input video and the prompt. In this way, the information processing device 1 according to Embodiment 1 enables frame selection that takes the prompt into consideration, making it possible to select a more appropriate frame for answering the prompt compared to conventional systems.

[0040] As described above, according to this embodiment 1, the information processing device 1 includes a frame selection unit 101 that outputs data indicating a selected frame, which is used to answer the prompt, from among the frames included in the video, based on the input video and the prompt, and an answer unit 103 that outputs data indicating the answer to the prompt, based on the data output by the frame selection unit 101. As a result, the information processing device 1 according to embodiment 1 can reduce the memory used to generate the answer to the prompt compared to the conventional method. Furthermore, the information processing device 1 according to embodiment 1 can improve the accuracy of the answer to the prompt compared to the conventional method.

[0041] Furthermore, according to this embodiment 1, the frame selection unit 101 also outputs data indicating reason information that shows at least one of the reasons for selecting a frame and the reasons for not selecting any other frames. As a result, in the information processing device 1 according to embodiment 1, at least one of the reasons for selecting a frame and the reasons for not selecting any other frames becomes clear, and further improvement in the accuracy of answer generation can be expected.

[0042] Furthermore, according to this embodiment 1, the reason information includes information indicating at least one of the following: a summary of the video and the position of each frame within the video. As a result, in the information processing device 1 according to embodiment 1, the information of at least one of the video information and the information of the frames contained in the video becomes clearer, and further improvement in the accuracy of answer generation can be expected.

[0043] Furthermore, according to this embodiment 1, the frame selection unit 101 also outputs data indicating the timestamp of the video. As a result, the information processing device 1 according to embodiment 1 provides clearer information about the video, and further improvements in the accuracy of answer generation can be expected.

[0044] Also, according to the first embodiment, the frame selection unit 101 directly performs frame selection using the MLLM. As a result, in the information processing apparatus 1 according to the first embodiment, by using the MLLM, reason information can be generated more freely in natural language, and further improvement in the accuracy of answer generation can be expected.

[0045] Also, according to the first embodiment, the frame selection unit 101 divides the video into specified numbers of frames and performs frame selection processing for each divided frame, which is a collection of the divided frames. As a result, in the information processing apparatus 1 according to the first embodiment, even when the number of frames of the video is large, it becomes possible to perform processing after dividing the number of frames that the frame selection unit 101 can handle.

[0046] Also, according to the first embodiment, the video is a plurality of videos, and the frame selection unit 101 performs frame selection processing for each video. As a result, in the information processing apparatus 1 according to the first embodiment, it becomes possible to handle a plurality of videos.

[0047] Also, according to the first embodiment, the information processing method includes a step in which the frame selection unit 101 outputs data indicating a selected frame, which is a frame used for answering the prompt among the frames included in the input video, based on the input video and the prompt, and a step in which the answer unit 103 outputs data indicating an answer to the prompt based on the data output by the frame selection unit 101. As a result, in the information processing method according to the first embodiment, it becomes possible to reduce the memory when generating an answer to the prompt compared to the prior art. Also, in the information processing method according to the first embodiment, it becomes possible to improve the accuracy of the answer to the prompt compared to the prior art.

[0048] Second Embodiment. In the first embodiment, the case where the frame selection unit 101 directly performs processing including frame selection using the MLLM has been shown. In contrast, in the second embodiment, the case where the frame selection unit 101b calls a Vision-based model using the MLLM or the LLM and causes this model to perform processing including frame selection will be described.

[0049] Figure 7 shows an example of the configuration of the information processing device 1 according to Embodiment 2. In the information processing device 1 according to Embodiment 2 shown in Figure 7, the frame selection unit 101 is changed to a frame selection unit 101b compared to the information processing device 1 according to Embodiment 1 shown in Figure 1. Other configuration examples in the information processing device 1 according to Embodiment 2 shown in Figure 7 are the same as the configuration example of the information processing device 1 according to Embodiment 1 shown in Figure 1, and only the different parts will be described using the same reference numerals.

[0050] The frame selection unit 101b outputs data indicating which frames from the video are used to answer the prompt, based on the input video and prompt.

[0051] The prompt input to the frame selection unit 101b is a prompt entered by the user, for example, via an input device (not shown), such as a keyboard, in response to the video input to the frame selection unit 101b. This prompt is text, but may also include an image in addition to text. The video input to the frame selection unit 101b is a video obtained by, for example, an imaging device (not shown). This video may be input directly to the information processing device 1, or it may be converted into still images frame by frame at a specified FPS before being input to the information processing device 1. Furthermore, the conversion to still images frame by frame may be performed by the information processing device 1. In addition, a timestamp may be added to the still images when they are converted to still images frame by frame.

[0052] In the following, a prompt input to the frame selection unit 101b may be referred to as an external input prompt to distinguish it from other prompts. Also, in the following, a frame used to respond to a prompt obtained by the frame selection unit 101b may be referred to as a selected frame to distinguish it from other frames.

[0053] Furthermore, the frame selection unit 101b may also output data indicating reason information that shows at least one of the reasons for selecting a frame and the reasons for not selecting any other frames.

[0054] Furthermore, the above reasoning information may include information indicating the reason for selecting a Vision-type model.

[0055] The frame selection unit 101b may also output data indicating the timestamp of the video.

[0056] In Embodiment 2, the frame selection unit 101b selects a frame by selecting a Vision-based model using MLLM or LLM. Furthermore, if the frame selection unit 101b outputs data indicating additional information which is at least one of reason information and a timestamp, the frame selection unit 101b selects a frame and generates the additional information by selecting a Vision-based model using MLLM or LLM.

[0057] As an example of the processing procedure of the frame selection unit 101b, first, the frame selection unit 101b receives video and external input prompts as input. Then, the frame selection unit 101b inputs the received video and external input prompts to the MLLM. The frame selection unit 101b then selects an appropriate Vision system model from the MLLM or LLM and requests that the MLLM or LLM use the selected Vision system model to select the frames necessary to answer the external input prompts from the video. Furthermore, if the frame selection unit 101b outputs data indicating additional information, which is at least one of reason information and a timestamp, it requests the MLLM or LLM to generate the additional information using the selected Vision system model, in addition to the frame selection request mentioned above. The frame selection unit 101b then receives the frames selected by the MLLM or LLM and outputs data indicating the selected frames. Furthermore, if additional information is generated by the MLLM or LLM, the frame selection unit 101b receives the additional information in addition to the selected frames and outputs data indicating the selected frames and the additional information.

[0058] The video input to the frame selection unit 101b may be one video or multiple videos. If the video input to the frame selection unit 101b is multiple videos, the frame selection unit 101b may perform the above processing for each video.

[0059] Furthermore, if the number of selected frames exceeds the specified number of frames (Dn), the frame selection unit 101b may repeat the frame selection process to reduce the number of frames. For example, the specified number of frames (Dn) may be specified by the user, or the information processing device 1 may automatically specify the number that can be processed from its own computing resources. Generally, it is thought that there is often a limit to the number that can be processed based on the GPU memory usage when generating answers using MLLM used in the answer unit 103.

[0060] Furthermore, if the number of selected frames exceeds the specified number of frames (Dn) even after repeating the selection process a predetermined number of times, the frame selection unit 101b reduces the number of selected frames to the specified number of frames (Dn) or less by reducing the number of frames from the selected frames using a predetermined method. In this case, for example, the frame selection unit 101b may reduce the number of selected frames to the specified number of frames (Dn) or less by selecting frames at equal intervals from the selected frames. The predetermined number of times can be set arbitrarily.

[0061] Furthermore, the above frame count reduction process is not limited to application to the information processing device 1 according to Embodiment 2, but can also be applied to the information processing device 1 according to other embodiments.

[0062] In the second embodiment, the response database 102 stores the data output by the frame selection unit 101b. In this case, it is desirable that the response database 102 stores the data output by the frame selection unit 101b in association with data indicating the corresponding external input prompt.

[0063] Furthermore, in Embodiment 2, the response unit 103 outputs data indicating the response to the external input prompt based on the data output by the frame selection unit 101b. At this time, the response unit 103 obtains the data output by the frame selection unit 101b by reading data from the response database 102 and performs the above processing.

[0064] The MLLM used in the frame selection unit 101b and the MLLM used in the answer unit 103 may be the same or different. Furthermore, the frame selection unit 101b and the answer unit 103 may use an internally held MLLM or an external MLLM.

[0065] Next, an example of the operation of the information processing device 1 according to Embodiment 2 shown in Figure 7 will be described with reference to Figures 8 to 11. In the following description, we will explain using as an example the case in which the frame selection unit 101b selects a Vision system model using MLLM or LLM to select frames and generate reason information and timestamps. Furthermore, in the following description, we will explain using as an example the case in which the frame selection unit 101b determines the number of selected frames.

[0066] In the information processing device 1 according to Embodiment 2 shown in Figure 7, for example, as shown in Figure 8, first, the frame selection unit 101b receives video and prompt input (step ST201). A specific example of the operation of the frame selection unit 101b in step ST201 is the same as that shown as a specific example of the operation of the frame selection unit 101 in step ST101 in Embodiment 1.

[0067] Next, the frame selection unit 101b, based on an external input prompt, selects a Vision-type model using MLLM or LLM for the video, selects a frame to be used as a response to the external input prompt, and generates reason information and a timestamp (step ST202). The reason information may include information indicating the reason for selecting the Vision-type model.

[0068] For example, the frame selection unit 101b inputs a prompt to the MLLM or LLM, such as, "You need to respond to an external input prompt regarding the video. We want to select frames in the video that are likely to be related to the question indicated by the external input prompt. Please select them using an appropriate Vision system model. Furthermore, please answer the following questions about the video, including the selected Vision system model and the reason for it: (1) Timestamp, (2) Relation to the question: Yes / No, (3) Reason for (2)." Then, for example, in response to the above prompt, the MLLM or LLM selects "Object Detection" as the Vision system model and answers, "Because we need frames that show people," as the reason. Also, as shown in Figure 9, for example, the MLLM or LLM provides answers for each frame of the video. Note that the example in Figure 9 is an illustrative diagram of the response by the MLLM or LLM, and in reality, the response by the MLLM or LLM may be composed of natural language.

[0069] Next, the frame selection unit 101b determines whether the number of selected frames is less than or equal to the number of frames (Dn) (step ST203).

[0070] In step ST203, if the frame selection unit 101b determines that the number of selected frames is greater than the number of frames (Dn), the sequence returns to step ST202, and the frame selection unit 101b repeats the frame selection process so that the number of frames decreases.

[0071] For example, suppose the specified number of frames (Dn) is 9. If the frame selection unit 101b determines that the number of selected frames is greater than 9, it will input a prompt to the MLLM or LLM, such as, "You previously provided a list of potentially relevant frames, but it exceeds Dn (=9). Please further refine the selection of frames." In response to this prompt, the MLLM or LLM will select "Tracking" as the Vision model and provide the reason, "Frames with small movements of the same person are unnecessary." Alternatively, as shown in Figure 10, the MLLM or LLM will provide a response for each frame of the video. In the example in Figure 9, there were 11 selected frames, whereas in the example in Figure 10, there are 9 selected frames, indicating a reduction in the number of selected frames. Note that the example in Figure 10 is an illustrative diagram of the response by the MLLM or LLM; in reality, the response by the MLLM or LLM may be composed in natural language.

[0072] On the other hand, in step ST203, if the frame selection unit 101b determines that the number of selected frames is less than or equal to the number of frames (Dn), the sequence proceeds to step ST204.

[0073] The data obtained by the frame selection unit 101b, including the selected frame, reason information, and timestamp, is stored in the response database 102. Figure 11 shows an example of the data stored in the response database 102. The example in Figure 11 shows the case where the specified number of frames (Dn) is 11 or greater. In this case, the above data is actually stored in the response database 102 in a format such as Json.

[0074] Next, the response unit 103 generates a response to the external input prompt using MLLM based on the selected frame, reason information, and timestamp (step ST204). A specific example of the operation of the response unit 103 in step ST204 is the same as that shown as a specific example of the operation of the response unit 103 in step ST104 in Embodiment 1.

[0075] As described above, according to this embodiment 2, the frame selection unit 101b selects a frame by selecting a Vision-type model using MLLM or LLM. As a result, the information processing device 1 according to embodiment 2 can generate reason information more freely in natural language by using MLLM or LLM, and further improvements in the accuracy of response generation can be expected.

[0076] Furthermore, according to this second embodiment, if the number of selected frames is greater than the specified number of frames, the frame selection unit 101b repeats the frame selection process to reduce the number of frames. As a result, the information processing device 1 according to the second embodiment can reduce the number of selected frames to a number that the response unit 103 can handle.

[0077] Furthermore, according to this second embodiment, if the number of selected frames exceeds the specified number of frames even after repeating the selection process a predetermined number of times, the frame selection unit 101b reduces the number of selected frames by a predetermined method so that the number of selected frames is less than or equal to the specified number of frames. As a result, the information processing device 1 according to the second embodiment can avoid a state in which it cannot exit a loop of repetitive processing.

[0078] Furthermore, in the case where processing including frame selection is performed directly using MLLM, as in the frame selection unit 101 in Embodiment 1, it is expected that reason information can be expressed more flexibly in natural language compared to the case of Embodiment 2. On the other hand, in the case where a Vision-type model is called using MLLM or LLM and processing including frame selection is performed by this model, as in the frame selection unit 101b in Embodiment 2, it is expected that processing can be performed faster without using memory because it is not necessary to pass images to MLLM or LLM compared to the case of Embodiment 1.

[0079] Embodiment 3. The information processing device 1 according to Embodiments 1 and 2 described the process from when a prompt is first input to a video until an answer to that prompt is obtained. In contrast, the information processing device 1 according to Embodiment 3 describes the process when a prompt is subsequently input in response to the answer.

[0080] Figure 12 shows an example of the configuration of the information processing device 1 according to Embodiment 3. In the information processing device 1 according to Embodiment 3 shown in Figure 12, the frame selection unit 101 is changed to a frame selection unit 101c compared to the information processing device 1 according to Embodiment 1 shown in Figure 1. Other configuration examples in the information processing device 1 according to Embodiment 3 shown in Figure 12 are the same as the configuration example of the information processing device 1 according to Embodiment 1 shown in Figure 1, and only the different parts will be described using the same reference numerals.

[0081] In addition to the functions of the frame selection unit 101 in Embodiment 1, the frame selection unit 101c has a function for determining whether a prompt needs to be re-entered (re-entry determination function) and a function for determining whether reprocessing is necessary in the frame selection unit 101c (reprocessing determination function).

[0082] The re-input determination function is a function in which the frame selection unit 101c determines whether a prompt has been input for the answer obtained by the answer unit 103. In other words, the frame selection unit 101c determines whether a second or subsequent prompt has been input consecutively for the video.

[0083] In the following, the external input prompt used for the response by the response unit 103 described above will be referred to as the previous external input prompt, and the external input prompt entered immediately after that response will be referred to as the current external input prompt.

[0084] Furthermore, the reprocessing determination function determines whether reprocessing by the frame selection unit 101c is necessary based on the answer, the previous external input prompt, and the video when the frame selection unit 101c determines that a prompt has been entered for the answer generated by the answer unit 103. If the frame selection unit 101c determines that reprocessing is necessary, it outputs data indicating the frame from the video that will be used for the answer to the current external input prompt, based on the current external input prompt. The frame selection unit 101c may also output data indicating additional information, which is at least one of reason information and a timestamp, along with the data indicating the frame.

[0085] In the following, the data output by the frame selection unit 101c in response to the previous external input prompt will be referred to as the previously output data, and the data output by the frame selection unit 101c in response to the current external input prompt will be referred to as the currently output data.

[0086] In Embodiment 3, the response database 102 stores the data output by the frame selection unit 101c. In this case, it is desirable that the response database 102 stores the data output by the frame selection unit 101c in association with data indicating the corresponding external input prompt.

[0087] Furthermore, in Embodiment 3, the response unit 103 outputs data indicating the response to the external input prompt based on the data output by the frame selection unit 101c. At this time, the response unit 103 obtains the data output by the frame selection unit 101c by reading data from the response database 102 and performs the above processing.

[0088] Furthermore, in addition to the above functions, the response unit 103 has the function of outputting data indicating the response to the current external input prompt. Here, if the frame selection unit 101c determines that reprocessing is necessary, the response unit 103 outputs data indicating the response to the current external input prompt based on the data output by the frame selection unit 101c. On the other hand, if the frame selection unit 101c determines that reprocessing is not necessary, the response unit 103 outputs data indicating the response to the current external input prompt based on the data output by the frame selection unit 101c in the previous instance.

[0089] Next, an example of the operation of the information processing device 1 according to Embodiment 3 shown in Figure 12 will be described with reference to Figure 13. In the information processing device 1 according to Embodiment 3 shown in Figure 12, for example, as shown in Figure 13, first the information processing device 1 performs the operations of steps ST101 to ST104 shown in Figure 2 (step ST301). Note that the "frame selection unit 101" in steps ST101 to ST104 is replaced with the "frame selection unit 101c".

[0090] Next, the frame selection unit 101c determines whether a prompt has been entered for the answer generated by the answer unit 103 (step ST302).

[0091] In step ST301, if the frame selection unit 101c determines that no prompt has been entered for the answer generated by the answer unit 103, the sequence ends.

[0092] On the other hand, in step ST302, if the frame selection unit 101c determines that a prompt has been entered for the answer generated by the answer unit 103, it determines whether reprocessing by the frame selection unit 101c is necessary based on the answer, the previous prompt, and the video (step ST303).

[0093] In step ST303, if the frame selection unit 101c determines that reprocessing is not necessary, the response unit 103 outputs data indicating the response to the current external input prompt based on the previously selected frame and previous additional information by the frame selection unit 101c (step ST304).

[0094] For example, after the response by the response unit 103 as shown in Embodiment 1, suppose a prompt such as "Please tell me specifically about the movement of the person in white" is input to the frame selection unit 101c. This prompt is related to the previous prompt. In this case, for example, the frame selection unit 101c inputs a prompt to the MLLM such as "Furthermore, you need to respond to the first user-added external input prompt. Do you need to re-select the frames to respond?" Then, for example, in response to the above prompt, the MLLM responds with "No, the current information is sufficient to respond," and the response unit 103 generates a response to the current external input prompt using the previous result.

[0095] On the other hand, in step ST303, if the frame selection unit 101c determines that reprocessing is necessary, it uses MLLM to select a frame from the video to be used as the answer to the current prompt, based on the current prompt, and generates additional information (step ST305). Next, the answer unit 103 outputs data indicating the answer to the current external input prompt based on the frame selected by the frame selection unit 101c and the current additional information (step ST304).

[0096] For example, suppose that after the response by the response unit 103 shown in Embodiment 1, a prompt such as "Are there any open shops in the shopping district?" is input to the frame selection unit 101c. This prompt is unrelated to the previous prompt. In this case, for example, the frame selection unit 101c inputs a prompt to the MLLM such as "Furthermore, you need to respond to a second user-added external input prompt. Do you need to re-select the frames to respond?" Then, for example, in response to the above prompt, the MLLM responds, "Yes. The information required for the response is different, so you need to re-select the frames," and the frame selection unit 101c executes the process again, and the response unit 103 generates a response to the current external input prompt using the results of the previous response.

[0097] In the above, we showed the case where the frame selection unit 101 in the information processing device 1 according to Embodiment 1 is changed to the frame selection unit 101c. However, we are not limited to this, and the frame selection unit 101b in the information processing device 1 according to Embodiment 2 may also be changed to the frame selection unit 101c, and the same effects as above can be obtained.

[0098] As described above, according to this embodiment 3, when a prompt for an answer obtained by the answer unit 103 is input to the frame selection unit 101c, and the answer unit 101c determines that reprocessing is necessary based on the answer, the previous prompt used to generate the answer, and the video, it outputs data indicating the selected frame, which is the frame among the frames included in the video used to answer the current prompt, based on the current prompt. The answer unit 103 outputs data indicating the answer to the current prompt based on the data output by the frame selection unit 101c if the frame selection unit 101c determines that reprocessing is necessary, and outputs data indicating the answer to the current prompt based on the data output by the frame selection unit 101c if the frame selection unit 101c determines that reprocessing is not necessary. As a result, the information processing device 1 according to embodiment 3 enables multi-turn communication.

[0099] Embodiment 4. The information processing device 1 according to Embodiments 1-3 shows the case where the input prompt consists only of a question. However, the above prompt is not limited to this, and may include constraint information in addition to a question. Examples of constraints include a physical model, building structure, equipment constraints, or vehicle trajectory. In this case, for example, the frame selection unit 101 and the answer unit 103 in Embodiment 1 process the information while also considering the constraints included in the prompt.

[0100] For example, in Figure 14A, a video of a road is input to the frame selection unit 101. Then, a prompt is input to the frame selection unit 101 that includes a question such as, "Detect any dangerous driving in the road video and provide details," as well as constraints such as, "The direction of vehicle movement on this road is as shown in Figure 14B. The area outside the road is a sidewalk, so vehicle entry is prohibited." This allows the information processing device 1 in Embodiment 1, for example, to provide an answer that takes the constraints into account. Note that the above example shows the constraints included in the prompt as text and images, but the constraints are not limited to this and may be expressed as text or images only.

[0101] In the above, we showed the case in the information processing device 1 according to Embodiment 1 where the prompt includes constraint information. However, the information processing device 1 according to Embodiments 2 and 3 may also include constraint information in the prompt, and the same effects as above can be obtained.

[0102] As described above, according to this embodiment 4, the prompt includes constraint information. As a result, the information processing device 1 according to embodiment 4 can generate a more accurate response because the prompt includes constraint information.

[0103] Embodiment 5. Embodiments 1-4 described the case where the information processing device 1 processes based on video. On the other hand, video may contain audio signals, and if the information processing device 1 is configured to handle audio signals, it is also possible to process based on both video and audio signals.

[0104] Figure 15 shows an example of the configuration of the information processing device 1 according to Embodiment 5. In the information processing device 1 according to Embodiment 5 shown in Figure 15, the frame selection unit 101 is changed to a frame selection unit 101d compared to the information processing device 1 according to Embodiment 1 shown in Figure 1. Other configuration examples in the information processing device 1 according to Embodiment 5 shown in Figure 15 are the same as the configuration example of the information processing device 1 according to Embodiment 1 shown in Figure 1, and only the different parts will be described using the same reference numerals.

[0105] The video input to the frame selection unit 101d includes an audio signal. The frame selection unit 101d then performs frame selection processing on the frame selection unit 101 in Embodiment 1, taking into account the audio signal included in the video. The MLLM used by the frame selection unit 101d is a model that can also accept audio signals as input.

[0106] For example, in Figure 16, a video showing a specific work scene is input to the frame selection unit 101d. This video includes an audio signal. Then, a prompt such as, "This is a video of a craftsman at work. Please tell me the important points of this work, along with the corresponding frames," is input to the frame selection unit 101d.

[0107] In this case, for example, in response to the above external input prompt, the MLLM of the frame selection unit 101d provides the following summary of the video: "A person is working and occasionally making loud noises by hitting something." Also, as shown in Figure 17, for example, the MLLM states that the first and second frames of the video are not relevant to the question, and gives the reason that "Based on the image and sound, it is thought that there is no significant movement of the subject." The MLLM also states that the third frame of the video is relevant to the question, and gives the reason that "There is no sound, but the subject is moving." The MLLM also states that the fourth frame of the video is relevant to the question, and gives the reason that "There is a loud clanging sound." Finally, the MLLM states that the fifth frame of the video is not relevant to the question, and gives the reason that "Based on the image and sound, it is thought that there is no significant movement of the subject." Furthermore, MLLM states that the sixth frame of the video is relevant to the question, and gives the reason that "there is no sound, but the subject is moving." MLLM also states that the seventh frame of the video is relevant to the question, and gives the reason that "there is a loud clanging sound." Finally, MLLM states that the eighth frame of the video is not relevant to the question, and gives the reason that "from the image and sound, it can be concluded that the subject is not moving significantly." Note that Figure 17 is an example illustrating MLLM's responses; in reality, MLLM's responses may be composed in natural language.

[0108] In the above, the case in which the frame selection unit 101 in the information processing device 1 according to Embodiment 1 is changed to the frame selection unit 101d was shown. However, the case is not limited to this, and the frame selection unit 101b or frame selection unit 101c in the information processing device 1 according to Embodiment 2-4 may be changed to the frame selection unit 101d, and the same effects as above can be obtained. When the frame selection unit 101d selects a Vision system model using MLLM or LLM, this Vision system model is an acoustic signal processing system model.

[0109] As described above, according to this embodiment 5, the video includes an audio signal, and the frame selection unit 101d performs frame selection processing based on the audio signal included in the video. As a result, the information processing device 1 according to embodiment 5 can generate answers using the audio signal as well, and further improvements in the accuracy of answer generation can be expected.

[0110] Finally, with reference to Figure 18, an example of the hardware configuration of the information processing device 1 according to Embodiment 1-5 will be described. Note that the following description will focus on the hardware configuration example of the information processing device 1 according to Embodiment 1, but the same applies to the hardware configuration example of the information processing device 1 according to Embodiment 2-5. The functions of the frame selection unit 101 and the answer unit 103 in the information processing device 1 are realized by the processing circuit 51. The processing circuit 51 may be dedicated hardware, as shown in Figure 18A, or it may be a CPU (Central Processing Unit, central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)) 52 that executes the program stored in the memory 53, as shown in Figure 18B.

[0111] If the processing circuit 51 is dedicated hardware, it may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The processing circuit 51 may implement the functions of each part of the frame selection unit 101 and the answer unit 103 individually, or it may implement the functions of each part together in the processing circuit 51.

[0112] When the processing circuit 51 is a CPU 52, the functions of the frame selection unit 101 and the response unit 103 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 53. The processing circuit 51 realizes the functions of each unit by reading and executing the programs stored in the memory 53. In other words, the information processing device 1 is equipped with a memory 53 for storing programs that, when executed by the processing circuit 51, result in the execution of each step shown in Figure 2, for example. These programs can also be said to cause the computer to execute the procedures and methods of the frame selection unit 101 and the response unit 103. Here, the memory 53 may be, for example, a non-volatile or volatile semiconductor memory such as RAM, ROM, flash memory, EPROM, EEPROM, magnetic disk, flexible disk, optical disk, compact disk, minidisc, or DVD.

[0113] Furthermore, the functions of the frame selection unit 101 and the response unit 103 may be partially implemented by dedicated hardware and partially by software or firmware. For example, the frame selection unit 101 can be implemented by a processing circuit 51 as dedicated hardware, and the response unit 103 can be implemented by the processing circuit 51 reading and executing a program stored in memory 53.

[0114] In this way, the processing circuit 51 can realize each of the above-mentioned functions through hardware, software, firmware, or a combination thereof.

[0115] Furthermore, it is possible to freely combine the embodiments, modify any component of each embodiment, or omit any component in each embodiment.

[0116] The information processing device described herein can reduce the amount of memory used to generate responses to prompts compared to conventional devices, making it suitable for use in information processing devices and the like.

[0117] 1 Information processing device, 51 Processing circuit, 52 CPU, 53 Memory, 101, 101b, 101c, 101d Frame selection unit, 102 Answer database, 103 Answer unit.

Claims

1. An information processing device comprising: a frame selection unit that outputs data indicating a selected frame from among the frames contained in the video that is used to answer the prompt, based on an input video and a prompt; and a response unit that outputs data indicating an answer to the prompt, based on the data output by the frame selection unit.

2. The information processing apparatus according to claim 1, characterized in that the frame selection unit also outputs data indicating reason information that shows at least one of the reasons for selecting the selected frame and the reasons for not selecting any frame other than the selected frame.

3. The information processing apparatus according to claim 2, characterized in that the reason information includes information indicating at least one of the following: a summary of the video and the position of each frame contained in the video.

4. The information processing apparatus according to any one of claims 1 to 3, characterized in that the frame selection unit also outputs data indicating the timestamp of the video.

5. The information processing apparatus according to any one of claims 1 to 4, characterized in that the frame selection unit directly performs frame selection using MLLM.

6. The information processing apparatus according to any one of claims 1 to 4, characterized in that the frame selection unit selects a frame by selecting a Vision-type model using MLLM or LLM.

7. The information processing apparatus according to any one of claims 1 to 6, characterized in that the frame selection unit divides the video into a specified number of frames and performs a frame selection process for each divided frame which is a group of divided frames.

8. The information processing apparatus according to any one of claims 1 to 7, characterized in that the frame selection unit repeats the frame selection process so as to reduce the number of frames when the number of selected frames is greater than the number of specified frames.

9. The information processing apparatus according to claim 8, characterized in that, if the number of selected frames exceeds a specified number of frames even after the frame selection unit has repeated selection a predetermined number of times, the number of selected frames is reduced from the selected frames in a predetermined manner so that the number of selected frames is less than or equal to the specified number of frames.

10. The information processing apparatus according to any one of claims 1 to 9, wherein the video is a plurality of videos, and the frame selection unit performs frame selection processing for each of the videos.

11. The information processing apparatus according to any one of claims 1 to 10, characterized in that the frame selection unit, when it receives a prompt for an answer obtained by the answer unit and determines that reprocessing is necessary based on the answer, the previous prompt used to generate the answer, and the video, outputs data indicating a selected frame, which is one of the frames included in the video used to answer the current prompt, based on the current prompt; the answer unit, when it determines that reprocessing is necessary by the frame selection unit, outputs data indicating an answer to the current prompt based on the data output by the frame selection unit; and when it determines that reprocessing is not necessary by the frame selection unit, it outputs data indicating an answer to the current prompt based on the data output by the frame selection unit in the previous instance.

12. The information processing apparatus according to any one of claims 1 to 11, characterized in that the prompt includes constraint information.

13. The information processing apparatus according to any one of claims 1 to 12, wherein the video includes an audio signal, and the frame selection unit performs frame selection processing based on the audio signal included in the video.

14. An information processing method comprising: a frame selection unit outputting data indicating a selected frame, which is used to answer the prompt, from among the frames included in the video, based on the input video and the prompt; and a response unit outputting data indicating the answer to the prompt, based on the data output by the frame selection unit.

Images