Information processing system, information processing method, and information processing program
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- NEC CORP
- Filing Date
- 2025-12-26
- Publication Date
- 2026-07-30
Smart Images
Figure JP2025045912_30072026_PF_FP_ABST
Abstract
Description
Information Processing System, Information Processing Method, and Information Processing Program
[0001] The present disclosure relates to an information processing system, an information processing method, and an information processing program.
[0002] Techniques have been developed to extract frame images containing a scene desired by a user, videos including the frame images, etc. from videos captured by imaging means (for example, Patent Document 1).
[0003] Japanese Unexamined Patent Application Publication No. 2024-166908
[0004] When extracting a frame image that explicitly shows the basis for an answer to a user's question sentence, for example, a portion where a person or object specified in the question sentence is detected is extracted from the video captured by the imaging means. In the case of a complex question sentence such as when a person or object is unclear or their relationship is ambiguous, there is a possibility that a frame image that explicitly shows the basis for an answer to the question sentence may not be extracted.
[0005] The present disclosure aims to provide an information processing system, an information processing method, and an information processing program for extracting a frame image that explicitly shows the basis for an answer to a question sentence.
[0006] The information processing system according to the present disclosure includes a question interpretation unit that specifies at least event identification information for identifying an event among the information included in the input question sentence, a question decomposition unit that generates one or more texts indicating the content of the event based on the event identification information, a question graph data generation unit that generates question graph data composed of a combination of a plurality of nodes indicating a person or object and one or more edges indicating the relationship between the plurality of nodes from the texts, a video graph data generation unit that generates video graph data composed of a combination of a plurality of nodes indicating a person or object and one or more edges indicating the relationship between the plurality of nodes from the input one or more videos, and a collation unit that collates the question graph data and the video graph data and extracts one or more frame images indicating the answer content to the question sentence from the video based on the collation result.
[0007] The information processing method relating to this disclosure includes the steps of: identifying event identification information that identifies an event from the information contained in the input question sentence; generating one or more texts indicating the content of the event based on the event identification information; generating question graph data from the texts, which consists of a plurality of nodes indicating a person or object and one or more edges indicating the relationships between the plurality of nodes; generating video graph data from one or more input video images, which consists of a plurality of nodes indicating a person or object and one or more edges indicating the relationships between the plurality of nodes; and comparing the question graph data with the video graph data and extracting one or more frame images from the video that indicate the content of the answer to the question sentence based on the comparison result.
[0008] The information processing program relating to this disclosure causes a computer to perform the following steps: identify event identification information that identifies an event from the information contained in the input question; generate one or more texts indicating the content of the event based on the event identification information; generate question graph data composed of a combination of a plurality of nodes indicating a person or object and one or more edges indicating the relationships between the plurality of nodes from the texts; generate video graph data composed of a combination of a plurality of nodes indicating a person or object and one or more edges indicating the relationships between the plurality of nodes from one or more input videos; and compare the question graph data with the video graph data and extract one or more frame images from the videos indicating the content of the answer to the question based on the comparison result.
[0009] This disclosure provides an information processing system, information processing method, and information processing program for extracting frame images that explicitly show the basis for the answer to a question.
[0010] This is a diagram showing an example of the configuration of the information processing system related to this disclosure. This is a diagram showing an example of graph data. This is a diagram showing another example of the configuration of the information processing system related to this disclosure. This is a diagram showing an example of the configuration of the video processing unit. This is a diagram showing another example of video graph data. This is a diagram showing another example of question graph data. This is a diagram showing an example of the data structure of video graph data associated with a frame ID. This is a diagram showing an example of the data structure of question graph data associated with a text ID. This is a flowchart showing an example of the information processing method related to this disclosure. This is a diagram showing another example of the configuration of the information processing system related to this disclosure. This is a flowchart showing another example of the information processing method related to this disclosure. This is a diagram showing another example of the configuration of the video processing unit. This is a diagram showing another example of the configuration of the information processing system. This is a diagram showing an example of the matching result output by the matching unit. This is a diagram showing another example of the configuration of the information processing system. This is a diagram showing an example of frame images before and after marking. This is a block diagram showing an example of the hardware configuration that realizes the information processing of the information processing system related to this disclosure.
[0011] (Embodiment 1) <Configuration of the Information Processing System> The configuration example of the information processing system 1 will be described below with reference to Figure 1. Figure 1 is a diagram showing an example of the configuration of the information processing system according to this disclosure.
[0012] The information processing system 1 relating to this disclosure takes one or more video images and a question as input. The information processing system 1 then extracts one or more frame images from the input video that represent the content of the answers to the input question. As shown in Figure 1, the information processing system 1 comprises a video graph data generation unit 201, a question interpretation unit 202, a question decomposition unit 203, a question graph data generation unit 204, and a matching unit 205.
[0013] The video graph data generation unit 201 generates graph data from the input video that shows the relationships between multiple nodes representing people or objects.
[0014] Here, we will explain graph data in detail. Figure 2 shows an example of graph data. The upper part of Figure 2 is graph data generated by the video graph data generation unit 201. The lower part of Figure 2 is graph data generated by the question graph data generation unit 204, which will be described later. Graph data is a graph structure that shows the relationships between nodes, composed of a plurality of nodes representing people or objects and one or more edges that show the relationships between the plurality of nodes. In this disclosure, nodes P1 to Pn (where n is an integer of 2 or more) represent "people", nodes O1 to On (where n is an integer of 2 or more) represent "objects", and edges E1 to En (where n is an integer of 2 or more) represent the relationships between nodes.
[0015] For example, in the upper part of Figure 2, node P1 is "Person 1", node P2 is "Person 2", node O1 is "Bag 1", node O2 is "Bag 2", edge E1 is "Hold", and edge E2 is "Talk". In this case, the graph data shown in the upper part of Figure 2 shows the relationships "Person 1 holds Bag 1", "Person 2 holds Bag 2", and "Person 1 talks with Person 2". Note that the relationships shown here are just examples. For example, edges E1 and E2 may not be limited to simple relationships such as "Hold", but may include complex relationships such as "Hold in the right hand". Furthermore, nodes P1 to Pn and O1 to On may be assigned attributes. For example, if node O1 "Bag 1" has the attribute "Red", node O1 will represent "Red Bag 1".
[0016] The video graph data generation unit 201 generates the graph data described above from the input video. In the following, the graph data generated by the video graph data generation unit 201 will be referred to as video graph data VG, and the graph data generated by the question graph data generation unit 204, which will be described later, will be referred to as question graph data QG.
[0017] The question interpretation unit 202 identifies at least event identification information from the information contained in the input question to identify an event. For example, if the question is "Was there a bag theft incident at the station?", the question interpretation unit 202 identifies "bag theft incident" as the event identification information.
[0018] The question decomposition unit 203 generates one or more texts indicating the content of an event based on the event identification information identified by the question interpretation unit 202. For example, if the event identification information is "theft," the question decomposition unit 203 generates the texts "Person A holds bag A," "Person A puts bag A down," and "Person B holds bag A." Even if a complex question is input, such as when people or objects are unclear or their relationships are ambiguous, the question interpretation unit 202 and the question decomposition unit 203 can decompose the question into one or more simple action-indicating sentences.
[0019] The question graph data generation unit 204 generates question graph data QG for each text generated by the question decomposition unit 203. The question graph data QG is preferably represented by two nodes and one edge, as shown in the lower diagram of Figure 2, but is not limited to this.
[0020] The matching unit 205 matches the question graph data QG with the video graph data VG. More specifically, first, the matching unit 205 compares the question graph data QG with the video graph data VG to confirm whether the question graph data QG is included in the video graph data VG. Next, the matching unit 205 extracts information as a matching result indicating the time point in the input video in which the question graph data QG is included in the video graph data VG. Based on the matching result, the matching unit 205 extracts one or more frame images from the input video that show the content of the answer to the input question. That is, since the frame image is an image that contains the question graph data QG generated from the question, it is an image that explicitly shows the basis for the answer to the question. If the video graph data VG does not contain the question graph data QG, the matching unit 205 may output the matching result as, for example, "Not applicable".
[0021] As described above, the information processing system 1, with its question interpretation unit 202 and question decomposition unit 203, decomposes the input question sentence into one or more sentences, and the question graph data generation unit 204 generates question graph data QG based on the decomposed sentences. The information processing system 1 then compares this question graph data QG with the video graph data VG generated from the input video. Based on the comparison result, the information processing system 1 extracts a frame image from the input video. That is, since the frame image contains the question graph data QG generated from the question sentence, it is an image that explicitly shows the basis for the answer to the question sentence. Therefore, according to Embodiment 1, the information processing system 1 can extract a frame image that explicitly shows the basis for the answer to the question sentence.
[0022] For example, the information processing system 1 may include a question interpretation unit 202, a question decomposition unit 203, and a question graph data generation unit 204, while the video graph data generation unit 201 and the matching unit 205 may be provided on other servers or devices. In this case, the information processing system 1 executes the process of generating question graph data QG from the input question sentence and transmits the generated question graph data QV to the other server or device. That is, the information processing system 1 may be configured to execute the process of generating question graph data QG from the input question sentence, but not to execute the video graph data generation process, matching process, or extraction process for video graph data VG.
[0023] (Embodiment 2) <Configuration of Information Processing System> Another example of the information processing system will be described below with reference to Figure 3. Figure 3 is a diagram showing another example of the configuration of the information processing system according to the present disclosure. As described in Embodiment 1, the information processing system 1 according to the present disclosure takes one or more video images and a question as input. The information processing system 1 then extracts one or more frame images from the input video that show the content of the answer to the input question. The information processing system 1 outputs the extracted one or more frame images. The information processing system 1 according to the present disclosure can be used, for example, for investigations in the event of an emergency in facility security, reporting of dangerous situations that occur in the workplace, or recording childcare at home. As shown in Figure 3, the information processing system 1 comprises an input unit 10, a processing unit 20, and an output unit 30.
[0024] The input unit 10 comprises a video input unit 100 and a question input unit 101. The video input unit 100 accepts one or more video inputs. If there are multiple video inputs, each of the multiple videos may be captured by a different imaging device, or it may be a video captured multiple times by a single imaging device. The video input unit 100 may receive video directly from the imaging device, for example, or it may receive video stored on a server.
[0025] The question input unit 101 accepts the input of a question. The input questions are complex, such as when people or objects are unclear or when their relationships are ambiguous. Examples of complex questions include, for example, "Was there a bag theft incident at the station?" or "Please find the scene of the bicycle theft incident." Of course, the question input unit 101 may also accept the input of simple questions.
[0026] In the following explanation, we will use the example of a case where the question entered into the question input unit 101 is "Have there been any bag theft incidents in the station?", and one or more frame images representing the answer to this question are extracted from three video feeds input to the video input unit 100 and output. In this case, the three video feeds input to the video input unit 100 are, for example, videos acquired from three surveillance cameras installed in the station.
[0027] The processing unit 20 includes a video processing unit 200, a video graph data generation unit 201, a question interpretation unit 202, a question decomposition unit 203, a question graph data generation unit 204, and a matching unit 205.
[0028] The video processing unit 200 performs video processing on the video input to the video input unit 100. Figure 4 shows an example of the configuration of the video processing unit. As shown in Figure 4, the video processing unit 200 includes a person / object detection unit 211, a relationship detection unit 212, a tracking unit 213, and an identification unit 214.
[0029] The person / object detection unit 211 detects people and objects from the video. The person / object detection unit 211 then manages the detected people and objects as nodes, which will be described later. A node represents the content of the subject or object in a sentence. Each detected node is assigned a node ID (Identification) as identification information.
[0030] The relationship detection unit 212 detects the relationships between nodes detected by the person / object detection unit 211 from the video. In this disclosure, relationships refer to actions such as "holding" or "talking," that is, the content of predicates in text. The relationship detection unit 212 manages the detected relationships as edges, which will be described later. Each detected edge is assigned an edge ID as identification information.
[0031] The tracking unit 213 tracks the time-series changes of nodes detected by the person / object detection unit 211 for each of the multiple video streams. More specifically, the tracking unit 213 performs a tracking process for each of the multiple video streams, tracking the time-series changes of nodes within a single video stream. The tracking unit 213 associates tracking information, which indicates the result of the tracking process for each node, with the node ID of each node.
[0032] The identification unit 214 performs identification processing across multiple video feeds. More specifically, the identification unit 214 first identifies the same node that appears in multiple video feeds. Then, the identification unit 214 performs identification processing across the multiple video feeds to track the time-series changes of the identified node. The identification unit 214 associates identification information, which indicates the result of the identification processing for each node, with the node ID of each node.
[0033] The node ID, edge ID, tracking information, and identification information obtained by the video processing unit 200 are managed, for example, in a database (not shown).
[0034] The video graph data generation unit 201 shown in Figure 3 generates video graph data based on the results of video processing performed by the video processing unit 200. The video graph data is a graph data composed of a combination of multiple nodes representing people or objects and one or more edges representing the relationships between the multiple nodes.
[0035] Figure 5 shows another example of video graph data. In Figure 5, the video graph data VG arranged vertically are graph data generated from different videos. That is, the video graph data VG1_1, VG2_1, and VG3_1 shown in Figure 5 are graph data generated from different videos. Also, in Figure 5, the video graph data VG arranged horizontally are graph data generated from the same video. That is, the video graph data VG1_1, VG1_2, and VG1_3 shown in Figure 5 are graph data generated from the same video. In Figure 5, the horizontal direction is the time direction, and the video graph data VG are arranged horizontally in chronological order. In this way, the video graph data generation unit 201 generates multiple video graph data VG in chronological order for each video.
[0036] As described in Embodiment 1, in this disclosure, nodes P1 to Pn (where n is an integer of 2 or more) represent "people", nodes O1 to On (where n is an integer of 2 or more) represent "objects", and edges E1 to En (where n is an integer of 2 or more) represent the relationships between nodes.
[0037] For example, in Figure 5, regarding nodes, node P1 is "Person 1", node P2 is "Person 2", node P3 is "Person 3", node O1 is "Bag 1", and node O2 is "Bag 2". Regarding edges, edge E1 is "Hold", edge E2 is "Talk", and edge E3 is "Place". In this case, the video graph data VG1_1 shows the relationships "Person 1 holds Bag 1", "Person 2 holds Bag 2", and "Person 1 talks to Person 2". The video graph data VG1_2 shows the relationship "Person 1 places Bag 1", the video graph data VG1_2 shows the relationship "Person 1 places Bag 1", and the video graph data VG1_3 shows the relationship "Person 3 holds Bag 1". Note that the relationships shown here are just examples. For example, edges E1 and E2 may not be limited to simple relationships such as "Hold", but may include more complex relationships such as "Hold in the right hand". Furthermore, nodes P1 to Pn and O1 to On may be assigned attributes. For example, if node O1 "Bag 1" has the attribute "Red", then node O1 will represent "Red Bag 1".
[0038] The video graph data generation unit 201 associates the generated video graph data VG with a frame ID. The frame ID includes, for example, video identification information indicating the identification of the video, and time information indicating the point in time in the video. The video graph data VG associated with the frame ID is managed, for example, in a database (not shown).
[0039] The question interpretation unit 202 and the question decomposition unit 203 shown in Figure 3 perform preprocessing of the question text so that the question graph data generation unit 204, which will be described later, can generate graph data of the question text.
[0040] The question interpretation unit 202 identifies at least event identification information from the information contained in the input question sentence to identify an event. For example, if the question sentence "Was there a bag theft incident at the station?" is input, the question interpretation unit 202 identifies "bag theft incident" as the event identification information. In addition to event identification information, the question interpretation unit 202 may also identify other things such as the location of the event, the person or object involved. For example, if the question sentence "Was there a bag theft incident at the station?" is input, the event identification information "bag theft incident" corresponds to the location of the event being "station" and the object involved being "bag". The question interpretation unit 202 performs identification processing using, for example, Large Language Models (LLM), natural language processing algorithms, rule-based algorithms, etc., either alone or in combination.
[0041] The question decomposition unit 203 generates one or more texts that describe the content of an event based on the event identification information identified by the question interpretation unit 202. Each text generated by the question decomposition unit 203 contains one subject, one predicate, and one object. When generating multiple texts, the question decomposition unit 203 generates the texts in chronological order according to the flow of the event. For example, if the event identification information is "theft incident," the question decomposition unit 203 generates the texts "Person A holds bag A," "Person A puts bag A down," and "Person B holds bag A." The question decomposition unit 203 executes the text generation process using, for example, LLM, natural language processing algorithms, rule-based algorithms, etc., either alone or in combination. The question decomposition unit 203 may execute the text generation process multiple times. Each generated text is assigned a text ID as identification information.
[0042] As explained above, even if a complex question is input, such as when people or objects are unclear or their relationships are ambiguous, the question interpretation unit 202 and the question decomposition unit 203 can decompose the question into one or more simple action-indicating sentences.
[0043] The question graph data generation unit 204 generates question graph data from the text generated by the question decomposition unit 203. The question graph data is graph data composed of a combination of a plurality of nodes indicating a person or an object and one or more edges indicating the relationship between the plurality of nodes. The question graph data generation unit 204 generates question graph data from the text alone or in combination, for example, using an LLM, a natural language processing algorithm, a rule-based algorithm, etc. Note that the question graph data generation unit 204 generates question graph data for each text.
[0044] FIG. 6 is a diagram showing another example of question graph data. In FIG. 6, the horizontal direction is the time direction, and the question graph data QG is arranged in chronological order in the horizontal direction. In this way, the question graph data generation unit 204 generates a plurality of question graph data QG in chronological order for each text generated by the question decomposition unit 203. In the example of FIG. 6, the question graph data QG1 indicates the text "Person A has bag A", the question graph data QG2 indicates the text "Person A places bag A", and the question graph data QG3 indicates the text "Person B has bag A".
[0045] The question graph data generation unit 204 associates the generated question graph data QG with a text ID. The question graph data QG associated with the text ID is managed, for example, in a database not shown.
[0046] Note that when generating the question graph data QG, the question graph data generation unit 204 may determine the information of the nodes and edges of the question graph data QG based on the information of the nodes and edges included in the video. For example, when the node O1 of the video is "handbag 1" and the node O1 of the question graph data QG is "bag A", the question graph data generation unit 204 may, for example, convert the node O1 of the question graph data QG to "handbag 1". In this way, by determining the nodes and edges of the question graph data QG in consideration of the nodes and edges included in the video, the matching unit 205 described later can perform the matching process with higher accuracy.
[0047] The matching unit 205 shown in FIG. 3 matches the question graph data QG and the video graph data VG. Then, based on the matching result, the matching unit 205 extracts one or more frame images indicating the answer content for the question sentence from the video. The matching unit 205 executes a matching process using an algorithm such as full search or GNN (Graph Neural Network), for example.
[0048] First, the matching unit 205 compares the question graph data QG and the video graph data VG, and checks whether the question graph data QG is included in the video graph data VG. When the question graph data QG is included in the video graph data VG, the matching unit 205 extracts the frame ID of the video graph data VG in which the question graph data QG is included as the matching result. Here, when there are multiple pieces of question graph data QG, the matching unit 205 checks whether the time series order of the question graph data QG matches the time series order of the video graph data VG in which the question graph data QG is included. When the time series orders match, the matching unit 205 extracts the frame ID of the video graph data VG in which the question graph data QG is included as the matching result. When the question graph data QG is not included in the video graph data VG or the time series orders do not match, the matching unit 205 may output, for example, the matching result as "not applicable".
[0049] For example, a case where the matching unit 205 matches the video graph data VG shown in FIG. 5 and the question graph data QG shown in FIG. 6 will be described. FIG. 7 is a diagram showing an example of the data structure of video graph data with frame IDs linked. FIG. 8 is a diagram showing an example of the data structure of question graph data with text IDs linked. Note that FIG. 7 is generated based on the video graph data shown in FIG. 5, and FIG. 8 is generated based on the question graph data QG shown in FIG. 6. Also, in FIGS. 7 and 8, the video graph data VG and the question graph data QG are expressed as "node - edge - node", and more specifically as "subject - predicate - object", such as "person 1 - has - bag 1".
[0050] From Figures 7 and 8, it can be seen that question graph data QG1 is included in at least video graph data VG1_1. Also, question graph data QG2 is included in video graph data VG1_2. Question graph data QG3 is included in video graph data VG1_3. Furthermore, in the time direction, question graph data QG2 is located after question graph data QG1, and question graph data QG3 is located after question graph data QG2. Similarly, in video graph data VG containing question graph data QG, video graph data VG1_2 is located after video graph data VG1_1, and video graph data VG1_3 is located after video graph data VG1_2. Therefore, the chronological order of video graph data VG containing question graph data QG matches the chronological order of question graph data QG. Accordingly, the matching unit 205 extracts the frame IDs of video graph data VG1_1, VG1_2, and VG1_3 as matching results.
[0051] The matching unit 205 then extracts a frame image from the video based on the extracted matching result, i.e., the frame ID. Therefore, the frame image extracted by the matching unit 205 is an image that contains the question graph data QG generated from the question text, and thus explicitly shows the basis for the answer to that question text.
[0052] The output unit 30 includes a frame image output unit 300. The frame image output unit 300 outputs the frame images extracted by the matching unit 205 to, for example, a display or server (not shown). The frame image output unit 300 may also output video for a predetermined time range based on the frame ID.
[0053] <Information Processing Method> Next, an example of an information processing method will be explained using Figure 9. Figure 9 is a flowchart showing an example of an information processing method related to this disclosure.
[0054] First, the video input unit 100 receives one or more video inputs (step S101). Following step S101, the video processing unit 200 performs video processing on the video input to the video input unit 100 (step S102). In step S102, the video processing unit 200 first detects nodes representing people or objects and edges that detect relationships between nodes. Next, the video processing unit 200 performs a tracking process to track the time-series changes of nodes within a single video, and an identification process to identify identical nodes appearing in multiple videos and track the time-series changes of those nodes.
[0055] Following step S102, the video graph data generation unit 201 generates video graph data VG based on the results of the video processing performed by the video processing unit 200 (step S103). More specifically, the video graph data generation unit 201 generates multiple video graph data VGs in chronological order, each composed of a combination of multiple nodes representing people or objects and one or more edges representing the relationships between the multiple nodes. The generated video graph data VGs are associated with frame IDs. The frame ID includes, for example, video identification information indicating the identification information of the video and time information indicating the point in time in the video.
[0056] Furthermore, the question input unit 101 accepts the input of a question (step S104). Note that step S104 may be performed before or after step S101, or simultaneously.
[0057] Following step S104, the question interpretation unit 202 identifies at least event identification information that identifies an event from the information contained in the question text entered into the question input unit 101 (step S105). In addition to event identification information, the question interpretation unit 202 may also identify, for example, the location of the event, the person or object involved, etc.
[0058] Following step S105, the question decomposition unit 203 generates one or more texts indicating the content of an event based on the event identification information identified by the question interpretation unit 202 (step S106). Each text generated by the question decomposition unit 203 contains one subject, one predicate, and one object. Each generated text is assigned a text ID as identification information.
[0059] Following step S106, the question graph data generation unit 204 generates question graph data QG from the text generated by the question decomposition unit 203 (step S107). More specifically, the question graph data generation unit 204 generates question graph data QG from the text generated by the question decomposition unit 203, which is composed of a combination of multiple nodes representing people or objects and one or more edges representing the relationships between the multiple nodes. The generated question graph data QG is associated with a text ID.
[0060] In step S106, the question graph data generation unit 204 may determine the node and edge information of the question graph data QG based on the node and edge information contained in the video. In this case, the question graph data generation unit 204 obtains the node and edge information contained in the video from the video processing unit 200, the video graph data generation unit 201, etc.
[0061] After both steps S103 and S107 are completed, the matching unit 205 matches the question graph data QG with the video graph data VG and extracts one or more frame images from the video that show the answers to the question text based on the matching result (step S108).
[0062] In step S108, the matching unit 205 first compares the question graph data QG with the video graph data VG to confirm whether the video graph data VG contains the question graph data QG. If the video graph data VG contains the question graph data QG, the matching unit 205 extracts the frame ID of the video graph data VG containing the question graph data QG as the matching result. If there are multiple question graph data QGs, the matching unit 205 checks whether the chronological order of the question graph data QGs matches the chronological order of the video graph data VGs containing the question graph data QGs. If the chronological order matches, the matching unit 205 extracts the frame ID of the video graph data VGs containing the question graph data QG as the matching result. If the video graph data VGs do not contain the question graph data QGs or the chronological order does not match, the matching unit 205 may output the matching result as, for example, "No match found".
[0063] The matching unit 205 then extracts a frame image from the video based on the extracted matching result, i.e., the frame ID. Therefore, the frame image extracted by the matching unit 205 is an image that contains the question graph data QG generated from the question text, and thus explicitly shows the basis for the answer to that question text.
[0064] Finally, the frame image output unit 300 outputs the frame images extracted by the matching unit 205 to, for example, a display or server (not shown). The frame image output unit 300 may also output video for a predetermined time range based on the frame ID.
[0065] As described above, the information processing system 1, with its question interpretation unit 202 and question decomposition unit 203, decomposes the input question sentence into one or more sentences, and the question graph data generation unit 204 generates question graph data QG based on the decomposed sentences. The information processing system 1 then compares this question graph data QG with the video graph data VG generated from the input video. Based on the comparison result, the information processing system 1 extracts a frame image from the input video. That is, since the frame image contains the question graph data QG generated from the question sentence, it is an image that explicitly shows the basis for the answer to the question sentence. Therefore, according to Embodiment 2, the information processing system 1 can extract a frame image that explicitly shows the basis for the answer to the question sentence.
[0066] For example, the information processing system 1 includes a question interpretation unit 202, a question decomposition unit 203, and a question graph data generation unit 204, while other components may be provided on other servers or devices. In this case, the information processing system 1 executes the process of generating question graph data QG from the input question sentence and transmits the generated question graph data QV to the other server or device. That is, the information processing system 1 executes the process of generating question graph data QG from the input question sentence, but may be configured not to execute the process of generating, matching, or extracting video graph data VG.
[0067] (Embodiment 3) <Configuration of the Information Processing System> Another example of the information processing system will be described below with reference to Figure 10. Figure 10 is a diagram showing another example of the configuration of the information processing system according to the present disclosure. As shown in Figure 10, the information processing system 1 according to Embodiment 3 comprises an input unit 10, a processing unit 20, and an output unit 30. The information processing system 1 according to Embodiment 3 differs from the information processing system 1 according to Embodiment 2 in that it comprises a conversion unit 206, a response text generation unit 207, and a response text output unit 301. The other configurations are the same as those of the information processing system 1 described in Embodiment 2, so redundant explanations will be omitted.
[0068] The conversion unit 206 converts graph data into natural language text based on the matching results of the matching unit 205. More specifically, the conversion unit 206 converts video graph data VG, which includes question graph data QG, into natural language text. The conversion unit 206 converts video graph data VG1_1, which includes question graph data QG1 as shown in Figures 7 and 8, into, for example, "It was detected that person 1 brought bag 1." The conversion unit 206 also converts video graph data VG1_2, which includes question graph data QG2, into, for example, "It was detected that person 1 placed bag 1," and converts video graph data VG1_3, which includes question graph data QG3, into, for example, "It was detected that person 3 was holding bag 1." The conversion unit 206 performs the above conversion process by using, for example, LLM, natural language processing algorithms, rule-based algorithms, etc., either alone or in combination.
[0069] The response generation unit 207 generates a response to a question based on the natural language text converted by the conversion unit 206 and the question entered into the question input unit 101. If the question is "Was there a bag theft incident at the station?" and the natural language text is "Person 1 was detected bringing bag 1", "Person 1 was detected putting bag 1 down", and "Person 3 was detected holding bag 1", the response generation unit 207 will generate, for example, the response "It can be confirmed that after Person 1 put bag 1 down, Person 3 was holding bag 1. This situation may be considered an act generally known as bag theft. Therefore, it is thought that a bag theft incident occurred at the station." The response generation unit 207 will execute the response generation process using, for example, LLM.
[0070] The response output unit 301 outputs the response generated by the response generation unit 207 to, for example, a display or server (not shown). The response output unit 301 may also output the natural language text converted by the conversion unit 206 along with the response.
[0071] <Information Processing Method> Next, another example of the information processing method will be explained using Figure 11. Figure 11 is a flowchart of another example of the information processing method according to this disclosure. As shown in Figure 11, the information processing method according to Embodiment 3 differs from the information processing method according to Embodiment 2 in that, after the matching process of the matching unit 205 (step S108), it includes a conversion step (step S110), a response text generation step (step S111), and a response text output step (step S112). The other steps are the same as those described in Embodiment 2, so redundant explanations will be omitted.
[0072] After the matching process of the matching unit 205 (step S108), the conversion unit 206 converts the graph data into natural language text based on the matching result of the matching unit 205 (step S110). More specifically, the conversion unit 206 converts the video graph data VG, which includes the question graph data QG, into natural language text. Note that step S110 may be performed either before or after step S109, or simultaneously, as long as it is after step S108.
[0073] Following step S110, the response text generation unit 207 generates a response text to the question text based on the natural language text converted by the conversion unit 206 and the question text entered into the question input unit 101 (step S111).
[0074] Following step S111, the response output unit 301 outputs the response generated by the response generation unit 207 to, for example, a display or server (not shown) (step S112). The response output unit 301 may also output the natural language text converted by the conversion unit 206 along with the response.
[0075] As described above, the information processing system 1 generates an answer to the input question, along with a frame image that explicitly shows the basis for the answer to the question. This answer is generated based on video graph data VG, which includes question graph data QG generated from the question. Therefore, the answer generated by the information processing system 1 is a document that explicitly shows the basis for the answer to the question. Thus, according to embodiment 3, the information processing system 1 can output a frame image and an answer that explicitly show the basis for the answer to the question.
[0076] (Embodiment 4) <Configuration of the Information Processing System> Below, another example of the information processing system will be described using Figures 10 and 12. Figure 12 is a diagram showing another example of the configuration of the video processing unit 200. As shown in Figure 10, the information processing system 1 according to Embodiment 4 comprises an input unit 10, a processing unit 20, and an output unit 30. The information processing system 1 according to Embodiment 4 differs from the information processing system 1 according to Embodiment 3 in that, as shown in Figure 12, the video processing unit 200 further comprises a video feature extraction unit 215. The other configurations are the same as those of the information processing system 1 described in Embodiments 2 and 3, so redundant explanations will be omitted.
[0077] The video feature extraction unit 215 extracts video features from the video input to the video input unit 100, including at least one of scene features and semantic features. The video feature extraction unit 215 extracts video features using, for example, a convolutional neural network or ViT (Vision Transformer). Examples of scene features include "the floor is wet" and "there is a bench." Examples of semantic features include "the person is smiling" and "the bag is red."
[0078] The video features extracted by the video feature extraction unit 215 are used by the answer text generation unit 207. That is, the answer text generation unit 207 may generate an answer text based on the natural language text converted by the conversion unit 206, the question text input to the question input unit 101, and the video features. This allows the answer text generation unit 207 to generate an answer text that includes justification considering the context of the entire video.
[0079] As described above, the information processing system 1 extracts video features from the video input unit 100, including at least one of scene features and semantic features. The information processing system 1 then generates an answer sentence based on the natural language text converted by the conversion unit 206, the question sentence input to the question input unit 101, and the video features. Thus, according to Embodiment 4, the information processing system 1 can generate an answer sentence that includes justification considering the context of the entire video.
[0080] (Embodiment 5) <Configuration of the Information Processing System> Another example of the information processing system will be described below using Figures 12 and 13. Figure 13 is a diagram showing another example of the configuration of the information processing system. As shown in Figure 13, the information processing system 1 according to Embodiment 5 comprises an input unit 10, a processing unit 20, and an output unit 30. The information processing system 1 according to Embodiment 5 differs from the information processing system 1 according to Embodiment 4 in that it further comprises a filtering unit 208, as shown in Figure 13. The other configurations are the same as those of the information processing system 1 described in Embodiments 2 to 4, so redundant explanations will be omitted.
[0081] The filtering unit 208 filters the frame images extracted by the matching unit 205. More specifically, the filtering unit 208 first compares the question text input to the question input unit 101 with the video features extracted by the video feature extraction unit 215 shown in Figure 12 and the matching results of the matching unit 205. Based on this comparison, the filtering unit 208 determines the degree of relevance between the question text and the matching results of the matching unit 205. From the matching results extracted by the matching unit 205, the filtering unit 208 extracts matching results that have a degree of relevance to the question text that is equal to or greater than a predetermined value. The predetermined value used as a standard may be set by the user as appropriate. Then, based on the matching results that have a degree of relevance to the question text that is equal to or greater than the predetermined value, the filtering unit 208 extracts frame images from the video.
[0082] Figure 14 shows an example of a matching result extracted by the matching unit. Both the video graph data VG1_3 and video graph data VG4_3 shown in Figure 14 are video graph data VG that include the question graph data Q3. However, the node related to the node "Bag 1" and the edge "Hold" is "Person 3" in video graph data VG1_3 and "Person 4" in video graph data VG4_3. In this case, it is possible that two people, Person 3 and Person 4, stole Bag 1 which Person 1 was holding. Now, suppose the filtering unit 208 identifies, for example, that Person 1 and Person 4 were smiling and talking, from the video features extracted by the video feature extraction unit 215 shown in Figure 12. Based on these video features and the question, the possibility that Person 4 stole the bag becomes low, so the filtering unit 208 extracts a frame image based on the matching result excluding the video graph data VG4_3 that includes "Person 4 - Hold - Bag 1". In other words, the filtering unit 208 extracts frame images from the video based on the frame IDs of the video graph data VG1_1, VG1_2, and VG1_3 shown in Figure 14.
[0083] The frame image output unit 300 outputs the frame images extracted by the filtering unit 208 to, for example, a display or server (not shown).
[0084] The conversion unit 206 converts the graph data into natural language text based on the matching results extracted by the filtering unit 208. The answer text generation unit 207 then generates an answer text to the question text based on the natural language text converted by the conversion unit 206 and the question text input to the question input unit 101. The answer text output unit 301 outputs the answer text generated by the answer text generation unit 207 to, for example, a display or server (not shown).
[0085] As described above, the information processing system 1 compares the input question sentence, the video features, and the matching results of the matching unit 205. The information processing system 1 then extracts matching results from the matching results extracted by the matching unit 205 that have a degree of relevance to the question sentence that is equal to or greater than a predetermined value. Based on the matching results that have a degree of relevance to the question sentence that is equal to or greater than a predetermined value, the information processing system 1 extracts frame images from the video. Thus, according to Embodiment 5, the information processing system 1 can generate frame images and answer sentences that include justification that takes into account the relationship between the overall context of the video and the question sentence.
[0086] (Embodiment 6) <Configuration of the Information Processing System> Another example of the information processing system will be described below with reference to Figure 15. Figure 15 is a diagram showing another example of the configuration of the information processing system. As shown in Figure 15, the information processing system 1 according to Embodiment 5 comprises an input unit 10, a processing unit 20, and an output unit 30. The information processing system 1 according to Embodiment 6 differs from the information processing system 1 according to Embodiment 5 in that it further comprises a marking unit 209, as shown in Figure 15. The other configurations are the same as those of the information processing system 1 described in Embodiments 2 to 5, so redundant explanations will be omitted.
[0087] The marking unit 209 marks the nodes included in the question graph data QG corresponding to the frame image extracted by the filtering unit 208.
[0088] First, the marking unit 209 obtains the frame image extracted by the filtering unit 208 from the filtering unit 208. The marking unit 209 also obtains one or more pieces of information from the conversion unit 206 that indicate a person or object to answer the question, which are contained in the natural language converted by the conversion unit 206. Here, the information indicating a person or object obtained from the conversion unit 206 is a node included in the question graph data QG corresponding to the frame image obtained from the filtering unit 208.
[0089] Then, the marking unit 209 performs marking processing on the frame image acquired from the filtering unit 208, using, for example, object detection, object tracking, or an object identification model, based on the information indicating a person or object acquired from the conversion unit 206.
[0090] Figure 16 shows an example of frame images before and after marking. The upper part of Figure 16 shows the frame image before marking, and the lower part of Figure 16 shows the frame image after marking. Figure 16 is a frame image extracted based on the frame ID of the video graph data VG1_1 shown in Figure 14, for example. Therefore, node P1 in the frame image shown in Figure 16 is "Person 1", and node O1 is "Bag 1". In addition, the video graph data VG1_1 contains the question graph data QG1, and nodes P1 and O1 are nodes that are included in both the video graph data VG1_1 and the question graph data QG1.
[0091] Therefore, the marking unit 209 marks nodes P1 and O1 on the frame image in the upper diagram of Figure 16, as shown in the lower diagram of Figure 16. To improve identifiability, the marking unit 209 may use different colors for the marking M1 surrounding node P1 and the marking M2 surrounding node O1. The frame image that has been marked by the marking unit 209 is output by the frame image output unit 300 to, for example, a display or server (not shown).
[0092] The answer generation unit 207 generates an answer to a question based on the natural language text converted by the conversion unit 206, the question input to the question input unit 101, and the result of the marking process by the marking unit 209. That is, the answer generation unit 207 generates an answer that includes information indicating the display mode of the marking. For example, when referring to the lower figure in Figure 16, the answer generation unit 207 generates the answer "Person 1 in the yellow mark is holding bag 1 in the red mark, ...". The answer generated by the answer generation unit 207 is output by the answer output unit 301 to, for example, a display or server (not shown).
[0093] As described above, the information processing system 1 marks the nodes included in the question graph data QG corresponding to the frame image extracted by the filtering unit 208. Therefore, according to embodiment 6, the information processing system 1 can output a frame image in which the basis for the answer is clearly shown. Furthermore, by generating the answer text including information indicating the display method of the marking, the information processing system 1 can output an answer text in which the correspondence with the frame image is easy to understand.
[0094] <Hardware Configuration for Realizing the Information Processing Functions of the Information Processing System> Some or all of the information processing realized by Information Processing System 1 can be realized by a general-purpose computer system. This will be briefly explained below using Figure 17.
[0095] Figure 17 is a block diagram showing an example of a hardware configuration for realizing information processing in an information processing system according to this disclosure. The computer 400 includes, for example, a CPU (Central Processing Unit) 401 which is a control device, RAM (Random Access Memory) 402, and ROM (Read Only Memory) 403. The computer 400 further includes an IF (Interface) 404 which is an interface to the outside world, and an HDD (Hard Disk Drive) 405 which is an example of a non-volatile storage device. Furthermore, the computer 400 may also include other configurations not shown, such as input devices such as a keyboard and mouse, and display devices such as a display.
[0096] HDD 405 stores an OS (Operating System) and a control program 406, which are not shown in the diagram. The control program 406 is a computer program (information processing program) in which the information processing of the information processing system 1 is implemented.
[0097] The CPU 401 controls various processes in the computer 400, including access to RAM 402, ROM 403, IF 404, and HDD 405. The computer 400 reads and executes the OS and control program 406 stored in HDD 405 by the CPU 401. In this way, the computer 400 realizes the information processing of the information processing system 1.
[0098] The program described above includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more of the functions described in this disclosure. The program may be stored in a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include RAM, ROM, flash memory, SSD (Solid-State Drive), or other memory technologies, CD-ROM, DVD (Digital Versatile Disc), Blu-ray® disc, or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include, a temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically, or otherwise propagating signals.
[0099] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.
[0100] Each drawing is merely illustrative to illustrate one or more embodiments. Each drawing may be associated with one or more other embodiments, rather than being associated with only one specific embodiment. As those skilled in the art will understand, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings, for example, to create embodiments not explicitly shown or described. Not all features or steps shown in any one drawing to illustrate an exemplary embodiment are necessarily required, and some features or steps may be omitted. The order of steps described in any of the drawings may be changed as appropriate.
[0101] Some or all of the above embodiments may also be described as follows, but are not limited to the following:
[0102] (Note 1) An information processing system that takes one or more video images and a question as input, comprising: a question interpretation unit that identifies at least event identification information that identifies an event from the information contained in the question; a question decomposition unit that generates one or more texts indicating the content of the event based on the event identification information; a question graph data generation unit that generates question graph data composed of a combination of a plurality of nodes indicating a person or object and one or more edges indicating the relationships between the plurality of nodes from the text; a video graph data generation unit that generates video graph data composed of a combination of a plurality of nodes indicating a person or object and one or more edges indicating the relationships between the plurality of nodes from the video; and a matching unit that compares the question graph data and the video graph data and extracts one or more frame images from the video indicating the answer content to the question based on the matching result.
[0103] (Note 2) The information processing system described in Note 1, wherein the text contains one subject, one predicate, and one object, and in the question graph data and the video graph data, the nodes indicate the content of the subject or object, and the edges indicate the content of the predicate.
[0104] (Note 3) The information processing system described in Note 1 or 2 generates multiple copies of the aforementioned question graph data and video graph data in chronological order.
[0105] (Appendix 4) The information processing system according to any one of Appendix 1 to 3, further comprising: a conversion unit that converts the video graph data including the question graph data into natural language text; and an answer text generation unit that generates an answer text to answer the question text based on the natural language text.
[0106] (Note 5) The information processing system according to any one of Notes 1 to 4, wherein the question graph data generation unit determines the identification information of nodes and edges included in the question graph data based on the identification information of nodes and edges included in the video.
[0107] (Appendix 6) The information processing system according to Appendix 4 or 5, further comprising a video feature extraction unit that extracts video features including at least one of scene features and semantic features from the video, and the response text generation unit generates the response text based on the natural language text and the video features.
[0108] (Note 7) The information processing system according to any one of Notes 1 to 6, further comprising a filtering unit that compares the question statement with the video features and the matching result, and extracts one or more frame images indicating the content of the answer to the question statement based on the matching result which has a degree of relevance to the question statement of a predetermined value or higher.
[0109] (Note 8) The information processing system according to Note 7, further comprising a marking unit that marks the nodes included in the question graph data corresponding to the frame image extracted by the filtering unit, and the answer sentence generation unit generates an answer sentence to the question sentence, including information indicating the display manner of the marking.
[0110] (Note 9) An information processing system that generates question graph data for comparison with video graph data generated from one or more videos, which are composed of a combination of a plurality of nodes representing people or objects and one or more edges representing the relationships between the plurality of nodes, from an input question sentence, comprising: a question interpretation unit that identifies at least event identification information that identifies an event from the information contained in the question sentence; a question decomposition unit that generates one or more texts that represent the content of the event based on the event identification information; and a question graph data generation unit that generates the question graph data composed of a combination of a plurality of nodes representing people or objects and one or more edges representing the relationships between the plurality of nodes from the texts.
[0111] (Note 10) An information processing method that takes one or more video images and a question as input, comprising: a step of identifying event identification information that identifies an event from the information contained in the question; a step of generating one or more texts that indicate the content of the event based on the event identification information; a step of generating question graph data from the texts, which is composed of a combination of a plurality of nodes that indicate a person or object and one or more edges that indicate the relationships between the plurality of nodes; a step of generating video graph data from the video, which is composed of a combination of a plurality of nodes that indicate a person or object and one or more edges that indicate the relationships between the plurality of nodes; and a step of comparing the question graph data and the video graph data, and extracting one or more frame images from the video that indicate the content of the answer to the question based on the comparison result.
[0112] (Note 11) An information processing program that takes one or more video images and a question as input, wherein the computer is made to perform the following steps: a process to identify event identification information that identifies an event from the information contained in the question; a process to generate one or more texts that indicate the content of the event based on the event identification information; a process to generate question graph data from the text, which consists of a combination of a plurality of nodes that indicate a person or object and one or more edges that indicate the relationships between the plurality of nodes; a process to generate video graph data from the video, which consists of a combination of a plurality of nodes that indicate a person or object and one or more edges that indicate the relationships between the plurality of nodes; and a process to compare the question graph data and the video graph data, and extract one or more frame images from the video that indicate the content of the answer to the question based on the comparison result.
[0113] Some or all of the elements (e.g., configuration and function) described in Appendices 2 to 8 that are dependent on Appendice 1 may also be dependent on Appendices 10 and 11 in the same way as those described in Appendices 2 to 8. Some or all of the elements described in any appendice may be applied to various hardware, software, recording means, systems, and methods for recording software.
[0114] This application claims priority based on Japanese Patent Application No. 2025-008434, filed on 21 January 2025, and incorporates all of its disclosures herein.
[0115] 1 Information Processing System 10 Input Unit 20 Processing Unit 30 Output Unit 100 Video Input Unit 101 Question Input Unit 200 Video Processing Unit 201 Video Graph Data Generation Unit 202 Question Interpretation Unit 203 Question Decomposition Unit 204 Question Graph Data Generation Unit 205 Matching Unit 206 Conversion Unit 207 Answer Text Generation Unit 208 Filtering Unit 209 Marking Unit 211 Person / Object Detection Unit 212 Relationship Detection Unit 213 Tracking Unit 214 Identification Unit 215 Video Feature Extraction Unit 300 Frame Image Output Unit 301 Answer Text Output Unit 400 Computer 401 CPU 402 RAM 403 ROM 404 IF 405 HDD 406 Control Program E1-En Edge M1, M2 Marking O1-On, P1-Pn Nodes QG, QG1, QG2, QG3 Question graph data VG, VG1_1, VG1_2, VG1_3, VG2_1, VG2_2, VG2_3, VG3_1, VG3_2, VG3_3, VG4_3 Video graph data
Claims
1. An information processing system comprising: a question interpretation unit that identifies at least event identification information that identifies an event from the information contained in the input question; a question decomposition unit that generates one or more texts indicating the content of the event based on the event identification information; a question graph data generation unit that generates question graph data composed of a combination of a plurality of nodes indicating a person or object and one or more edges indicating the relationships between the plurality of nodes from the texts; a video graph data generation unit that generates video graph data composed of a combination of a plurality of nodes indicating a person or object and one or more edges indicating the relationships between the plurality of nodes from one or more input videos; and a matching unit that compares the question graph data and the video graph data and extracts one or more frame images from the video indicating the answer content to the question based on the matching result.
2. The information processing system according to claim 1, wherein the text contains one subject, one predicate, and one object, and in the question graph data and the video graph data, the nodes indicate the content of the subject or object, and the edges indicate the content of the predicate.
3. The information processing system according to claim 1 or 2, wherein multiple question graph data and video graph data are each generated in chronological order.
4. The information processing system according to claim 1 or 2, further comprising: a conversion unit that converts the video graph data, including the question graph data, into natural language text; and an answer text generation unit that generates an answer text to answer the question text based on the natural language text.
5. The information processing system according to claim 4, wherein the question graph data generation unit determines the identification information of nodes and edges included in the question graph data based on the identification information of nodes and edges included in the video.
6. The information processing system according to claim 5, further comprising a video feature extraction unit that extracts video features including at least one of scene features and semantic features from the video, wherein the response text generation unit generates the response text based on the natural language text and the video features.
7. The information processing system according to claim 6, further comprising a filtering unit that compares the question statement with the video features and the matching result, and extracts one or more frame images indicating the content of the answer to the question statement based on the matching result which has a degree of relevance to the question statement of a predetermined value or higher.
8. The information processing system according to claim 7, further comprising a marking unit that marks the nodes included in the question graph data corresponding to the frame image extracted by the filtering unit, wherein the answer sentence generation unit generates an answer sentence to the question sentence, including information indicating the display manner of the marking.
9. An information processing method comprising: identifying event identification information that identifies an event from the information contained in an input question; generating one or more texts indicating the content of the event based on the event identification information; generating question graph data from the texts, which consists of a combination of a plurality of nodes indicating a person or object and one or more edges indicating the relationships between the plurality of nodes; generating video graph data from one or more input videos, which consists of a combination of a plurality of nodes indicating a person or object and one or more edges indicating the relationships between the plurality of nodes; and comparing the question graph data and the video graph data, and extracting one or more frame images from the video indicating the content of the answer to the question based on the comparison result.
10. An information processing program that causes a computer to perform the following steps: a process of identifying event identification information that identifies an event from the information contained in an input question; a process of generating one or more texts indicating the content of the event based on the event identification information; a process of generating question graph data composed of a combination of a plurality of nodes indicating a person or object and one or more edges indicating the relationships between the plurality of nodes from the texts; a process of generating video graph data composed of a combination of a plurality of nodes indicating a person or object and one or more edges indicating the relationships between the plurality of nodes from one or more input videos; and a process of comparing the question graph data and the video graph data, and extracting one or more frame images from the videos indicating the content of the answer to the question based on the comparison result.