Scene information extraction device, program, and scene information extraction method

Abstract

A scene information extraction device (100) comprises: an object recognition unit (102) that recognizes one or more objects from a video; an object posture identification unit (103) that identifies the posture of said one or more objects from the video; a skeleton extraction unit (104) that extracts a skeleton including a plurality of parts of a subject from the video; an interaction identification unit (106) that identifies, as a target combination, a combination having a predetermined interaction from among combinations of one object and one part or combinations of two objects; a motion identification unit (108) that identifies a plurality of motions of the subject from the target combination; a behavior identification unit (110) that identifies a behavior of the subject from the plurality of motions; and an information integration unit (111) that generates scene information in which the identified behavior and the target combination are associated with each other.

Description

Scene Information Extraction Device, Program, and Scene Information Extraction Method

[0007] ,

[0001] The present disclosure relates to a scene information extraction device, a program, and a scene information extraction method.

[0002] Examples of tasks for extracting scene information from video include work content analysis in factories, purchase behavior analysis in stores, and detection of suspicious behavior in the street. Conventionally, in these tasks, methods of learning a learning model based on rules or for each site have been adopted. Specifically, characteristic postures resulting from actions are detected, or analysis is performed based on the presence or absence of keypoints within a set area. Such analysis methods are highly dependent on the shooting angle or environment, leading to an increase in costs during introduction and update.

[0003] In this regard, by using SGG (Scene Graph Generation) described in Non-Patent Document 1, it is possible to reduce the increase in costs during introduction and update.

[0004] Guangming Zhu, et. al., "Scene Graph Generation: A Comprehensive Survey", Neurocomputing, Volume 566, 2024

[0005] However, although SGG is a technology that can combine image analysis and natural language processing to convert video into a scene graph expressed linguistically, it cannot extract detailed information such as the coordinates of objects and the paths of actions.

[0006] Therefore, one or more aspects of the present disclosure aim to enable the extraction of various types of information with different specificities from video. <0A scene information extraction device according to one aspect of the present disclosure is characterized by comprising: an object recognition unit that recognizes one or more objects from video footage of a subject performing an action; a posture identification unit that identifies the posture of the one or more objects from the video footage; a skeleton extraction unit that extracts a skeleton including multiple parts of the subject from the video footage; an interaction identification unit that identifies a combination having a predetermined interaction from any one object included in the one or more objects and any one part included in the multiple parts, or any two objects included in the multiple objects, as a target combination; an action identification unit that identifies multiple actions of the subject from at least one of the one object and one part in the target combination, and the two objects in the target combination; and an information integration unit that generates scene information relating the identified action to at least one of the one object and one part in the target combination, and the two objects in the target combination.

[0008] A program according to one aspect of the present disclosure is characterized in that it causes a computer to function as: an object recognition unit that recognizes one or more objects from video footage of a subject performing an action; a posture identification unit that identifies the posture of the one or more objects from the video footage; a skeleton extraction unit that extracts a skeleton including multiple parts of the subject from the video footage; an interaction identification unit that identifies a combination having a predetermined interaction from any combination of one object and any one part included in the one or more objects, or any combination of any two objects included in the multiple objects, as a target combination; an action identification unit that identifies multiple actions of the subject from at least one of the one object and one part in the target combination, and the two objects in the target combination; an action identification unit that identifies the action from the multiple actions; and an information integration unit that generates scene information relating the identified action to at least one of the one object and one part in the target combination, and the two objects in the target combination.

[0009] A scene information extraction method according to one aspect of the present disclosure is characterized by: recognizing one or more objects from video footage of a subject performing an action; identifying the posture of the one or more objects from the video footage; extracting the skeleton including multiple parts of the subject from the video footage; identifying a combination having a predetermined interaction from any one object included in the one or more objects and any one part included in the multiple parts, or from any two objects included in the multiple objects, as a target combination; identifying multiple actions of the subject from at least one of the one object and one part in the target combination, and the two objects in the target combination; identifying the action from the multiple actions; and generating scene information relating the identified action to at least one of the one object and one part in the target combination, and the two objects in the target combination.

[0010] According to one or more aspects of this disclosure, various types of information with different levels of specificity can be extracted from video footage.

[0011] This is a schematic block diagram showing the configuration of the scene information extraction device according to the embodiment. (A) to (E) are schematic diagrams for explaining the orientation of an object detected by the object orientation identification unit. This is a schematic diagram for explaining the various types of information handled by the scene information extraction device. This is a schematic block diagram showing the configuration of the PC. This is a flowchart showing the operation of the scene information extraction device according to the embodiment.

[0012] Embodiment. Figure 1 is a block diagram schematically showing the configuration of a scene information extraction device 100 according to an embodiment. The scene information extraction device 100 comprises an image acquisition unit 101, an object recognition unit 102, an object posture identification unit 103, a skeleton extraction unit 104, an interaction information storage unit 105, an interaction identification unit 106, an operation information storage unit 107, an operation identification unit 108, an action information storage unit 109, an action identification unit 110, and an information integration unit 111.

[0013] The video acquisition unit 101 acquires video data showing images of a subject performing an action. For example, the video acquisition unit 101 may acquire video data from an imaging device such as a camera via a communication unit (not shown) realized by a communication I / F (Interface) 14, which will be described later, or it may acquire video data from a server (not shown) connected to a network such as the Internet via the communication unit, or it may acquire video data by reading video data stored in a storage unit (not shown) realized by a storage 11, which will be described later. The acquired video data is provided to the object recognition unit 102, the object posture identification unit 103, and the skeleton extraction unit 104.

[0014] The object recognition unit 102 recognizes one or more objects from the video data. Here, the object recognition unit 102 detects objects such as parts and tools from each frame included in the video data and recognizes the detected objects. For example, the object recognition unit 102 can use a known general object recognition model such as YOLOX (You Only Look Once X) to detect and recognize objects. If the types of objects that can be recognized are few, additional training may be performed on the general object recognition model beforehand.

[0015] The object orientation identification unit 103 shown in Figure 1 is an orientation identification unit that identifies the orientation of one or more objects recognized by the object recognition unit 102 from the video data. Here, the object orientation identification unit 103 identifies the orientation of the object recognized by the object recognition unit 102 for each frame. Here, the orientation is identified by the position and orientation of the object. For example, the object orientation determination unit 103 can determine the orientation of the object using known methods such as Epro-PnP (Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation) or HOISDF (Constructing 3D Hand-Object Pose Estimation with Global Signed Distance Fields).

[0016] Figures 2(A) to 2(E) are schematic diagrams illustrating the orientation of an object detected by the object orientation identification unit 103. As shown in Figure 2(A), the position and orientation of the first object OB1 are determined in a certain frame using the XYZ coordinate system CS. Also, as shown in Figure 2(B), the position and orientation of the second object OB2 are determined in a different frame using the XYZ coordinate system CS.

[0017] Then, the position and orientation of the first object OB1 are changed as shown in the XYZ coordinate system CS in Figure 2(C), and the position and orientation of the second object OB2 are also changed as shown in the XYZ coordinate system CS in Figure 2(D), so that the first object OB1 and the second object OB2 are assembled as shown in Figure 2(E). Thus, the movement paths of the first object OB1 and the second object OB2 during their assembly can be identified.

[0018] The object orientation identification unit 103 can generate object information 120, as shown in Figure 3, which includes "id," the identification information of the recognized object; "category," indicating the type of the recognized object; "objectPose," indicating the orientation of the recognized object; and "objectShape," indicating the shape of the recognized object. Here, although not shown in the figure, "objectPose," which indicates the orientation of the recognized object, is assumed to include "position," indicating the position of the recognized object; and "orientation," indicating the orientation of the recognized object.

[0019] The skeleton extraction unit 104 shown in Figure 1 extracts the skeleton, including multiple parts of the subject, from the video data. Here, the skeleton extraction unit 104 extracts skeletal information from each frame contained in the video data, indicating the 3D coordinates of feature points on the subject's body, hands, face, and feet. For example, the skeleton extraction unit 104 may extract the skeletal information using a known method such as RTMW (Real-Time Multi-Person 2D and 3D Whole-body Pose Estimation).

[0020] Then, as shown in Figure 3, the skeleton extraction unit 104 can generate skeleton information 121 by identifying a "humanPose" that indicates the coordinates of the feature points for each "id," which is worker identification information that identifies the subject included in the frame.

[0021] The interaction information storage unit 105 stores information necessary to identify interactions between parts of a person's body and an object, or between objects. For example, if the interaction identification unit 106, described later, identifies interactions on a rule basis, the interaction information storage unit 105 stores the rules for identifying the interactions.

[0022] Specifically, rules may be stored to determine that contact is occurring if the position of a body part and the position of an object are within a predetermined range for a predetermined period of time or longer. Rules may be stored to determine that contact is occurring if the position of an object and the position of another object are within a predetermined range for a predetermined period of time or longer. Rules may be stored to determine that an object is in the direction of the subject's line of sight if its position is within a predetermined range relative to the direction of the face for a predetermined period of time or longer. Rules may be stored to determine that an object is in the direction of the subject's line of sight if its position is within a predetermined range relative to the direction of the subject's line of sight for a predetermined period of time or longer. Rules may be stored to determine that an object is the object of action if its position is within a predetermined range relative to the direction of the subject's arm. Rules may be stored to determine that an object is the object of action if its position is within a predetermined range relative to the direction the subject's index finger is pointing.

[0023] Furthermore, when the interaction identification unit 106, described later, identifies interactions on a learning basis, the interaction information storage unit 105 stores the learned model used to identify interactions. Specifically, it is predetermined that interactions occur when there are predetermined relationships such as contact relationships, gaze direction relationships, or object relationships. Then, a learning model is pre-trained that takes the positions of parts of the subject and objects, or the positions of objects themselves, as input data, and outputs whether or not an interaction exists based on such input data. The interaction information storage unit 105 then stores the learned model thus acquired.

[0024] The interaction identification unit 106 identifies combinations that have predetermined interactions from a combination of any one object included in one or more objects recognized by the object recognition unit 102 and any one part included in multiple parts extracted by the skeleton extraction unit 104, or a combination of any two objects included in multiple objects recognized by the object recognition unit 102, as target combinations. Here, the interaction identification unit 106 identifies skeletons and objects that can be determined to have an interaction as described above, as well as combinations of objects and objects, and generates interaction information indicating that combination for each such combination.

[0025] For example, as shown in Figure 3, the interaction information 122 includes "id," which is the identification information of the identified interaction; "time," which is the time corresponding to the frame in which the interaction was identified; "subject," which identifies the subject or object; "object," which identifies the object with which interaction with the subject or object identified by "subject" was observed; and "interaction," which identifies the identified interaction.

[0026] The operation information storage unit 107 stores information necessary to identify an operation from time-series data in a combination of interactions. For example, if the operation identification unit 108, described later, identifies interactions in a rule-based manner, the operation information storage unit 107 stores the rules for identifying the operation.

[0027] Specifically, there are rules for each action that are predetermined using the positions of body parts and objects, or the positions of two objects. For example, a rule that determines "extending" when the hand is gradually moving toward an object, a rule that determines "grasping" when the hand reaches the same position as the object after the "extending" action is performed and remains in that position for a predetermined time or longer, and a rule that determines "carrying" when the hand and the object move at the same distance for a predetermined time or longer after the "grasping" action is performed. These rules are predetermined for each action and stored in the action information storage unit 107.

[0028] Furthermore, when the operation identification unit 108, described later, identifies interactions on a learning basis, the operation information storage unit 107 stores the learned model that was learned to identify the operation.

[0029] Specifically, the learning model should be pre-trained using time-series data that shows the positions of combinations that are judged to have interactions, and training data that considers the actions in such time-series data as correct.

[0030] The motion identification unit 108 identifies multiple actions of a subject from at least one of the following: one object and one body part in the target combination identified by the interaction identification unit 106, and two objects in the target combination identified by the interaction identification unit 106. Here, the motion identification unit 108 identifies the actions of the subject frame by frame using time-series data in the combinations in which interaction is determined to exist, generated by the interaction identification unit 106. The actions identified by the motion identification unit 108 are predetermined. For example, the motion identification unit 108 can identify actions from time-series data in the combinations in which interaction exists using a rule-based or learning-based method.

[0031] Specifically, if the time-series data for a combination that the interaction identification unit 106 has determined to have an interaction satisfies a rule stored in the operation information storage unit 107, the operation identification unit 108 should identify the operation corresponding to that rule.

[0032] Furthermore, the operation identification unit 108 may acquire an operation corresponding to the input data by inputting time-series data of combinations that have been determined to have interactions identified by the interaction identification unit 106 as input data to the learning model stored in the operation information storage unit 107. Here, the operation identification unit 108 may identify an operation from the learning model for each time-series data. If there are multiple operations identified in this way at the same time, the operation identification unit 108 may identify the operation with the highest confidence score among the identified operations as the operation of the subject.

[0033] Then, as shown in Figure 3, the motion identification unit 108 generates motion information 123 which includes "id" to identify the identified motion, "fromtime" indicating the time when the motion started, "totime" indicating the time when the motion ended, "person" indicating the person who performed the motion, "object" indicating the object that was the target of the motion, and "motion" indicating the type of motion.

[0034] As described above, the operation identification unit 108 identifies multiple operations from at least one of the time series of the positions of one object and one part in the target combination, and the time series of the positions of two objects in the target combination.

[0035] The action information storage unit 109 stores information necessary to identify an action from time-series data showing the time series of actions identified by the action identification unit 108. For example, if the action identification unit 110, which will be described later, identifies interactions on a rule basis, the action information storage unit 107 stores the rules for identifying the actions.

[0036] Specifically, if the sequence of actions is "grab the container," "carry the container," "release the container," "grab the lid," "carry the lid," "release the lid at the container's position," "grab the screw," "carry the screw," "release the screw at the lid's position," "grab the screwdriver," "carry the screwdriver," and "rotate the screwdriver at the screw's position," a rule may be predetermined to identify that the action is "screwing." If the sequence of actions is "hold a white shirt in the left hand," "hold a yellow shirt in the right hand," "stand in front of a mirror," "look in the mirror," "stop in place," and "compare clothes," a rule may be predetermined to identify that the action is "choosing clothes." If the sequence of actions is "multiple people approach an ATM (Automatic Teller Machine)," "crouch in front of the ATM," "touch the bottom of the ATM," and "touch for a longer time than usual," a rule may be predetermined to identify that the action is "operating an ATM other than the control panel."

[0037] Furthermore, when the behavior identification unit 110, which will be described later, identifies a behavior based on learning, the behavior information storage unit 109 stores the learning model that has been learned to identify the behavior.

[0038] Specifically, the learning model should be pre-trained using time-series data that shows the time series of actions as an example, and training data that considers the correct actions in such time-series data as the correct answers.

[0039] The behavior identification unit 110 identifies the actions performed by the subject from among the multiple actions identified by the action identification unit 108. Here, the behavior identification unit 110 identifies the subject's actions using the time-series data of the actions identified by the action identification unit 108. For example, the behavior identification unit 110 can identify actions from the time-series data of actions using a rule-based or learning-based approach.

[0040] Specifically, if the time-series data of an action identified by the action identification unit 108 satisfies a rule stored in the action information storage unit 109, the action identification unit 110 should identify the action corresponding to that rule.

[0041] Alternatively, the action identification unit 110 may acquire an action corresponding to the input data by inputting the time-series data of the action identified by the action identification unit 108 as input data to the learning model stored in the action information storage unit 109.

[0042] Then, as shown in Figure 3, the action identification unit 110 generates action information 124 which includes "id" to identify the identified action, "fromtime" indicating the time the action started, "totime" indicating the time the action ended, "person" indicating the person who performed the action, "object" indicating the object that was the target of the action, and "action" indicating the type of action.

[0043] The information integration unit 111 shown in FIG. 1 generates scene information that associates at least one of the actions specified by the action specification unit 110, one object and one part in the target combination, and two objects in the target combination. Note that the information integration unit 111 may also associate, in the scene information, interaction information indicating the interaction of the target combination generated by the interaction specification unit 106 with them.

[0044] Here, the information integration unit 111 generates scene information by associating information with different levels of abstraction generated by the object pose specification unit 103, the skeleton extraction unit 104, the interaction specification unit 106, the motion specification unit 108, and the action specification unit 110 so that these pieces of information can be mutually referred to.

[0045] Specifically, the information integration unit 111 associates action information with the motion information that is the basis of the action information by means of an ID or the like, and associates the motion information with the interaction information that is the basis of the motion information by an ID or the like. Further, scene information is generated by associating at least one of the skeleton information and the object information in which an interaction is assumed by the interaction information by an ID or the like.

[0046] By generating scene information in this way, applications that utilize the scene information can associate and refer to action information with a high level of abstraction and interaction information, skeleton information, or object information with a low level of abstraction according to the application.

[0047] The scene information extraction device 100 described above can be realized by a computer such as the PC 10 shown in FIG. 4, for example. The PC 10 includes a storage 11 such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive), a memory 12, a processor 13 such as a CPU (Central Processing Unit), a communication I / F (Interface) 14 such as a NIC (Network Interface Card), an input I / F 15 such as a keyboard and a mouse, and a display 16.

[0048] For example, the interaction information storage unit 105, the action information storage unit 107, and the behavior information storage unit 109 can be realized by the storage 11 or the memory 12. The video acquisition unit 101, the object recognition unit 102, the object pose identification unit 103, the skeleton extraction unit 104, the interaction identification unit 106, the action identification unit 108, the behavior identification unit 110, and the information integration unit 111 can be realized by the processor 13 executing a program stored in the storage 11.

[0049] The program may be downloaded from a recording medium (not shown) via a reader / writer (not shown) or from a network via the communication I / F 14 to the storage 11, and then loaded onto the memory 12 and executed by the processor 13. Alternatively, it may be directly loaded onto the memory 12 from a recording medium via a reader / writer (not shown) or from a network via the communication I / F 14 and executed by the processor 13. In other words, the program may be provided by a computer program product such as a recording medium.

[0050] Note that at least a part of the video acquisition unit 101, the object recognition unit 102, the object pose identification unit 103, the skeleton extraction unit 104, the interaction identification unit 106, the action identification unit 108, the behavior identification unit 110, and the information integration unit 111 may also be configured by a processing circuit such as a single circuit, a composite circuit, a processor operating with a program, a parallel processor operating with a program, an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array). As described above, the scene information extraction device 100 can be realized by a processing circuit network.

[0051] FIG. 5 is a flowchart showing the operation of the scene information extraction device 100 according to the embodiment. First, the video acquisition unit 101 acquires video data indicating a video of a subject performing an action (S10). The acquired video data is provided to the object recognition unit 102, the object pose identification unit 103, and the skeleton extraction unit 104.

[0052] Next, the object recognition unit 102 detects objects such as parts and tools from each frame included in the video data and recognizes the detected objects (S11).

[0053] Next, the object orientation identification unit 103 identifies the orientation of the object recognized by the object recognition unit 102 for each frame (S12). Then, the object orientation identification unit 103 generates object information indicating the object recognized by the object recognition unit 102 and the orientation of that object.

[0054] Next, the skeleton extraction unit 104 extracts skeletal information from each frame contained in the video data, indicating the 3D coordinates of feature points in the skeleton of the subject, such as the body, hands, face, and feet (S13).

[0055] Next, the interaction identification unit 106 identifies at least one of the skeletons and objects, as well as the objects and combinations of objects, that can be determined to have an interaction (S14).

[0056] Next, the motion identification unit 108 uses the time-series data of the combinations in which interaction was determined to exist, generated by the interaction identification unit 106, to identify the subject's actions for each frame (S15).

[0057] The action identification unit 110 identifies the subject's actions using the time-series data of actions identified by the motion identification unit 108 (S16).

[0058] The information integration unit 111 generates scene information by associating information of different levels of abstraction generated by the object posture identification unit 103, the skeleton extraction unit 104, the interaction identification unit 106, the motion identification unit 108, and the action identification unit 110, and making this information mutually referential (S17).

[0059] As described above, according to this embodiment, detailed semantic information such as movements and actions can be extracted from video footage of a subject performing an action.

[0060] 100 Scene information extraction device, 101 Video acquisition unit, 102 Object recognition unit, 103 Object posture identification unit, 104 Skeleton extraction unit, 105 Interaction information storage unit, 106 Interaction identification unit, 107 Motion information storage unit, 108 Motion identification unit, 109 Action information storage unit, 110 Action identification unit, 111 Information integration unit

Claims

1. A scene information extraction device comprising: an object recognition unit that recognizes one or more objects from video footage of a subject performing an action; a posture identification unit that identifies the posture of the one or more objects from the video footage; a skeleton extraction unit that extracts a skeleton including multiple parts of the subject from the video footage; an interaction identification unit that identifies a combination having a predetermined interaction from any one object included in the one or more objects and any one part included in the multiple parts, or any two objects included in the multiple objects, as a target combination; an action identification unit that identifies multiple actions of the subject from at least one of the one object and one part in the target combination, and the two objects in the target combination; an action identification unit that identifies the action from the multiple actions; and an information integration unit that generates scene information relating the identified action to at least one of the one object and one part in the target combination, and the two objects in the target combination.

2. The scene information extraction device according to claim 1, characterized in that the operation identification unit identifies the plurality of operations from at least one of the time series of the position of one object and the position of one part in the target combination, and the time series of the positions of the two objects in the target combination.

3. The scene information extraction device according to claim 1 or 2, characterized in that the interaction identification unit generates interaction information indicating the interaction of the target combination, and the information integration unit associates the interaction information in the scene information with at least one of the identified action, one object and one part in the target combination, and the two objects in the target combination.

4. A program characterized in that the computer functions as: an object recognition unit that recognizes one or more objects from video footage of a subject performing an action; a posture identification unit that identifies the posture of the one or more objects from the video footage; a skeleton extraction unit that extracts the skeleton including multiple parts of the subject from the video footage; an interaction identification unit that identifies a combination having a predetermined interaction from any combination of one object and any one part included in the one or more objects, or any combination of any two objects included in the multiple objects, as a target combination; an action identification unit that identifies multiple actions of the subject from at least one of the one object and one part in the target combination, and the two objects in the target combination; an action identification unit that identifies the action from the multiple actions; and an information integration unit that generates scene information relating the identified action to at least one of the one object and one part in the target combination, and the two objects in the target combination.

5. A scene information extraction method characterized by: recognizing one or more objects from video footage of a subject performing an action; identifying the posture of the one or more objects from the video footage; extracting the skeleton including multiple parts of the subject from the video footage; identifying a combination of any one object included in the one or more objects and any one part included in the multiple parts, or a combination of any two objects included in the multiple objects, that has a predetermined interaction, as a target combination; identifying multiple actions of the subject from at least one of the one object and one part in the target combination, and the two objects in the target combination; identifying the action from the multiple actions; and generating scene information that associates the identified action with at least one of the one object and one part in the target combination, and the two objects in the target combination.

Images