Image-based parent-child interaction evaluation system

Abstract

The present invention relates to a system for evaluating parent-child interaction by quantitatively analyzing non-verbal communication based on an image, and a learning system and a method for evaluating parent-child interaction based on an image. The learning system for image-based parent-child interaction evaluation according to the present invention comprises: an image processing unit that extracts at least one parent-child non-verbal interaction element from a frame constituting a training image; an interaction input score collection unit that collects at least one interaction input score for the training image; and an artificial neural network module that is trained with the at least one parent-child non-verbal interaction element and the at least one interaction input score for the training image so as to learn to derive a parent-child interaction evaluation score on the basis of the image.

Description

Video-based parent-child interaction evaluation system

[0001] The present invention relates to a video-based parent-child interaction evaluation system, and more specifically, to a system that evaluates parent-child interaction by quantitatively analyzing non-verbal communication based on video. Furthermore, the present invention relates to a learning system and method for evaluating parent-child interaction based on video.

[0002] This application is a priority application for Korean Patent Application No. 10-2024-0163227 filed on November 15, 2024, and all contents disclosed in the specification and drawings of said application are incorporated into this application by reference.

[0003] This achievement is the result of research conducted with funding from the government of the Republic of Korea (Ministry of Science and ICT) and supported by the National Research Foundation of Korea (No. RS-2023-00218176).

[0004] Among the methods for evaluating child development, there is a method for assessing parent-child interaction. Self-report questionnaires or third-party observation methods are primarily used to evaluate parent-child interaction. A self-report questionnaire is a method in which a respondent provides answers to questions based on their own thoughts and experiences, whereas a third-party observation method is a method in which a trained observer directly observes and evaluates the natural interaction between parents and children.

[0005] While self-report assessments allow for simple and rapid data collection, they have the disadvantage of potentially introducing subjective biases from respondents and failing to evaluate subtle nonverbal cues (e.g., facial expressions, eye contact) observed in actual parent-child interactions.

[0006] While the third-party observation method allows a third party to observe the actual interactions between parents and children and evaluate them based on objective criteria, it requires a trained observer, and reliable evaluations may not be possible because parents and children may exhibit behaviors different from usual if they are aware that they are being observed.

[0007] For this reason, videos observing the daily behavior of parents and children are filmed for extended periods, and experts analyze the nonverbal communication and interactions captured in the footage; however, this approach presents a problem in that accurate evaluation is difficult due to the long duration of the analysis and the severe shortage of trained experts.

[0008] For this reason, there is a need to develop a system that automatically evaluates non-verbal communication and interaction between parents and children based on video footage of the parents and children.

[0009] [Prior Art Literature]

[0010] (Patent Document 1) Republic of Korea Published Patent No. 10-2021-0020723 (Published Feb. 24, 2021)

[0011] (Patent Document 2) Republic of Korea Published Patent No. 10-2023-0039468 (Published March 21, 2023)

[0012] The objective of the present invention is to provide a system and method for analyzing image data to extract human objects, tracking parent and child objects, and extracting interaction elements such as the physical distance between the parent and child objects, the movement speed of the child object, and the gaze direction of the parent and child objects.

[0013] Another objective of the present invention is to provide a system and method that learns interaction elements extracted from a training video and interaction input scores input by experts who observed the training video, and learns to predict non-verbal parent-child interaction evaluation scores from an evaluation video.

[0014] Another objective of the present invention is to provide a system and method for extracting interaction elements from evaluation image data and predicting an interaction evaluation score.

[0015] The present invention may be implemented in various ways, including a device (system), a method, a computer program stored on a computer-readable medium, or a computer-readable medium on which a computer program is stored.

[0016] A learning system for evaluating parent-child interaction based on video according to the present invention comprises: a video processing unit that extracts at least one non-verbal interaction element between parent and child from a frame constituting a video for learning; an interaction input score collection unit that collects at least one interaction input score for a video for learning; and an artificial neural network module that is trained with at least one non-verbal interaction element between parent and child and at least one interaction input score for a video for learning, and is trained to derive a parent-child interaction evaluation score based on the video.

[0017] Preferably, nonverbal interaction elements include at least one of the physical distance between the parent and the child, the child's movement speed, and the direction of the parent and the child's gaze.

[0018] Preferably, the image processing unit includes an image data preprocessing module that derives the positions of a parent object and a child object from a plurality of frames constituting a training image, tracks the movement path for each object, and outputs the result, and a non-verbal context analysis module that derives parent-child interaction elements by analyzing the non-verbal context based on the movement paths of the parent object and child object derived from each frame.

[0019] More preferably, the image data preprocessing module comprises: an object recognition unit that recognizes human objects in frames constituting a training image; an object bounding box derivation unit that derives a parent object bounding box and a child object bounding box for each human object recognized by the object recognition unit; an object-specific location tracking unit that derives the locations of the parent object bounding box and the child object bounding box; and an object-specific movement path output unit that tracks and outputs the movement paths of the parent object bounding box and the child object bounding box during a plurality of frames based on the location information of the parent object bounding box and the location information of the child object bounding box derived for each of the plurality of frames.

[0020] More preferably, the non-verbal context analysis module comprises: a contact point derivation unit that calculates the top-view contact point of the parent object bounding box and the top-view contact point of the child object bounding box, respectively; a distance calculation unit that calculates the distance between the top-view contact point of the parent object bounding box and the top-view contact point of the child object bounding box; a movement speed estimation unit that estimates the movement speed of the child object based on location information of the top-view contact point of the child object bounding box in at least two frames; and a gaze direction estimation unit that derives gaze end point coordinates based on the rotation matrix of the object head for each parent object and child object derived from the frames, and estimates the gaze vectors of the parent object and child object based on the gaze start point coordinates and gaze end point coordinates of the head center point.

[0021] More preferably, the top view contact point of the parent object bounding box is defined as the center point between the feet of the parent object, and the top view contact point of the child object bounding box is defined as the center point between the feet of the child object.

[0022] More preferably, the top view contact point is calculated using the slope view contact point extracted from the frame and the transformation matrix M.

[0023] More preferably, the movement speed estimation unit estimates the movement speed of a child object by dividing the movement distance of the top view contact point of the child object bounding box during at least two frames in a time interval between at least two frames.

[0024] Preferably, the artificial neural network module is composed of a combination of a Gated Recurrent Unit (GRU) model and a Convolutional Neural Network (CNN) model from the Recurrent Neural Network (RNN) family that process time-series data.

[0025] A learning method for evaluating parent-child interaction based on images, implemented by a processor of a computer system according to the present invention, comprises the steps of: extracting at least one non-verbal interaction element between a parent and a child from a frame constituting a learning image; collecting at least one interaction input score for the learning image; and training an artificial neural network module with at least one non-verbal interaction element between a parent and a child and at least one interaction input score to learn to derive a parent-child interaction evaluation score based on the image.

[0026] Preferably, nonverbal interaction elements include at least one of the physical distance between the parent and the child, the child's movement speed, and the direction of the parent and the child's gaze.

[0027] Preferably, the method includes the step of deriving the positions of a parent object and a child object from each of the multiple frames constituting the training video and tracking the movement path of each object, and the step of analyzing the non-verbal context based on the movement paths of the parent object and child object derived from each frame to derive non-verbal interaction elements between the parent and child.

[0028] More preferably, the method includes the step of recognizing a person object in a frame constituting a training video; the step of deriving a parent object bounding box and a child object bounding box for each recognized person object; the step of deriving the locations of the parent object bounding box and the child object bounding box; and the step of tracking the movement path of the parent object bounding box and the child object bounding box during a plurality of frames based on the location information of the parent object bounding box and the location information of the child object bounding box tracked for each of the plurality of frames.

[0029] More preferably, the method includes the steps of calculating the top-view contact point of the parent object bounding box and the top-view contact point of the child object bounding box in each frame, respectively; calculating the distance between the top-view contact point of the parent object bounding box and the top-view contact point of the child object bounding box; estimating the movement speed of the child object based on the location information of the top-view contact point of the child object bounding box in at least two frames; and for each parent object and child object derived in each frame, deriving the gaze endpoint coordinates based on the rotation matrix of the object head and estimating the gaze vectors of the parent object and the child object based on the gaze start point coordinates and gaze end point coordinates of the head center point.

[0030] More preferably, the top view contact point of the parent object bounding box is defined as the center point between the feet of the parent object, and the top view contact point of the child object bounding box is defined as the center point between the feet of the child object.

[0031] More preferably, the top view contact point is calculated using the slope view contact point extracted from the frame and the transformation matrix M.

[0032] More preferably, the movement speed of a child object is estimated by dividing the movement distance of the top view contact point of the child object bounding box over at least two frames during a time interval between at least two frames.

[0033] Preferably, the artificial neural network module is composed of a combination of a Gated Recurrent Unit (GRU) model and a Convolutional Neural Network (CNN) model from the Recurrent Neural Network (RNN) family that process time-series data.

[0034] A video-based parent-child interaction evaluation system according to the present invention comprises a video processing unit that extracts at least one non-verbal parent-child interaction element from a frame constituting a video for evaluation, and an artificial neural network module trained by the learning method for video-based parent-child interaction evaluation described above.

[0035] According to the present invention, non-verbal interaction elements between parent and child, such as the physical distance between the parent object and the child object, the movement speed of the child object, and the gaze direction of the parent object and the child object, can be extracted by extracting and tracking a parent object and a child object from a video frame.

[0036] According to the present invention, an artificial neural network module learns non-verbal interaction elements between parents and children and interaction input scores input from experts, and can predict an interaction evaluation score by evaluating non-verbal interaction between parents and children from evaluation video data.

[0037] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art to which the present invention pertains (referred to as "person skilled in the art") from the description in the claims.

[0038] Embodiments of the present invention will be described with reference to the accompanying drawings described below, wherein similar reference numerals indicate similar elements, but are not limited thereto.

[0039] FIG. 1 is a block diagram illustrating a learning system for evaluating video-based parent-child interaction according to one embodiment of the present invention.

[0040] FIG. 2 is a block diagram illustrating an image data preprocessing module according to the present invention.

[0041] FIG. 3 is a block diagram illustrating a non-verbal context analysis module according to the present invention.

[0042] FIG. 4 is a flowchart illustrating a learning method for evaluating video-based parent-child interaction according to the present invention.

[0043] FIG. 5 is a block diagram illustrating a video-based parent-child interaction evaluation system according to the present invention.

[0044] Hereinafter, specific details for implementing the present invention will be described in detail with reference to the attached drawings. However, in the following description, specific descriptions regarding widely known functions or configurations will be omitted if there is a risk of unnecessarily obscuring the essence of the present invention.

[0045] In the attached drawings, identical or corresponding components are assigned the same reference numerals. Additionally, in the description of the following embodiments, the description of identical or corresponding components may be omitted. However, even if a description of a component is omitted, it is not intended that such component is not included in any embodiment.

[0046] The advantages and features of the embodiments disclosed in this specification, and the methods for achieving them, will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms, and these embodiments are provided merely to fully inform a person skilled in the art of the scope of the invention.

[0047] Unless otherwise defined, all terms used in this specification (including technical and scientific terms) may be used in a meaning commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0048] For example, the term “technique” may refer to systems, methods, computer-readable instructions, modules, algorithms, hardware logic, and / or operations throughout the document as permitted by the context described above.

[0049] The terms used in this specification will be briefly explained, and the disclosed embodiments will be described in detail. The terms used in this specification have been selected to be as generally used as possible, taking into account their functions in the present invention; however, these terms may vary depending on the intent of those skilled in the relevant field, case law, or the emergence of new technologies. Additionally, in specific cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should be defined not merely by their names, but based on their meanings and the overall content of the invention.

[0050] In this specification, singular expressions include plural expressions unless the context clearly specifies them as singular. Additionally, plural expressions include singular expressions unless the context clearly specifies them as plural. Throughout the specification, when a part is described as including a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0051] In the present invention, terms such as 'comprising', 'comprising', etc. may indicate the presence of features, steps, actions, elements and / or components, but do not exclude the addition of one or more other functions, steps, actions, elements, components and / or combinations thereof.

[0052] In the present invention, where a specific component is described as being 'combined,' 'combined,' 'connected,' 'associated,' or 'reacted' to any other component, the specific component may be directly combined, combined, connected, and / or associated with, or reacted to the other component, but is not limited thereto. For example, one or more intermediate components may exist between the specific component and the other component. Additionally, in the present invention, "and / or" may include each of the one or more listed items or a combination of at least some of the one or more items.

[0053] In the present invention, terms such as 'first', 'second', etc., are used to distinguish a specific component from another component, and the components described above are not limited by these terms. For example, the 'first' component may be used to refer to an element of the same or similar form as the 'second' component.

[0054] Artificial Intelligence (AI) refers to the field of researching artificial intelligence or the methodologies to create it, while Machine Learning refers to the field of researching methodologies to define and solve various problems addressed within the field of AI. Machine Learning is also defined as an algorithm that improves performance on a task through continuous experience.

[0055] An Artificial Neural Network (ANN) is a model used in machine learning that can refer to any model capable of problem-solving, composed of artificial neurons (nodes) that form a network through the connection of synapses. An artificial neural network can be defined by connection patterns between neurons in different layers, a learning process that updates learning parameters within the model, and an activation function that generates output values.

[0056] An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer may include one or more neurons, and the artificial neural network may include synapses connecting the neurons. In an artificial neural network, each neuron may output input signals input through the synapses, weights of each layer, and function values ​​of an activation function for biases.

[0057] Model learning parameters refer to parameters determined through learning, including synaptic connection weights and neuron biases. Hyperparameters, on the other hand, refer to parameters that must be set prior to training in a machine learning algorithm, including the learning rate, number of iterations, mini-batch size, and initialization function.

[0058] The objective of training an artificial neural network can be viewed as determining the learning parameters that minimize the loss function. The loss function can be used as an indicator to determine the optimal learning parameters during the training process of an artificial neural network.

[0059] Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning depending on the learning method.

[0060] Supervised learning refers to a method of training an artificial neural network with labels provided for the training data; a label can refer to the correct answer (or result) that the neural network must infer when the training data is input. Unsupervised learning refers to a method of training an artificial neural network without labels provided for the training data. Reinforcement learning refers to a learning method in which an agent defined within an environment is trained to select an action or sequence of actions that maximizes the cumulative reward in each state.

[0061] Machine learning implemented by a Deep Neural Network (DNN) containing multiple hidden layers among artificial neural networks is also called Deep Learning, and Deep Learning is a part of Machine Learning. Hereinafter, Machine Learning is used in a sense that includes Deep Learning. Artificial intelligence can be performed by artificial neural network modules.

[0062] Generally, video-based assessments of parent-child interaction are conducted by trained professionals. Specifically, parents and children are filmed together over an extended period, and a trained professional reviews the footage to evaluate the level of nonverbal interaction between them and assigns a score.

[0063] The present invention proposes a method in which a computer system analyzes video of parents and children and automatically assigns a non-verbal interaction evaluation score.

[0064] To this end, the present invention derives interaction elements for evaluating parent-child interaction from a learning video and collects interaction input scores assigned by experts who have watched the learning video, thereby learning the interaction elements derived from the learning video and the interaction input scores.

[0065] FIG. 1 is a block diagram illustrating a learning system for evaluating image-based parent-child interaction according to one embodiment of the present invention. FIG. 2 is a block diagram illustrating an image data preprocessing module according to the present invention, and FIG. 3 is a block diagram illustrating a non-verbal context analysis module according to the present invention.

[0066] A learning system for evaluating image-based parent-child interaction according to the present invention will be described with reference to FIGS. 1 to 3.

[0067] A learning system for evaluating parent-child interaction based on video according to the present invention includes an image processing unit (120) that extracts at least one parent-child interaction element from a frame screen constituting a learning video (110), an interaction input score collection unit (130) that collects interaction input scores for the learning video (110), and an artificial neural network module (140) that is trained to derive a parent-child interaction evaluation score based on video by inputting at least one non-verbal interaction element between parent and child extracted by the image processing unit (120) and an interaction input score collected by the interaction input score collection unit (130) for the learning video.

[0068] Nonverbal interaction elements between parents and children may include at least one of the physical distance between the parent and child, the child's speed of movement, and the direction of gaze between the parent and child.

[0069] The image processing unit (120) includes an image data preprocessing module (121) that derives the positions of a parent object and a child object in a frame screen constituting a learning image (110), derives the positions of a parent object and a child object from a plurality of frames, and outputs a movement path for each object, and a non-verbal context analysis module (122) that derives parent-child interaction elements by analyzing the non-verbal context based on the parent object and child object derived from each frame.

[0070] As illustrated in FIG. 2, the image data preprocessing module (121) includes an object recognition unit (210) that recognizes a person object in the frame of a learning image, an object bounding box derivation unit (220) that derives a parent object bounding box and a child object bounding box for each person object recognized by the object recognition unit (210), an object-specific location tracking unit (230) that derives the location of the parent object bounding box and the child object bounding box for each frame of the learning image and tracks the location of the parent object bounding box and the child object bounding box in a plurality of image frames, and an object-specific movement path output unit (240) that outputs the movement path of the parent object bounding box and the movement path of the child object bounding box during a plurality of image frames based on the location information of the parent object bounding box and the location information of the child object bounding box tracked for each of the plurality of image frames by the object-specific location tracking unit (230).

[0071] The object recognition unit (210) recognizes a person object among the foreground object instances in the video frame. The person object instances appearing in the video frame may include a parent object instance and a child object instance.

[0072] The object bounding box derivation unit (220) extracts feature information for a person object recognized in an image frame to distinguish between a parent object and a child object, and extracts a parent object bounding box and a child object bounding box, respectively. This bounding box can be defined as a rectangular area of ​​minimum area that includes the two feet and head of the person object, and each bounding box has a minimum coordinate value (x min , y min ) and maximum coordinate value(x max , y max It can be specified as ). Each bounding box may include information on minimum coordinate values, maximum coordinate values, and object type (parent object or child object).

[0073] The object-specific location tracking unit (230) tracks the locations of the parent object bounding box and the child object bounding box in each video frame for multiple video frames of the training video (110). The location of each bounding box can be indicated by the minimum and maximum coordinate values ​​of each bounding box. The object-specific location tracking unit (230) collects location information of the parent object bounding box extracted from each video frame and collects location information of the child object bounding box.

[0074] The object-specific movement path output unit (240) tracks and outputs the movement path of each object bounding box by arranging the location information of each object bounding box collected by the object-specific location tracking unit (230) in the order of the frames' time. The movement path information of each object bounding box over multiple frames is {B(t1), B(t2), ..., B(t n It can be expressed as )}. The parent object bounding box in the i-th frame is B parent (t i Represented as ), the child object bounding box in the i-th frame is B child (t i It can be represented as ), and each bounding box is the minimum coordinate value (x min , y min ) and maximum coordinate value(x max , y max It can be represented as ).

[0075] Movement path of the parent object bounding box (T parent ) and the movement path of the child object bounding box (T child ) can be expressed as in Equation 1. The movement path of the parent object bounding box and the movement path of the child object bounding box derived in this way can be provided to a non-verbal context analysis module (122).

[0076] [Formula 1]

[0077] T parent = {B parent (t1), B parent (t2), ..., Bparent (t n )}

[0078] T child = {B child (t1), B child (t2), ..., B child (t n )}

[0079] The non-verbal context analysis module (122) includes a contact point derivation unit (310) that calculates the top view contact point of the parent object bounding box and the top view contact point of the child object bounding box in each video frame as illustrated in FIG. 3, a distance calculation unit (320) that calculates the distance between the top view contact point of the parent object bounding box and the top view contact point of the child object bounding box, a movement speed estimation unit (330) that estimates the movement speed of the child object based on location information of the top view contact point of the child object bounding box in at least two video frames, and a gaze direction estimation unit (340) that derives the gaze end point coordinates based on the rotation matrix of the object head for each parent object and child object derived in the video frame, and estimates the gaze vector based on the gaze start point coordinates and gaze end point coordinates of the head center point.

[0080] The contact point derivation unit (310) calculates the contact point of each object bounding box to calculate information related to the non-verbal communication context between the parent object and the child object by utilizing the bounding box of each object. The contact point of each object bounding box can be defined as the center point between the feet of each object, and can be set as the midpoint between the two feet in an image frame observed at an oblique angle.

[0081] The contact point (P) of the object can be expressed as in Equation 2.

[0082] [Equation 2]

[0083]

[0084] The contact point (P) can be transformed into a top-view perspective using a transformation matrix M, and this is defined as the top-view contact point. The transformation matrix M can be calculated using the fixed point of the oblique view and the corresponding fixed point of the top view. The fixed point of the oblique view is an anchor. oblique (i) = (x i , y i Represented as ), and the fixed point of the top view is anchor top (i) = (x' i , y' i It can be expressed as ).

[0085] The relationship between slope view coordinates and top view coordinates using the transformation matrix M can be expressed as Equation 3. Here, [x i ,y i ,1] T means a two-dimensional coordinate system.

[0086] [Equation 3]

[0087]

[0088] The transformation matrix M is a 3×3 homogeneous transformation matrix that facilitates the transformation from oblique view to top view, and the transformation matrix M can be estimated using anchor point pairs (anchor_oblique, anchor_top) in the image. Top view coordinates (x' i , y' i ) can be obtained by normalizing the coordinates obtained from Equation 3 through the homogeneous coordinate normalization process as shown in Equation 4. The top view contact point of each object bounding box can be calculated as 2D coordinates.

[0089] [Equation 4]

[0090]

[0091] As described above, the contact point (P) of each object bounding box can be converted from a sloped view to a top view.

[0092] In this invention, the distance and movement speed between objects are estimated based on top-view contact points calculated for each object bounding box, and context for analyzing non-verbal communication between parents and children and interaction elements between parents and children are extracted.

[0093] The distance calculation unit (320) estimates the distance between the parent object and the child object by calculating the Euclidean distance between the top view contact point of the parent object bounding box and the top view contact point of the child object bounding box using Equation 5.

[0094] [Formula 5]

[0095]

[0096] Here, P' parent (t i ), P' child (t i ) are each at time t i It refers to the top view contact point of the parent object bounding box and the top view contact point of the child object bounding box.

[0097] The movement speed estimation unit (330) estimates the movement speed of the parent object based on the position movement information of the top view contact point of the parent object bounding box in at least two frames, and estimates the movement speed of the child object based on the position movement information of the top view contact point of the child object bounding box in at least two frames, and this can be expressed as Equation 6.

[0098] [Equation 6]

[0099]

[0100] Here, Δt represents the time interval between frames, and P' parent (t i ), P' child (t i ) are each at time t i In this, it refers to the top view contact point of the parent object bounding box and the top view contact point of the child object bounding box, and P' parent (t i-1 ), P'child (t i-1 ) are each at time t i-1 In this, it refers to the top-view contact point of the parent object bounding box and the top-view contact point of the child object bounding box. PS(t i ) and CS(t i ) are each at time t i It refers to the movement speed of the parent object and the movement speed of the child object.

[0101] In other words, dividing the displacement distance of the top-view contact point of the child object bounding box between frames by the time interval between frames gives time t i The movement speed of the child object can be estimated, and by dividing the distance traveled by the top-view contact point of the parent object bounding box between frames by the time interval between frames, the time t i The movement speed of the parent object can be estimated.

[0102] The gaze direction estimation unit (340) estimates a gaze vector representing the gaze direction and gaze distance of the parent object and the child object. The gaze direction of the parent and child is an important indicator for evaluating the quality of parent-child interaction, and the concentration and joint attention of the two objects can be estimated by estimating the gaze direction. The gaze of each object can be calculated by estimating the posture of the object's head.

[0103] The gaze direction estimation unit (340) derives the gaze end point coordinates based on the rotation angle of each body head rotated around the yaw axis, pitch axis, and roll axis centered on the center point of the object head derived from the image, sets the center point of the object head as the gaze start point coordinates, and estimates the gaze vector based on the gaze start point coordinates and the gaze end point coordinates. The rotation angle of the object head rotated around the yaw axis, pitch axis, and roll axis centered on the center point of the object head can be expressed as a rotation matrix of the object head as shown in Equation 7.

[0104] [Equation 7]

[0105]

[0106] Set the center point of each object's head as the starting point of the gaze. The starting point of the gaze is the starting point of the gaze x-coordinate (G start ,x) and the y-coordinate of the starting point of the view (G start It can be composed of ,y). Applying the rotation matrix of the object head in Equation 7 to Equation 8 gives the x-coordinate of the gaze endpoint (G end ,x) and the y-coordinate of the view endpoint (G end Each of ,y) can be calculated.

[0107] [Equation 8]

[0108] G end ,x = |R yaw |×R yaw [2,1] + G start ,x

[0109] G end ,y = |R yaw |×{R yaw [1,1] × R pitch [1,3]}+ G start ,y

[0110] Here, |R yaw | is the magnitude (norm) of the rotation matrix, and R yaw [2,1] is the (2,1) component of the rotation matrix, and G start ,x is the x-coordinate of the starting point of the view, and R yaw [1,1] is the (1,1) component of the rotation matrix, and R pitch [1,3] is the (1,3) component of the pitch rotation matrix.

[0111] According to Equation 8, the x and y coordinates of the end point of the gaze can be derived, and by applying the x and y coordinates of the end point of the gaze and the x and y coordinates of the start point of the gaze to Equation 9, the gaze vector (G) can be obtained.

[0112] [Formula 9]

[0113] G = G end - G start

[0114] In this way, non-verbal interaction elements between parent and child can be derived through the image processing unit (120). These non-verbal interaction elements between parent and child may include the distance between the parent object and the child object, the movement speed of the child object, and the direction of gaze between the parent object and the child object.

[0115] Meanwhile, the learning video (110) may be provided to at least one expert, and the expert may give input scores for parent-child interaction while watching the learning video (110). The interaction input score collection unit (130) may collect interaction input scores for the parent and child captured in the learning video (110) from at least one expert.

[0116] At least one parent-child non-verbal interaction element extracted by the image processing unit (120) for a training video (110) and an interaction input score collected by the interaction input score collection unit (130) can be learned in an artificial neural network module (140). Through this, the artificial neural network module (140) can be trained to output an interaction evaluation score based on the parent-child non-verbal interaction element derived from the video.

[0117] The artificial neural network module (140) can be constructed by combining a Gated Recurrent Unit (GRU), a model of the Recurrent Neural Network (RNN) family that processes time-series data, and a Convolutional Neural Network (CNN), which is strong in analyzing unstructured image data. First, non-verbal interaction elements calculated for each frame, such as the physical distance between the parent object and the child object, the movement speed of the child object, and the direction of gaze, are provided to the GRU as input data in a frame-by-frame sequence. Then, an attention mechanism is applied to the input sequence data to identify the element data that has the greatest influence among the provided non-verbal interaction elements, and then the CNN model is used to extract features of the upper layer.

[0118] FIG. 4 is a flowchart illustrating a learning method for evaluating video-based parent-child interaction according to the present invention. The learning method for evaluating video-based parent-child interaction according to the present invention can be implemented by a processor in a conventional computer system.

[0119] The computer system recognizes human objects in the frame screens that make up the training video and derives a parent object bounding box and a child object bounding box for each recognized human object (S401).

[0120] The computer system extracts the locations of the parent object bounding box and the child object bounding box from multiple frames that make up the training video (S402).

[0121] The computer system tracks and outputs the movement path of the parent object bounding box and the movement path of the child object bounding box during multiple video frames based on the location information of the parent object bounding box and the location information of the child object bounding box extracted for each of the multiple video frames (S403).

[0122] The computer system calculates the top view contact point of the parent object bounding box and the top view contact point of the child object bounding box for each of the multiple image frames (S404).

[0123] The computer system derives at least one non-verbal interaction element using the top-view contact point of the parent object bounding box and the top-view contact point of the child object bounding box (S405). The non-verbal interaction element may include the physical distance between the parent object and the child object, the movement speed of the child object, and the gaze direction of the parent object and the child object. The computer system calculates the physical distance between the parent object and the child object by calculating the distance between the top-view contact point of the parent object bounding box and the top-view contact point of the child object bounding box. The computer system estimates the movement speed of the child object in at least two frames based on the location information of the top-view contact point of the child object bounding box in at least two video frames. For each parent object and child object derived from the video frames, the computer system derives the gaze endpoint coordinates based on the rotation matrix of the object's head and estimates the gaze vector based on the gaze start point coordinates and gaze end point coordinates of the head center point.

[0124] Meanwhile, the learning video (110) may be provided to at least one expert, and the expert may give input scores for parent-child interaction while watching the learning video (110). The computer system may collect input scores for parent-child interaction captured in the learning video from at least one expert (S406).

[0125] The computer system trains an artificial neural network module using the parent-child nonverbal interaction elements derived in step S405 and the interaction input scores collected in step S406 (S407). In this way, the artificial neural network module can be trained to output an interaction evaluation score based on the parent-child nonverbal interaction elements derived from the video.

[0126] The image-based parent-child interaction evaluation system according to the present invention may be implemented by including an artificial neural network module trained by the learning method for image-based parent-child interaction evaluation described above.

[0127] FIG. 5 is a block diagram illustrating a video-based parent-child interaction evaluation system according to the present invention.

[0128] The image-based parent-child interaction evaluation system according to the present invention includes an image processing unit (520) that extracts at least one parent-child interaction element from a frame screen constituting an evaluation image (510), and an artificial neural network module (530) that derives a parent-child interaction evaluation score based on at least one non-verbal interaction element between the parent and child extracted by the image processing unit (520) for the evaluation image. The non-verbal interaction element between the parent and child may be at least one of the physical distance between the parent and child, the child's movement speed, and the direction of gaze between the parent and child.

[0129] The image processing unit (520) includes an image data preprocessing module (521) that derives the positions of a parent object and a child object in the frame of the evaluation image (510) and derives the positions of a parent object and a child object from a plurality of frames and outputs a movement path for each object, and a non-verbal context analysis module (522) that derives parent-child interaction elements by analyzing the non-verbal context based on the parent object and child object derived from each frame.

[0130] The image processing unit (520), image data preprocessing module (521), and non-verbal context analysis module (522) of FIG. 5 may be configured with the same functions as the image processing unit (120), image data preprocessing module (121), and non-verbal context analysis module (122) of FIG. 1.

[0131] That is, the image data preprocessing module (521) may include an object recognition unit, an object boundary box derivation unit, an object-specific location tracking unit, and an object-specific movement path output unit as shown in FIG. 2. Additionally, the non-verbal context analysis module (522) may include a contact point derivation unit, a distance calculation unit, a movement speed estimation unit, and a gaze direction estimation unit as shown in FIG. 3.

[0132] The artificial neural network module (530) of FIG. 5 may be an artificial neural network module (140) learned by the learning system of FIG. 1.

[0133] All of the aforementioned methods and processes may be implemented as software code modules executed by one or more general-purpose computers or processors and may be fully automated. The code modules may be stored on any type of computer-readable storage medium or other computer storage device. Some or all of the methods may be implemented in specialized computer hardware.

[0134] Any ordinary description, element, or block of a flowchart described in this specification and / or illustrated in the accompanying drawings should be understood as potentially representing code, modules, segments, or parts containing one or more executable instructions for implementing a specific logical function or element. Alternative examples are included within the scope of the examples described herein and, depending on the function understood herein, elements or functions may be executed substantially synchronously or in reverse order from those omitted, illustrated, or discussed.

[0135] Many variations and modifications may be made to the embodiments described above, and such elements should be understood as being one of other acceptable examples. All such modifications and variations are intended to be included within the scope of this disclosure and protected by the following claims. The embodiments according to the invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. A computer-readable recording medium may include program instructions, data files, data structures, etc., either alone or in combination. Program instructions recorded on a computer-readable recording medium may be those specifically designed and configured for the invention or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

[0136] Although the present invention has been described above with specific details such as specific components, limited embodiments, and drawings, this is provided only to aid in a more comprehensive understanding of the invention, and the invention is not limited to the above embodiments, and a person skilled in the art to which the invention belongs can make various modifications and variations from this description.

[0137] Accordingly, the scope of the present invention is not limited to the embodiments described above, and all modifications equivalent to or equivalent to the claims set forth below, as well as the claims described below, shall be considered to fall within the scope of the scope of the present invention.

Claims

1. An image processing unit configured to extract at least one non-verbal parent-child interaction element from a frame constituting a learning video; An interaction input score collection unit configured to collect at least one interaction input score for the above-mentioned learning video; and An artificial neural network module trained with the above-mentioned training video using the above-mentioned at least one parent-child non-verbal interaction element and the above-mentioned at least one interaction input score, and trained to derive a parent-child interaction evaluation score based on the video; comprising Learning system for evaluating video-based parent-child interaction.

2. In Paragraph 1, The above nonverbal interaction element includes at least one of the physical distance between the parent and the child, the child's movement speed, and the direction of gaze between the parent and the child. Learning system for evaluating video-based parent-child interaction.

3. In Paragraph 1, The image processing unit comprises an image data preprocessing module configured to derive the positions of a parent object and a child object from a plurality of frames constituting the training image, respectively, and to track and output the movement path for each object, and A non-verbal context analysis module configured to derive parent-child interaction elements by analyzing the non-verbal context based on the movement paths of parent and child objects derived from each frame, Learning system for evaluating video-based parent-child interaction.

4. In Paragraph 3, The above image data preprocessing module is, An object recognition unit configured to recognize human objects in frames constituting the above-mentioned training video, and An object bounding box derivation unit configured to derive a parent object bounding box and a child object bounding box for each person object recognized by the object recognition unit above, An object-specific location tracking unit configured to derive the locations of the parent object bounding box and the child object bounding box, and Includes an object-specific movement path output unit configured to track and output the movement path of the parent object bounding box and the movement path of the child object bounding box during a plurality of frames based on the position information of the parent object bounding box and the position information of the child object bounding box derived for each of the plurality of frames. Learning system for evaluating video-based parent-child interaction.

5. In Paragraph 3, The above non-verbal context analysis module is, A contact point derivation unit configured to calculate the top view contact point of the parent object bounding box and the top view contact point of the child object bounding box, respectively, and A distance calculation unit configured to calculate the distance between the top view contact point of the parent object bounding box and the top view contact point of the child object bounding box, A movement speed estimation unit configured to estimate the movement speed of a child object based on location information of the top view contact point of the child object bounding box in at least two frames, and A gaze direction estimation unit configured to derive gaze endpoint coordinates based on the rotation matrix of the object's head for each parent object and child object derived from a frame, and to estimate gaze vectors of the parent object and child object based on the gaze start point coordinates of the head center point and the said gaze endpoint coordinates, Learning system for evaluating video-based parent-child interaction.

6. In Paragraph 5, The top view contact point of the parent object bounding box is determined as the center point between the feet of the parent object, and the top view contact point of the child object bounding box is determined as the center point between the feet of the child object. Learning system for evaluating video-based parent-child interaction.

7. In Paragraph 5, The above top view contact point is calculated using the slope view contact point extracted from the frame and the transformation matrix M, Learning system for evaluating video-based parent-child interaction.

8. In Paragraph 5, The above movement speed estimation unit estimates the movement speed of the child object by dividing the movement distance of the top view contact point of the child object bounding box during the at least two frames in the time interval between the at least two frames. Learning system for evaluating video-based parent-child interaction.

9. In paragraph 1, the artificial neural network module is configured by combining a Gated Recurrent Unit (GRU) model of the Recurrent Neural Network (RNN) family and a Convolutional Neural Network (CNN) model for processing time-series data. Learning system for evaluating video-based parent-child interaction.

10. In a learning method for evaluating image-based parent-child interaction implemented by a processor of a computer system, A step of extracting at least one parent-child nonverbal interaction element from a frame constituting a learning video; A step of collecting at least one interaction input score for the above-mentioned learning video; and A step comprising training an artificial neural network module with at least one parent-child non-verbal interaction element and at least one interaction input score to learn to derive a parent-child interaction evaluation score based on video; Learning method for evaluating video-based parent-child interaction.

11. In Paragraph 10, The above nonverbal interaction element includes at least one of the physical distance between the parent and the child, the child's movement speed, and the direction of gaze between the parent and the child. Learning method for evaluating video-based parent-child interaction.

12. In Paragraph 10, The step of extracting at least one non-verbal parent-child interaction element from a frame constituting the above-mentioned learning video is: A step of deriving the positions of a parent object and a child object from each of the multiple frames constituting the above-mentioned training video and tracking the movement path for each object, and A step comprising deriving non-verbal interaction elements between parent and child by analyzing the non-verbal context based on the movement paths of parent and child objects derived from each frame, Learning method for evaluating video-based parent-child interaction.

13. In Paragraph 12, The step of deriving the positions of the parent object and the child object from each of the multiple frames constituting the above-mentioned training video and tracking the movement path for each object is: A step of recognizing a human object in a frame constituting the above-mentioned training video, and A step of deriving a parent object bounding box and a child object bounding box for each of the above-mentioned recognized person objects, The step of deriving the positions of the parent object bounding box and the child object bounding box, and A method comprising the step of tracking the movement path of a parent object bounding box and a child object bounding box during a plurality of frames based on the location information of the parent object bounding box and the location information of the child object bounding box tracked for each of the plurality of frames, Learning method for evaluating video-based parent-child interaction.

14. In Paragraph 12, The step of deriving non-verbal interaction elements between parent and child by analyzing the non-verbal context based on the movement paths of the parent object and child object derived from each of the above frames is: The step of calculating the top view contact point of the parent object bounding box and the top view contact point of the child object bounding box in each of the above frames, respectively, and A step of calculating the distance between the top view contact point of the parent object bounding box and the top view contact point of the child object bounding box, A step of estimating the movement speed of a child object based on location information of the top view contact point of the child object bounding box in at least two frames, and A step comprising deriving gaze endpoint coordinates based on the rotation matrix of the object head for each parent object and child object derived from each of the above frames, and estimating gaze vectors of the parent object and child object based on the gaze start point coordinates of the head center point and the gaze endpoint coordinates. Learning method for evaluating video-based parent-child interaction.

15. In Paragraph 14, The top view contact point of the parent object bounding box is defined as the center point between the feet of the parent object, and the top view contact point of the child object bounding box is defined as the center point between the feet of the child object, Learning method for evaluating video-based parent-child interaction.

16. In Paragraph 14, The above top view contact point is calculated using the slope view contact point extracted from the frame and the transformation matrix M, Learning method for evaluating video-based parent-child interaction.

17. In Paragraph 14, The step of estimating the movement speed of a child object based on the location information of the top view contact point of the child object bounding box in at least two frames is: Estimating the movement speed of the child object by dividing the movement distance of the top view contact point of the child object bounding box during the at least two frames in the time interval between the at least two frames. Learning method for evaluating video-based parent-child interaction.

18. In paragraph 10, the artificial neural network module is configured by combining a Gated Recurrent Unit (GRU) model of the Recurrent Neural Network (RNN) family and a Convolutional Neural Network (CNN) model for processing time-series data, Learning method for evaluating video-based parent-child interaction.

19. An image processing unit that extracts at least one parent-child non-verbal interaction element from a frame constituting an evaluation video, and A module comprising an artificial neural network trained by the learning method of any one of paragraphs 10 to 18, Video-based parent-child interaction evaluation system.

20. In Paragraph 19, The above nonverbal interaction element includes at least one of the physical distance between the parent and the child, the child's movement speed, and the direction of gaze between the parent and the child. Video-based parent-child interaction evaluation system.

Images