Emotion recognition apparatus and method for recognizing facial emotion

Abstract

This emotion recognition apparatus comprises: an image processing module for generating a cropped image obtained by cropping an object of interest from an input image, a masking image in which a background excluding the object of interest from the input image is emphasized, and facial landmarks required to understand a facial structure and facial expression of the object of interest in the input image; an image and context feature extraction module for extracting image features for the object of interest from the cropped image and extracting context features for the background from the masking image; a landmark feature extraction module for extracting landmark features for a facial expression of the object of interest from the facial landmarks; a feature fusion module for combining the extracted image features, context features, and landmark features; and an emotion classification module for classifying a facial emotion of the object of interest on the basis of the combined features.

Description

Emotion recognition device and method for recognizing facial emotions

[0001] The present invention relates to an emotion recognition device and method for recognizing facial emotions.

[0002] With the recent advancement of artificial intelligence, research on emotion recognition technology utilizing various methods such as facial recognition, natural language processing, and voice analysis is actively underway. Emotion recognition technology refers to a technology that automatically detects and interprets human emotional states by utilizing diverse information such as video, audio, and biosignals; in particular, visual channels that utilize video information possess the advantages of high information density and a non-contact nature.

[0003] Facial expressions, a key element of communication through visual channels, rely heavily on facial landmarks—key points on the face such as the eyes, nose, mouth, eyebrows, and jawline—and these facial landmarks are essential for detecting fine muscle movements that indicate major emotions such as happiness, sadness, fear, surprise, disgust, and anger.

[0004] However, existing emotion recognition technologies fail to sufficiently integrate facial landmark data and image context attributes, and tend to overlook background information within images that could significantly improve the accuracy of emotion recognition.

[0005] For example, even in a photograph of a person smiling, considering the person's location and situation can provide richer information about their emotional state; however, many current emotion recognition technologies ignore background information or infer emotions by analyzing only facial expressions, which leads to problems with low reliability and accuracy.

[0006] In addition, existing emotion recognition technologies struggle to handle ambiguity when similar facial expressions, such as surprise and fear, can represent different emotions, and tend to overlook fine facial features that convey subtle emotions.

[0007] Therefore, to overcome the limitations of such emotion recognition technologies, research is needed on technologies that can more accurately recognize emotional states by integrating characteristics of the object of interest with contextual features.

[0008] The present invention was devised to solve the above-mentioned problems, and the objective of the present invention is to provide an emotion recognition device and method for recognizing facial emotions.

[0009] An emotion recognition device for recognizing facial emotions of an object of interest from an input image according to an embodiment of the present invention for achieving the above objective comprises: an image processing module that generates a cropped image in which an object of interest is cut out from the input image, a masked image that emphasizes the background excluding the object of interest in the input image, and facial landmarks necessary for understanding the facial structure and expression of the object of interest in the input image; an image and context feature extraction module that extracts image features for the object of interest from the cropped image and extracts context features for the background from the masked image; a landmark feature extraction module that extracts landmark features for the facial expression of the object of interest from the facial landmarks; a feature combining module that combines the extracted image features, context features, and landmark features; and an emotion classification module that classifies facial emotions of the object of interest based on the combined features.

[0010] A method for an emotion recognition device according to an embodiment of the present invention to achieve the above objective, which recognizes facial emotions of an object of interest from an input image, comprises: a step in which an image processing module generates a cropped image in which an object of interest is cut out from the input image, a masked image in which the background excluding the object of interest is emphasized from the input image, and facial landmarks necessary for understanding the facial structure and expression of the object of interest in the input image; a step in which an image and context feature extraction module extracts image features for the object of interest from the cropped image and extracts context features for the background from the masked image; a step in which a landmark feature extraction module extracts landmark features for the facial expression of the object of interest from the facial landmarks; a step in which a feature combining module combines the extracted image features, context features, and landmark features; and a step in which an emotion classification module classifies the facial emotions of the object of interest based on the combined features.

[0011] According to one aspect of the present invention described above, by providing an emotion recognition device and method for recognizing facial emotions, the accuracy and relevance of facial emotion recognition can be significantly improved, and it can play an important role in enhancing the driving safety of a vehicle driver by improving Human-Computer Interaction, and it can personalize the in-vehicle experience so that passengers can have a more comfortable and enjoyable time.

[0012] In addition, medical quality can be significantly improved by providing real-time emotional state information of patients on remote medical platforms, and customer service interactions can be enhanced by enabling digital assistants and customer service bots to respond in consideration of the user's emotional state.

[0013] Furthermore, by integrating emotion recognition technology into smart devices such as smartphones, home assistants, and virtual and augmented reality devices, a more personalized and highly responsive user experience based on the user's real-time emotional state can be provided. Additionally, emotion recognition technology can significantly improve the performance of security and surveillance systems by enhancing the ability to identify and respond to potential threats in public places.

[0014] Furthermore, emotion recognition systems can precisely identify a learner's emotional state in real time, enabling teachers or educational software to provide immediate, situation-appropriate feedback to improve learner engagement and learning efficiency.

[0015] FIG. 1 is a diagram showing the internal blocks of an emotion recognition device according to an embodiment of the present invention.

[0016] Figure 2 is a diagram illustrating the operation of the emotion recognition device of Figure 1 classifying emotion labels and predicting VAD attributes.

[0017] Figure 3 is a diagram illustrating the detailed operation of the image and context feature learning module of Figure 1.

[0018] Figure 4 is a diagram illustrating the detailed operation of the landmark feature learning module of Figure 1.

[0019] Figure 5 is a diagram illustrating the detailed operation of the attention module of Figure 1.

[0020] FIG. 6 is a flowchart illustrating the operation of an emotion recognition device recognizing facial emotions according to an embodiment of the present invention.

[0021] The following detailed description of the invention refers to the accompanying drawings, which illustrate specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that various embodiments of the invention are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in relation to one embodiment. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the following detailed description is not intended to be limiting, and the scope of the invention is limited only by the appended claims, including all equivalents to those claimed therein, provided appropriately described. Similar reference numerals in the drawings refer to the same or similar functions across various aspects.

[0022] The components according to the present invention are defined by functional distinction rather than physical distinction, and can be defined by the functions each performs. Each component may be implemented as hardware or as program code and processing units that perform each function, and the functions of two or more components may be included and implemented in a single component. Therefore, it should be noted that the names assigned to the components in the following embodiments are not intended to physically distinguish each component but are assigned to imply the representative function performed by each component, and that the technical concept of the present invention is not limited by the names of the components.

[0023] Preferred embodiments of the present invention will be described in more detail below with reference to the drawings.

[0024] FIG. 1 is a diagram showing the internal blocks of an emotion recognition device according to an embodiment of the present invention, FIG. 2 is a diagram explaining the operation of the emotion recognition device of FIG. 1 classifying emotion labels and predicting VAD attributes, FIG. 3 is a diagram explaining the detailed operation of the image and context feature learning module of FIG. 1, FIG. 4 is a diagram explaining the detailed operation of the landmark feature learning module of FIG. 1, and FIG. 5 is a diagram explaining the detailed operation of the attention module of FIG. 1.

[0025] An emotional recognition device is a device for recognizing facial emotions of an object of interest from an input image, and includes an image processing module (110), an image and context feature extraction module (120), a landmark feature extraction module (130), an attention module (140), a feature combining module (150), and an emotion classification module (160).

[0026] The image processing module (110) generates a cropped image in which an object of interest (person) is cut out from an input image, a masked image in which the background excluding the object of interest is emphasized from the input image, and face landmarks necessary to understand the face structure and expression of the object of interest in the input image.

[0027] The image and context feature extraction module (120) extracts image features for the object of interest from the cropped image and extracts context features for the background excluding the object of interest from the masked image.

[0028] As explained in more detail through FIG. 3, the image and context feature extraction module (120) is composed of an image feature extraction unit (122) that extracts image features and a context feature extraction unit (124) that extracts context features, and each of the image feature extraction unit (122) and the context feature extraction unit (124) includes at least one convolution layer (Conv), a pooling layer, and a specialized block.

[0029] The convolutional layer extracts image features by inputting a cropped image and extracts context features by inputting a masked image. Here, the image features include facial expressions, posture, movement information, etc., of the object of interest, and the context features include background locations, surrounding objects, situational information, etc. In particular, the context features enable a more accurate and multifaceted analysis of the emotion recognition device through information obtained from the background.

[0030] In an embodiment of the present invention, an efficient Convolutional Neural Network (CNN), such as Depthwise Separable Convolution, is used to significantly reduce computational load and memory usage, and to extract features faster and more resource-efficiently compared to a general CNN.

[0031] The pooling layer reduces the dimensionality of the feature map generated through the convolution operation of the convolution layer, thereby allowing feature information to be preserved while minimizing the computational load. In an embodiment of the present invention, max pooling is used to extract important information (maximum values) from the feature map and remove unnecessary information.

[0032] The special block applies higher weights to important features based on the feature map learned in the preceding layer and adaptively readjusts the said weights.

[0033] Thus, the feature information (Learnt feature) output through the convolutional layer, pooling layer, and special block represents both the image feature extracted from the image feature extraction unit (122) and the context feature extracted from the context feature extraction unit (124).

[0034] The landmark feature extraction module (130) extracts landmark features for the facial expression of an object of interest from the face landmarks generated by the image processing module (110). Here, the landmark features include the distance and ratio between landmarks, the change in angle according to the change in landmark position, the tilt and direction of major parts of the face, the symmetry of the facial expression, and the change in landmark position over time.

[0035] As explained in more detail with reference to FIG. 4, the landmark feature extraction module (130) extracts landmark features by applying a compound scaling method that dynamically adjusts the number of channels and the number of layers. In an embodiment of the present invention, by applying the compound scaling method, face landmarks are processed at various sizes and resolutions to extract richer landmark features, thereby optimizing the performance of the landmark feature extraction. Here, the number of channels and the number of layers represent the width and depth of the model, respectively.

[0036] Additionally, the landmark feature extraction module (130) outputs the extracted landmark features through a multi-scale fully connected layer (MS-FC). The multi-scale fully connected layer processes the input landmark features at various scales to capture both fine-scale features and coarse-scale features, thereby improving the performance of emotion recognition at various scales. Here, the fine-scale features may be, for example, a smile, eye movements, or subtle changes in the lips, and the coarse-scale features may be, for example, the overall shape of the face or significant changes in facial expressions.

[0037] In these multi-scale fully connected layers, scaling refers to processing input data at various levels of abstraction using different receptive fields. Here, a small receptive field captures details—that is, fine features—while a large receptive field captures broader features, such as facial symmetry. The size of the receptive field can be adjusted by changing the kernel size, dilation rate, or pooling operation, and this multi-scale approach provides a comprehensive feature representation that combines local and global contexts to improve landmark location accuracy.

[0038] The attention module (140) applies an attention mechanism to the image features and context features extracted by the image and context feature extraction module (120), and assigns a higher weight to the features that have higher importance in classifying the facial emotions of the object of interest among the image features and context features.

[0039] As explained in more detail through FIG. 5, the attention module (140) includes at least one spatial attention (SA) block and at least one channel attention (CA) block.

[0040] The spatial attention block emphasizes the face region by assigning a higher weight to the face region within the input image, and the channel attention block emphasizes the important feature channel by evaluating the importance of each feature channel and assigning a higher weight to the more important feature channel.

[0041] In this way, the attention module (140) applies a dual attention mechanism through a spatial attention block and a channel attention block, thereby enabling effective utilization of both spatial information and channel-based information.

[0042] The feature combination module (150) combines image features and context features extracted from the image and context feature extraction module (120) with landmark features extracted from the landmark feature extraction module (130), thereby enabling the emotion recognition device to understand facial emotions more comprehensively. At this time, the image features and context features refer to features to which an attention mechanism has been applied through the attention module (140).

[0043] The emotion classification module (160) classifies the facial emotion of the object of interest based on the features combined through the feature combining module (150). The emotion classification module (160) can be implemented as a classifier model trained to classify the facial emotion of the object of interest when combined features are input, and the classifier model can process continuous emotion attribute prediction such as VAD along with multi-label emotion classification using a loss function combining Binary Cross Entropy and Mean Square Error (MSE).

[0044] That is, the emotion classification module (160) classifies the facial emotion of the object of interest into a discrete emotion and outputs one of the predefined emotion labels. The term "discrete emotion" refers to clearly distinct individual emotional states, and the type of emotion can be defined by multiple emotion labels. Here, the emotion labels may be, for example, happiness, sadness, anger, fear, disgust, surprise, etc.

[0045] Additionally, the emotion classification module (160) outputs a predicted value of a continuous emotion attribute for the classified discrete emotions. The continuous emotion attribute can be represented as a VAD attribute including valence, arousal, and dominance, wherein valence indicates the positive or negative degree of the emotion, arousal indicates the intensity or activation degree of the emotion, and dominance indicates the degree of control or dominance felt over the emotion.

[0046] FIG. 6 is a flowchart illustrating the operation of an emotion recognition device recognizing facial emotions according to an embodiment of the present invention.

[0047] The image processing module of the emotion recognition device generates a cropped image in which an object of interest is cut out from an input image, a masked image in which the background excluding the object of interest in the input image is emphasized, and face landmarks necessary to understand the face structure and expression of the object of interest in the input image. (S601)

[0048] Then, the image and context feature extraction module extracts image features for the object of interest from the cropped image and context features for the background from the masked image (S603), and the landmark feature extraction module extracts landmark features for the facial expression of the object of interest from the face landmarks (S605).

[0049] Subsequently, the feature combining module combines image features, context features, and landmark features extracted from S603 and S605 (S607), and the emotion classification module classifies the facial emotion of the object of interest based on the features combined in S607 (S609).

[0050] The facial emotion recognition method of the present invention, as described above, can be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either individually or in combination.

[0051] The program instructions recorded on the above-mentioned computer-readable recording medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software.

[0052] Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, and flash memory.

[0053] Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

[0054] Although various embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention.

[0055] [Explanation of the symbol]

[0056] 110: Image processing module

[0057] 120: Image and Context Feature Extraction Module

[0058] 122 : Image feature extraction unit

[0059] 124 : Context Feature Extraction Unit

[0060] 130 : Landmark Feature Extraction Module

[0061] 140 : Attention Module

[0062] 150 : Feature combination module

[0063] 160: Emotion Classification Module

Claims

1. An emotion recognition device that recognizes the facial emotion of an object of interest from an input image, An image processing module that generates a cropped image in which an object of interest is cut out from the input image, a masked image in which the background excluding the object of interest is emphasized from the input image, and face landmarks necessary to understand the face structure and expression of the object of interest in the input image; An image and context feature extraction module that extracts image features for the object of interest from the cropped image and extracts context features for the background from the masked image; A landmark feature extraction module that extracts landmark features for the facial expression of the object of interest from the above face landmarks; A feature combining module that combines the extracted image features, context features, and landmark features; and An emotion recognition device comprising an emotion classification module that classifies the facial emotion of the object of interest based on the combined features described above.

2. In Paragraph 1, The above image and context feature extraction module is, It is composed of an image feature extraction unit for extracting the above image features and a context feature extraction unit for extracting the above context features, and An emotion recognition device comprising, wherein each of the image feature extraction unit and the context feature extraction unit comprises at least one convolution layer for extracting each of the image feature and context feature, at least one pooling layer for reducing the dimensionality of a feature map, and at least one specialized block for applying higher weights to important features according to the feature map.

3. In Paragraph 1, The above landmark feature extraction module is, The above landmark features are extracted by applying a compound scaling method that dynamically adjusts the number of channels and layers, and An emotion recognition device that outputs the above-mentioned extracted landmark features through a multi-scale fully connected layer.

4. In Paragraph 1, It further includes an attention module that applies an attention mechanism to the above image features and context features to assign higher weights to features with higher importance among the above image features and context features in classifying the facial emotion of the object of interest, and The above attention module is, An emotion recognition device comprising at least one spatial attention block that assigns a higher weight to a face region within the input image and at least one channel attention block that assigns a higher weight to an important feature channel.

5. In Paragraph 1, The above emotion classification module is, Classify the facial emotion of the object of interest as a discrete emotion and output one of the predefined emotion labels, and An emotion recognition device that outputs predicted values ​​of continuous emotion attributes for the discrete emotions classified above.

6. A method for an emotion recognition device to recognize the facial emotion of an object of interest from an input image, A step in which an image processing module generates a cropped image in which an object of interest is cut out from the input image, a masked image in which the background excluding the object of interest is emphasized from the input image, and face landmarks necessary for understanding face structure and expression; A step in which an image and context feature extraction module extracts image features for the object of interest from the cropped image and extracts context features for the background from the masked image; A landmark feature extraction module extracts landmark features for the facial expression of the object of interest from the face landmarks; A feature combining module combining the extracted image features, context features, and landmark features; and A face emotion recognition method comprising the step of classifying the face emotion of the object of interest based on the combined features of the emotion classification module.

7. In Paragraph 6, The above image and context feature extraction module is, It is composed of an image feature extraction unit for extracting the above image features and a context feature extraction unit for extracting the above context features, and A face emotion recognition method comprising, wherein each of the image feature extraction unit and the context feature extraction unit comprises at least one convolution layer for extracting each of the image feature and context feature, at least one pooling layer for reducing the dimensionality of a feature map, and at least one specialized block for applying higher weights to important features according to the feature map.

8. In Paragraph 6, The step of extracting the above landmark features is as follows: A step of extracting the above landmark features by applying a compound scaling method that dynamically adjusts the number of channels and layers, and A face emotion recognition method comprising the step of outputting the extracted landmark features through a multi-scale fully connected layer.

9. In Paragraph 6, After the step of extracting the image features and context features, the method further includes a step in which an attention module applies an attention mechanism to the image features and context features to assign a higher weight to the features that have higher importance in classifying the facial emotion of the object of interest among the image features and context features. The above attention module is, A face emotion recognition method comprising at least one spatial attention block that assigns a higher weight to a face region within the input image and at least one channel attention block that assigns a higher weight to an important feature channel.

10. In Paragraph 6, The step of classifying the facial emotions of the object of interest above is as follows: A step of classifying the facial emotion of the object of interest into a discrete emotion and outputting one of the predefined emotion labels, and A face emotion recognition method comprising the step of outputting a predicted value of a continuous emotion attribute for the discrete emotions classified above.

Images