Machine learning methods and systems for emotion recognition in a child

Abstract

Methods and systems for emotion recognition in a child are disclosed A machine learning method of emotion recognition in a child comprises: receiving video data of a scene including a child subject, the video data comprising video image data and video audio data; tracking facial images of the child subject in image frames of the video image data; identifying segments of the video audio data corresponding to sounds generated by the child subject; extracting audio features from the segments of the video audio data corresponding to sounds generated by the child subject; temporally aligning the extracted audio features with the facial images of the child subject to generate temporally aligned child audio-visual features; and classifying an emotion of the child subject by applying a machine learning classifier to the temporally aligned child audio-visual features.

Description

[0001] MACHINE LEARNING METHODS AND SYSTEMS FOR EMOTION RECOGNITION IN A CHILD

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to machine learning systems and methods for emotion recognition in a child. In particular, the present disclosure relates to using video data comprising video image data and video audio data to classify an emotion of a child subject.

[0004] BACKGROUND

[0005] Mental disorders, including Autism Spectrum Disorder (ASD), Attention Deficit Hyperactivity Disorder (ADHD), and developmental delays, are prevalent among young children. According to the Centers for Disease Control and Prevention (CDC), approximately 1 in 36 children in the United States is diagnosed with ASD. Additionally, the CDC reports that about 9.8% of children aged 3-17 years have received an ADHD diagnosis. Developmental delays, affecting various aspects of a child's growth, are also common, with studies suggesting that 1 in 6 children between the ages of 3 and 17 experience developmental disabilities. These statistics highlight the significant impact of mental disorders on the paediatric population, underscoring the need for effective interventions and treatments.

[0006] Mental disorders in children often present through characteristic emotional behaviours, offering a potential pathway for early diagnosis. Children with ASD often lack typical facial expressions to communicate their emotions, complicating the interpretation of their emotional states by others. Those with ADHD may demonstrate impulsive emotional responses, resulting in exaggerated reactions to minor frustrations or disappointments. Additionally, children with developmental delays may exhibit emotional behaviours that do not correspond with their chronological age, aligning more closely with those of younger children.

[0007] Machine learning (ML) has immense potential to enhance child emotion recognition (IER) and address the need for fundamental spatiotemporal dynamics analysis.Indeed, ML has been applied to affective computing for decades and has made tremendous progress. However, ML-based IER is still an underexplored area, primarily due to the lack of publicly available databases. So far, the largest public facial expression image dataset for children is Child Facial Expression in real World (CFEW), which contains 5.3K static images of children that are 0-3 years old. To deal with the data scarcity, one may circumvent the obstacle by directly utilizing off-the-shelf ML toolkits that are trained using adult-dominant databases, or proceed to establish a child-specific database followed by developing child-specific models. However, both options could suffer from their own issues.

[0008] Directly utilizing off-the-shelf ML toolkits could result in degraded performance due to domain shift. There are morphological differences between young children’s and adults’ faces. Children’s faces tend to have larger eyes, fatty cheek pads, smoother skin, fainter brows, and no beards. A direct employment of these tools often leads to false positives / negatives in face tracking and misclassification in emotion recognition.

[0009] Whereas establishing a child-specific database introduces a number of challenges that do not commonly arise. First and foremost, in a spontaneous setting, the unpredictable nature of child subjects often brings a greater degree of head pose variability and may require extra experimenters for caregiving. Second, the spontaneous activities are usually loosely structured, in which the interested emotional events could possess large variability in terms of quantity and temporal length across different children. This inevitable noisiness makes it challenging for any downstream ML-based processing. Last but not least, on top of the two options, child’s emotion expression is inherently a multimodal process, which requires the utilization of the complementarity of different modalities.

[0010] SUMMARY

[0011] According to a first aspect of the present disclosure a machine learning method of emotion recognition in a child is provided. The method comprises: receiving video data of a scene including a child subject, the video data comprising video image data and video audio data; tracking facial images of the child subject in image frames of the video image data; identifying segments of the video audio data corresponding tosounds generated by the child subject; extracting audio features from the segments of the video audio data corresponding to sounds generated by the child subject; temporally aligning the extracted audio features with the facial images of the child subject to generate temporally aligned child audio-visual features; and classifying an emotion of the child subject by applying a machine learning classifier to the temporally aligned child audio-visual features.

[0012] In an embodiment, identifying segments of the video data corresponding to sounds generated by the child subject comprises detecting child vocalisations.

[0013] In an embodiment, detecting child vocalisations comprises segmenting the video audio data into a plurality of time windows and classifying the audio data of time windows of the plurality of time windows.

[0014] In an embodiment, extracting audio features comprises generating a spectrogram.

[0015] In an embodiment, the spectrogram is a log-mel spectrogram.

[0016] In an embodiment the method further comprises capturing a calibration image of the child subject during a calibration sequence, and using the calibration image of the child subject to track facial images of the child subject.

[0017] IIn an embodiment, the machine learning classifier is trained to classify the emotion of the child subject as positive, negative or neutral.

[0018] In an embodiment, the child subject has an age of 0 to 36 months.

[0019] In an embodiment, the child subject has an age of 0 to 18 months.

[0020] According to a second aspect of the present disclosure, a computer readable medium carrying processor executable instructions which when executed on a processor cause the processor to carry out a method as described above is provided.According to a third aspect of the present disclosure a system for emotion recognition in a child is provided. The system comprises: a processor and a data storage device storing computer program instructions operable to cause the processor to: receive video data of a scene including a child subject, the video data comprising video image data and video audio data: track facial images of the child subject in image frames of the video image data; identify segments of the video audio data corresponding to sounds generated by the child subject; extract audio features from the segments of the video audio data corresponding to sounds generated by the child subject; temporally align the extracted audio features with the facial images of the child subject to generate temporally aligned child audio-visual features; and classify an emotion of the child subject by applying a machine learning classifier to the temporally aligned child audio-visual features.

[0021] In an embodiment, the data storage device further stores computer program instructions operable to cause the processor to identify segments of the video data corresponding to sounds generated by the child subject by detecting child vocalisations.

[0022] In an embodiment, detecting child vocalisations comprises segmenting the video audio data into a plurality of time windows and classifying the audio data of time windows of the plurality of time windows.

[0023] In an embodiment, the data storage device further stores computer program instructions operable to cause the processor to extract audio features by generating a spectrogram.

[0024] In an embodiment, the spectrogram is a log-mel spectrogram.

[0025] In an embodiment, the data storage device further stores computer program instructions operable to cause the processor to capture a calibration image of the child subject during a calibration sequence, and use the calibration image of the child subject to track facial images of the child subject.In an embodiment, the machine learning classifier is trained to classify the emotion of the child subject as positive, negative or neutral.

[0026] In an embodiment, the child subject has an age of 0 to 36 months.

[0027] In an embodiment, the child subject has an age of 0 to 18 months.

[0028] BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In the following, embodiments of the present invention will be described as non-limiting examples with reference to the accompanying drawings in which:

[0030] FIG.1 is a block diagram showing an overview of a machine learning system for emotion recognition in a child according to an embodiment of the present invention;

[0031] FIG.2 is a block diagram showing an emotion recognition system according to an embodiment of the present invention;

[0032] FIG.3 is a flowchart showing a machine learning method of emotion recognition in a child according to an embodiment of the present invention;

[0033] FIG.4 shows a machine learning network used in embodiments of the present invention;

[0034] FIG.5 is a table showing results of emotion recognition in child subjects using an embodiment of the present invention;

[0035] FIG.6 is a table showing how complementarity and attention modules affect the accuracy of an emotion recognition in an embodiment of the present invention;

[0036] FIG.7 is a table showing data demography after a split of events into subsets when training a machine learning model;FIG.8A to FIG.8C show visualizations of attention scores for three representative events using hierarchical attention layers;

[0037] FIG.9 is a table showing validation set performance across dyad-level 5-fold cross-validation of an embodiment of the present invention; and

[0038] FIG.10 is a table showing test set performance on child emotion recognition.

[0039] DETAILED DESCRIPTION

[0040] The present disclosure proposes an automatic emotion recognition system specifically designed for children, addressing the unique challenges associated with child emotion detection. The system effectively handles the dynamic movement patterns of children by employing a robust face tracking mechanism. It also isolates the child’s vocalizations from overlapping audio sources, tackling the complexities of childspecific vocal communication. Furthermore, the system integrates multimodal information using an attention-based deep learning model, enhancing the complementary strengths of visual and acoustic modalities for more accurate emotion recognition.

[0041] The system takes a video recording as input, processing both visual and audio modalities. For the visual modality, a face tracking system is utilized that includes a face template self-collection process, a face detector, and face recognition components. This enables clear facial images of the child during free interactions with parents to be obtained, despite their dynamic movements. For the audio modality, an effective audio tagging component is utilized to isolate the child’s utterances — both verbal and non-verbal — from severely overlapped audio recordings. Finally, an attention-based deep learning model fuses the visual and audio data, recognizing the child's emotional state over time by leveraging both modalities and temporal dynamics.

[0042] The system can be used to assess children’s emotions during natural interactions with parents or when the child is alone, providing objective, quantifiable scores to inform clinicians, caregivers, or parents of the child's emotional status. It is envisaged thatthe system will be particularly beneficial with young children, for example those aged 0-3 years or more preferably 0-18 months.

[0043] FIG.1 is a block diagram showing an overview of a machine learning system for emotional recognition in a child according to an embodiment of the present invention.

[0044] As shown in FIG.1, the emotion recognition system 100 takes video data 10 of a child subject and a calibration sequence 20 as inputs. The video data 10 comprises video image data and video audio data. The video data 10 comprises a video of a scene which includes the child subject. The scene may also include other individuals such as an adult interacting with the child subject. The calibration sequence 20 is designed to collect a diverse set of facial images for each participant under various head poses, which is essential for accurate face identification and tracking. The calibration sequence 20 may be captured by the same camera as the video data and functions as an interactive application that uses the camera to capture photos of the participant's face when specific pose requirements are met. As the participant adjusts their head to satisfy certain degree angles, the system automatically records images, resulting in a collection of facial images with varied orientations. The calibration sequence 20 therefore includes is a personalized face template for each participant, consisting of numerous images that enhance the robustness of face detection and recognition during free interactions, despite dynamic movements.

[0045] Let Icalibrepresent the set of calibration images captured for a participant Icalib=

[0046]

[0047] where each image lkis a color image of size h x w pixels with c color channels: Ik∈ Rh×w×c. Each image Ikis associated with a head pose angle θk, ensuring coverage over a range of poses: θk∈ [θmin, θmax].

[0048] These calibration images serve as reference templates in the face recognition process, facilitating accurate identification even when the participant's head orientation varies during interaction.

[0049] The emotion recognition system 100 comprises a face tracking module 122, a face feature generation module 124, a child vocalization detection module 126, an audioextraction / spectrum generator module 128, a temporal alignment module 130 and an emotion classifier module 132.

[0050] The face tracking module 122 is designed to detect and track the participant's face across video frames despite dynamic movements and occlusions. Spatially, the system first detects all faces in the current video frame using a face detector. It then compares the similarity of face embeddings and locations against the collected templates from the calibration sequence 20 and faces from previous frames to identify the target subject. Temporally, it utilizes tracking algorithms such as an Intersection over Union (loU) tracker to ensure consistency across frames. This approach allows the system to handle disappear-reappear scenarios, regaining correct identification when the participant re-enters the camera's field of view.

[0051] Let Ftdenote the current video frame at time t. The face detector identifies a set of face bounding boxes Bt= bt,1, bt,2, ..., bt,N, where each btnrepresents the bounding box coordinates of a detected face. For each detected face, a face embedding etnis extracted using a face recognition model. The similarity between the embedding etnand the stored templates Etempiateis computed using a similarity function S(et,n, Etemplate). The face with the highest similarity exceeding a threshold is identified as the target child subject. To maintain tracking over time, the loU tracker associates detected faces between consecutive frames based on the overlap of bounding boxes. The loU between bounding boxes

[0052]

[0053] and btis calculated as:

[0054]

[0055] By combining spatial face recognition with temporal tracking, the system effectively handles situations where the participant disappears and reappears in the video. When the face reappears, the system re-identifies it by matching the face embeddings with the stored templates, ensuring continuous and accurate face tracking.

[0056] The face feature generation module 124 refines and aligns the detected faces to standardize facial images for further analysis and recognition. This componentensures that facial features are consistently positioned across all images, enhancing the performance of subsequent emotion detection algorithms.

[0057] For each detected face bounding box btnat time t, the region of interest (ROI) is extracted from the video frame Ft, defined as:

[0058] ROIt,n= Ft[yt,n:yt,n+ ht,n, xt,n:xt,n+ wt,n]

[0059] where (xt,n, yt,n) is the top-left coordinate, and (wt,n, ht,n) are the width and the height of the bounding box. A facial landmark detector then identifies key points on the face within the ROI, resulting in a set of landmarks Lt,n= {(x1,y1), (x2,y2),...,(xK,yK)}, where K is the number of landmarks (e.g., 68 landmarks). An affine transformation Ttnis computed to align the landmarks Ltnto a reference shape Lref. The transformation is applied to the ROI to obtain the aligned face image:

[0060] It,n= Tt,n(ROIt,n)

[0061] This process includes scaling, rotation, and translation to ensure that the facial features are in consistent positions across all images. The output is a set of aligned facial images ltnready for emotion recognition.

[0062] The child vocalization detection module 126 utilizes a deep learning audio tagging model trained on a large-scale dataset comprising a wide array of common audio classes and child-specific vocalizations. This comprehensive training enables the model to effectively distinguish between the child’s non-verbal vocalizations and the parent's child-directed speech (baby talk), despite their acoustic similarities.

[0063] The component processes the continuous audio stream A = {a1,a2,...,aT}, where atrepresents the audio sample at time t. The audio is segmented into short-time frames using a sliding window approach, resulting in frames F = {fx,f2,...,fN}. For each frame fn, acoustic features such as Mel-frequency cepstral coefficients (MFCCs) are extracted to form feature vectors Xn. These feature vectors are input into the deeplearning model, which outputs a probability distribution over the set of audio classes C, including classes specific to child vocalizations and adult speech:

[0064] Pn= 퓜(Xn), Pn∈ R|C|

[0065] The model assigns higher probabilities to the classes that best represent the audio content of each frame. To distinguish between the child's vocalizations and the parent's baby talk, the model leverages distinctive acoustic patterns inherent in nonverbal child sounds — such as fundamental frequency, formant structures, and temporal dynamics — that differ from adult speech even when modified by child-directed speech.

[0066] By aggregating the frame-level predictions, the component identifies segments where the child's vocalizations are present. These segments are then isolated to produce the child-only audio stream A* = {a*1, a*2,...,a*T*}, which can be further processed for emotion recognition.

[0067] The audio extraction / spectrum generator module 128 processes the extracted child-only audio segments to generate log-mel spectrograms. These spectrograms provide a time-frequency representation of the audio signal, essential for various audio analysis tasks. The input to this component is the set of child-only audio segments denoted as A* = {a*1, a*2,

[0068]

[0069] where each amrepresents an audio segment. Each segment is windowed into frames of 960 ms duration with an appropriate step size. For each frame, a short-time Fourier transform (STFT) is applied, followed by mapping the resulting spectrogram onto 64 mel-spaced frequency bins and log-transforming the magnitudes to produce the log-mel spectrograms, denoted as S* = {s*1,...,s*M}.

[0070] The temporal alignment module 130 prepares and synchronizes input data from different modalities — visual and audio — according to the requirements of the multimodal emotion recognition model. It aligns the facial images and corresponding audio features (such as log-mel spectrograms) in time, ensuring that each visual frame matches the appropriate audio segment. This temporal synchronization is crucial foreffective multimodal fusion, allowing the downstream model to accurately integrate visual and auditory cues for reliable emotion recognition.

[0071] The emotion classifier module 132 processes chunks of sampled visual and audio data points to detect emotions in a three-class classification framework. Each chunk consists of 12 video frames and 12 corresponding log-mel spectrograms, representing 12 data points.

[0072] The input to this component is a set of synchronized visual and audio data chunks, {(!□., Si), (l2, S2),..., 0i2, S12)}, where each pair (lt, St) represents a video frame and its corresponding log-mel spectrogram.

[0073] The deep learning model for emotion detection has two branches: one for visual data and one for audio data. Each branch consists of a backbone network followed by linear layers. The backbone network in the visual branch processes the facial images, while the backbone network in the audio branch processes the log-mel spectrograms. The outputs of the backbone networks can be denoted as Vtfor visual features and Atfor audio features at time t. Mathematically, the processing in each branch can be represented as:

[0074] Vt= Backbonev(It)

[0075] At= BackboneA(St)

[0076] The outputs from each branch are then passed through linear layers to further transform the features. Let Lv(Vt) and LA(At) represent the outputs of the linear layers for the visual and audio branches, respectively. These outputs are then concatenated and fused by the modality fusion layer:

[0077] Et= Fuse(Lv(Vt), LA(At))

[0078] The fused features Etare further integrated across the temporal dimension by the temporal fusion layer, which considers the temporal context of the chunk. This can be represented as:Etemp = TemporalFuse({E1, E2,..., E12})

[0079] Finally, the classifier takes the temporally fused features Ftempand predicts the emotion class for the chunk. The classifier's output is a three-class prediction representing the detected emotion. The output of this component is the emotional status for each chunk of 12 data points.

[0080] FIG.2 is a block diagram showing an emotion recognition system according to an embodiment of the present invention. The emotion recognition system 100 is a computer system with memory that stores computer program modules which implement methods for emotion recognition in a child according to embodiments of the present invention.

[0081] The emotion recognition system 100 comprises a processor 110, a working memory 112, an input interface 114, a user interface 116, and program storage 120. The processor 110 may be implemented as one or more central processing unit (CPU) and I or graphics processing unit (GPU) chips. The program storage 120 is a non-volatile storage device such as a hard disk drive which stores computer program modules. The computer program modules are loaded into the working memory 112 for execution by the processor 110. The input interface 114 is an interface which allows data, including the video data 10 and calibration sequence 20 shown in FIG.1 to be received by the emotion recognition system 100. The user interface 116 allows results of the emotion recognition processing to a user and may also allow the user to make input or selections relating to the processing.

[0082] The program storage 120 stores a face tracking module 122, a face feature generation module 124, a child vocalization detection module 126, an audio extraction / spectrum generator module 128, a temporal alignment module 130 and an emotion classifier module 132. The computer program modules cause the processor 110 to execute emotion recognition processing which is described in more detail below. The program storage 120 may be referred to in some contexts as computer readable storage media and / or non-transitory computer readable media. As depicted in FIG.2, the computerprogram modules are distinct modules which perform respective functions implemented by the emotion recognition system 100. It will be appreciated that the boundaries between these modules are exemplary only, and that alternative embodiments may merge modules or impose an alternative decomposition of functionality of modules. For example, the modules discussed herein may be decomposed into sub-modules to be executed as multiple computer processes, and, optionally, on multiple computers. Moreover, alternative embodiments may combine multiple instances of a particular module or sub-module. It will also be appreciated that, while a software implementation of the computer program modules is described herein, these may alternatively be implemented as one or more hardware modules (such as field-programmable gate array(s) or application-specific integrated circuit(s)) comprising circuitry which implements equivalent functionality to that implemented in software.

[0083] Although the emotion recognition system 100 is described with reference to a computer, it should be appreciated that the emotion recognition system 100 may be formed by two or more computers in communication with each other that collaborate to perform a task. For example, but not by way of limitation, an application may be partitioned in such a way as to permit concurrent and / or parallel processing of the instructions of the application. Alternatively, the data processed by the application may be partitioned in such a way as to permit concurrent and / or parallel processing of different portions of a data set by the two or more computers. In an embodiment, virtualization software may be employed by the emotion recognition system 100 to provide the functionality of a number of servers that is not directly bound to the number of computers in the emotion recognition system 100. In an embodiment, the functionality disclosed above may be provided by executing the application and / or applications in a cloud computing environment. Cloud computing may comprise providing computing services via a network connection using dynamically scalable computing resources. A cloud computing environment may be established by an enterprise and / or may be hired on an as-needed basis from a third-party provider.

[0084] FIG.3 is a flowchart showing a machine learning method of emotion recognition in a child according to an embodiment of the present invention. The method 300 shown in FIG.3 is carried out by the emotion recognition system 100 shown in FIG.2.In step 302, the input interface 114 of the emotion recognition system 100 receives video data 10. The video data 10 comprises video image data and video audio data of a scene including the child subject. As discussed above, the scene may also include other individuals.

[0085] In step 304, face tracking module 122 is executed by the processor 110 to track facial images of the child subject in image frames of the video image data.

[0086] In step 306, the child vocalization detection module 126 is executed by the processor 110 to identify segments of the video audio data which correspond to sounds generated by the child subject.

[0087] In step 308, the audio extraction module 128 is executed by the processor 110 to extract audio features from the segments of the video audio data corresponding to sounds generated by the child subject. The audio features may be extracted by generating spectrograms such as mel-spectrograms.

[0088] In step 310, the temporal alignment module 130 is executed by the processor 110 to temporally align the extracted audio features with the facial images of the child subject to generate temporally aligned child audio-visual features.

[0089] In step 312, the emotion classifier module 132 is executed to classify the emotion of the child subject. The classification of the emotion of the child subject may be output to the user interface 116 of the emotion recognition system. The classification of the emotion of the child subject may be a classification of whether the emotion of the child subject is negative, positive or neutral.

[0090] FIG.4 shows a machine learning network used in embodiments of the present invention. The attention weights are denoted in grey scales. A darker colour denotes a higher weight. For a conceptual demonstration, the last video frame misses the child’s face, and its corresponding weight from the modal and temporal attention modules are drawn with lighter colour, indicating a lower saliency.As shown in FIG.4, the machine learning network 400 comprises a multimodal feature extractor 410, a cross-modal attention module 430 and a temporal attention module 450.

[0091] The multimodal feature extractor 410 takes video frames 412 and audio log-mel spectrograms 414 as input.

[0092] Children’s emotion expression is inherently a multimodal process, in which different modalities complement each other. Thus, to facilitate downstream multimodal representation learning, extracting features from multimodal input becomes both natural and imperative. The proposed design, a pretrained CNN (ResNet50 network) 416 and a pretrained CNN (VGG network) 418 are employed as the visual and audio backbones, respectively, to extract visual and audio features. These backbones were pretrained using large-scale facial image and audio databases.

[0093] Specifically, given T facial images 412 and their corresponding log-mel spectrograms 414, the extractor initially produces T x F dimensional visual features 420 and T x F dimensional audio features 422, respectively. These are then concatenated 424 to form a T x 2F multimodal feature vector 426.

[0094] To pivot attention to the available modality and extract meaningful representation, the modal attention module 430 is designed, incorporating a linear layer 432 followed by a SoftMax activation layer 434. Specifically, given the T x 2F multimodal feature vector, the module calculates a T x 2 attention score 436 for each time step, with higher values signifying a greater degree of information. The weighted sum 438 between the multimodal feature vector 426 and the attention score 436 effectively fuses the features, resulting in a T x F fused feature vector 440.

[0095] Utilizing a sequence of data points, rather than a single static point, offers a richer understanding of temporal dynamics and helps resolve ambiguities inherent in static data points. In the present context, it is crucial for the model to focus on the most informative temporal segments associated with an emotional event, such as prioritizing a sequence of smiling frames over neutral ones when learning about apositive event. Therefore, T consecutive data points are consumed to make one prediction, which is based on the most informative temporal segment following the same attention mechanism. The proposed temporal attention module 450 also consists of a linear layer 452 and a soft-max activation layer 454. Specifically, given the T x F fused feature 440, the temporal attention module 450 produces a T x 1 attention score 456, with higher values indicating more informative temporal segments. After the weighted sum 458, an 1 x F feature vector 460 is obtained, which then facilitates the 3-class prediction 462. Note that in this work, T - 12 is used as the sequence length. A sliding window is used to sample the event data into samples, with both the window size and stride equalling 12.

[0096] The visual backbone takes as input the facial video frames (25 fps), obtained by the cascaded use of YOLO and OpenFace. The audio backbone takes as input the audio log-mel spectrograms (100 fps). The audio data is prepared through the following steps: Given a 960ms mono audio clip sampled at 16K Hz, it undergoes decomposition using a short-time Fourier transform (STFT) with a 25ms sliding window and a 10ms step. The resulting spectrogram is then mapped onto 64 mel-spaced frequency bins, producing a log-mel spectrogram with dimensions of 96 x 64. Given the 10ms step size for the sliding window, the resulting spectrogram has a frame rate of 100 fps, which is four times the frame rate of the video. To synchronize the video and spectrogram frames, the data loader employs the relationship y - 4x + b, where y and x denote the indices of the video and spectrogram frames, respectively, and 0 < b < 4 is a random offset that is shared within a minibatch.

[0097] To accelerate the process, the 512-dimensional and 128-dimensional features from the visual and audio backbones, respectively, are extracted offline and saved to disk. During training, these features are fed into the downstream layers. Given that we used the leave-one-out protocol, this resulted in 107 folds for the complete Subset U. The training is conducted over 20 epochs for each fold, with a batch size of 32. The Adam optimizer is utilized, starting with an initial learning rate of le - 3. A linear warm-up strategy is applied during the first 5 epochs. Subsequently, the PyTorch ReduceOn Plateau scheduler is employed to adjust the learning rate, with the minimum learning rate set to le - 4, the factor to le - 1, and the patience to 3. To mitigate therisk of overfitting, the best model state from the previous epoch is reloaded at the beginning of each new epoch.

[0098] The system has been tested using the dataset collected from 107 children (48 Singaporean and 59 Brazilian). Two scenarios are employed to evaluate the models' generalization ability. In the normal scenario, the model is trained and tested using all the children’ data. In the cross-cultural scenario, the model is trained using children from one country and tested with children from another country. The two scenarios both employ leave-one-out protocol for model training. The test set always contains samples from one child, and the remaining samples are randomly shuffled, then split with an 80% / 20% ratio to comprise the training and validation sets, respectively.

[0099] FIG.5 is a table showing results of emotion recognition in child subjects using an embodiment of the present invention. The results were produced on 3-class emotion recognition in normal and cross-cultural scenarios. Acc: the accuracy. F1: the f1 -score. Bold fonts indicate the best results. SG: Singaporeans. BR: Brazilians.

[0100] The method End2you corresponds to Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B. W., & Zafeiriou, S. (2017). End-to-end multimodal emotion recognition using deep neural networks. IEEE Journal of selected topics in signal processing, 11(8), 1301-1309.

[0101] The method MulT corresponds to Tsai, Y. H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L. P., & Salakhutdinov, R. (2019, July). Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for computational linguistics. Meeting (Vol. 2019, p. 6558). NIH Public Access.

[0102] The method ShaSpec corresponds to Wang, H., Chen, Y., Ma, C., Avery, J., Hull, L., & Carneiro, G. (2023). Multi-modal learning with missing modality via shared-specific feature modelling. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 15878-15887).

[0103] In the normal scenario, the proposed multimodal emotion detection method achieved the highest accuracy and F1 -score, demonstrating its superior performance in emotionrecognition compared to state-of-the-art methods. Specifically, we achieved an accuracy of 70.7% and an F1 -score of 72.6%, outperforming ShaSpec, which had the second-best results with an accuracy of 67.9% and an F1 -score of 68.6%. Both End2you and MulT exhibited lower performance metrics with higher standard deviations, indicating greater sensitivity to the dataset's latent structure.

[0104] In the cross-cultural scenario, the proposed method showed the highest performance, highlighting its promising generalization ability. When trained with Singaporean children and tested with Brazilian children, our method achieved an accuracy of 67.4% and an F1 -score of 67.6%, outperforming ShaSpec, which achieved an accuracy of 67.2% and an F1 -score of 66.3%. Additionally, when trained with Brazilian children and tested with Singaporean children, our method achieved an accuracy of 61.5% and an F1 -score of 61.3%, again outperforming ShaSpec and other methods.

[0105] Overall, while there is a general trend of decreased performance in cross-cultural scenarios across all methods, our method demonstrates robustness and superior performance in both normal and cross-cultural scenarios. These promising preliminary results indicate the potential effectiveness of our method in emotion recognition tasks across diverse cultural settings.

[0106] FIG.6 is a table showing how complementarity and attention modules affect the accuracy of an emotion recognition in an embodiment of the present invention.

[0107] It was also tested how the complementarity and attention modules affect the accuracy and F1 -score of our emotion recognition model. The scenarios include Audio-only, Visual-only, without the temporal attention module (W / o Temp), without both modal and temporal modules (W / o Mod. And Temp.), and the complete model that integrates all features and layers. In the Audio-only and Visual-only scenarios, a dummy zero tensor would be employed to replaces the input from one modality. In the scenario without the temporal attention layer, the model would infer in a frame-wise manner. When omitting both modal and temporal attention layers, the features from both modalities are simply concatenated.It can be seen that when the model is fed only with audio data, it produces the lowest accuracy and f1 -score among all the scenarios, which indicates that while audio information contributes to emotion recognition, it is less effective when used in isolation. Whereas the results from visual-only suggests that visual cues are more informative for emotion recognition than audio cues when used alone. And when the model is composed without the modal and temporal attention modules, the performance both decrease. Overall, the results show a clear trend: the integration of both audio and visual modalities, along with the use of attention mechanisms (both modal and temporal), results in the highest performance for emotion recognition.

[0108] In this section, the acquisition of the database is introduced, which serves as the foundation for the IER model training and cross-cultural analyses. The acquisition comprises three major steps: i) videotaping and event annotation, ii) child face tracking, and iii) data splitting.

[0109] To efficiently induce spontaneous emotions in babies, a mother- child interactive task is employed. The context of dyadic interaction not only provides a familiar caregiver to interact but is also proven to be helpful in enhancing the child’s affective expressions. Overall, 59 Brazilian and 48 Singaporean mother-child dyads were recruited for data acquisition. The children range from 7.2 to 25.7 months old (M (mean) =14.2m, SD (standard deviation) =4.4m). Among them, the Brazilian children range from 7.2 to 16.2 months old (M=12.1m, SD=2.3m), and the Singaporean children from 11.2 to 25.7 months old (M=16.7m, SD=5.0m). Specifically, the database is acquired by videotaping the mother-child interaction during the A-not-B (AnB). The task consists of 12 trials. In each trial the mother hides a toy inside one of two tubs and covers both tubs simultaneously with identical lids, then asks her child to guess where the toy is hidden. A camera with 4K resolution and 25 fps is used to videotape the frontal view of the child. The collected video recordings are then annotated into positive, negative, and neutral emotional events. A positive emotional event can refer to smiling, grunts, cooing, etc., while a negative emotional event can refer to anger, sadness, crying, whining, fussing noises, etc. An event could last from seconds to minutes. There is no fixed number and duration of labelled events for each child.To obtain child facial images from the video events, in which the large head pose variability and presence of experimenters / caregivers are expected, YOLOv7 is employed as the child face tracker. 50 video frames for each child are sampled and manually labelled with bounding boxes for finetuning the YOLOv7 model. Finally, with the child facial bounding boxes detected by the finetuned YOLO, OpenFace which was pretrained using multiple databases, is used to further align them in a consistent coordinate system and mask out the background. Overall, 74.3% of the video frames from the entire database are successfully extracted, resulting in 1.5 million cropped and aligned child facial images.

[0110] Inspired by research indicating that human facial expressions usually last for 0.5-4s, the annotated events are divided into two subsets, U and V. Subset U comprises short events that are 0.5-1 Os long, suitable for IER model training due to its consistency. The decision to extend the duration up to 10 seconds, rather than limiting it to 4 seconds, represents a trade-off between achieving consistency and retaining a sufficient amount of data for effective IER model training. Subset V comprises long events that are longer than 10s, suitable for validating the trained IER model as well as conducting cross-cultural analysis due to its rich temporal information.

[0111] FIG.7 is a table showing data demography after a split of events into subsets when training a machine learning model. The demography of the data after the split. No. S.: number of subjects. No. E.: number of events. No. F.: number (K) of video frames. M. STD: the mean length and standard deviation of events.

[0112] FIG.7 provides the data demography after the split. It can be seen that Subset U features a consistent duration of events, as both the mean and standard deviation are single-digit, whereas Subset V possesses two- or even three-digit counterparts.

[0113] FIG.8A to FIG.8C show visualizations of attention scores for three representative events using hierarchical attention layers. The thumbnail images indicate the key frame of the emotional events. The line graphs show the visual, audio, and temporal attention scores in percentage. And the log-mel spectrograms demonstrate the information contained in the corresponding audio segments. The attention scores are able to attend to the most informative modalities and temporal segment.In FIG.8A, the selected event was labelled as a negative event, in which a child typically presents crying, whining, and unhappy facial expressions. It can be seen that before 1.7 seconds, the child was showing a neutral face with vocal expressions (as can be seen from the wave-like structure in the log-mels), and the model attends to the latter, which is a more informative modality towards the event category. From 1.7 to 3.4 seconds, the child turns to an obviously unhappy face, during which the visual attention score is prioritized. When the child turns to a neutral face again in the end, the audio attention score rises again. Throughout the whole event, the temporal scores show a strong response towards the informative log-mel segments.

[0114] In FIG.8B, initially, the child exhibits a neutral expression, and the visual attention score is at a moderate level (approximately 0.6). Starting from 0.6s, as the child smiles, the visual attention score escalates to a high level (around 0.9). Then, from 3s onward, when the child returns to a neutral expression, the visual attention score decreases accordingly.

[0115] In FIG.8C, up until 2s, the child’s mouth is obscured, resulting in a low visual attention score (ranging from 0.2 to 0.5). Simultaneously, the log-mel spectrogram reveals substantial information due to crying, characterized by its wave-like structure, and consequently, the audio attention score is high (approximately 0.8). Beginning at 2s, as the child’s mouth becomes visible, there is a noticeable increase in the visual attention score.

[0116] In the three examples shown in FIG.8A to FIG.8C, it is evident that segments with high temporal attention scores typically align with segments of the spectrograms rich in audio information, indicating the model’s tendency to focus on audio-enriched segments in the temporal dimension.

[0117] As described above, the present disclosure provides a novel multi-dataset, multi-task training methodology that substantially improves the robustness and generalization capability of the visual emotion recognition component. The training framework leverages four complementary datasets spanning both adult and child populations. The categorical labels represent three emotion classes (Positive, Neutral, Negative),while the continuous labels represent Valence and Arousal values in the range [-1, +1], A key technical advancement is the model's ability to perform multi-task learning, simultaneously handling both discrete emotion classification and continuous valencearousal regression within a single unified architecture.

[0118] FIG.9 is a table showing validation set performance across dyad-level 5-fold cross-validation of an embodiment of the present invention.

[0119] AFF: AffectNet; Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2017). Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 10(1), 18-31.

[0120] LEAP: an adult-infant emotion dataset with 3-class categorical labels.

[0121] CFEW-CLS: Tang, C„ Li, S„ Zheng, W„ Zong, Y„ Zhang, S„ Lu, C„ & Zhao, Y. (2023). CFEW: A large-scale database for understanding child facial expression in real world. IEEE Transactions on Affective Computing, 15(3), 990-1003.

[0122] F1: F1 -score; CCC: Concordance Correlation Coefficient. The complete AffectNet, CFEW 3-cls, and CFEW VA datasets are combined with each LEAP fold. LEAP folds are split at the dyad level, ensuring that test dyads are unseen during training.

[0123] FIG.10 is a table showing test set performance on LEAP (child emotion recognition).

[0124] The results shown in FIG.9 and FIG.10 demonstrate consistent cross-fold performance (F1 range: 0.67–0.73), effective transfer learning from adult emotion datasets to child emotion recognition, and high multi-task accuracy achieving strong performance on both classification (F1 up to 0.97) and continuous VA prediction (CCC up to 0.95) simultaneously.

[0125] Whilst the foregoing description has described exemplary embodiments, it will be understood by those skilled in the art that many variations of the embodiments can be made within the scope and spirit of the present invention.

Claims

CLAIMS1. A machine learning method of emotion recognition in a child, the method comprising:receiving video data of a scene including a child subject, the video data comprising video image data and video audio data;tracking facial images of the child subject in image frames of the video image data;identifying segments of the video audio data corresponding to sounds generated by the child subject;extracting audio features from the segments of the video audio data corresponding to sounds generated by the child subject;temporally aligning the extracted audio features with the facial images of the child subject to generate temporally aligned child audio-visual features; and classifying an emotion of the child subject by applying a machine learning classifier to the temporally aligned child audio-visual features.

2. The method according to claim 1, wherein identifying segments of the video data corresponding to sounds generated by the child subject comprises detecting child vocalisations.

3. The method according to claim 2, wherein detecting child vocalisations comprises segmenting the video audio data into a plurality of time windows and classifying the audio data of time windows of the plurality of time windows.

4. The method according to any preceding claim, wherein extracting audio features comprises generating a spectrogram.

5. The method according to claim 4, wherein the spectrogram is a log-mel spectrogram.

6. The method according to any preceding claim, further comprising capturing a calibration image of the child subject during a calibration sequence, and using the calibration image of the child subject to track facial images of the child subject.

7. The method according to any preceding claim, wherein the machine learning classifier is trained to classify the emotion of the child subject as positive, negative or neutral.

8. The method according to any preceding claim, wherein the child subject has an age of 0 to 36 months.

9. The method according to claim 8, wherein the child subject has an age of 0 to 18 months.

10. A computer readable medium carrying processor executable instructions which when executed on a processor cause the processor to carry out a method according to any one of claim 1 to 9.

11. A system for emotion recognition in a child, the system comprising: a processor and a data storage device storing computer program instructions operable to cause the processor to:receive video data of a scene including a child subject, the video data comprising video image data and video audio data;track facial images of the child subject in image frames of the video image data;identify segments of the video audio data corresponding to sounds generated by the child subject;extract audio features from the segments of the video audio data corresponding to sounds generated by the child subject;temporally align the extracted audio features with the facial images of the child subject to generate temporally aligned child audio-visual features; and classify an emotion of the child subject by applying a machine learning classifier to the temporally aligned child audio-visual features.

12. The system according to claim 11, wherein the data storage device further stores computer program instructions operable to cause the processor to identify segmentsof the video data corresponding to sounds generated by the child subject by detecting child vocalisations.

13. The system according to claim 12, wherein detecting child vocalisations comprises segmenting the video audio data into a plurality of time windows and classifying the audio data of time windows of the plurality of time windows.

14. The system according to any one of claims 11 to 13, wherein the data storage device further stores computer program instructions operable to cause the processor to extract audio features by generating a spectrogram.

15. The system according to claim 14, wherein the spectrogram is a log-mel spectrogram.

16. The system according to any one of claims 11 to 15, wherein the data storage device further stores computer program instructions operable to cause the processor to capture a calibration image of the child subject during a calibration sequence, and use the calibration image of the child subject to track facial images of the child subject.

17. The system according to any one of claims 11 to 16, wherein the machine learning classifier is trained to classify the emotion of the child subject as positive, negative or neutral.

18. The system according to any one of claims 11 to 17, wherein the child subject has an age of 0 to 36 months.

19. The system according to claim 18, wherein the child subject has an age of 0 to 18 months.

Images