The invention discloses a bimodal
emotion recognition method and
system based on 3D convolutional neural networks. According to the method, first, two 3D convolutional neural networks for face emotionrecognition and body gesture
emotion recognition are constructed respectively, and
network model parameters are optimized based on a
training set and a
verification set of a bimodal face and body gesture
database; second, the two neural networks obtained after optimization are tested based on a
test set of the bimodal face and body gesture
database to obtain a face
emotion recognition confusion matrix and a body gesture emotion recognition
confusion matrix; and last, priori knowledge of the face emotion recognition
confusion matrix and the body gesture emotion recognition
confusion matrix isutilized to fuse bimodal recognition results of a newly input face
video sequence and body gesture
video sequence, and a bimodal
emotion classification result is obtained. Through the method, the 3D convolutional neural networks and a bimodal fusion
algorithm are adopted, the subjectivity of manual
feature design is avoided, the limitation of single-
modal emotion recognition is overcome, and the accuracy and robustness of emotion recognition can be effectively improved.