The invention relates to the technical field of
robot perception, in particular to an intelligent accompanying human-type
robot based on multi-mode emotion interaction and a
perception method, and the method comprises the following steps: collecting
facial expression and
body posture data of a user through a
visual perception unit, collecting intonation, speed, volume and keyword data through a voice
perception unit, and carrying out the recognition of the speech, speed, volume and keyword data; data preprocessing and
feature extraction are carried out; secondly, carrying out
time sequence modeling on the extracted visual and voice features based on a
recurrent neural network, and generating visual and voice
time sequence feature vectors; the method comprises the steps that firstly,
time sequence features are extracted, then, a multi-mode fusion model based on an attention mechanism is adopted, the time sequence features are fused into a comprehensive emotion
feature vector used for emotion state classification, and finally, personalized interaction content such as voice, expressions and limb actions matched with the emotion state of a user is generated according to an
emotion classification result. According to the method, the accuracy of
robot emotion perception and the naturalness of man-
machine interaction are remarkably improved, and the method is suitable for multi-scene application of education,
medical treatment, old-age care and the like.