The invention discloses a personalized audio-visual fusion intelligent interaction
system based on multi-
modal data and a use method thereof. The
system comprises a multi-
modal data receiving module, a facial
feature extraction module, a three-dimensional model generation module, an audio
feature extraction module, a virtual
sound generation module, a
virtual image integration module and an interaction module. A user uploads pictures, videos and audios, the
system extracts face and audio features, virtual images and virtual sounds are generated, and the virtual images, the virtual sounds and the virtual sounds are adjusted and matched. And the interaction module displays the
virtual image, the user inputs the dialogue content, and the system generates a reply through an
algorithm and performs video and audio interaction through the
virtual image. The virtual image and
sound generation and interaction based on the multi-
modal data can be realized, so that the highly personalized and intelligent user experience is provided. According to the invention, not only can highly vivid virtual images and sounds be generated, but also natural and smooth user interaction can be realized through the intelligent interaction module, so that the quality and satisfaction of user experience are greatly improved.