The invention provides a multi-mode based emotion recognition method, comprising a data collection device, an output device and an emotion analysis software system, wherein the emotion analysis software system performs comprehensive analysis and reasoning on data obtained by the data collection device and finally outputs a result to the output device. The specific steps are as follows: an emotionrecognition step based on facial image expressions, an emotion recognition step based on voice signals, an emotion analysis step based on text semantic, an emotion recognition step based on human gestures, an emotion recognition step based on physiological signals, a semantic comprehension step based on multi-round dialogues, and a multi-mode emotion semantic fusion association judgment step basedon timing sequence. The multi-mode based emotion recognition method provided by the invention has the advantages of breaking through the five kinds of single-mode emotion recognition, innovatively performing comprehensive judgment on the information of multiple single modes by using a deep neural network through neural network coding, deep correlation and understanding, greatly improving the accuracy and being suitable for most general inquiry interaction application scenes.