The invention relates to a shaking detection algorithm based on multi-feature fusion. The algorithm comprises the following steps of: initializing an undistorted video frame and a video frame to be detected; then, sequentially extracting a brightness similarity factor, a contrast similarity factor and a structure similarity factor to obtain a similarity metric function, wherein the contrast similarity factor is obtained after the average brightness is removed; and meanwhile, judging that the video frame of a signal to be detected does not shake, or judging that the video frame of the signal to be detected shakes. According to the shaking detection algorithm, the brightness similarity factor, the contrast similarity factor and the structure similarity factor of the video frame of a reference signal and the video frame of the signal to be detected are sequentially extracted, the similarities of the three kinds of extracted variable information are compared, and finally, the three comparison results are combined, so that a similarity index is obtained and used as an evaluation criterion for image quality, i.e., indicating whether shaking happens or not.