The invention relates to the field of computer vision and intelligent human-machine interaction, in particular to a human-machine interaction system based on machine vision and an interaction method of the system, and provides a method for carrying out static sign language letter recognition based on an improved SURF algorithm by combining a Kinect sensor. The Kinect sensor collects the depth image of a target area to carry out hand pixel area division, and interference caused by illumination changes and complex backgrounds can be eliminated. The improved SURF algorithm is used for extracting feature points, meanwhile, the self-adaption radius r is set, SURF feature points are screened grade by grade in the neighbourhood with r as the radius by comparing the number of the feature points and the feature point distance, the recognition rate is greatly improved, and the robustness of recognition work on the skin color, the illumination changes, the complex backgrounds and other environmental factors, angle changes and scale changes is guaranteed. To solve the problem that SURF feature vector dimensions are high, an SVM one-to-one classification method is adopted, SURF feature descriptors are classified and trained, and a recognition result is obtained.