The invention relates to a gesture interaction system based on computer visions. The gesture interaction system comprises a video collection module, an image processing module, an identification and positioning module, and an interaction design module, wherein the video collection module is used for collecting the video images of a hand by a camera in a real-time way, the image processing module is used for processing the video information through a Gaussian filtration method, a binaryzation method and the like and obtaining hand contours as binary image for identification and tracking, the identification and positioning module is used for carrying out gesture identification on the hand contours through a PCA (principal component analysis) algorithm, and simultaneously tracking the position of a palm through a particle filtration algorithm, and the interaction design module is used for calling a mouse API (application programming interface) according to the gesture identification results and the position where the palm is located, to enable a mouse cursor to realize operation functions, such as move, left click, right double click, right click or no operation. The gesture interaction system has the advantages that the accuracy is high, the response speed is high, multiple interaction functions are realized, and the identification accuracy is high. The gesture interaction system can be used for the interaction of human-computer gestures inputted by the computer, and can also be used for other fields of system monitoring, mode identification and the like.