The invention provides a pedestrian detecting and tracking method based on multi-stage detection. The method comprises the following steps that 1), background frames are extracted and preprocessed; 2), foreground regions are extracted and labeled through a rectangular frame, current frames and the background image frames are used for carrying out difference, foreground information is extracted, binarization processing is carried out on the foreground information, all the communicated foreground regions are labeled through the outer contour, a minimum bounding rectangle is drawn for each contour region, size information of all the rectangles is recorded, a differential image gray value and a threshold value T are compared, parts larger than T are determined as motion object parts, or else the parts are background parts; 3), the external characteristics of pedestrians are utilized for judging target regions preliminarily; 4), statistics is made on the number of the rectangular regions Rn judged to be the similar target regions, a pedestrian detecting algorithm based on the HOG characteristics is adopted for ruling out non-target regions, and then the optical flow method is adopted for tracking the remaining target regions. According to the method, the calculation speed is high, the instantaneity is good, and the practicality is high.