The invention discloses an object scale self-adaption tracking method based on a spatial-temporal model, and the method comprises the following steps: beginning a video, reading in a first frame of image, manually assigning the rectangle position of a tracked object; then based on context airspace, initializing the spatial-temporal model and a multiscale history object template library; next reading in a next frame of image, building the spatial-temporal model by iteration, calculating a confidence map, and estimating an object center position; then according to the history object template library, judging a template optimal scale, determining the rectangle position of the object, completing tracking of the current frame of object, and updating scale parameters of the spatial-temporal model and the multiscale history object template library; finally detecting whether the video is finished or not, continuously reading in a next frame if the video is not finished, or completing tracking. By adopting the object scale self-adaption tracking method, the object appearance scale change is effectively coped with under the conditions of illumination change, partial blocking and swift moving, and the robust tracking is realized.