The invention belongs to the technical field of photoelectricity, and particularly relates to a multi-target data association algorithm based on a feature matching matrix. The algorithm includes the following steps that a current video sequence image is collected in the stationary state of shooting equipment; a target, closest to the center of a moving target at the moment of t-1, in the center of a jth moving area at the moment of t is calculated, and the closest matching distance is recorded as d (i, j); the intersecting area of a moving target at a t-1 frame and a moving target at a t frame is calculated, and if the intersecting area of a target at the moment t and a target in the current area is larger than a certain threshold, the intersecting area is recorded as s (i, j); when the center association distance and the area intersecting area are both larger than a certain threshold, an element R (i, j) of the matching matrix is set as 1; an association matching matrix is judged, and association tracking is performed on emerged new targets and an ideal tracking state through a centroid matching algorithm; when combination of the targets occurs, movement information and color information before combination of the targets are reserved respectively, and combination tracking of Kalman filtering and histogram fusion is performed. According the multi-target data association algorithm based on the feature matching matrix, multi-target data association under the condition of sheltering can be achieved.