The invention relates to a face automatic labeling method and a face automatic labeling system. Firstly, faces are detected from a captured video, a face image set is acquired, then the face image set is filtered, an HSV (Hue, Saturation, Value) color histogram difference value of an image of an adjacent frame is simultaneously acquired, shot segmentation is carried out by adopting a spatial color histogram shot edge detecting algorithm, angular points in a target region of a first frame are detected for the faces from the adjacent frame, the angular points are deferred to a next frame by using a local matching method, corresponding updating is carried out, a matched number is subjected to statistics, and according to a threshold value of the matched number, the operation is sequentially carried out to acquire face sequences; then speakers and unspeaking people are detected by a lip motion detection module according to lip motion of the speakers in the face sequences and the speakers, talk contents and talk time are fused to be labeled; and finally, the faces on each sequence are read in and are gradually positioned, then affine transformation is carried out according to a positioning result and pixel gray values in a round region with a fixed size near a transformed feature point are extracted to be used as features of the faces. The face automatic labeling method and the face automatic labeling system which are disclosed by the invention are convenient to use and have high accuracy.