The invention provides a speaker identification method base on a simple direct tolerance learning algorithm. The method comprises the following steps: acquiring voice samples of multiple speakers, extracting i-vectors of all the samples, performing channel compensation processing by use of an LDA or WCCN method, performing length normalizing, and forming a training sample set; according to the i-vectors of the training sample set and speaker identity, constructing a similar sample pair set and a non-similar sample pair set; by use of a KISS algorithm, obtaining a tolerance matrix by performing training on the similar sample pair set and the non-similar sample pair set; and for two pieces of new voice, their i-vectors are extracted firstly, the channel compensation processing is carried out by use of the LDA or WCCN method, the length normalizing is performed, by use of the previously calculated tolerance matrix, a Mahalanobis distance between the two i-vectors is calculated and compared with a threshold, and thus whether the two pieces of new voice belong to the same speaker is determined. According to the invention, the obtained Mahalanobis distance tolerance matrix can better truly reflect similarities and distinctions of a sample space so as to improve the performance of a speaker identification system.