The invention discloses a data storage method for a sparse matrix and a method for realizing sparse matrix-vector multiply (SpMV) based on the same. The data storage method comprises the following steps of: A, scanning the sparse matrix according to the line number with preset queue unit size, and storing a non-zero element value of a line of a non-zero element into a non-zero element array val when the non-zero element is scanned; and B, calculating a queue value difference, judging the type of the queue value difference of the non-zero element queue unit, taking the queue units with the same queue value difference type as the same queue group, and storing the queue group information to an index array. The method for realizing the SpMV comprises the following steps of: 1) calculating the queue value of the queue unit according to the queue value difference type; 2) reading a corresponding x variable according to the queue value, and multiplying the corresponding x variable by an element in the val; 3) accumulating a result which is calculated by each queue unit in the queue group, and saving in a result array; and 4) continuously reading the queue value difference type of the next queue group. By the method, zero element filling is reduced, and operation of realizing the SpMV and operation efficiency of storing access operation are improved.