The invention provides a multi-mode ultrasonic image classification method and a breast cancer diagnosis device, and the method comprises the steps: S1, segmenting a region-of-interest image from an original gray-scale ultrasonic-elastic imaging image pair, and obtaining a pure elastic imaging image according to the segmented region-of-interest image; s2, extracting single-mode image features of the gray-scale ultrasonic image and the elastic imaging image by using a DenseNet network; s3, constructing a resistance loss function and an orthogonality constraint function, and extracting shared features between the gray-scale ultrasonic image and the elastic imaging image; and S4, constructing a multi-task learning framework, splicing the inter-modal shared features obtained in the S3 and thesingle-modal features obtained in the S2, inputting the inter-modal shared features and the single-modal features into a plurality of classifiers together, and performing benign and malignant classification respectively. According to the method, benign and malignant classification can be carried out on the gray-scale ultrasonic image, the elastic imaging image and the two modal images at the sametime, and the method has excellent performance of high accuracy and wide application range.