The invention provides a breast lesion quantitative image evaluation (CAD) system and an application method thereof. The breast lesion quantitative image evaluation system adopts image data of a fractal technology, pattern analysis, and the like to extract an excavating mean and a mathematical modeling algorithm, combines clinical data to establish a breast lesion grown diffusion nonlinear data model and is applied to tumor medical image analysis and tumor disease risk evaluation. The nonlinear data model comprises the clinical parameters of breast tumor grown diffusion state characteristic parameters, calcific state characteristic parameters, breast surface characteristic limitation asymmetric compaction comparison, nipple retraction, pachyderma, structural distortion, and the like. The breast lesion quantitative image evaluation system has a full-graphical interface, can lead a molybdenum target, nuclear magnetism and ultrasonic image data in and is convenient and quick in operation(one-key operation). By the CAD system, a benign-malignant forecasting numerical value and a tumor classification forecasting value of breast molybdenum target piece (nuclear magnetism and ultrasonic) lesions can be calculated, and results of the benign-malignant forecasting numerical value and the tumor classification forecasting value can be applied to breast image auxiliary diagnosis and breast filming screening.