The invention belongs to the field of depth learning, and provides a rail surface defect detection method based on the depth learning network, aiming at solving various problems existing in the priorrail detection methods. The depth learning method first automatically resets the input rail image to 416*416, and then extracts and processes the image. Image extraction mainly by Darknet-53 model complete. The processing output is mainly accomplished by the FPN-like network model. Firstly, the rail image is divided into cells. According to the position of the defects in the cells, the width, height and coordinates of the center point of the defects are calculated by dimension clustering method, and the coordinates are normalized. At the same time, we use logistic regression to predict the fraction of boundary box object, use binary cross-entropy loss to predict the category contained in the boundary box, calculate the confidence level, and then process the convolution in the output, up-sampling, network feature fusion to get the prediction results. The invention can accurately identify defects and effectively improve the detection and identification rate of rail surface defects.