The invention discloses an image classification method based on a bi-directional neural network structure, comprising the following steps: 1. The Directional layer replaces the full-connection layer in a traditional convolution network, the image classification model based on bi-directional neural network is built, 2. network forward propagation is performed, and by adding transformation matricesL and R, the rectangular structure of the last convolution layer is preserved, steps 2, 3 are repeated, the bi-directional neural network is fine-tuned until the classification network converges. 5, the class number of the image is obtained through forward propagation on the trained model. A bidirectional neural network structure is used on the premise of keeping the depth and width of the networkconstant, the dimension of the feature is transformed by designing the transformation matrix, which ensures the matrix form of convolution, effectively preserves the structure information of the original feature space of the image, and avoids the loss of spatial structure information in the process of matrix drawing into vectors in the whole connection layer.