The invention discloses a firework identification method and a firework identification system based on deep learning of an image. The firework identification method comprises the following steps of step 1, acquiring a label-free sample image set and a label sample image set; step 2, obtaining a label-free training data set and a label training data set; step 3, performing whitening preliminary processing on training data; step 4, based on the label-free training data subjected to the whitening preliminary processing, constructing a deep neutral network based on sparse self coding by adopting unsupervised learning, and extracting a basic image feature set of the label-free training data; step 5, convolving basic image features and pooling image data; step 6, training a Softmax classifier based on the convolved and pooled label training data set; step 7, inputting the convolved and pooled images to be identified into the trained Softmax classifier to obtain the identification result. According to the firework identification method and the firework identification system disclosed by the invention, the visual identification rate of fireworks and a similar object can be effectively improved, and automatic identification with higher precision for the fireworks can be realized.