The application discloses a two-stage speculative decoding optimization method and
system for
large model inference acceleration. First, a general domain dataset and a specific task domain
distillation dataset are constructed, and a frozen
verification model and a candidate generation model to be trained are established. Based on the general domain dataset, the first-stage preliminary capability training of the candidate generation model is performed, so that the candidate generation model enters the initial interval effectively aligned with the
verification model. Based on the task domain
distillation data, the second-stage key mark optimization training is performed. The ratio of the
verification model probability to the candidate generation model probability is calculated for each to-be-predicted mark to obtain the potential contribution index of the candidate generation model. The
dynamic screening threshold and the sparse
mask are generated by using
global statistics, and the loss optimization is only performed on the key contribution mark. Finally, the trained candidate generation model and the verification model are used to perform speculative decoding
inference. The method improves the acceptance length of the candidate mark and the
inference acceleration effect, and reduces the training cost.