The invention discloses a pre-training model low-rank adaptive parameter high-efficiency
fine tuning method based on
triangular decomposition, which comprises the following steps: for a pre-training
large model, selecting levels needing parameter
fine tuning from the
large model according to actual needs, and respectively performing
triangular decomposition on a weight matrix of each layer to be subjected to
fine tuning, and taking the obtained scaling factor and the
diagonal sub-matrix as trainable parameters, freezing other
model parameters, inputting an input sample in the training sample set into the
large model to update the trainable parameters, and reconstructing by adopting the updated trainable parameters to obtain a weight matrix, thereby completing efficient fine tuning of the pre-trained large
model parameters. According to the method, the number of trainable parameters is remarkably reduced through three-solution
decomposition, meanwhile, an orthogonal constraint mechanism is introduced, pre-training knowledge is kept, and meanwhile the performance level close to full-amount adjustment and optimization is achieved. The method not only reduces the consumption of computing resources, but also improves the optimization speed of the model.