Pre-Trained Model Adaptation with MPO Weight Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Adapting large-scale pre-trained machine learning models to specific tasks requires significant computational resources and energy expenditure due to the large number of parameters, especially when training data is limited.
Innovation Solution
Utilizing quantum circuits to implement a matrix product operator (MPO) representation for task-specific parameter increments, which reduces the number of trainable parameters by constructing updates through a tensor network, and optionally leveraging variational quantum circuits to determine low-rank weight matrices for further parameter reduction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full fine-tuning is used to adapt pre-trained models to downstream tasks, then model accuracy for the specific task is improved, but computational cost and energy expenditure increase significantly
Solution Approach 1:
The patent extracts only the essential update information needed for task adaptation by representing weight updates as low-rank matrices. Instead of updating all parameters, the method extracts and trains only the low-rank update matrices ΔW that capture the necessary task-specific adjustments, significantly reducing computational cost while maintaining accuracy.
Solution Approach 2:
The patent changes the parameter representation from full-rank weight matrices to low-rank factorized form. By expressing weight updates as products of low-rank matrices, the method transforms the optimization problem into one with fewer effective parameters, reducing computational complexity while preserving the essential information needed for task performance.
2Adaptability or versatility
If conventional fine-tuning techniques are used with limited training data, then model adaptation is achieved, but accuracy is limited due to overfitting or insufficient learning
Solution Approach 1:
The patent changes the parameter representation to low-rank factorized form, which regularizes the optimization problem. This parameter transformation effectively reduces the search space and prevents overfitting when training data is limited, while still capturing task-specific patterns through the low-rank update matrices.
3Reliability
If all model parameters are updated during adaptation, then complete model retraining is achieved, but training time increases significantly
Solution Approach 1:
The patent extracts only the essential update information needed for task adaptation by representing weight updates as low-rank matrices. Instead of updating all parameters, the method extracts and trains only the low-rank update matrices ΔW that capture the necessary task-specific adjustments, significantly reducing training time while maintaining accuracy.
Solution Approach 2:
The patent applies partial action by updating only a subset of parameters (the low-rank update matrices) rather than all parameters. This partial parameter update approach is sufficient for task adaptation and avoids the time-consuming process of retraining the entire model.
Data Source
Figure 1
Figure 2A~2C
Figure 3
AI summary
A computer-implemented method for adapting a pre-trained machine learning model to a learning task, comprising receiving the pre-trained machine learning model comprising a plurality of learned weights for transforming an input towards an output, wherein the plurality of learned weights can be expressed with a weight matrix; performing a training process for adapting the pre-trained machine learning model to the learning task by updating a task-specific parameter increment added to the weight matrix, which is constructed from an MPO representation with a plurality of tensors each having an uncontracted first and second index, and a contracted index of a bond dimension, wherein a product of dimensions of the uncontracted first and second indices of the tensors are at least equal to a first dimension and a second dimension of the weight matrix, respectively, and wherein entries of the tensors are trainable parameters of the training process.