Parallel Neural Network Finetuning With Block-Diagonal Matrices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Finetuning large neural networks, such as foundation models, requires significant computational resources and time, even when using existing methods like Orthogonal Finetuning (OFT), which can degrade performance with increased block counts.
Innovation Solution
The method employs a block-diagonal matrix, particularly using Householder transformations, to parallelize the computation of neural network layers, maintaining performance by freezing the weight matrix and adapting the block-diagonal matrix elements, enabling parallel computation across threads or distributed environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If finetuning is performed on large neural networks using existing methods, then adaptation to downstream tasks is achieved, but computational resources and training time remain excessive
Solution Approach 1:
The weight matrix is segmented into multiple blocks, and a block-diagonal matrix is constructed where each block corresponds to a specific block of the weight matrix. This segmentation allows parallel computation across different blocks, significantly reducing finetuning time and computational resource requirements while maintaining adaptation performance.
2Productivity
If the number of blocks in the block-diagonal matrix is increased, then parallelization and finetuning speed are improved, but performance degradation may occur
Solution Approach 1:
The patent optimizes the parameters of the block-diagonal matrix, specifically the blocks corresponding to Householder transformations, to maintain neural network performance while enabling parallel computation. By carefully selecting and optimizing these parameters, the system achieves both speedup and performance maintenance.
3Use of energy by moving object
If the weight matrix is frozen and only the block-diagonal matrix is adapted, then computational resources are reduced, but adaptation capabilities may be limited
Solution Approach 1:
The patent extracts the essential adaptation information into the block-diagonal matrix while keeping the weight matrix frozen. This extraction allows the system to maintain the bulk of the pre-trained knowledge while introducing necessary adaptations through the block-diagonal matrix, achieving efficient resource usage without sacrificing adaptation capabilities.
Data Source
AI summary
A computer-implemented method for finetuning a neural network. The method includes: providing an input to a layer of the neural network; determining a block-diagonal matrix; determining a first matrix by multiplying the block-diagonal matrix with a weight matrix of the layer, wherein the result of the multiplication is obtained by multiplying at least a plurality of blocks of the block-diagonal matrix with a respective part of the weight matrix in parallel computing operations and combining the result to form the first matrix; determining an output of the layer by multiplying the first matrix with the input of the layer; determining an output of the neural network based on the output of the layer; adapting elements of the block-diagonal matrix based on a difference of the output of the neural network and a desired output with respect to the input datum.