Low-Rank NAS Model Compression for Reduced Network Dimensions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for compressing large machine learning models are inefficient in terms of computational resources and memory usage, particularly when applied to embedded systems, and lack effective techniques for scalable and adaptable network architecture optimization.
Innovation Solution
A method combining low-rank matrix approximation (LoRA) with one-shot neural architecture search (NAS) is employed, adding low-rank matrices to pre-trained weight matrices of a base model to facilitate the extraction of submodels with reduced network dimensions, optimizing for computational efficiency and memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If low-rank matrices are added to pre-trained weight matrices to reduce network dimension, then computational effort and memory requirements are reduced, but model accuracy may deteriorate
Solution Approach 1:
The patent applies parameter changes by modifying the weight matrices through low-rank decomposition, transforming the original high-dimensional parameters into a compact form with reduced dimensions while preserving essential information for accurate predictions
Solution Approach 2:
The patent uses composite materials analogy by combining the pre-trained weight matrix with low-rank matrices to form a composite weight structure that maintains the benefits of pre-training while achieving dimensionality reduction and improved efficiency
2Quantity of substance
If low-rank matrices are added to pre-trained weight matrices to reduce network dimension, then memory requirements are reduced, but model complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the weight matrix into multiple components (pre-trained weights and low-rank matrices), allowing separate optimization and management of different parts of the model to reduce overall memory requirements while maintaining functionality
3Productivity
If neural architecture search is performed to extract submodels with reduced network dimension, then model efficiency is improved, but training time increases
Solution Approach 1:
The patent applies preliminary action by performing neural architecture search and extracting optimized submodels in advance before deployment, allowing the system to identify efficient architectures beforehand and avoid time-consuming optimization during actual operation
Solution Approach 2:
The patent uses copying by creating multiple candidate submodels with different network dimensions through neural architecture search, then selecting the optimal copy that balances efficiency and performance requirements for the specific application
Data Source
AI summary
A method for reducing a network dimension of a base model. The method including: providing the base model, which has pre-trained weight matrices and is trained to solve a target task; converting the base model into a one-shot model which has weight matrices; adding at least one, in particular network dimension-specific, low-rank matrix to each weight matrix of the one-shot model; carrying out a neural network search for the one-shot model to extract at least one submodel of the one-shot model having a reduced network dimension on the basis of the low-rank matrices and the weight matrices of the one-shot model until a termination criterion is reached; and providing the at least one submodel having a reduced network dimension, in particular for implementation on an embedded system.