Neural Network Parameter Compression via Subspace Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural networks require significant storage space and bandwidth for transmission due to their large number of parameters, making them inefficient for devices with limited resources and slow to train and transmit.
Innovation Solution
A system that generates a compressed representation of neural networks by training them in a reduced subspace, using a projection matrix to map the original parameter vector to a smaller subspace, allowing for efficient storage and transmission while maintaining high accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If neural networks use tens or hundreds of millions of parameters to achieve good performance, then accuracy is improved, but storage space requirement increases
Solution Approach 1:
The patent creates a compressed copy of the neural network parameters by training in a lower-dimensional subspace. Instead of storing the full high-dimensional parameter vector, the system stores a compressed representation in a lower-dimensional space that can be transformed back to generate the full parameter set, significantly reducing storage requirements while maintaining model accuracy
Solution Approach 2:
The patent transforms the parameter representation from high-dimensional to low-dimensional by changing the parameter space dimensionality. By training the neural network in a compressed subspace and using projection matrices to map between spaces, the system reduces the number of parameters that need to be stored and transmitted while preserving the essential information needed for accurate predictions
2Reliability
If neural networks use tens or hundreds of millions of parameters, then performance is improved, but transmission bandwidth requirement increases
Solution Approach 1:
The system transmits a compressed copy of the neural network parameters instead of the full parameter set. By sending only the low-dimensional trained parameters and projection matrices, the bandwidth required for transmission is dramatically reduced while the receiving system can reconstruct the full parameter set for deployment
Solution Approach 2:
The patent changes the parameter dimensionality to reduce transmission bandwidth. By transforming parameters from high-dimensional to low-dimensional space for transmission, and then transforming back at the receiving end, the system minimizes bandwidth consumption during network transmission while maintaining full model performance
3Reliability
If neural networks process a very large number of parameters, then accuracy is improved, but training time increases
Solution Approach 1:
The patent extracts the essential information from the high-dimensional parameter space by training in a lower-dimensional subspace. By identifying and training only the critical parameters in a compressed space, the system reduces the computational burden during training while still capturing the essential patterns needed for accurate predictions
Solution Approach 2:
The system changes the parameter space dimensionality during training by working in a compressed low-dimensional subspace. This parameter transformation reduces the number of parameters that need to be processed during training, significantly decreasing training time while maintaining the ability to achieve high accuracy through proper subspace selection and projection
Data Source
AI summary
Machine learning based models, for example, neural network models employ large numbers of parameters, from a few million to hundreds of millions or more. A machine learning based model is trained using fewer parameters than specified. An initial parameter vector is initialized, for example, using random number generation based on a seed. During training phase, the parameter vectors are modified in a subspace around the initial vector. The trained model can be stored or transmitted using seed values and the trained parameter vector in the subspace. The neural network model can be uncompressed using the seed values and the trained parameter vector in the subspace. The compressed representation of neural networks may be used for various applications such as generating maps, object recognition in images, processing of sensor data, natural language processing, and others.