Entropy Penalized Weight Reparameterization for Neural Network Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine-learned model parameters require large storage space, making them impractical for deployment on low-memory or low-bandwidth devices.
Innovation Solution
The method involves representing model parameters in a reparameterization space during training, using entropy penalized reparameterization techniques, and applying entropy coding compression to generate a highly compressible representation of the machine-learned model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If machine-learned model parameters are stored in their original form, then model accuracy is maintained, but storage space requirements increase significantly
Solution Approach 1:
The patent transforms model parameters from their original high-precision format into a compressed representation using entropy penalized reparameterization. This changes the parameter representation space, allowing the same model functionality to be achieved with fewer bits per parameter while maintaining accuracy through the learned reparameterization mapping.
Solution Approach 2:
The patent introduces an intermediary reparameterization layer that maps original parameters to a compressed representation space. This intermediary transformation enables lossless or near-lossless compression by learning a more efficient parameterization that preserves model accuracy while reducing storage requirements.
2Quantity of substance
If model parameters are compressed to reduce storage space, then storage and transmission requirements decrease, but model performance may deteriorate
Solution Approach 1:
The patent incorporates feedback through the entropy penalty term in the loss function during training. This feedback mechanism guides the reparameterization process to maintain information essential for model performance while achieving compression, ensuring that the compressed parameters retain the necessary predictive capability.
Solution Approach 2:
The patent performs preliminary reparameterization learning during the model training phase before deployment. By pre-learning the optimal compressed parameter representation and its mapping to the original parameter space, the system ensures that when compressed parameters are used at inference time, model performance is preserved without requiring the original large parameters.
Data Source
AI summary
Example aspects of the present disclosure are directed to systems and methods that learn a compressed representation of a machine-learned model (e.g., neural network) via representation of the model parameters within a reparameterization space during training of the model. In particular, the present disclosure describes an end-to-end model weight compression approach that employs a latent-variable data compression method. The model parameters (e.g., weights and biases) are represented in a “latent” or “reparameterization” space, amounting to a reparameterization. In some implementations, this space can be equipped with a learned probability model, which is used first to impose an entropy penalty on the parameter representation during training, and second to compress the representation using arithmetic coding after training. The proposed approach can thus maximize accuracy and model compressibility jointly, in an end-to-end fashion, with the rate-error trade-off specified by a hyperparameter.