Neural Network Weight Compression with Variational Dropout Smoothing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for compressing Deep Neural Network (DNN) models are inefficient in reducing storage size and computational requirements, particularly for deployment on mobile devices and chips, as they do not effectively remove unimportant weight coefficients and rely on greedy pruning or sparsity-promoting regularization.
Innovation Solution
The method involves quantizing and entropy-coding weight coefficients, followed by applying variational dropout and smoothness regularization to locally smooth the coefficients, which are then compressed, improving the efficiency of DNN model compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If greedy pruning or sparsity-promoting regularization is used to remove weight coefficients, then model size is reduced, but compression efficiency is insufficient for resource-constrained devices
Solution Approach 1:
The patent transforms the discrete pruning problem into a continuous optimization problem by introducing variational dropout parameters that continuously adjust weight coefficients. This allows for more fine-grained control over which weights to retain or remove, improving compression efficiency compared to traditional greedy pruning methods that operate in discrete steps.
Solution Approach 2:
The patent applies smoothness regularization during the training phase to pre-smooth the weight coefficients before the actual pruning process. This preliminary smoothing action creates a more structured weight distribution that makes subsequent compression more efficient and effective, allowing for better identification of important vs. unimportant weights.
2Reliability
If more weight coefficients are retained to maintain model accuracy, then model performance is preserved, but storage requirements and computational load increase
Solution Approach 1:
The patent applies different treatment to different regions of the weight matrix by using smoothness regularization that operates locally on neighborhoods of weights. This allows the model to preserve important local patterns and structures while compressing less important regions, achieving better accuracy-compression trade-offs compared to uniform pruning approaches.
Solution Approach 2:
The patent introduces variational dropout as an intermediary mechanism that smoothly transitions between retaining and removing weights. Instead of binary keep/remove decisions, the variational dropout parameters provide a continuous intermediary that allows for gradual compression while monitoring accuracy, enabling fine-tuned control over the accuracy-size trade-off.
3Quantity of substance
If traditional quantization and entropy-coding are applied directly to weight coefficients, then compression is achieved, but the presence of unsmoothed coefficients reduces compression effectiveness
Solution Approach 1:
The patent applies smoothness regularization as a preliminary processing step before quantization and entropy-coding. This pre-smoothing of weight coefficients creates a more regular and predictable weight distribution that is more amenable to compression, improving the effectiveness of subsequent quantization and entropy-coding operations compared to applying these operations directly to unprocessed weights.
Data Source
AI summary
A method, computer program, and computer system is provided for compressing a deep neural network model. Weight coefficients associated with a deep neural network are quantize and entropy-coded. The quantized and entropy-coded weight coefficients are locally smoothed. The smoothed weight coefficients are compressed based on applying a variational dropout to the weight coefficients.