Neural Network Training via Cyclical Learning Rates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models require significant operations and memory, making it difficult to achieve high performance in embedded systems with limited hardware resources, which is alleviated by lightened deep learning models like quantization techniques that limit weight values.
Innovation Solution
A neural network training method involving cyclical learning rates, where the model is trained with full precision initially, then quantized, and retrained based on the lowest learning rate, with weights averaged and quantized again to improve precision and reduce complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If quantization technique is applied to reduce operation complexity, then device complexity is reduced, but manufacturing precision deteriorates
Solution Approach 1:
The patent applies preliminary action by performing multiple training phases before final deployment. The model is first trained with full precision weights, then quantized, and subsequently retrained with quantized weights using cyclical learning rates. This preliminary training sequence ensures the model adapts to quantization effects beforehand, maintaining precision while achieving reduced operation complexity in the final deployed model.
Solution Approach 2:
The patent employs parameter changes by systematically varying the learning rate through cyclical patterns during the retraining phase. The learning rate oscillates between high and low values, allowing the model to escape local minima and achieve better convergence with quantized weights. This dynamic parameter adjustment compensates for the precision loss from quantization while maintaining reduced computational complexity.
2Manufacturing precision
If multiple training phases are applied to improve precision, then manufacturing precision is improved, but productivity deteriorates
Solution Approach 1:
The patent applies periodic action through cyclical learning rates that oscillate between high and low values during the retraining phase. This periodic variation in learning rate allows the model to efficiently navigate the loss landscape, achieving good convergence without requiring excessively long training periods. The cyclical pattern enables the model to make rapid progress during high learning rate phases and fine-tune during low learning rate phases, balancing precision improvement with training efficiency.
Solution Approach 2:
The patent uses copying by reusing the architecture and initial weights from the full-precision trained model when performing quantized retraining. Instead of training from scratch, the quantized model copies the structure and learned patterns from the full-precision version, then adapts to quantization constraints. This copying approach significantly reduces the training time required compared to training a new model, while still achieving the precision benefits of multiple training phases.
Data Source
AI summary
Disclosed is a neural network training method and apparatus. The neural network training method includes a neural network training method, including receiving a neural network model that is first trained based on a first weight, second training the first trained neural network model based on learning rates to obtain second weights from a second trained neural network, and third training the second trained neural network model based on the second weights.