Neural Network Batch Normalization Sparsification for Speed
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network training methods face challenges in improving computation speed and reducing data transmission delays, primarily due to high data volumes associated with feature maps, which necessitate increased on-chip storage and complex hardware designs.
Innovation Solution
The proposed method involves training a neural network to determine indicator parameters and updating batch normalization layer parameters based on preset conditions to increase zero elements in feature map outputs, thereby implementing sparsity and reducing data transmission volumes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the quantity of computation units is increased to improve computation speed, then computation speed is improved, but device complexity increases
Solution Approach 1:
The patent extracts and removes redundant data from feature maps through sparsification, keeping only significant data elements. This reduces the volume of data that needs to be processed and transmitted, thereby improving computation speed without requiring additional computation units, thus avoiding increased device complexity
Solution Approach 2:
The patent changes the density parameter of feature maps by applying sparsification techniques, transforming dense feature maps into sparse ones. This parameter change reduces data transmission volume and computation load, improving speed without adding hardware complexity
2Loss of time
If bandwidth is increased to reduce data transmission delay, then data transmission delay is reduced, but device complexity and cost increase
Solution Approach 1:
The patent extracts only the most important data elements from feature maps through sparsification, removing redundant information. This reduces the data volume that needs to be transmitted, thereby reducing transmission delay without requiring increased bandwidth or more complex transmission hardware
Solution Approach 2:
The patent changes the data volume parameter by transforming dense feature maps into sparse feature maps. This parameter change directly reduces transmission delay without requiring hardware upgrades, avoiding increased device complexity
3Quantity of substance
If on-chip storage is increased to handle larger data volumes, then data transmission capacity is improved, but device complexity and cost increase
Solution Approach 1:
The patent extracts and removes redundant data from feature maps, keeping only significant elements. This reduces the total data volume that needs to be stored and processed, allowing smaller on-chip storage to handle the reduced data load, thereby avoiding increased device complexity and cost
Solution Approach 2:
The patent changes the storage requirement parameter by sparsifying feature maps, reducing the volume from dense to sparse representation. This parameter change allows using smaller, simpler storage hardware while maintaining data transmission capacity for the essential information
Data Source
AI summary
Disclosed are a neural network training method, a neural network training device and an electronic device. The neural network training method includes: training a first neural network to be trained by using sample data; determining an indicator parameter of the first neural network in a current training process; determining an update manner corresponding to a preset condition if the indicator parameter meets the preset condition; and updating a parameter of a batch normalization layer in the first neural network based on the update manner. In this way, sparsing of a feature map output by a neural network is implemented, thereby reducing an amount of data to be transmitted and improving computation speed of a chip.