Independent Mini-Batch Normalization for CNN Learning Speed
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deep learning techniques using convolutional neural networks (CNNs) face challenges in improving learning speed due to difficulties in independent normalization processes for individual data pieces within a mini-batch, hindering further performance enhancements.
Innovation Solution
A data processing method and device that apply a first CNN layer to each data piece in a mini-batch, calculate independent statistics, and use normalization parameters based on these statistics and cumulative statistics to normalize feature maps, allowing for independent normalization processes while maintaining generalization performance, enabling parallel execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional batch normalization is used with mean and standard deviation of all data in mini-batch, then normalization is applied to all data together, but independent normalization process for individual data pieces cannot be achieved, hindering learning speed improvement
Solution Approach 1:
The patent divides the mini-batch into individual data pieces and performs normalization independently for each piece. Instead of calculating mean and standard deviation across the entire mini-batch, the patent calculates statistics for each data piece separately, enabling independent normalization processes that can be executed in parallel, thus improving learning speed while maintaining normalization effectiveness
Solution Approach 2:
The patent introduces a new dimension of independence by allowing each data piece to have its own normalization parameters (mean and standard deviation) rather than sharing common parameters across the mini-batch. This dimensional separation enables parallel processing of normalization operations for individual data pieces, resolving the contradiction between independent processing and overall productivity
2Productivity
If normalization is performed using statistics from all data in mini-batch, then generalization performance is maintained, but parallel execution and processing efficiency are limited
Solution Approach 1:
By segmenting the normalization process into individual data piece operations with separate statistic calculations, the patent enables parallel execution without compromising the normalization quality that ensures generalization performance. Each data piece receives appropriate normalization based on its own characteristics while maintaining consistency with the overall learning objective
Solution Approach 2:
Each data piece performs self-service normalization by calculating its own mean and standard deviation independently. This self-service approach allows parallel processing of multiple data pieces while maintaining the statistical properties necessary for good generalization performance, as each piece is normalized according to its own distribution characteristics
Data Source
AI summary
A data processing device applies a first convolutional neural network layer to pieces of data included in a mini-batch to obtain a first feature map of each of the pieces of data, independently calculates a first statistic for each of the pieces of data based on the first feature maps, calculates a normalization parameter for each of the pieces of data based on the first statistic of each of the pieces of data and a cumulative statistic, normalizes the first feature map of each of the pieces of data by using a normalization parameter of each of the pieces of data to obtain a normalized feature map of each of the pieces of data, and applies a second convolutional neural network layer to the normalized feature map of each of the pieces of data to obtain a second feature map of each of the pieces of data.