CNN Training via Input Downsampling to Cut Multiplication Load
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current CNN training procedures are inefficient due to the high number of multiplication operations required, leading to lengthy training times and increased hardware costs associated with using Fast Fourier Transforms to speed up the process.
Innovation Solution
The method exploits the spatial scaling property of convolution operations to reduce the number of multiplication operations by downsampling input structures and kernels during an initial pre-training phase, followed by upsampling and continued training with original sizes, thereby reducing computational load without degrading performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If Fast Fourier Transforms are used to speed up CNN training, then training time is reduced, but hardware costs increase
Solution Approach 1:
The patent changes the parameter of input data resolution by using downscaled versions during pre-training. This reduces the computational complexity of convolution operations without requiring FFT hardware, thereby reducing training time while avoiding increased hardware costs.
2Measurement precision
If convolution operations are performed with full-resolution input structures and kernels, then classification accuracy is maintained, but the number of multiplication operations increases
Solution Approach 1:
The patent segments the training process into two distinct phases: pre-training with downscaled input structures and kernels, and fine-tuning with full-resolution data. This segmentation allows the system to benefit from reduced computational load in the first phase while ensuring high accuracy in the second phase.
Solution Approach 2:
The patent performs preliminary action by conducting pre-training with downscaled data before the main training phase. This preliminary step initializes the neural network weights in a way that reduces the computational burden of subsequent full-resolution training, improving overall training efficiency.
3Productivity
If downscaled versions of input structures and kernels are used during pre-training, then the number of multiplication operations is reduced, but data resolution is temporarily degraded
Solution Approach 1:
The patent applies dynamics by making the input data resolution adaptive throughout the training process. The system starts with lower resolution data for computational efficiency, then transitions to full-resolution data for fine-tuning, allowing the resolution parameter to change dynamically based on the training phase.
Data Source
Figure 1A
Figure 1B
Figure 2A~2B
AI summary
A method is disclosed. Said method comprises implementing a convolutional neural network in a processing circuit. Said convolutional neural network is configured to receive an input data structure comprising a group of values corresponding to signal samples and to generate a corresponding classification output indicative of a selected one among a plurality of predefined classes. Said convolutional neural network comprises an ordered sequence of layers. Each layer of the sequence is configured to receive a corresponding layer input data structure comprising a group of input values, and generate a corresponding layer output data structure comprising a group of output values by convolving said layer input data structure with at least one corresponding filter comprising a corresponding group of weights. The layer input data structure of the first layer of the sequence corresponds to said input data structure. The layer input data structure of a generic layer of the sequence different from the first layer corresponds to the layer output data structure generated by the previous layer in the sequence. The method further comprises training the convolutional neural network to update the weights of the filters of the layers by exploiting a training set of training input data structures belonging to known predefined classes. Said training comprises the following phases a), b), c), d): a) generating a modified convolutional neural network by downscaling, for at least one layer of the sequence of layers of the convolutional neural network, the at least one corresponding filter to obtain a downscaled filter comprising a reduced number of weights; b) downscaling the training input data structures to obtain corresponding downscaled training input data structures comprising a reduced number of values; c) for each downscaled training input data structure of at least a subset of the training set, providing such downscaled training input data structure to the modified convolutional neural network to generate a corresponding classification output, and comparing said classification output with the predefined class the training input data structure corresponding to said downscaled training input data structure belongs to; d) updating the weights of the filters of the layers based on said comparisons.