CNN Training Acceleration via Non-Zero Index Skipping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current convolutional neural networks (CNNs) face significant challenges in speeding up training due to the repetitive and time-consuming nature of convolution calculations, particularly in identifying non-zero elements for efficient backward propagation, which hinders the acceleration of the entire training process.
Innovation Solution
A training program that utilizes a fusion calculation between the convolution layer and the pooling layer to skip unnecessary convolution calculations by generating an index for non-zero elements during forward propagation, allowing only non-zero element convolutions in backward propagation, thereby speeding up the training process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If conventional convolution calculations are performed for all elements during backward propagation, then calculation completeness is maintained, but training time is excessively long
Solution Approach 1:
The patent applies preliminary action by generating the index map during forward propagation that identifies positions of non-zero elements after pooling operations. This pre-computed index is then used during backward propagation to directly access only the relevant non-zero elements for convolution calculations, eliminating the need to process zero elements. This preliminary identification of useful calculation positions resolves the contradiction by preparing necessary information in advance, thereby reducing training time while maintaining calculation completeness for all necessary elements.
2Productivity
If convolution calculations are skipped for zero elements, then calculation efficiency is improved, but identification of non-zero elements becomes complex
Solution Approach 1:
The patent merges the pooling operation with the generation of the index map in a unified process. During the pooling layer forward propagation, the same computation that determines which elements become non-zero in the pooled output also populates the index map with these positions. This merging of functions eliminates the need for separate complex identification processes, as the index is naturally generated as part of the pooling operation itself, thus improving calculation efficiency without significantly increasing device complexity.
3Reliability
If all convolution calculations are performed, then accuracy is maintained, but computational load is excessive
Solution Approach 1:
The patent extracts and processes only the essential non-zero elements for convolution calculations during backward propagation by utilizing the pre-generated index map. Instead of performing convolution on all elements including zeros, the method extracts specifically the non-zero elements identified by the index, processes only those necessary computations, and maintains gradient flow accuracy. This selective extraction maintains recognition accuracy while dramatically reducing computational load by eliminating redundant operations on zero elements.
Data Source
AI summary
A recording medium stores a program for causing a computer to execute processing including: causing a convolution layer to execute a convolution calculation of forward propagation on first data output from a layer closer to an input side than the convolution layer; generating, when a pooling layer is caused to execute a pooling calculation of forward propagation on output data, an index in which a position of a non-zero element is set for each element of the output data; and causing, when the convolution layer is caused to execute a convolution calculation of backward propagation of the first data and second data that is output from a layer closer to an output side than the pooling layer, the convolution layer to execute a convolution calculation of a non-zero element based on the index, the input data, and the second data, and to skip a convolution calculation of a zero element.