Sparse Code Generation for Convolutional Neural Network Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network execution methods are inefficient due to the need to perform costly operations on zero elements during convolutional calculations, leading to suboptimal performance and increased computational time.
Innovation Solution
The system generates code that only executes instructions for non-zero elements in convolutional neural networks, using fused multiply-add instructions and vector broadcast operations to optimize computations, thereby reducing unnecessary operations and memory footprint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional neural network execution methods are used that process all elements including zeros, then complete computational coverage is achieved, but computational efficiency deteriorates due to unnecessary operations on zero elements
Solution Approach 1:
The patent extracts and processes only the non-zero elements from the convolutional kernel, separating them from the zero elements. This is achieved by identifying the indices of non-zero elements and generating specialized code that operates exclusively on these extracted elements, thereby eliminating unnecessary operations on zero elements and improving computational efficiency
Solution Approach 2:
The patent segments the convolutional computation into distinct parts: identifying non-zero elements, extracting their indices, and processing them separately from zero elements. This segmentation allows the system to generate optimized code that handles non-zero elements with specialized instructions while skipping zero elements entirely, thus reducing computational time
2Productivity
If code is generated for all kernel elements including zeros, then comprehensive computation is performed, but memory footprint increases due to storing and processing unnecessary zero data
Solution Approach 1:
The patent extracts only the non-zero kernel elements and their corresponding indices into compact data structures. This extraction eliminates the need to store and process zero elements in memory, reducing the memory footprint while maintaining execution efficiency by focusing computational resources on the essential non-zero data
3Productivity
If standard convolution operations are executed without optimization, then simplicity of implementation is maintained, but computational performance deteriorates due to processing all elements uniformly
Solution Approach 1:
The patent performs preliminary actions during the code generation phase by identifying non-zero elements, extracting their indices, and generating optimized instructions before execution. This preliminary processing creates specialized code that efficiently handles non-zero elements during inference, improving performance while keeping the runtime complexity manageable through pre-computed optimization
Data Source
AI summary
A system and method may generate code to be used when executing neural networks (NNs), for example convolutional neural networks (CNNs) which may include one or more convolutional layers. For at least one convolutional layer, for each non-zero element in a kernel tensor or matrix associated with the convolutional layer, instructions may be generated or issued. For example, for each non-zero element, a vector broadcast instruction may be generated, and a fused multiply-add (FMA) instruction may be generated, having as parameters a register representing a portion of the output for the convolutional layer, a register storing input data for the convolutional layer, and a register or reference to memory storing the non-zero element. The software or code produced may be executed during convolutional operations, for example as part of a larger application such as a NN inference application.