Sparse Code Generation for Convolutional Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network execution methods are inefficient due to the need to perform calculations on zero elements during convolutional operations, leading to reduced computational performance and increased memory usage.
Innovation Solution
The system generates code that only includes instructions for non-zero elements in convolutional kernel tensors, using fused multiply-add instructions and vector broadcast operations to optimize computations, thereby avoiding calculations on zero elements and reducing memory footprint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional neural network execution methods are used to perform calculations on all elements including zero elements during convolutional operations, then computational completeness is maintained, but computational performance is reduced and memory usage is increased
Solution Approach 1:
The patent extracts and removes zero elements from the computational process by generating sparse code that only includes instructions for non-zero elements in convolutional kernel tensors. This is achieved by analyzing the kernel tensor to identify non-zero elements and generating optimized instructions specifically for those elements, thereby eliminating unnecessary calculations and improving computational performance.
Solution Approach 2:
The patent segments the convolutional computation into separate handling for zero and non-zero elements. By dividing the kernel tensor into sparse (non-zero) and dense (zero) portions, the system generates specialized code paths that process only the non-zero segments, reducing overall computational complexity while maintaining correctness.
2Quantity of substance
If calculations are performed on all elements including zero elements during convolutional operations, then correctness is maintained, but memory usage is increased
Solution Approach 1:
The patent extracts zero elements from the computational process by generating sparse code that exclusively processes non-zero elements. This extraction is achieved through static analysis of the kernel tensor to identify non-zero positions, then generating optimized instructions that reference only those positions, thereby reducing memory footprint while preserving computational correctness.
Solution Approach 2:
The patent performs preliminary analysis of the kernel tensor during code generation to identify and mark non-zero elements before generating execution instructions. This preliminary action allows the system to prepare optimized sparse code that inherently knows which elements to process, eliminating the need for runtime checks and reducing memory usage without compromising correctness.
3Productivity
If optimized sparse code is generated to skip zero elements during convolutional operations, then computational efficiency is improved, but code generation complexity is increased
Solution Approach 1:
The patent performs preliminary analysis of the kernel tensor during the code generation phase to identify non-zero elements and their positions. This preliminary action enables the generation of optimized sparse code that directly references only non-zero elements, achieving computational efficiency while containing code generation complexity through systematic analysis and structured instruction emission.
Solution Approach 2:
The patent introduces an intermediate code generation stage that acts as a mediator between the kernel tensor definition and the final execution code. This intermediary process analyzes the kernel structure, identifies non-zero elements, and generates optimized instructions, thereby bridging the gap between raw data and efficient computation without requiring complex runtime processing.
4Loss of time
If traditional convolutional operations are executed with full kernel tensors, then implementation simplicity is maintained, but inference time is increased
Solution Approach 1:
The patent extracts and eliminates unnecessary zero-element calculations from the inference process by generating sparse code during compilation. This extraction shifts the complexity from runtime execution to compile-time code generation, resulting in significantly faster inference times while the generated code remains relatively simple and structured.
Solution Approach 2:
The patent performs preliminary optimization of the convolutional kernel during code generation by identifying and marking non-zero elements. This preliminary action creates optimized execution code that skips zero elements during inference, thereby reducing inference time while maintaining implementation simplicity through systematic pre-processing of the kernel data.
Data Source
AI summary
A system and method may generate code to be used when executing neural networks (NNs), for example convolutional neural networks (CNNs) which may include one or more convolutional layers. For at least one convolutional layer, for each non-zero element in a kernel tensor or matrix associated with the convolutional layer, instructions may be generated or issued. For example, for each non-zero element, a vector broadcast instruction may be generated, and a fused multiply-add (FMA) instruction may be generated, having as parameters a register representing a portion of the output for the convolutional layer, a register storing input data for the convolutional layer, and a register or reference to memory storing the non-zero element. The software or code produced may be executed during convolutional operations, for example as part of a larger application such as a NN inference application.