Sparse Activation Convolution Using Non-Zero Weight Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional convolution operations in neural networks waste computational resources due to processing zero-valued elements in activation tensors, leading to inefficiency and increased energy consumption without improving model accuracy.
Innovation Solution
Desparsified convolution techniques dynamically identify and process only non-zero elements in activation tensors, using sparsity look-ahead to achieve mathematically equivalent results with reduced computational complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If conventional convolution operations process all elements in activation tensors, then complete computational coverage is achieved, but computational resources and energy consumption are wasted on zero-valued elements
Solution Approach 1:
The patent extracts and processes only the non-zero elements from activation tensors, separating them from zero-valued elements. This is achieved through dynamic sparsity identification that locates non-zero element positions and values, then performs convolution operations exclusively on these extracted elements, eliminating wasted computation on zeros while maintaining complete coverage of meaningful data
Solution Approach 2:
The patent applies partial action by performing convolution operations on only a subset of elements (specifically, only non-zero elements) rather than all elements in the activation tensor. This partial processing approach reduces computational workload and energy consumption while still achieving the necessary computational coverage for accurate results
2Productivity
If sparse activation elements are dynamically identified and processed, then computational resources are reduced, but additional processing steps are required
Solution Approach 1:
The patent performs preliminary action by dynamically identifying and locating non-zero elements in the activation tensor before executing convolution operations. This pre-processing step captures the positions and values of non-zero elements, allowing subsequent convolution operations to proceed efficiently without needing to re-evaluate sparsity patterns during computation
Solution Approach 2:
The patent introduces an intermediary mechanism (sparsity identification module) that acts as a mediator between the input activation tensor and the convolution operation. This intermediary dynamically determines which elements require processing and prepares the necessary data structures, simplifying the overall system architecture while enabling efficient sparse computation
3Loss of time
If all elements including zeros are convolved, then mathematical completeness is maintained, but processing time increases unnecessarily
Solution Approach 1:
The patent extracts only the non-zero elements from the activation tensor for convolution processing, taking out the essential computational content while eliminating redundant zero-valued elements. This extraction maintains mathematical completeness because zero elements contribute nothing to the convolution sum, while significantly reducing processing time by focusing computation only on meaningful data
Data Source
AI summary
Certain aspects of the present disclosure provide techniques for desparsified convolution. An activation tensor is received, and a convolution output is generated for the activation tensor, comprising: selecting a subset of weight elements, corresponding to a set of non-zero elements in the activation tensor, from a weight tensor, and multiplying the set of non-zero elements and the set of weight elements.