Demasking Engine for Pruned Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural networks with large tensors are ill-suited for real-time applications due to slow processing speeds, and pruning them does not significantly accelerate inference time despite reducing network size.
Innovation Solution
A technique is introduced to remove masks from pruned neural networks by replacing masked tensors with denser versions, allowing for faster operations and propagating scatter operations to maintain output dimensionality, thereby optimizing neural network performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If pruning is applied to reduce network size, then memory footprint is reduced, but inference speed is not significantly accelerated
Solution Approach 1:
The patent extracts and removes the mask tensors from the pruned neural network, keeping only the active (non-zero) weights. This eliminates the computational overhead of processing zeroed-out elements while maintaining the reduced network size, thereby achieving both memory efficiency and speed improvement.
Solution Approach 2:
Instead of keeping the full network and zeroing out elements via masks, the patent inverts the approach by directly representing only the active elements. This inversion transforms the data structure from a sparse masked tensor to a dense tensor containing only non-zero values, enabling faster computation without requiring mask application during inference.
2Productivity
If masks are applied to zero out elements, then tensor operations become less meaningful, but removing masks restores original tensor dimensions
Solution Approach 1:
The patent resolves the dimensionality issue by changing how the network parameters are organized. Instead of maintaining original tensor dimensions with zeroed elements, it reorganizes parameters into a compressed format that only includes active elements. The scatter operation then restores the expected output dimensions by placing results in appropriate positions, effectively managing dimensionality without sacrificing operational efficiency.
3Productivity
If dense tensors are used, then operations can be performed quickly, but memory consumption increases
Solution Approach 1:
The patent applies local quality by treating different parts of the network differently - active weights are stored in a dense format for fast processing, while inactive weights are completely excluded from storage. This localized approach to density ensures that only the necessary portions of the network consume memory, achieving fast processing without proportional memory increase.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A demasking engine removes masks from a pruned neural network that is represented by a graph of nodes. The demasking engine analyzes a tensor and a mask associated with a given node in the graph of nodes to determine portions of the tensor that are zeroed by the mask. The demasking engine then removes these portions from the tensor to generate a densified tensor that has a smaller dimensionality than the original tensor. A function associated with the node can be evaluated more quickly based on the densified tensor than the original tensor. The demasking engine adds a scatter operation subsequent to the node in order to scale the dimensionality of the densified tensor to the dimensionality associated with the original tensor.