Neural Network Weight Reordering for Balanced Sparse Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deployment of deep-neural networks in memory and compute constrained environments, such as embedded devices, is limited due to their large size and resource requirements, hindering the ability to perform inference operations at the edge or on mobile devices.
Innovation Solution
Transforming the weight space of neural network layers to adapt sparsity patterns without retraining, balancing compute load across processing elements and reducing idle time by rearranging weights to fit the hardware compute model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deep-neural networks with complex architectures are deployed, then model accuracy and capability are improved, but memory and computational resource requirements increase
Solution Approach 1:
The patent extracts and removes redundant or less important weights from the neural network model through pruning techniques. By identifying and eliminating unnecessary connections between neurons, the model maintains its essential functionality and accuracy while significantly reducing the quantity of weights that need to be stored and processed, thus decreasing memory and computational resource requirements.
Solution Approach 2:
The patent applies different sparsity patterns to different regions or layers of the neural network based on their specific characteristics and importance. Critical layers or connections maintain higher density to preserve accuracy, while less critical regions are pruned more aggressively. This localized approach allows the model to maintain overall accuracy while reducing total resource requirements.
2Quantity of substance
If model pruning is applied to reduce network size, then memory and computational resources are reduced, but compute load balancing and idle time among processing elements deteriorate
Solution Approach 1:
The patent applies different sparsity patterns (such as row-sparsity, column-sparsity, or block-sparsity) to different layers or groups of weights based on their specific characteristics. This localized sparsity approach ensures that each processing element receives a balanced workload, as the sparsity pattern is tailored to the local structure of the weights rather than uniformly applied throughout the entire network.
Solution Approach 2:
The patent transforms the weight space by applying mathematical transformations that reorganize the weights according to specific sparsity patterns. These transformations change the parameters of the weight representation (such as ordering, grouping, or density distribution) to optimize for both memory efficiency and compute load balancing across processing elements, thereby reducing idle time while maintaining reduced network size.
3Quantity of substance
If sparsity patterns are applied to reduce computational overhead, then resource requirements are reduced, but hardware adaptability and deployment flexibility are limited
Solution Approach 1:
The patent develops sparsity transformation techniques that can be applied across different hardware platforms and network architectures. The weight space transformation methods are designed to be platform-agnostic, allowing the same pruned and transformed model to be deployed on various types of hardware (GPUs, TPUs, embedded processors, FPGAs) without requiring retraining or significant modification, thus maintaining hardware adaptability while reducing computational overhead.
Solution Approach 2:
The patent employs mathematical transformations of the weight space that preserve the functional behavior of the network while changing the representation parameters. These transformations (such as permutations, block reorganizations, or density-based reordering) allow the model to maintain its accuracy and adaptability across different hardware configurations while achieving reduced computational overhead through efficient sparsity exploitation.
Data Source
AI summary
Systems, methods, computer program products, and apparatuses to transform a weight space of an inference model to increase the compute efficiency of a target inference platform. A density of a weight space can be determined, and a transformation parameter derived based on the determined density. The weight space can be re-ordered based on the transformation parameter to balance the compute load between the processing elements (PEs) of the target platform, and as such, reduce the idle time and/or stalls of the PEs.