Structured Sparsity in Neural Networks for Memory Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks (ANNs) face challenges in reducing power consumption and memory requirements, particularly due to the inherent coupling between the control plane and data plane, which limits efficient memory access and interlayer memory usage.
Innovation Solution
The implementation of structured sparsity mechanisms that leverage prior knowledge of weight or activation patterns, allowing for reduced memory accesses and interlayer memory size by decoupling the data plane from the control plane, and guiding training to maximize sparsity in the weight domain.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional neural network architecture is used, then computational functionality is maintained, but memory requirements and power consumption are high
Solution Approach 1:
The patent applies parameter changes by transforming the data representation from full-precision floating-point to sparse formats with reduced bit-width. Specifically, it represents activation values and weights using fewer bits while maintaining computational functionality through structured sparsity patterns, thereby reducing memory bandwidth and storage requirements without significant loss in computational accuracy
Solution Approach 2:
The patent segments the neural network computation into distinct phases: a training phase where full-precision data is processed to learn weight matrices, and an inference phase where sparse representations are used for efficient computation. This temporal segmentation allows the system to use high precision during training and reduced precision during inference, optimizing the trade-off between accuracy and resource consumption
2Use of energy by moving object
If conventional neural network architecture is used, then computational functionality is maintained, but power consumption is high
Solution Approach 1:
The patent changes data representation parameters from full-precision to sparse, low-bit formats during inference, reducing the energy required for data movement and processing. The structured sparsity patterns enable efficient hardware implementation that consumes significantly less power compared to conventional dense matrix operations while maintaining computational effectiveness
Solution Approach 2:
The patent applies partial action by processing only the non-zero elements of weight matrices and activation vectors during inference, rather than processing all elements. This selective computation approach reduces the computational workload and associated power consumption while maintaining the essential computational functionality through the structured sparsity patterns
3Quantity of substance
If structured sparsity mechanisms are implemented, then memory requirements are reduced, but implementation complexity increases
Solution Approach 1:
The patent applies preliminary action by performing full-precision training computations in advance to learn and store weight matrices in a compressed sparse format. During inference, the system uses these pre-computed sparse representations, avoiding the need for complex real-time compression algorithms and reducing the operational complexity of the system
Solution Approach 2:
The patent extracts and removes zero elements from weight matrices and activation vectors, storing only the non-zero values along with their position information. This extraction process simplifies the data structure and reduces memory requirements while the position metadata maintains the structural information needed for correct computation, balancing complexity reduction with functionality
Data Source
AI summary
A novel and useful system and method of improved power performance and lowered memory requirements for an artificial neural network based on packing memory utilizing several structured sparsity mechanisms. The invention applies to neural network (NN) processing engines adapted to implement mechanisms to search for structured sparsity in weights and activations, resulting in a considerably reduced memory usage. The sparsity guided training mechanism synthesizes and generates structured sparsity weights. A compiler mechanism within a software development kit (SDK), manipulates structured weight domain sparsity to generate a sparse set of static weights for the NN. The structured sparsity static weights are loaded into the NN after compilation and utilized by both the structured weight domain sparsity mechanism and the structured activation domain sparsity mechanism. The application of structured sparsity lowers the span of search options and creates a relatively loose coupling between the data and control planes.