Neural Network Neuron Removal for Hardware Resource Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks require significant computational resources and memory, making them unsuitable for embedded systems with limited power and processing capacity, such as wearables and smartphones, due to their high computational intensity and memory bandwidth demands.
Innovation Solution
The method involves determining ranges for neurons in a trained neural network model based on training datasets, removing neurons with negative values to decrease hardware resources, and generating an optimized circuit layout for hardware implementation, which reduces computational resources while maintaining minimal loss in accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural networks are implemented with high computational complexity to achieve state-of-the-art classification rates, then classification accuracy is improved, but hardware resource consumption and energy usage increase significantly
Solution Approach 1:
The patent extracts and removes redundant neurons from the neural network model based on range analysis. By determining the output range of each neuron and identifying neurons with always-negative ranges (which are always zeroed by ReLU), the patent selectively removes these neurons to reduce hardware resources while maintaining classification accuracy.
Solution Approach 2:
The patent changes the structural parameters of the neural network by removing specific neurons identified through range analysis. This parameter change reduces the number of computational units (adders and multipliers) in the hardware implementation, thereby reducing device complexity and energy consumption while preserving the essential classification functionality.
2Measurement precision
If the number of neurons in the neural network is increased to improve model capacity, then classification performance is improved, but memory bandwidth requirements and latency increase
Solution Approach 1:
The patent removes neurons that contribute minimally to classification performance but increase inference latency. By analyzing neuron output ranges and removing always-zero neurons, the patent reduces the number of MAC operations required, thereby reducing memory bandwidth requirements and inference latency while maintaining model capacity for accurate classification.
3Reliability
If full precision neural network computation is maintained to ensure accuracy, then classification reliability is improved, but energy consumption and computational overhead increase
Solution Approach 1:
The patent removes redundant computational elements (neurons with always-negative ranges) that consume energy but do not contribute to classification reliability. This extraction reduces the number of active computational units in the hardware, thereby reducing energy consumption while maintaining the reliability of classifications performed by the remaining essential neurons.
Data Source
AI summary
Methods and systems for range-based hardware optimization of a neural network model are disclosed. The methods and systems include: obtaining a trained neural network model, the trained neural network model comprising: a plurality of neurons; determining a plurality of ranges for the plurality of neurons based on a plurality of training datasets, the plurality of ranges corresponding to the plurality of neurons; removing a first neuron from the trained neural network model based on a first range of the plurality of ranges to decrease hardware computational resources utilized for the first neuron; and generating an optimized neural network model based on the plurality of neurons without the first neuron. Other aspects, embodiments, and features are also claimed and described.