Analog Activation Circuits for Neural Network Energy Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) face inefficiencies due to the need for frequent data conversion between digital and analog domains for activation functions, which consumes significant energy and complicates the implementation of analog in-memory computing, especially with requirements for high-speed, high-linearity analog-digital converters that are power and area-intensive.
Innovation Solution
Implementing analog circuits using differential pairs of transistors, such as BJTs or MOSFETs, to compute activation functions like tanh and ReLU directly in the analog domain, reducing the need for digital conversions and simplifying hardware requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data conversion between digital and analog domains is performed for activation functions, then DNN computation can be implemented, but energy consumption increases significantly
Solution Approach 1:
The patent extracts the activation function computation from the digital domain and implements it directly in the analog domain using specialized analog circuits (tanh circuit and ReLU circuit). This separates the activation function operation from the general-purpose digital processing, eliminating the need for repeated analog-digital conversion and reducing energy consumption associated with data format transformation.
Solution Approach 2:
The patent replaces the mechanical/digital system of analog-to-digital conversion and digital processing with an analog electronic system that directly computes activation functions using continuous voltage signals. The tanh circuit uses operational amplifiers and resistors to compute hyperbolic tangent, while the ReLU circuit uses diodes and operational amplifiers, substituting digital conversion mechanisms with analog electronic computation.
2Measurement precision
If analog-digital converters are used for activation functions, then computation accuracy can be maintained, but hardware complexity and area increase
Solution Approach 1:
The patent extracts the activation function computation from the digital domain and implements it directly in the analog domain using specialized analog circuits (tanh circuit and ReLU circuit). This separates the activation function operation from the general-purpose digital processing, eliminating the need for repeated analog-digital conversion and reducing energy consumption associated with data format transformation.
Solution Approach 2:
The patent replaces the mechanical/digital system of analog-to-digital conversion and digital processing with an analog electronic system that directly computes activation functions using continuous voltage signals. The tanh circuit uses operational amplifiers and resistors to compute hyperbolic tangent, while the ReLU circuit uses diodes and operational amplifiers, substituting digital conversion mechanisms with analog electronic computation.
3Adaptability or versatility
If frequent data conversion between digital and analog domains is performed, then activation functions can be implemented, but power consumption increases
Solution Approach 1:
The patent merges the activation function computation with the analog neural network processing by implementing both tanh and ReLU activation functions directly in the analog domain. The tanh circuit and ReLU circuit are integrated into the analog processing pipeline, allowing activation functions to be computed alongside the main neural network operations without requiring separate digital conversion stages, thereby reducing overall power consumption.
Solution Approach 2:
The patent replaces the mechanical/digital system of analog-to-digital conversion and digital processing with an analog electronic system that directly computes activation functions using continuous voltage signals. The tanh circuit uses operational amplifiers and resistors to compute hyperbolic tangent, while the ReLU circuit uses diodes and operational amplifiers, substituting digital conversion mechanisms with analog electronic computation.
4Use of energy by moving object
If analog circuits are used for activation functions, then energy efficiency improves, but implementation complexity increases
Solution Approach 1:
The patent designs universal analog circuits that can implement multiple activation functions. The tanh circuit and ReLU circuit are designed as standardized building blocks that can be used across different neural network layers and configurations. These circuits use common analog components (operational amplifiers, resistors, diodes) and can be fabricated using standard analog IC processes, making them easier to manufacture despite the increased complexity of analog implementation.
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
An analog neural network including a hardware activation function is provided. A layer of the analog neural network includes a sequence of processing elements that receives analog signals, perform MAC operations on the analog signals, and generates analog outputs. The analog outputs are provided to an analog circuitry that can apply an activation function on the analog outputs. The output of the analog circuitry are also analog signals, which can further be provided to the next layer in the network. The analog circuitry may include a differential pair of transistors to compute the tanh activation function. Alternatively, the analog circuitry may include a comparator and multiplexer to compute the ReLU activation function. Compared with digital implementation of activation functions, the analog circuitry eliminates the need of converting the analog outputs of the layer to digital signals and the need of converting the result of the activation function to analog signals.