Analog Memory-Based Activation Function for Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning neural networks rely on digital circuits for activation functions, which are not energy-efficient and require data movement from memory to processors for computation, limiting performance and energy efficiency.
Innovation Solution
Implementing analog memory-based activation functions using non-volatile memory devices connected in parallel with diodes or field effect transistors, where parameters defining the activation function slopes are stored and used directly in the crossbar array for in-memory computing, eliminating the need for digital-to-analog conversions and enhancing energy efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If digital circuits are used for activation functions, then computation accuracy is maintained, but energy efficiency deteriorates and data movement requirements increase
Solution Approach 1:
The patent merges the activation function computation with the memory storage by implementing analog activation functions directly in non-volatile memory devices. This combines the functions of storage and computation, eliminating the need for separate digital circuit processing and reducing data movement between memory and processor.
Solution Approach 2:
The patent replaces the digital mechanical system (digital circuits with ADCs) with an analog physical system (analog activation functions implemented through non-volatile memory conductance properties). This substitution enables direct computation in memory, reducing energy consumption and eliminating data movement requirements.
2Measurement precision
If analog-to-digital converters are used for activation functions, then digital processing accuracy is achieved, but energy consumption increases
Solution Approach 1:
The patent eliminates the need for analog-to-digital converters by implementing activation functions directly in the analog domain within non-volatile memory. The conductance-based analog activation functions perform computations directly without requiring digital conversion, thereby maintaining accuracy while eliminating the energy consumption associated with ADCs.
Solution Approach 2:
The non-volatile memory devices perform the activation function computation themselves through their inherent analog properties, without requiring external digital processing circuits. The memory devices serve dual purposes: storing data and performing the activation function, thereby eliminating the need for separate energy-intensive processing components.
3Productivity
If data is moved from memory to processors for computation, then digital processing capability is utilized, but performance and energy efficiency deteriorate
Solution Approach 1:
The patent combines memory storage and computation into a single integrated system where non-volatile memory devices simultaneously perform both functions. This merging eliminates the need to move data between memory and processor, thereby improving performance by eliminating data transfer bottlenecks while reducing energy consumption by eliminating the need for separate processing hardware.
Solution Approach 2:
The patent creates an equipotential system where memory and computation occur at the same location without requiring data movement. The analog activation functions in non-volatile memory operate at the same potential level as the stored data, eliminating the need for data transfer and associated energy consumption while maximizing computational efficiency.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach significantly improves energy efficiency and performance by performing computations directly in memory, reducing data movement and power consumption, while maintaining the accuracy of activation functions like Leaky ReLU and ReLU in analog memory-based neural networks.
Implementation Method 1
non-volatile memory devices connected in parallel with diodes or field effect transistors, where parameters defining the activation function slopes are stored
Implementation Method 2
non-volatile memory devices connected in parallel with diodes or field effect transistors
Implementation Method 3
non-volatile memory devices connected in parallel with diodes or field effect transistors
Data Source
AI summary
Analog memory-based activation function for an artificial neural network can be provided. An apparatus can include at least two non-volatile memory devices connected in parallel such that the current can flow through one of the two non-volatile memory devices depending on the voltage level driving the current. To control which branch an input current flows through, each of the two non-volatile memory devices can be connected to a circuit element that can function as a switch, for example, a diode such as a semiconductor diode, a transistor, or another circuit element. Such apparatus can implement an analog memory-based activation function, for example, for an analog memory-based artificial neural network.