Approximating Neural Network Activation Functions via Baseline Difference Lookup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing activation functions in deep learning, such as GELU and SWISH, are difficult to implement in computation hardware due to their complexity, which hinders their adoption in practical applications despite their advantages in speed and accuracy.
Innovation Solution
The approach involves using a predefined baseline function, like ReLU, to store difference-functions between activation functions and the baseline in memory as lookup tables, allowing for efficient interpolation and approximation of activation functions, enabling the implementation of complex activation functions like GELU and SWISH in hardware.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If complex activation functions (GELU, SWISH) are implemented directly in computation hardware, then accuracy and speed of neural network evaluation are improved, but device complexity and difficulty of implementation increase significantly
Solution Approach 1:
The patent introduces a baseline function (typically ReLU) as an intermediary that is easy to implement in hardware. The complex activation function is decomposed into the baseline function plus a difference function. The baseline function handles the bulk of the computation with simple hardware, while the difference function (stored in lookup tables) provides the necessary complexity correction with minimal additional hardware overhead.
Solution Approach 2:
The difference functions are pre-computed and stored in lookup tables before runtime. This preliminary action transfers the computational complexity from runtime hardware operations to pre-processing and memory storage, allowing the hardware to simply perform memory retrieval and addition operations rather than complex real-time calculations.
2Productivity
If complex activation functions are implemented in computation hardware, then neural network performance is improved, but ease of manufacture and implementation deteriorates
Solution Approach 1:
The activation function implementation is segmented into two independent components: a baseline function module that can be manufactured using standard simple hardware circuits, and a difference function module that uses lookup tables. This segmentation allows each component to be manufactured and tested independently, significantly improving ease of manufacture while maintaining high neural network performance.
3Ease of manufacture
If lookup tables for difference functions are used to approximate activation functions, then ease of hardware implementation is improved, but manufacturing precision and approximation accuracy may be affected
Solution Approach 1:
The patent allows for adjustable parameters in the baseline function (such as the slope parameter in leaky ReLU) and supports variable precision in the lookup tables. By changing these parameters, the system can be optimized for different precision requirements while maintaining ease of hardware implementation. The lookup table precision can be tuned based on the specific application needs.
Data Source
AI summary
A computational apparatus for implementing a neural network model having multiple neurons that evaluate an activation function, the apparatus including a memory and circuitry. The memory is configured to hold values of a difference-function, each value being a respective difference between the activation function and a predefined baseline function. The circuitry is configured to evaluate the neural network model, including, for at least one of the neurons: evaluate the baseline function at the argument, retrieve from the memory one or more values of the difference-function responsively to the argument, and evaluate the activation function at the argument based on the baseline function at the argument and on the one or more values of the difference-function.