Programmable LUT for Neural Network Activation Approximation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current solutions for approximating activation functions in deep neural networks (DNNs) are inflexible and inefficient, particularly in supporting new activation functions and reducing computational overhead, as they rely on Digital Signal Processor (DSP) based implementations that fail to support various activation functions effectively.
Innovation Solution
A programmable look-up table (LUT) architecture is used to approximate non-linear activation functions using piece-wise linear or reciprocal approximations, allowing for flexible support of multiple activation functions by classifying input segments and storing configuration parameters in a LUT for efficient computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If DSP based implementations are used to approximate activation functions, then computational accuracy can be maintained, but flexibility in supporting new activation functions and computational efficiency deteriorates
Solution Approach 1:
The activation function is divided into multiple linear segments, each characterized by specific slope and intercept parameters. These segments are stored in a lookup table (LUT) structure, allowing the system to approximate non-linear activation functions by selecting appropriate linear segments based on input ranges. This segmentation enables flexible support for various activation functions while maintaining computational efficiency through pre-computed linear approximations.
Solution Approach 2:
The system changes parameters (slope and intercept values) in the linear segments to accommodate different activation functions. By storing configuration parameters in the LUT, the same hardware architecture can adapt to support multiple activation functions by simply loading different parameter sets, thereby improving flexibility without sacrificing computational efficiency.
2Productivity
If piece-wise linear approximation is used to approximate non-linear activation functions, then computational efficiency improves, but approximation accuracy for complex activation functions deteriorates
Solution Approach 1:
The input range of the activation function is divided into multiple segments, each approximated by a linear function with specific slope and intercept. By storing these segmented approximations in a lookup table, the system achieves computational efficiency through simple table lookup operations while maintaining sufficient accuracy for the segmented regions. The segmentation allows complex non-linear functions to be approximated using multiple simpler linear functions.
Solution Approach 2:
The system uses partial linear approximations that may not perfectly capture the entire non-linear activation function, but provides sufficient accuracy for practical purposes. By using more linear segments (excessive segmentation), the approximation accuracy improves while maintaining the computational benefits of linear operations, accepting that no single linear function can perfectly represent complex non-linear activation functions.
Data Source
AI summary
An activation function in a neural network may be approximated by one or more linear functions. A linear function may correspond to a segment of the input range of the activation function, e.g., a linear segment. A programmable look-up table may store slopes and intercepts of linear functions. A post processing engine (PPE) array executing the activation function may determine that an input data element of the activation function falls into the linear segment and compute an output of the linear function using the input data element. The output of the linear function may be used as the approximated output of the activation function. Alternatively, the PPE array may determine that the input data element is in a saturation segment and use a fixed value associated with the saturation segment as the approximated output of the activation function.