Programmable Mapping Table for Neural Network Activation Functions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware implementations of activation functions in neural networks face challenges in balancing accuracy and speed, with lookup tables improving speed but reducing accuracy due to quantization errors, and the inflexibility of fixed accuracy limiting further improvements.
Innovation Solution
A programmable mapping table system that maps input boundary values to mathematical function parameters, allowing for a two-stage process to estimate activation function outputs, enabling improved accuracy and flexibility in trading off speed and accuracy by adjusting the granularity of input boundary values and function parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If lookup tables are used to implement activation functions, then processing speed is improved, but accuracy deteriorates due to quantization errors
Solution Approach 1:
The patent implements a dynamic approach where the system can switch between different activation function implementation methods (lookup tables vs. computational methods) based on operational requirements. The neural network processor includes multiple activation function units that can be selectively activated, allowing the system to adapt between speed-optimized and accuracy-optimized modes depending on the specific processing task and input data characteristics.
Solution Approach 2:
The patent segments the activation function processing into multiple specialized units (first activation function unit, second activation function unit, etc.), each handling different aspects or types of activation functions. This segmentation allows parallel processing paths where lookup tables can be used for simple cases while more accurate computational methods are applied to complex cases, resolving the contradiction between speed and accuracy.
2Device complexity
If fixed accuracy is used in activation function implementation, then device complexity is reduced, but adaptability deteriorates
Solution Approach 1:
The patent designs a universal activation function processing architecture that can handle multiple different activation functions (sigmoid, tanh, ReLU, Leaky ReLU, ELU, etc.) through a unified hardware structure. The neural network processor includes configurable parameters (alpha, lambda, gamma) that can be programmed to implement various activation function types, providing high adaptability without proportionally increasing device complexity.
Solution Approach 2:
The patent uses programmable parameters (alpha, lambda, gamma) to control the behavior of activation functions. By changing these parameters, the same hardware can adapt to different activation function types and accuracy requirements. This parameter-based control allows flexible adaptation while maintaining a relatively simple fixed hardware structure.
Data Source
AI summary
Methods and systems for performing hardware computations of mathematical functions are provided. In one example, a system comprises a mapping table that maps each base value of a plurality of base values to parameters related to a mathematical function; a selection module configured to select, based on an input value, a first base value and first parameters mapped to the first base value in the mapping table; and arithmetic circuits configured to: receive, from the mapping table, the first base value and the first plurality of parameters; and compute, based on a relationship between the input value and the first base value, and based on the first parameters, an estimated output value of the mathematical function for the input value.