Taylor Series Activation Function Unit for DNN Accelerators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current implementations of activation functions in deep neural networks (DNNs) face inefficiencies due to high computation costs and limited resources, particularly in resource-constrained devices, as they often rely on Look-up-Table (LUT) or Digital Signal Processor (DSP) cores, which require additional area and incur performance overheads.
Innovation Solution
The proposed solution involves a DNN accelerator with an activation function unit that computes approximations of activation functions using Taylor series, allowing for programmable accuracy-performance trade-offs by controlling the number of Taylor series terms, thereby achieving high accuracy with minimal performance penalty.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If Look-up-Table (LUT) or Digital Signal Processor (DSP) cores are used for activation functions, then computation accuracy is maintained, but device area increases and performance overhead occurs
Solution Approach 1:
The patent changes the computational approach from LUT/DSP-based exact computation to polynomial approximation, transforming the activation function computation into a series of simpler arithmetic operations (multiplications and additions) that can be performed with lower hardware complexity while maintaining acceptable accuracy through controlled polynomial degree selection
2Measurement precision
If traditional activation function computation methods are used, then accuracy is achieved, but computation time and clock cycles increase
Solution Approach 1:
The patent replaces the traditional LUT/DSP-based mechanical computation system with a polynomial approximation system that uses simpler arithmetic operations, significantly reducing the computational complexity and clock cycles required while maintaining acceptable accuracy through the mathematical properties of polynomial expansions
3Productivity
If polynomial approximation is used for activation functions, then computation efficiency improves, but accuracy may be compromised
Solution Approach 1:
The patent introduces dynamic controllability to the polynomial approximation by allowing the degree of the polynomial to be adjusted based on the required accuracy and performance trade-offs, enabling the system to adapt between higher accuracy (higher degree) and higher efficiency (lower degree) as needed
Data Source
AI summary
An activation function unit can compute activation functions approximated by Taylor series. The activation function unit may include a plurality of compute elements. Each compute element may include two multipliers and an accumulator. The first multiplier may compute intermediate products using an activation, such as an output activation of a DNN layer. The second multiplier may compute terms of Taylor series approximating an activation function based on the intermediate products from the first multiplier and coefficients of the Taylor series. The accumulator may compute a partial sum of the terms as an output of the activation function. The number of the terms may be determined based on a predetermined accuracy of the output of the activation function. The activation function unit may process multiple activations. Different activations may be input into different compute elements in different clock cycles. The activation function unit may compute activation functions with different accuracies.