Exponential Approximation Using Bit-Shift Taylor Factorials
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The computation of the exponential function, particularly in the softmax function of neural networks, is computationally expensive due to the need for significant floating-point operations and large lookup tables, which increases processing time and hardware requirements.
Innovation Solution
Approximate the exponential function using a Taylor series expansion with factorials replaced by nearest powers of 2, and decompose the argument into an integer and a power of 2, allowing bit-shift operations to replace divisions and multiplications, thereby reducing hardware complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the exponential function is computed using a Taylor series expansion with factorial division, then the approximation accuracy is improved, but the computational complexity and hardware requirements increase significantly
Solution Approach 1:
The patent changes the parameter representation by expressing the argument x as x = q + Δx where q is an integer and Δx is a fractional part. This decomposition allows the exponential computation to be separated into integer and fractional components, enabling the use of bit-shift operations for the integer part while maintaining accuracy through the fractional part handling.
Solution Approach 2:
The patent substitutes the mechanical division operation (which requires complex hardware circuitry) with bit-shift operations. By approximating factorial terms as powers of 2 and using the decomposition x = q + Δx, the division by factorial is replaced with right bit-shifts, significantly simplifying the hardware implementation while maintaining acceptable accuracy.
2Measurement precision
If the exponential function is computed using conventional methods with division operations, then the computational accuracy is maintained, but the processing time increases
Solution Approach 1:
The patent replaces slow division operations with fast bit-shift operations. By representing factorial terms as powers of 2 and using the argument decomposition, the computationally expensive division is substituted with simple bit-manipulation operations that execute much faster on digital processors.
Solution Approach 2:
The patent segments the computation into two parts: the integer part q handled by bit-shift operations for speed, and the fractional part Δx handled separately to maintain accuracy. This segmentation allows the time-critical parts to use fast operations while the accuracy-critical parts receive appropriate computational resources.
3Measurement precision
If large lookup tables are used to compute the exponential function, then the computation accuracy is improved, but the memory requirements and hardware size increase
Solution Approach 1:
The patent substitutes the memory-intensive lookup table approach with computation-based approximation using bit-shift operations. Instead of storing pre-computed values in large tables, the method computes exponential values on-the-fly using simple arithmetic operations, dramatically reducing memory requirements while maintaining acceptable accuracy.
Solution Approach 2:
The patent changes from a storage-based approach (lookup tables) to a computation-based approach (bit-shift operations). This parameter change in the computational methodology eliminates the need for large static memory structures, reducing hardware area while enabling dynamic computation of exponential values.
4Measurement precision
If complex circuitry is implemented to perform exact exponential computation, then the computational accuracy is improved, but the power consumption increases
Solution Approach 1:
The patent substitutes power-intensive complex circuitry with simple bit-shift and addition circuits. By replacing division operations and complex exponential computation logic with basic bit-manipulation operations, the hardware power consumption is dramatically reduced while maintaining sufficient computational accuracy for neural network applications.
Data Source
AI summary
A method for computing an approximate value A of the exponential function ex of an argument x. The method includes: approximating ex with a Taylor expansion T around x=0 that includes a predetermined number n of terms with i-th powers xi of the argument x divided by the respective factorial of i, with i=1, . . . , n, and in the computation of each term, approximating the factorial of i to the nearest power of 2, p(i!).