Hyperbolic Tangent Function Approximation Using Base-2 Bit Shifts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing approximations of sigmoid, hyperbolic tangent, and exponential linear unit functions are computationally expensive due to the use of division and exponentiation operations, which are inefficient and resource-intensive, especially in applications requiring derivatives of these functions.
Innovation Solution
The proposed solution involves approximating these functions using simplified arithmetic operations and bit shifts, eliminating division and replacing exponentiations of base e with exponentiations of base 2, which can be efficiently calculated using bit shifts. This approach allows for the implementation of these functions on lower-power processors, such as those found in mobile devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing approximations of sigmoid, hyperbolic tangent, and exponential linear unit functions are used, then the functions can be applied in image processing and artificial intelligence, but the computational cost is high due to division and exponentiation operations
Solution Approach 1:
The patent changes the base of exponentiation from e to 2, and uses piecewise linear approximations instead of continuous nonlinear functions. This parameter change allows the use of bit shift operations instead of expensive exponentiation, while maintaining sufficient accuracy for machine learning applications
Solution Approach 2:
The patent replaces the mathematical mechanism of division and exponentiation with bitwise operations and simple arithmetic. Specifically, it substitutes costly floating-point operations with integer-based bit shifts and additions, dramatically improving computational efficiency
2Reliability
If existing approximations meeting required accuracy levels are used, then the functions work correctly in applications, but the computational resources and time are excessive
Solution Approach 1:
The patent segments the domain of the functions into different regions (positive and negative inputs) and applies different simplified formulas to each region. This segmentation allows each piece to be computed efficiently while collectively maintaining the overall function behavior
Solution Approach 2:
The patent uses piecewise linear approximations that are simpler than the exact functions but sufficient for the required accuracy in machine learning applications. The approximation is intentionally simplified to prioritize speed over exact precision
3Productivity
If simplified approximations are used to reduce computational cost, then execution speed improves, but accuracy may be compromised
Solution Approach 1:
The patent changes the base of exponentiation from e to 2, and uses piecewise linear approximations instead of continuous nonlinear functions. This parameter change allows the use of bit shift operations instead of expensive exponentiation, while maintaining sufficient accuracy for machine learning applications
Data Source
AI summary
Accelerated approximations of functions, including: approximating, by a computing device, a hyperbolic tangent function applied to an input by: where the input is less than zero: performing a first exponentiation comprising raising a first base of two to a first exponent equal to double the input; and subtracting one from a result of the first exponentiation; and where the input is greater than zero, subtracting from one a result of a second exponentiation comprising raising a second base of two to a second exponent equal to a negative of double the input.