Tanh and Sigmoid Function Execution via Hardware Instructions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning neural networks face performance bottlenecks due to complex activation functions like tanh and sigmoid, which are computationally intensive and lack native hardware instructions for acceleration, leading to inefficiencies in training and deployment.
Innovation Solution
The implementation of dedicated instructions for tanh and sigmoid functions, such as math·tanh and math·sigmoid, which use piecewise quadratic interpolation to achieve high throughput and accuracy, reducing the computational burden by generating intermediate results and post-processing them with arithmetic logic unit instructions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If complex activation functions like tanh and sigmoid are used in deep learning neural networks, then the network can perform complex associative tasks with high accuracy, but the computational complexity and processing time increase significantly
Solution Approach 1:
The patent segments the complex tanh and sigmoid activation functions into multiple simpler computational steps. By breaking down the activation function computation into discrete operations that can be executed sequentially, the system maintains high accuracy while reducing the overall computational burden and enabling parallel processing optimization
Solution Approach 2:
The patent transforms the computational parameters of activation functions by using algebraic manipulations to convert transcendental function evaluations into polynomial-based computations. This parameter transformation maintains numerical accuracy while enabling the use of efficient hardware instructions for polynomial evaluation
2Measurement precision
If complex activation functions like tanh and sigmoid are used in deep learning neural networks, then the network can perform complex associative tasks with high accuracy, but the processing speed decreases due to lack of native hardware instructions
Solution Approach 1:
The patent substitutes the traditional software-based evaluation of complex activation functions with hardware-accelerated polynomial evaluation instructions. By replacing the mechanical computation of transcendental functions with optimized polynomial arithmetic that can leverage native CPU instructions, the system achieves both high accuracy and improved processing speed
Solution Approach 2:
The patent changes the computational parameters from direct transcendental function evaluation to polynomial approximation, enabling the use of efficient hardware instructions and improving processing speed while maintaining accuracy through careful selection of polynomial coefficients and evaluation methods
3Adaptability or versatility
If complex activation functions like tanh and sigmoid are used in deep learning neural networks, then the network can perform complex associative tasks, but the power consumption increases due to computational intensity
Solution Approach 1:
The patent segments the computation of activation functions into discrete, optimized steps that can be executed efficiently. By breaking down the computation into smaller operations that leverage hardware acceleration, the system maintains the capability to perform complex tasks while reducing overall power consumption through more efficient resource utilization
Solution Approach 2:
The patent substitutes power-intensive transcendental function evaluations with more energy-efficient polynomial computations that can utilize optimized hardware instructions. This substitution maintains the network's adaptability and versatility while significantly reducing the energy required for each activation function evaluation
4Adaptability or versatility
If complex activation functions like tanh and sigmoid are used in deep learning neural networks, then the network can perform complex associative tasks, but the training time increases due to computational bottleneck
Solution Approach 1:
The patent segments the activation function computation into optimized discrete operations that can be parallelized across multiple processing units. This segmentation maintains the network's capability to perform complex tasks while reducing training time by enabling efficient parallel execution of activation functions across large batches of data
Solution Approach 2:
The patent transforms the computational parameters to use polynomial approximations that can be evaluated more quickly than original transcendental functions. This parameter change maintains task capability while significantly reducing the time required for each forward and backward pass during training
Data Source
AI summary
Examples described herein relate to instructions to request performance of tanh and sigmoid instructions. For example, a compiler can generate native tanh instructions to perform tanh. In some examples, a tanh function can be compiled into instructions that include an instruction to perform either tanh(input) or tanh(input)/input depending on a value of the input to generate an intermediate output; an instruction to cause a performance of generation of scale factor based on the input; and an instruction to cause performance of a multiplication operation on the intermediate result with the scale factor. For example, a sigmoid function can be compiled to cause a math pipeline to perform a range check and performs operations based on a range.