Neural Network Activator Circuit With Segmented SiLU Approximation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
DNNs face high computational demands and inefficiencies in computing activation functions, leading to significant power consumption and latency, especially in resource-constrained environments like mobile devices and IoT applications.
Innovation Solution
Embedding DNNs on IC devices with optimized activation function units that utilize linear approximation, segmentation, and error correction to reduce computational complexity and resource requirements, including hardware implementations of SiLU activation functions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If DNNs are used to achieve high accuracy in AI applications, then accuracy is improved, but computational cost and power consumption increase significantly
Solution Approach 1:
The patent transforms the nonlinear activation function computation into linear operations by changing the mathematical parameters and representation. Specifically, it uses piecewise linear approximation to replace complex nonlinear functions, and employs bitwise operations and lookup tables to compute activation functions with simplified parameters, thereby reducing power consumption while maintaining accuracy
Solution Approach 2:
The patent creates precomputed lookup tables that store activation function values, allowing the system to copy and retrieve precalculated results instead of performing complex computations in real-time. This copying approach significantly reduces computational power requirements during inference while preserving the accuracy of activation function outputs
2Measurement precision
If DNNs are used to achieve high accuracy, then accuracy is improved, but computation time and latency increase
Solution Approach 1:
The patent performs preliminary computation by precalculating and storing activation function values in lookup tables during the offline phase. This preliminary action allows the system to retrieve precomputed results during real-time inference, dramatically reducing computation time and latency while maintaining accuracy
Solution Approach 2:
The patent segments the activation function computation into discrete intervals and represents each segment with linear approximations. By dividing the continuous nonlinear function into multiple linear segments, the system can quickly determine which segment applies and use simple linear calculations instead of complex nonlinear computations, reducing latency
3Measurement precision
If complex activation functions are computed in hardware, then accuracy is maintained, but device complexity increases
Solution Approach 1:
The patent replaces complex hardware computation circuits with simpler lookup table structures and control logic. By copying precomputed values into accessible memory structures, the system maintains computational accuracy while dramatically simplifying the hardware architecture and reducing the number of logic gates required
Solution Approach 2:
The patent substitutes complex mathematical computation mechanisms with simpler data retrieval and bitwise operation mechanisms. Instead of implementing hardware circuits that perform floating-point arithmetic and nonlinear function evaluation, the system uses memory access and integer bitwise operations, which are simpler to implement in hardware
4Adaptability or versatility
If resource-constrained environments are used for deployment, then portability is improved, but computational efficiency decreases
Solution Approach 1:
The patent changes the computational parameters from floating-point arithmetic to integer bitwise operations, which are more efficiently executed on resource-constrained devices. This parameter transformation enables deployment on portable devices while maintaining computational efficiency through simplified arithmetic operations
Solution Approach 2:
The patent uses lookup tables stored in compact memory structures to copy precomputed activation values, enabling resource-constrained devices to achieve efficient computation without requiring complex processing units. This approach allows portable devices to maintain high computational efficiency despite limited hardware resources
Data Source
AI summary
An integrated circuit (IC) device may implement a deep neural network (DNN). The IC device may include an activator unit that implements a nonlinear activation function in the DNN. The nonlinear activation function may be decomposed into a rectified linear unit (ReLU) function and a symmetric function. After receiving an input value, the activator unit may apply the ReLU function on the input value to compute a first value. The input range of the nonlinear activation function may be partitioned into segments. The activator unit may determine which segment the input value falls into. The activator unit may apply a linear function, which approximates the symmetric function within the segment, on the input value to compute a second value. The activator unit may correct an error in the second value and compute an output of the nonlinear activation function based on the first value and the second value.