Configurable Activation Function Circuit for Low-Latency AI Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning model processing systems face inefficiencies in handling nonlinear activation functions due to latency, power consumption, and memory usage when these functions are distributed across different processing devices, and conventional hardware implementations are limited in supporting a wide range of activation functions.
Innovation Solution
A configurable nonlinear activation (CNLA) function circuit that can be implemented in hardware, utilizing polynomial approximations and pipelined digital blocks with configurable bypasses and look-up tables to efficiently process a variety of nonlinear activation functions, co-located with other processing circuits like CIM arrays and DMACs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If fixed activation functions are used in neural network layers, then hardware implementation is simpler, but flexibility and adaptability are reduced
Solution Approach 1:
The patent implements configurable activation function circuits that can dynamically switch between different activation functions (ReLU, Leaky ReLU, Sigmoid, Tanh, etc.) based on training data characteristics. This dynamic configurability resolves the contradiction by allowing the system to adapt to different neural network layer requirements while maintaining a unified hardware architecture, thus achieving flexibility without proportionally increasing complexity.
Solution Approach 2:
The activation function circuit is designed as a universal module capable of performing multiple activation functions through configurable parameters. By making the circuit multi-functional, the patent eliminates the need for separate dedicated circuits for each activation function, thereby achieving adaptability while controlling device complexity through resource sharing.
2Measurement precision
If activation functions are configured based on training data characteristics, then model accuracy improves, but computational overhead increases
Solution Approach 1:
The system performs preliminary analysis of training data characteristics (such as data distribution, activation patterns) during the training phase to determine optimal activation function configurations. This preliminary action allows the inference phase to use pre-determined configurations without repeated computational overhead, thus achieving high accuracy while minimizing runtime computational costs.
Solution Approach 2:
The patent utilizes configurable parameters within activation function circuits to adapt to different data characteristics. By changing parameters (such as the alpha parameter in Leaky ReLU) based on training data analysis, the system achieves accurate modeling without requiring complex computational structures, thereby balancing accuracy with computational efficiency.
3Productivity
If different activation functions are implemented for different neural network layers, then model performance improves, but hardware complexity increases
Solution Approach 1:
The patent employs universal activation function circuits that can be configured to implement different activation functions in different neural network layers. This multi-functional design allows the same hardware module to serve multiple purposes, enabling optimized performance for each layer without proportionally increasing hardware complexity through duplication.
Solution Approach 2:
The system dynamically configures activation function parameters and types based on the specific requirements of each neural network layer during operation. This dynamic adaptability allows the hardware to optimize performance for different layers (e.g., using ReLU for convolutional layers and Tanh for recurrent layers) while maintaining a unified and manageable hardware architecture.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Certain aspects of the present disclosure provide a processor, comprising: a configurable nonlinear activation function circuit configured to: determine, based on a selected nonlinear activation function, a set of parameters for the nonlinear activation function; and generate output data based on application of the set of parameters for the nonlinear activation function, wherein: the configurable nonlinear activation function circuit comprises at least one nonlinear approximator comprising at least two successive linear approximators, and each linear approximator of the at least two successive linear approximators is configured to approximate a linear function using one or more function parameters of the set of parameters.