Activation Function LUT Coefficient Switching for Smaller AI Circuits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for implementing activation functions in neural networks using look-up tables (LUTs) face challenges in balancing circuit scale with accuracy and throughput, as increasing the number of pieces for better accuracy leads to larger tables and more complex wiring, while reducing pieces compromises approximation accuracy.
Innovation Solution
A data processing device and method that uses a look-up table with polynomial approximation, storing more coefficients than table stages, and selectively updates these coefficients to perform polynomial operations efficiently, reducing circuit scale and update frequency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the number of pieces for polynomial approximation is increased to improve approximation accuracy, then the table size of the LUT and the complexity of wiring increase, leading to an increase in circuit scale
Solution Approach 1:
The patent divides the input domain into multiple pieces and performs polynomial approximation on each piece separately. By segmenting the approximation task, the system can achieve high overall accuracy without requiring an excessively large LUT, as each segment uses a simpler polynomial model.
Solution Approach 2:
The patent pre-calculates and stores polynomial coefficients for each piece in the LUT before runtime. This preliminary action allows the runtime operation to simply retrieve and apply coefficients rather than performing complex calculations, reducing the circuit scale while maintaining accuracy.
2Device complexity
If the number of pieces for polynomial approximation is reduced to decrease table size and wiring complexity, then the approximation accuracy decreases
Solution Approach 1:
The patent changes the polynomial degree parameter for different pieces based on their characteristics. By adjusting the polynomial order dynamically across different input ranges, the system achieves high accuracy with fewer pieces, thereby reducing LUT size and circuit complexity.
3Adaptability or versatility
If the LUT is updated frequently to accommodate different activation functions, then the update delay increases and processing throughput is reduced
Solution Approach 1:
The patent designs a universal LUT structure that can store polynomial coefficients for multiple different activation functions simultaneously. This multi-functional design allows the system to switch between different activation functions without frequent updates, maintaining high throughput while supporting versatility.
Solution Approach 2:
The patent pre-loads polynomial coefficients for multiple activation functions into the LUT during initialization or idle periods. This preliminary preparation eliminates the need for frequent updates during processing, reducing update delays and maintaining high processing throughput.
Data Source
AI summary
A data processing device includes a processing unit. A processing unit selects an input value included in an input domain of a processing LUT from among a plurality of the input values that are values of inputs, selects only an approximation coefficient of a piece necessary for an operation from a total coefficient storage unit, stores the selected approximation coefficient in the processing LUT, outputs an approximation coefficient corresponding to the selected input value from the processing LUT, and performs polynomial approximation operation by using the selected input value and the output approximation coefficient.