Programmable Lookup Tables for Adaptive DNN Activation Functions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep neural networks (DNNs) face challenges in efficiently implementing activation functions in resource-constrained environments where power consumption, processing capabilities, or silicon area are limited, and their definition can change over time due to additional training.
Innovation Solution
A method involving a programmable lookup table in an activation module for DNNs, which generates and loads lookup data to approximate activation functions over a determined range of input values, allowing flexible and efficient implementation of multiple activation functions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a fixed lookup table is used for activation functions, then hardware area and power consumption are reduced, but adaptability to changing DNN definitions is lost
Solution Approach 1:
The patent implements a dynamic lookup table that can be reconfigured at runtime based on the specific activation function required. Instead of a fixed static table, the system monitors input ranges and dynamically loads appropriate lookup data into the table, allowing the same hardware to adapt to different DNN definitions and activation functions without requiring additional hardware area for multiple fixed tables.
2Measurement precision
If a large lookup table covers all possible input ranges, then accuracy is improved, but silicon area and power consumption increase
Solution Approach 1:
The patent segments the activation function approximation into multiple smaller lookup tables, each optimized for a specific input range. Instead of using one large table covering all possible inputs, the system divides the input space and creates specialized tables for different segments. The monitoring mechanism identifies which segment is currently active and loads only the corresponding table, maintaining high accuracy for each segment while keeping individual table sizes manageable.
Solution Approach 2:
The patent applies local quality by creating lookup tables with different characteristics optimized for specific local regions of the input space. Each lookup table is tailored to accurately represent the activation function within its specific input range, rather than using a single uniform table. This allows high precision where needed while minimizing overall hardware area by only loading the locally appropriate table.
3Adaptability or versatility
If multiple lookup tables are maintained for different activation functions, then versatility is improved, but device complexity increases
Solution Approach 1:
The patent implements a universal lookup table structure that can serve multiple activation functions through dynamic reconfiguration. Instead of maintaining separate dedicated tables for each activation function, the system uses a single lookup table infrastructure that can be programmed with different activation function data as needed. The monitoring and loading mechanism enables this one table to universally support multiple functions by loading the appropriate data before each activation operation.
Data Source
AI summary
Hardware for implementing a Deep Neural Network (DNN) for performing an activation function includes, at an activation module for performing an activation function, a programmable lookup table for storing lookup data approximating the activation function over a first range of input values to the activation module, the method comprising: providing calibration data to a representation of the hardware; monitoring an input to an activation module of the representation of the hardware so as to determine a range of input values to the activation module; generating lookup data for the lookup table representing the activation function over the determined range of input values; and loading the generated lookup data into the lookup table of the hardware, thereby configuring the activation module of the hardware for performing the activation function over the determined range of input values.