Activation Function Lookup Tables for Resource-Constrained DNN Hardware
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Neural Networks (DNNs) face challenges in efficient implementation in resource-constrained environments where power consumption, processing capabilities, or silicon area are limited, and their definition can change over time due to additional training.
Innovation Solution
A method for configuring hardware to implement DNNs by using a programmable lookup table to approximate activation functions, involving calibration, monitoring input ranges, and generating lookup data to efficiently perform activation functions, with features like interpolation and extrapolation to handle out-of-range inputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a fixed lookup table is used for activation functions, then the hardware implementation is simple and fast, but it cannot adapt to changing DNN definitions and additional training requirements
Solution Approach 1:
The patent implements a dynamically reconfigurable lookup table that can be updated with new activation function data during runtime. The system monitors input ranges and automatically regenerates lookup table entries when the DNN definition changes or additional training is performed, allowing the hardware to adapt to new requirements without redesign.
Solution Approach 2:
The system performs preliminary monitoring of input ranges to determine when lookup table regeneration is needed. By detecting changes in input distributions before they affect processing accuracy, the system can proactively update the lookup table entries, ensuring continued adaptability while maintaining efficient operation.
2Reliability
If a large lookup table covers all possible input ranges, then the activation function can handle any input value, but the silicon area and power consumption increase
Solution Approach 1:
The patent implements a localized lookup table approach that only stores activation function values for the currently relevant input range. By monitoring which input values actually occur during processing, the system maintains lookup table entries only for those local regions of the input domain, dramatically reducing silicon area while ensuring all actual inputs are covered.
Solution Approach 2:
The system dynamically changes the parameter range of the lookup table based on monitored input distributions. When the input range shifts due to additional training or different DNN configurations, the lookup table is regenerated to cover the new relevant range, maintaining full coverage adaptively without requiring a permanently large table.
3Measurement precision
If the lookup table is regenerated for each activation function in a series, then each function is accurately represented, but the processing time increases
Solution Approach 1:
The patent combines multiple activation function representations into a single shared lookup table structure. By monitoring the union of input ranges across all activation functions in the series and generating a unified lookup table that covers this combined range, the system eliminates the need to regenerate separate tables for each function, maintaining accuracy while improving throughput.
Solution Approach 2:
The system ensures continuous accurate activation function evaluation by maintaining the lookup table in a ready state that covers the expected input range. Rather than interrupting processing to regenerate tables, the lookup table is kept updated continuously or pre-updated based on monitored ranges, ensuring uninterrupted accurate processing of all activation functions.
Data Source
AI summary
Hardware is configured for implementing a Deep Neural Network (DNN) for performing an activation function. A programmable lookup table for storing lookup data approximating the activation function is provided at an activation module for performing the activation function. Training data is provided to an input layer of a representation of the hardware, wherein the representation of the hardware is configured to implement the DNN, to configure the DNN by using the training data, wherein configuring the DNN comprises determining lookup data for the lookup table representing the activation function. The lookup data is loaded into the lookup table of the hardware, thereby configuring the activation module of the hardware for performing the activation function during post-training operation.