Neural Network Arithmetic Device Activation Function Look-Up Table
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network arithmetic devices face inefficiencies in processing and storing activation functions, which hinders the speed and versatility of neural network operations.
Innovation Solution
The implementation of an arithmetic device with a command decoder, AF circuit, and mode register that stores activation functions in a look-up table form, allowing for rapid selection and application of various activation functions like sigmoid, Tanh, ReLU, and Maxout without the need for complex arithmetic operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If activation functions are processed using complex arithmetic operations, then computational accuracy is maintained, but processing speed deteriorates
Solution Approach 1:
The patent pre-calculates activation function values and stores them in look-up tables before runtime. During neural network inference, the system simply retrieves pre-computed values from the tables based on input indices, eliminating the need for real-time complex arithmetic operations. This preliminary preparation of data resolves the contradiction by trading off memory storage for computational speed.
Solution Approach 2:
The patent creates simplified copies of activation function computations in the form of look-up tables. Instead of performing actual mathematical computations during inference, the system uses indexed access to copied pre-computed results. This copying approach maintains computational accuracy while dramatically improving processing speed by replacing complex arithmetic with simple memory access operations.
2Adaptability or versatility
If multiple activation functions are supported through separate circuit implementations, then functional versatility is improved, but device complexity increases
Solution Approach 1:
The patent implements a universal look-up table structure that can store multiple different activation function types (sigmoid, tanh, ReLU, Maxout, etc.) within a single unified circuit design. By using a single versatile lookup mechanism rather than separate dedicated circuits for each activation function, the system achieves multi-functionality while reducing overall device complexity. The same hardware structure adapts to different functions by loading appropriate pre-computed tables.
3Productivity
If look-up tables are used for activation functions, then processing speed is improved, but memory usage increases
Solution Approach 1:
The patent segments the look-up table storage into multiple separate tables, each dedicated to a specific activation function type. This segmentation allows the system to load only the required table for the current operation, reducing the amount of memory actively used at any given time. The segmented approach maintains high processing throughput by having optimized tables ready while managing memory consumption through selective loading and storage organization.
Data Source
AI summary
An arithmetic device includes an AF circuit including a first table storage circuit. The AF circuit stores a table input signal into one variable latch selected based on an input selection signal among variable latches included in the first table storage circuit in a look-up table form when a table set signal is activated. The AF circuit extracts a result value of a first activation function realized by a look-up table based on an input distribution signal to output the extracted result value as a fist table output signal for generating an output distribution signal.