Neural Network Arithmetic Device Activation Function Circuit
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network arithmetic devices face challenges in efficiently storing and applying various activation functions, leading to increased complexity and storage requirements due to the need for specific design changes for different activation functions.
Innovation Solution
The proposed arithmetic device includes a function storage circuit and an activation function (AF) circuit that utilize a function selection signal to select and apply different activation functions, such as ReLU, Leaky ReLU, and ReLU-MAX, by setting slope and maximum values based on specific function information signals, allowing for flexible activation function application without requiring specific design changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If specific design changes are made for different activation functions, then the activation function can be accurately implemented, but the device complexity and storage requirements increase
Solution Approach 1:
The patent implements a universal activation function processing mechanism that can handle multiple activation functions (ReLU, Leaky ReLU, ReLU-MAX, etc.) through a single unified design. The function storage circuit stores multiple sets of function information signals (slope values, maximum values), and the selection circuit selects appropriate parameters based on the input data characteristics, eliminating the need for separate hardware designs for each activation function while maintaining implementation accuracy
Solution Approach 2:
The patent introduces dynamic parameter selection where the activation function parameters (slope, maximum value) are not fixed but are selected based on the input data characteristics. The selection circuit dynamically chooses the appropriate function information signal set from the function storage circuit, allowing the system to adapt to different activation functions without structural changes, thus reducing design complexity while maintaining accuracy
2Reliability
If multiple activation functions are stored separately, then each function can be accurately stored, but the storage space increases
Solution Approach 1:
The patent merges multiple activation function parameter sets into a single function storage circuit. Instead of having separate storage units for ReLU, Leaky ReLU, and other functions, the invention combines all function information signals (slope values, maximum values for different functions) into one unified storage circuit with selection logic, significantly reducing total storage space while maintaining accurate storage of each function's parameters
Solution Approach 2:
The patent segments the activation function parameters into key components (slope values, maximum values) that can be independently stored and selected. By dividing the function parameters into these fundamental elements, the system can store only the essential information needed for each activation function and reconstruct the full function behavior through parameter combination, reducing redundant storage
3Ease of manufacture
If the arithmetic device is designed for a specific activation function, then the design is simpler, but the adaptability to different functions decreases
Solution Approach 1:
The patent introduces a function storage circuit and selection circuit as intermediary components between the input data and the activation function processing unit. These intermediary circuits act as a bridge that translates different activation function requirements into unified parameter sets, allowing the core processing unit to remain simple while gaining the ability to handle multiple activation functions through parameter variation rather than structural complexity
Data Source
AI summary
An arithmetic device includes a function storage circuit and an activation function (AF) circuit. The function storage circuit stores and outputs a function selection signal, a first function information signal, and a second function information signal. The AF circuit generates an activation function result data by applying a slope value and a maximum value to a multiplication/accumulation (MAC) result data in a function setting mode that is activated by the function selection signal. The slope value is set based on the first function information signal, and the maximum value is set based on the second function information signal.