Softmax Hardware Mapping for Numerically Stable Neural Accelerators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Implementing a softmax layer in neural network accelerators (NNAs) is challenging due to its complexity and numerical instability issues, especially when dealing with large input values, which can lead to overflow or underflow, and existing NNAs are not designed to efficiently perform this operation.
Innovation Solution
The softmax layer is implemented using a series of elementary neural network operations, including maximum, subtraction, exponential, and division, which are mapped to existing hardware modules in NNAs, such as convolution engines and activation units, to perform the necessary calculations efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If softmax layer is implemented using traditional dedicated hardware, then computational accuracy is improved, but device complexity and hardware cost increase
Solution Approach 1:
The patent implements softmax layer using existing universal hardware components (convolution engines, activation units, pooling units) that can perform multiple functions. Instead of dedicated softmax hardware, the system reuses standard NNAs components to achieve the same computational accuracy, thereby reducing hardware complexity and cost while maintaining functionality.
2Adaptability or versatility
If softmax layer is implemented using existing NNA hardware, then hardware utilization is improved, but computational efficiency deteriorates
Solution Approach 1:
The patent segments the softmax layer computation into distinct operational steps (maximum value calculation, subtraction, exponential, division) that can be mapped to different existing hardware modules. This segmentation allows each step to be optimized and executed by appropriate existing components, improving overall computational efficiency while maximizing hardware utilization.
Solution Approach 2:
The system dynamically maps different parts of the softmax computation to different hardware modules based on the specific computational requirements. The controller flexibly assigns operations to convolution engines, activation units, or pooling units as needed, enabling adaptive hardware utilization that maintains high computational efficiency across different scenarios.
3Device complexity
If softmax layer is implemented without dedicated hardware, then device complexity is reduced, but computational performance deteriorates
Solution Approach 1:
The patent ensures continuous and efficient computation by mapping softmax operations to existing hardware that can process data continuously. The segmented approach allows overlapping operations across different hardware modules, maintaining productive action flow without the need for dedicated hardware, thus achieving both low complexity and high performance.
4Adaptability or versatility
If softmax layer is implemented using elementary operations mapping, then hardware utilization is improved, but implementation complexity increases
Solution Approach 1:
The system uses a controller that automatically handles the mapping of softmax operations to appropriate hardware modules based on the computational graph. This self-service approach abstracts the implementation complexity from the user, allowing high hardware utilization through automated resource allocation while keeping the user interface simple and the overall system manageable.
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
Methods are disclosed for implementing an exponential operation, and a softmax neural network layer, in neural network accelerator hardware. Also disclosed are a data processing system for implementing the exponential operation and a data processing system for implementing the softmax layer. The exponential operation or softmax layer is mapped to a plurality of elementary neural network operations, and the neural network accelerator hardware evaluates these operations, to produce the result of the operation or layer respectively.