Parallel Softmax Fragment Processing to Cut Memory Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The efficiency of Softmax activation function computation is hindered by memory access overhead and the risk of memory overflow due to large input vectors, particularly in parallel computing environments where accessing off-core or off-chip memory is slow and floating-point representation can lead to overflow errors and roundoff errors.
Innovation Solution
A method is introduced to compute the Softmax activation function by determining the natural logarithm of the sum of exponentials, reducing memory access by dividing the input vector into fragments processed in parallel, and using a max* operation to approximate the natural logarithm of the sum of exponentials without requiring a global maximum value, thus minimizing memory access and overflow risks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the input vector is processed in parallel using multiple processing units, then the computational speed is improved, but the memory access overhead increases due to the need to access off-core or off-chip memory
Solution Approach 1:
The input vector is divided into multiple fragments, with each fragment assigned to a different processing unit. This segmentation allows parallel processing of vector elements while reducing the amount of data each processing unit needs to access from off-chip memory, thereby minimizing memory access overhead while maintaining computational speed.
2Reliability
If the maximum value of the input vector is determined to prevent overflow, then the reliability is improved, but the memory access overhead increases due to requiring global maximum value
Solution Approach 1:
The determination of the maximum value is segmented across processing units. Each processing unit determines the maximum value of its local fragment independently in parallel, eliminating the need for one processing unit to access all vector elements from memory. This approach maintains overflow prevention reliability while significantly reducing memory access overhead.
3Loss of time
If the input vector is divided into fragments for parallel processing, then the memory access overhead is reduced, but the device complexity increases due to coordination requirements
Solution Approach 1:
The input vector is divided into fragments that can be processed independently by different processing units. Each processing unit operates autonomously on its fragment without requiring complex coordination with other units, simplifying the overall system architecture while reducing memory access overhead.
Solution Approach 2:
Each processing unit independently determines the maximum value of its own fragment and uses this local maximum for normalization, eliminating the need for complex inter-unit communication and coordination mechanisms.
Data Source
AI summary
A computer comprising a plurality of processing units, each processing unit having an execution unit and access to computer memory which stores code executable by the execution unit and input values of an input vector to be processed by the code, the code, when executed, configured to access the computer memory to obtain multiple pairs of input values of the input vector, determine a maximum or corrected maximum input value of each pair as a maximum result element, determine and store in a computer memory a maximum or corrected maximum result of each pair of maximum result elements as an approximation to the natural log of the sum of the exponents of the input values and access the computer memory to obtain each input value and apply it to the maximum or corrected maximum result to generate each output value of a Softmax output vector.