Triangular Mask SoftMax Circuit for Higher-Throughput FPGA Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware accelerators for machine learning tasks, such as FPGAs, face inefficiencies in memory storage, bandwidth, and computational overhead due to the application of triangular masks for autoregressive constraints, leading to wasted computations and increased resource consumption.
Innovation Solution
Implementing a triangular mask within the hardware accelerator logic and combining unmasked data rows to perform operations like SoftMax in a single pass, eliminating the need for transferring masks and reducing computational load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a triangular mask is applied to input data for autoregressive constraints, then the model learns to make predictions based only on earlier tokens, but computational resources are wasted on masked data values and memory bandwidth is increased
Solution Approach 1:
The patent extracts only the unmasked data values from the input matrix and processes them, while completely excluding the masked values from computation. This is achieved by identifying the triangular mask pattern and selectively processing only the valid data portion, thereby eliminating wasted computations on masked values while maintaining the autoregressive constraint.
Solution Approach 2:
The patent segments the computation process into distinct phases: mask application, unmasked value extraction, and processing. By dividing the data processing into segments that handle masked and unmasked values separately, the system avoids performing operations on masked data while maintaining the integrity of the autoregressive constraint.
2Reliability
If a triangular mask is applied to input data for autoregressive constraints, then the model learns to make predictions based only on earlier tokens, but memory bandwidth and storage requirements are increased
Solution Approach 1:
The patent extracts only the necessary unmasked data values from memory and processes them directly, avoiding the need to load and process the entire masked matrix. This extraction approach reduces memory bandwidth requirements by transferring only the essential data portions while maintaining the autoregressive constraint through selective processing.
3Ease of operation
If mask data is transferred from host to hardware accelerator, then the mask can be applied to input data, but bandwidth usage and computational overhead are increased
Solution Approach 1:
The patent performs preliminary action by pre-configuring the hardware accelerator with the triangular mask logic and organizing the input data structure to inherently reflect the mask pattern. This allows the system to directly process only unmasked values without requiring runtime mask data transfer or explicit mask application operations, thereby reducing computational overhead while maintaining ease of operation.
4Ease of operation
If computations are performed on all input data values including masked ones, then the hardware accelerator can process data in a straightforward manner, but energy consumption and computational load are increased
Solution Approach 1:
The patent extracts only the unmasked data values for processing, eliminating the need to perform computations on masked values. This extraction approach maintains processing simplicity by directly feeding only the necessary data into the computational pipeline while significantly reducing energy consumption by avoiding unnecessary operations on masked data.
Solution Approach 2:
The patent applies partial action by performing computations only on the subset of unmasked data values rather than all input values. This partial processing approach reduces computational load and energy consumption by exactly the amount corresponding to the masked portion, while still achieving the complete autoregressive constraint enforcement.
Data Source
AI summary
A field programmable gate array including a configurable interconnect fabric connecting logic blocks implementing a circuit to: receive input data including data values organized into rows and columns, each row having N data values; select R[i] unmasked data values of a row of the input data in accordance with a mask and an index i of the row; select N−[i] unmasked data values of another row of the input data in accordance with the mask and an index of the another row; merge the R[i] unmasked data values of the row and the N−[i] data values of the another row into a combined data vector of N data values; and compute R[i] normalized values based on the R[i] unmasked data values of the combined data vector and N−[i] normalized values based on the N−[i] data values of the combined data vector to generate N normalized data values.