FPGA Data Masking Circuit for Low-Bandwidth Transformer Acceleration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing accelerator designs for machine learning models, such as FPGAs, face challenges with memory storage overhead, memory bandwidth overhead, and arithmetic overhead during data masking operations, particularly in autoregressive transformer models, leading to increased energy consumption and reduced throughput.
Innovation Solution
Implementing a data masking circuit on an FPGA that generates masks on-device without floating-point arithmetic, using multiplexers to apply masks and optionally receiving external masks, thereby reducing memory and bandwidth overhead and computational load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data masking is performed using existing accelerator designs with floating-point arithmetic, then masking functionality is achieved, but memory storage overhead and memory bandwidth overhead increase
Solution Approach 1:
The patent extracts the masking operation from traditional floating-point arithmetic units and implements it as a dedicated data masking circuit that operates directly on tensor data. This circuit generates mask values and applies them through selective addition operations, eliminating the need for separate mask storage and reducing memory bandwidth requirements by performing masking in-place during tensor processing.
2Reliability
If data masking is performed using existing accelerator designs with floating-point arithmetic, then masking functionality is achieved, but arithmetic overhead increases
Solution Approach 1:
The patent merges the masking operation with the tensor processing pipeline by integrating a data masking circuit that operates concurrently with existing arithmetic units. The circuit generates mask values and applies them through selective addition, combining multiple operations into a unified hardware structure that reduces overall arithmetic overhead and eliminates separate masking computation stages.
3Measurement precision
If data masking operations are performed with high computational load, then masking accuracy is maintained, but energy consumption increases
Solution Approach 1:
The patent replaces complex floating-point arithmetic operations with a specialized data masking circuit that uses simpler integer-based mask generation and selective addition. This substitution maintains masking accuracy by preserving the binary mask structure while reducing computational complexity and energy consumption through optimized hardware logic that operates directly on tensor data without requiring full floating-point processing units.
Data Source
AI summary
A field programmable gate array (FPGA) including a configurable interconnect fabric connecting a plurality of logic blocks, the configurable interconnect fabric and the logic blocks being configured to implement a data masking circuit configured to: receive input data including data values at a plurality of indices of the input data; select between a data value of the data values and an alternative value using a masking multiplexer to generate masked data, the masking multiplexer being controlled by a mask value of a plurality of mask values at indices corresponding to the indices of the input data; and output the masked data. In some examples, the configurable interconnect fabric and the logic blocks are further configured to implement a mask generation circuit configured to generate the mask values. In some examples, the mask values are received from external memory.