Asymmetric FMA Instructions for Variable-Precision Deep Learning Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional CPU and GPU instruction set architectures limit performance gains in deep learning by requiring symmetric, 32-bit lanes for all operations, preventing the use of low-precision operands that could enhance throughput in deep learning algorithms.
Innovation Solution
Implementing asymmetric fused multiply-add (FMA) instructions that allow variable precision inputs, such as 8, 4, 2, and 1-bit operands, using single-instruction multiple-data (SIMD) processing lanes with 32-bit widths to execute operations efficiently, thereby avoiding the restriction of symmetric operands and maximizing throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional symmetric 32-bit FMA operations are used, then operational consistency and simplicity are maintained, but throughput in deep learning is limited
Solution Approach 1:
The patent implements asymmetric FMA operations where the first operand (activation) and second operand (weight) can have different precisions. The execution circuitry includes separate processing paths: a first FMA circuit for symmetric operations and a second FMA circuit for asymmetric operations with variable precision inputs. This allows low-precision weights (2-bit, 4-bit) to be combined with higher-precision activations (8-bit) to achieve higher throughput while maintaining sufficient training performance.
Solution Approach 2:
The patent introduces dynamic precision selection where the precision of operands can be varied based on the specific deep learning task requirements. The instruction set architecture includes encoding fields that specify the precision of each operand, allowing the system to adaptively choose between different precision levels (1-bit, 2-bit, 4-bit, 8-bit) to optimize throughput for different layers and operations in the neural network.
2Productivity
If low-precision operands (2-bit, 4-bit weights) are used, then throughput is enhanced, but traditional architectures cannot process them due to symmetric operand requirements
Solution Approach 1:
The patent changes the precision parameter of the operands to enable low-precision computing. The instruction encoding includes fields that specify the bit-width of each operand (e.g., 1-bit, 2-bit, 4-bit, 8-bit), allowing the execution circuitry to configure its processing paths accordingly. This parameter change enables the system to process low-precision weights and activations while maintaining compatibility with existing deep learning models through instruction-level abstraction.
Solution Approach 2:
The execution circuitry is segmented into multiple specialized FMA circuits: a first FMA circuit for symmetric operations and a second FMA circuit for asymmetric operations with variable precision. Each circuit is optimized for specific precision requirements, allowing parallel processing of different precision formats without interference. This segmentation enables efficient handling of mixed-precision deep learning workloads.
3Productivity
If variable precision inputs are allowed, then deep learning performance is improved, but instruction set complexity increases
Solution Approach 1:
The patent creates a universal FMA instruction that can handle multiple precision formats through a single instruction template. The instruction encoding includes precision specification fields that allow the same instruction format to represent symmetric FMA, asymmetric FMA with 4-bit weights, asymmetric FMA with 2-bit weights, and other variations. This multi-functionality reduces the need for separate instruction formats while maintaining support for variable precision operations.
Data Source
AI summary
Disclosed embodiments relate to instructions for fused multiply-add (FMA) operations with variable-precision inputs. In one example, a processor to execute an asymmetric FMA instruction includes fetch circuitry to fetch an FMA instruction having fields to specify an opcode, a destination, and first and second source vectors having first and second widths, respectively, decode circuitry to decode the fetched FMA instruction, and a single instruction multiple data (SIMD) execution circuit to process as many elements of the second source vector as fit into an SIMD lane width by multiplying each element by a corresponding element of the first source vector, and accumulating a resulting product with previous contents of the destination, wherein the SIMD lane width is one of 16 bits, 32 bits, and 64 bits, the first width is one of 4 bits and 8 bits, and the second width is one of 1 bit, 2 bits, and 4 bits.