Hybrid Floating Point Format for Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current floating point representation formats in FPUs are resource-intensive and complex, leading to long compute times and high power consumption, especially in neural network training, where high precision is often not necessary, and existing low precision formats do not efficiently reduce hardware size and power usage while maintaining accuracy.
Innovation Solution
Implementing a hybrid very low precision floating point format with 8-bit representation, dividing the bits into a sign bit, exponent bits, and mantissa bits, where the exponent bit is less in the first format and more in the second, allowing for efficient representation of numbers and handling of special conditions like infinity and NaN, and using a modified FPU configuration for neural network training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional floating point representation formats are used in FPU, then computation accuracy is maintained, but hardware area and power consumption increase significantly
Solution Approach 1:
The patent changes the parameters of floating point representation by using different precision formats (e.g., FP8, FP6) for different computational stages of neural network training. During forward propagation, lower precision formats are used where accuracy requirements are relaxed, while maintaining higher precision where needed. This dynamic parameter adjustment reduces overall power consumption while maintaining sufficient computation accuracy.
Solution Approach 2:
The patent segments the neural network training process into different stages (forward propagation, backward propagation, weight updates) and applies different floating point precision formats to each stage. This segmentation allows the system to use lower precision formats during stages where accuracy is less critical, thereby reducing power consumption without significantly compromising overall training accuracy.
2Measurement precision
If traditional floating point representation formats are used in FPU, then computation accuracy is maintained, but hardware area increases
Solution Approach 1:
The patent changes the precision parameters of floating point formats from traditional FP32/FP64 to compressed formats like FP8 and FP6. This parameter change reduces the number of bits required to represent each floating point number, thereby reducing the hardware area required for registers, memory, and computational units in the FPU, while maintaining sufficient accuracy for neural network training.
Solution Approach 2:
The patent segments the data representation into different precision levels, using lower precision formats for most computations and reserving higher precision for critical operations. This segmentation allows the hardware to be designed with smaller, more area-efficient units that handle the majority of computations at reduced precision.
3Area of moving object
If low precision formats are used to reduce hardware size, then area and power are reduced, but computation accuracy deteriorates
Solution Approach 1:
The patent implements dynamic precision selection where the floating point format precision is adjusted based on the computational stage and data type. During forward propagation, lower precision (FP8) is used for activations and weights. During backward propagation, precision is increased for gradient computations. This dynamic adaptation maintains computation accuracy where needed while maximizing area and power efficiency where possible.
Solution Approach 2:
The patent applies different precision qualities to different parts of the neural network training process. Critical computations like gradient calculations and weight updates use higher precision formats, while less critical operations like activation functions and intermediate computations use lower precision formats. This local quality differentiation ensures accuracy is maintained only where necessary.
4Measurement precision
If high precision floating point computations are performed, then accuracy is maintained, but training time increases
Solution Approach 1:
The patent changes the precision parameter from high (FP32/FP64) to low (FP8/FP6) for the majority of neural network training computations. This parameter change enables faster computation speeds and higher throughput in the FPU, as lower precision formats require fewer computational cycles and allow for more parallel operations, thereby reducing overall training time while maintaining sufficient accuracy.
Data Source
AI summary
In an embodiment, a method includes configuring a specialized circuit for floating point computations using numbers represented by a hybrid format, wherein the hybrid format includes a first format and a second format. In the embodiment, the method includes operating the further configured specialized circuit to store an approximation of a numeric value in the first format during a forward pass for training a deep learning network. In the embodiment, the method includes operating the further configured specialized circuit to store an approximation of a second numeric value in the second format during a backward pass for training the deep learning network.