Mixed Precision Deep Learning Acceleration Hardware
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models, particularly convolutional neural networks (CNNs), require significant computing resources and time due to their large number of parameters and operations, which can be inefficient in terms of memory, processing power, and energy consumption, especially when higher precision is needed for accuracy but not required for all operations.
Innovation Solution
Implementing mixed precision computing within CNNs, where operations can switch between high and low precision dynamically based on the specific requirements of each operation, allowing for parallel processing and precision mode selection to optimize resource usage and speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high precision is used for all operations in CNN, then accuracy is maintained, but computing resources and processing time increase significantly
Solution Approach 1:
The patent applies different precision levels to different parts of the neural network computation. Specifically, weights are stored and processed at lower precision (e.g., 8-bit integers) while activations maintain higher precision (e.g., 16-bit floats). This local differentiation allows the system to reduce overall computing resource consumption and increase processing speed while maintaining accuracy in critical computation paths.
Solution Approach 2:
The patent dynamically changes precision parameters based on operation type and network layer requirements. The system selectively applies quantization to weight matrices while preserving full precision for activation values and gradient computations. This parameter adaptation enables the network to achieve faster inference times with acceptable accuracy trade-offs in non-critical operations.
2Measurement precision
If high precision is used for all operations in CNN, then accuracy is maintained, but memory usage and energy consumption increase
Solution Approach 1:
The patent implements precision differentiation where weight parameters are stored at lower precision (reducing memory bandwidth requirements and energy for data transfer) while activation parameters maintain higher precision. This local quality approach reduces the energy consumption associated with memory access and computation in layers where full precision is not critical, while preserving accuracy where needed.
3Productivity
If low precision is used for all operations in CNN, then computing resources and processing time are reduced, but accuracy deteriorates
Solution Approach 1:
The patent applies a hierarchical precision strategy where different components of the neural network operate at different precision levels. Weight matrices use lower precision (e.g., INT8) to accelerate computation, while activation values and gradient computations use higher precision (e.g., FP16) to maintain numerical stability and accuracy. This selective application prevents accuracy deterioration in critical paths while achieving speedup in computation-heavy operations.
Solution Approach 2:
The patent employs dynamic precision selection where the precision level is adjusted based on the specific operation being performed and the requirements of different network layers. The system can switch between precision modes (e.g., INT8 for convolution, FP16 for activation functions) to optimize the balance between speed and accuracy for each computational step.
Data Source
AI summary
A device for deep learning acceleration with mixed precision may include a first precision mode port to receive an indication of an input precision mode and a second precision mode port to receive an indication of an output precision mode. The device may include a first data port to receive map data and a second data port to receive kernel data. The device may include multiply-accumulate (MAC) components that are each configured to generate a MAC output based on the input precision mode, the map data, and the kernel data. The device may include an adder component to generate an adder component output based on the input precision mode and one or more MAC outputs. The device may include a rounding component to round the adder component output, based on the output precision mode, to generate a rounded output, and an output port to output the rounded output.