Mixed Precision Deep Learning Acceleration via Dynamic Bit-Width Switching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models, particularly convolutional neural networks (CNNs), require significant computing resources and time due to their large number of parameters and operations, which can be inefficient in terms of memory, processing power, and energy consumption, especially when high precision is needed for accurate output.
Innovation Solution
Implementing mixed precision computing within CNNs, where operations can switch between high and low precision dynamically based on the specific requirements of each operation, allowing for parallel processing and precision mode selection to optimize resource usage and speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high precision computing is used throughout the CNN, then output accuracy is improved, but computational resources and processing time increase significantly
Solution Approach 1:
The patent applies different precision levels to different parts of the neural network computation. Specifically, weights are stored and processed in 8-bit precision while activations use 16-bit precision, and intermediate calculations are performed in 32-bit precision. This local differentiation of precision quality allows the system to maintain accuracy where needed while reducing resource consumption in other areas.
Solution Approach 2:
The patent dynamically changes precision parameters based on the specific computational requirements of different operations and data types. The system adjusts precision levels for weights, activations, and intermediate results according to their impact on final output accuracy, thereby optimizing the balance between computational efficiency and result precision.
2Measurement precision
If high precision computing is used throughout the CNN, then output accuracy is improved, but memory usage and energy consumption increase
Solution Approach 1:
The patent implements asymmetric precision where weights are stored in 8-bit format reducing memory bandwidth requirements, while activations use 16-bit precision to maintain computational accuracy. This localized application of different precision levels reduces overall memory usage and energy consumption compared to uniform high-precision processing.
Solution Approach 2:
The system dynamically adjusts precision parameters for different data types (weights vs. activations) and computational stages, using lower precision for storage and transmission where accuracy requirements are less stringent, and higher precision for critical computations, thereby optimizing energy efficiency.
3Productivity
If low precision computing is used, then computational resources and processing time are reduced, but output accuracy deteriorates
Solution Approach 1:
The patent applies different precision levels to different computational elements: 8-bit precision for weights, 16-bit for activations, and 32-bit for intermediate calculations. This ensures that low precision is used only where appropriate (weight storage) while maintaining higher precision where it impacts output accuracy (activations and intermediate results).
Solution Approach 2:
The system uses a composite precision approach, combining multiple precision levels (8-bit, 16-bit, 32-bit) within the same computational pipeline. This composite strategy allows the system to leverage the benefits of low-precision processing (reduced memory bandwidth, faster computation) while maintaining accuracy through strategic use of higher precision in critical paths.
4Device complexity
If uniform precision is used throughout the CNN, then implementation is simpler, but resource efficiency is suboptimal
Solution Approach 1:
The patent implements asymmetric precision processing where different data types (weights and activations) are handled with different precision levels. This requires separate processing paths and data structures but optimizes resource efficiency by using lower precision for weights and higher precision for activations, achieving better overall performance than uniform precision approaches.
Data Source
AI summary
A device for deep learning acceleration with mixed precision may include vector-vector (VV) components that are each configured to generate a VV output based on an input precision mode, an output precision mode, and at least one accumulation of products. Each accumulation of products may be calculated by adding products based on the input precision mode. Each product may be calculated by multiplying a map word and a kernel word based on the input precision mode. The input precision mode may indicate an input word length for the map word and for the kernel word, and the output precision mode may indicate an output word length for the VV output. The device may include one or more components configured to concatenate VV outputs, corresponding to the VV components, to generate a concatenated VV output. The device may include an output port configured to output the concatenated VV output.