Mixed-Precision CNN Compute Architecture for Faster Accurate Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional neural networks (CNNs) require significant computing resources and time due to their large number of parameters and operations, and using less precise values to reduce these demands results in less accurate model output.
Innovation Solution
Implementing mixed precision operations within CNNs, where high precision is used for critical operations and lower precision for others, combined with parallel processing and coordination modes, to optimize resource usage and speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high precision operations are used throughout the CNN, then output accuracy is improved, but computing resources and processing time increase significantly
Solution Approach 1:
The patent applies different precision levels to different parts of the neural network computation. Specifically, weight parameters are stored and processed with lower precision (e.g., 8-bit or 16-bit integers) while activation values maintain higher precision (e.g., 32-bit floating point). This local differentiation allows the system to reduce computing resource demands and increase processing speed in the weight processing paths while preserving output accuracy in the activation paths that directly affect the final result.
2Productivity
If lower precision values are used to reduce computing resources, then processing speed increases, but output accuracy decreases
Solution Approach 1:
The patent dynamically changes precision parameters based on the computational context. Weight parameters are converted to lower precision representations (such as quantized integers) to speed up multiplication operations, while activation parameters maintain higher precision to ensure accurate propagation of information through the network. The system selectively applies precision conversion to different parameter types, achieving both speed improvement and accuracy preservation.
Solution Approach 2:
The precision configuration is not static but dynamically adjusted based on the operational phase and parameter type. During forward propagation, the system dynamically switches between low-precision weight processing and high-precision activation processing. This dynamic precision management allows the network to adapt its computational precision to the specific requirements of each operation, optimizing both speed and accuracy.
3Power
If more computing resources are allocated to CNN operations, then processing capacity increases, but system complexity and resource requirements worsen
Solution Approach 1:
The patent fundamentally changes the precision parameters of the computational data representations. By storing weights in lower precision formats (8-bit or 16-bit integers instead of 32-bit floating point), the system reduces the bit-width required for data storage and transmission. This parameter change directly decreases memory bandwidth requirements, reduces activation memory usage, and simplifies the computational hardware requirements while maintaining adequate processing capacity through optimized computation paths.
Data Source
AI summary
A device for deep learning acceleration with mixed precision may include matrix-vector (MV) components that each include vector-vector (VV) components that are each configured to generate a respective VV output based on an input precision mode, an output precision mode, and an accumulation of products. The accumulation of products may be calculated by adding products based on the input precision mode. Each product may be calculated by multiplying, based on the input precision mode, a map data segment and a kernel data segment. Each MV component may include one or more components configured to concatenate VV outputs to generate a concatenated VV output. The device may include activation function components that are each configured to receive a corresponding concatenated VV output, generate an activation function output based on the corresponding concatenated VV output and the output precision mode, and output the activation function output.