Winograd Convolution With Dynamic Scaling for Precision Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Winograd transform operations in convolutional neural networks (CNNs) suffer from numerical instability and precision loss due to intermediate computations, particularly when using reduced precision arithmetic, which degrades the overall accuracy of the network.
Innovation Solution
A system and method that dynamically scales intermediate values in the Winograd transform based on overflow conditions, applying shift operations to maintain precision and adjust the final output, using bitwise operations and integer arithmetic to reduce precision loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If Winograd transform operation is performed using reduced precision arithmetic, then computational efficiency is improved, but precision loss occurs and accuracy degrades
Solution Approach 1:
The patent implements dynamic scaling factors that are adjusted based on the magnitude of intermediate computation results. The scaling factor is modified adaptively during the Winograd transform operation to prevent overflow and maintain numerical stability, allowing the system to switch between different precision levels dynamically rather than using a fixed precision approach throughout the computation
Solution Approach 2:
The patent changes the precision parameter dynamically during computation by adjusting the number of bits used to represent intermediate values. When intermediate results exceed a threshold, the system increases the bit width or applies scaling to maintain precision, thereby changing the numerical representation parameters adaptively to balance computational efficiency and accuracy
2Device complexity
If Winograd transform operation is performed using reduced precision arithmetic, then computational complexity is reduced, but numerical instability increases
Solution Approach 1:
The patent applies pre-computed scaling factors to intermediate results before they can overflow or become unstable. By anticipating potential numerical instability and applying compensatory scaling in advance, the system prevents numerical errors from accumulating, thereby maintaining reliability without significantly increasing computational complexity
3Measurement precision
If dynamic scaling with overflow compensation is applied, then precision loss is reduced, but computational overhead increases
Solution Approach 1:
The patent implements self-monitoring of intermediate computation results that automatically trigger scaling operations only when necessary. The system monitors the magnitude of intermediate values and applies dynamic scaling selectively based on actual numerical conditions, rather than applying scaling unconditionally, thereby reducing unnecessary computational overhead while maintaining precision when needed
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods for performing a Winograd transform operation with reduced precision loss. The Winograd transform operation is performed by iterating over each matrix element of a first matrix (D) and a second matrix (V) of the Winograd transform operation, wherein for each element: when a respective matrix element value exceeds one or more maximum threshold values or one or more minimum threshold values, right-shifting the respective matrix element value by at least one bit, performing a matrix multiplication of the first matrix (D) and second matrix (V) using one or more bitwise operations to obtain a result matrix, iterating over each matrix element of the result matrix, and for each matrix element: multiplying corresponding matrix elements from the first matrix (D) and second matrix (V), left-shifting a multiplication result of the matrix multiplication based on the right-shifting to adjust the result matrix, outputting the adjusted result matrix.