Dynamic Precision Adjustment in Neural Network Dot Product Engines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network processing methods require manual adjustment of precision levels, which is time-consuming and inefficient, and once precision is reduced for production deployment, it becomes fixed, limiting dynamic adjustment based on real-time factors like power consumption and processing speed.
Innovation Solution
Implementing a memristor crossbar architecture with a dot product engine that allows for dynamically adjustable precision across different stages of a multi-stage compute process, automatically determining precision levels at runtime based on competing factors such as power consumption, processing accuracy, and resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If precision is reduced to increase processing speed, then processing speed is improved, but precision is degraded
Solution Approach 1:
The patent implements dynamic precision adjustment where the precision level of neural network parameters is changed at runtime based on processing requirements. Different precision levels (e.g., 32-bit, 16-bit, 8-bit) can be selectively applied to different layers or operations during execution, allowing the system to optimize between speed and precision dynamically rather than being fixed at a single precision level.
Solution Approach 2:
The patent applies different precision levels to different parts of the neural network processing pipeline. Specifically, certain layers or operations can use higher precision while others use lower precision, allowing localized optimization where high precision is needed for accuracy-critical operations and lower precision can be used for speed-critical operations.
2Adaptability or versatility
If fixed precision is used for production deployment, then resource allocation is simplified, but adaptability to changing conditions is reduced
Solution Approach 1:
The system transitions from static precision configuration to dynamic precision adjustment, where precision levels can be modified at runtime based on processing requirements, power constraints, and performance targets. This allows the same neural network model to adapt to different deployment conditions without requiring multiple fixed-precision versions.
Solution Approach 2:
The patent changes the precision parameter of neural network weights and activations dynamically during execution. By adjusting bit-width parameters (e.g., changing from 32-bit float to 16-bit float or 8-bit integer), the system can optimize resource usage and performance while maintaining the same underlying model architecture and weights.
3Reliability
If high precision is used throughout the processing pipeline, then processing accuracy is improved, but resource consumption increases
Solution Approach 1:
The patent applies high precision selectively to only those parts of the neural network where it is most beneficial for accuracy, while using lower precision for other parts where the impact on accuracy is minimal. This localized application of high precision reduces overall power consumption while maintaining necessary accuracy levels.
Solution Approach 2:
Instead of applying high precision uniformly across all operations, the patent uses high precision only partially - specifically for critical operations that most impact accuracy - and accepts lower precision for less critical operations, achieving a balance between accuracy and power consumption.
Data Source
AI summary
Disclosed techniques provide for dynamically changing precision of a multi-stage compute process. For example, changing neural network (NN) parameters on a per-layer basis depending on properties of incoming data streams and per-layer performance of an NN among other considerations. NNs include multiple layers that may each be calculated with a different degree of accuracy and therefore, compute resource overhead (e.g., memory, processor resources, etc.). NNs are usually trained with 32-bit or 16-bit floating-point numbers. Once trained, an NN may be deployed in production. One approach to reduce compute overhead is to reduce parameter precision of NNs to 16 or 8 for deployment. The conversion to an acceptable lower precision is usually determined manually before deployment and precision levels are fixed while deployed. Disclosed techniques and implementations address automatic rather than manual determination or precision levels for different stages and dynamically adjusting precision for each stage at run-time.