Mixed Precision Neural Network Training via Data Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current approaches to training neural networks for deep learning are computationally intensive and time-consuming, often requiring extensive resources, and traditional reduced precision computing can lead to accuracy losses due to inconsistent precision usage across operations.
Innovation Solution
Implementing data-driven mixed precision learning by replicating neural networks with varying precision levels and using a pre-processing step to route input data to appropriate replicas based on content and context, allowing for dynamic precision adjustment and batch formation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If traditional reduced precision computing is used to speed up neural network training, then training time is reduced, but accuracy is lost due to inconsistent precision usage across operations
Solution Approach 1:
The patent applies local quality by assigning different precision levels to different neural network operations based on their specific requirements. Critical operations use higher precision while less sensitive operations use lower precision, optimizing the balance between speed and accuracy at the operational level rather than uniformly across the entire network
Solution Approach 2:
The system dynamically adjusts precision levels during training based on the importance and sensitivity of each operation. The precision configuration is not static but adapts to the specific needs of different layers and operations, allowing the system to optimize performance characteristics in real-time
2Manufacturing precision
If high precision is used throughout the neural network to maintain accuracy, then model performance is maintained, but computational resources and training time increase significantly
Solution Approach 1:
Instead of uniformly applying high precision across all operations, the patent identifies specific operations that require high precision and applies it only there. This localized approach maintains model accuracy where needed while reducing computational overhead in less critical areas
Solution Approach 2:
The system changes the precision parameter dynamically based on operation characteristics. By adjusting the precision parameter for different operations rather than maintaining a fixed high precision setting, the system achieves both accuracy and efficiency
3Ease of manufacture
If uniform precision is applied to all neural network operations, then implementation is simplified, but both accuracy and efficiency cannot be optimized simultaneously
Solution Approach 1:
The patent moves away from uniform precision implementation by introducing localized precision control at the operational level. This allows the system to achieve both simplicity through automated assignment and optimization through differentiated precision levels
Data Source
AI summary
Embodiments for implementing mixed precision learning for neural networks by a processor. A neural network may be replicated into a plurality of replicated instances and each of the plurality of replicated instances differ in precision used for representing and determining parameters of the neural network. Data instances may be routed to one or more of the plurality of replicated instances for processing according to a data pre-processing operation.