Surrogate Gradient Replacement for Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training large neural networks is hindered by errors in data transmission between multiple GPUs or processors, leading to significant slowdowns as errors are corrected, and these issues cascade with increased network complexity.
Innovation Solution
Implementing a system that identifies corrupted neural network gradient values or network packets during data transmission and replaces them with corresponding surrogate values, using a processor with circuits performing cyclic redundancy checks and data replacement operations based on predefined policies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data transmission between multiple GPUs or processors is performed during neural network training, then training capability and network scale are improved, but transmission errors occur more frequently and cause significant slowdowns
Solution Approach 1:
The patent applies preliminary action by performing cyclic redundancy check (CRC) validation on transmitted data packets before they are fully processed. This early detection mechanism identifies corrupted data in advance, allowing the system to switch to backup values without complete transmission failures, thus maintaining training reliability while supporting multi-GPU scalability.
Solution Approach 2:
The patent changes the state of data transmission by introducing surrogate gradient values as alternative parameters. When transmission errors are detected through CRC checks, the system switches from using original gradient values to using surrogate values (such as zero values or previously valid values), thereby maintaining training continuity and reducing the impact of transmission errors on overall reliability.
2Reliability
If transmission errors are detected and corrected through traditional methods, then data accuracy is maintained, but training speed slows down significantly
Solution Approach 1:
The patent employs disposable surrogate gradient values that can be quickly generated and discarded. When transmission errors are detected, the system replaces corrupted gradient values with inexpensive surrogate values (such as zero vectors or cached previous values) rather than performing costly retransmission or complex error correction, thus maintaining data accuracy while minimizing impact on training speed.
Solution Approach 2:
The patent applies skipping by allowing the training process to continue with surrogate values when errors are detected, rather than pausing to perform complete error correction or retransmission. This rushing through of error handling using pre-prepared surrogate values maintains training speed while still preserving data accuracy through the CRC validation mechanism.
3Productivity
If more processors are added to the network to increase training capacity, then processing power is improved, but transmission errors cascade and increase
Solution Approach 1:
The patent applies segmentation by dividing the gradient transmission process into independently validated segments. Each data packet transmitted between processors is individually checked using CRC validation, and errors are contained to specific packets rather than cascading through the entire network. This segmentated approach allows the network to scale to more processors while limiting the propagation of transmission errors.
Solution Approach 2:
The patent introduces CRC validation and surrogate value mechanisms as intermediary layers between processors. These intermediaries detect and handle transmission errors before they can cascade through the network, allowing the system to accommodate more processors without proportionally increasing error propagation. The intermediary validation layer isolates errors to individual transmission events.
Data Source
AI summary
Apparatuses, systems, and techniques to replace one or more corrupt neural network gradient values with one or more corresponding surrogate neural network gradient values. In at least one embodiment, faulty gradient values are identified in a transmitted data packet and are replaced according to a replacement policy with a suitable surrogate value.