Detached Subgraph Detection in Neural Network Training Graphs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning models can be adversely affected by unintentionally detached subgraphs, which are difficult to identify and correct, leading to inaccurate or underperforming models.
Innovation Solution
A system and method for automatically detecting and correcting detached subgraphs in neural networks using a combination of runtime detection and on-the-fly analysis, employing a pre-trained machine learning model to monitor weight gradients and provide location and correction suggestions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a deep learning program is manually inspected to detect detached subgraphs, then detection accuracy can be achieved, but time consumption and complexity increase significantly
Solution Approach 1:
The system enables self-service by allowing the deep learning program to automatically detect its own detached subgraphs through runtime monitoring of gradient flow and computational graph analysis, eliminating the need for manual inspection while maintaining high detection accuracy
Solution Approach 2:
The patent replaces the mechanical manual inspection process with an automated computational system that uses gradient flow analysis and computational graph traversal to detect detached subgraphs, significantly reducing time consumption while maintaining detection precision
2Reliability
If runtime monitoring of gradient flow is implemented to detect detached subgraphs, then detection capability is improved, but computational overhead increases
Solution Approach 1:
The system applies partial monitoring by focusing gradient flow analysis only on critical computational paths and nodes where detachment is most likely to occur, rather than monitoring the entire computational graph uniformly, thus improving detection capability while reducing computational overhead
Solution Approach 2:
The patent implements preliminary detection mechanisms by analyzing gradient flow patterns and computational graph structure before training completes, allowing early identification of detached subgraphs and avoiding unnecessary computational overhead during the training process
Data Source
AI summary
Detecting an anomaly in deep learning programming can include receiving a deep learning program with neural network. A pre-trained machine learning model can be run with the deep learning program with neural network as input. The pre-trained machine learning model detects whether the neural network includes a detaching subgraph. Responsive to detecting that the neural network includes a detaching subgraph, a location of the deep learning program causing the detaching subgraph can be output. The neural network of the deep learning program can be run in training mode and weight gradients associated with training of the neural network can be monitored. Based on the monitoring, occurrence of one or more detaching subgraphs can be detected. Responsive to detecting a detaching subgraph, the detaching subgraph can be output. A suggestion to correct the deep learning program can also be output.