ML Training Debugging via Cross-Platform Output Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning model training often encounters correctness issues such as generation of NaN values and parameter non-convergence, which can depend on the computing system configuration, making it difficult to identify and resolve these problems efficiently.
Innovation Solution
A debugging system that compares training outputs from different computing systems with varying configurations, using similarity measures to identify discrepancies and generate debugging data for quick resolution of correctness issues, including the use of AI accelerators and CPUs for training operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If training is performed on a first computing system with a first configuration, then training can be completed using specific hardware (e.g., AI accelerators), but correctness issues such as NaN values or parameter non-convergence may occur that do not appear when training on a second computing system with a different configuration (e.g., CPUs)
Solution Approach 1:
The patent creates a copy of the training computation graph and executes it on a different computing system configuration (e.g., CPU instead of AI accelerator). By comparing the outputs of the original and copied training executions, the system identifies correctness issues specific to the first configuration without sacrificing training speed, as the copy execution can be performed selectively for debugging purposes.
2Measurement precision
If debugging is performed by manual inspection of training outputs, then detailed analysis is possible, but significant time is consumed in identifying correctness issues
Solution Approach 1:
The patent implements an automated feedback mechanism where training outputs from the first and second computing systems are automatically compared using similarity measures. When discrepancies are detected (e.g., NaN values or divergent parameter values), the system provides immediate feedback about the correctness issue, eliminating the need for time-consuming manual inspection while maintaining high debugging accuracy.
Solution Approach 2:
The patent replaces the mechanical process of manual debugging with an automated computational system. Instead of manually inspecting training outputs and comparing results across different configurations, the system uses automated similarity measurements and computational comparison to identify correctness issues, dramatically reducing debugging time while maintaining or improving detection accuracy.
3Reliability
If training outputs from different computing systems are compared in detail, then correctness issues can be identified, but the complexity of comparing multiple configurations increases
Solution Approach 1:
The patent extracts only the critical comparison elements from the full training outputs - specifically focusing on key metrics such as parameter values, loss functions, and convergence behavior. By extracting and comparing only these essential elements rather than performing detailed comparison of all training outputs, the system maintains high correctness verification capability while reducing the overall complexity of the debugging process.
Data Source
AI summary
A method includes training, using a first computing system having a first configuration, a first machine learning model having a machine learning model architecture, and training, using a second computing system having a different second configuration, a second machine learning model having the machine learning model architecture. The method also includes determining, for a shared training operation performed by both the first computing system and the second computing system, a similarity measure that represents a similarity between: a first training output generated by the first computing system during performance of the shared training operation during training of the first machine learning model; and a second training output generated by the second computing system during performance of the shared training operation during training of the second machine learning model. The method further includes displaying, to a user, a graphical representation based on the similarity measure determined for the shared training operation.