ML Network Fail-Slow Detection Using Offered Load Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Fail-slow failures in network fabrics are difficult to detect due to their manifestation as slowdowns without hard stops or explicit failure logging, making it challenging to identify the source of the issue.
Innovation Solution
Detect fail-slow conditions by comparing the offered load of machine learning applications with network utilization over short intervals, using network measurements such as latency and throughput, and evaluating these against maximum bandwidth thresholds to identify potential failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If network monitoring is performed to detect fail-slow conditions, then detection capability is improved, but system complexity and resource consumption increase
Solution Approach 1:
The system pre-calculates and stores the maximum bandwidth for each physical network connection. This preliminary action enables rapid fail-slow detection by allowing the monitoring component to simply compare current load against the pre-stored maximum bandwidth threshold, avoiding complex real-time analysis and reducing system complexity during operation.
Solution Approach 2:
The patent introduces an intermediary monitoring component that sits between the network applications and the physical network connection. This intermediary automatically collects offered load data from applications, compares it against pre-stored maximum bandwidth values, and detects fail-slow conditions without requiring complex modifications to existing network infrastructure or applications.
2Measurement precision
If offered load is measured by instrumenting machine learning applications, then measurement accuracy is improved, but implementation complexity increases
Solution Approach 1:
The patent creates a universal instrumentation mechanism that can be applied to any machine learning application through a common interface. The monitoring component is designed to work with multiple applications simultaneously, collecting offered load data through standardized methods. This universal approach reduces implementation complexity compared to creating application-specific monitoring solutions for each machine learning workload.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Aspects of the disclosure are directed to detecting fail-slow conditions in a network executing machine learning applications. Detection may be performed by comparing, over short intervals of time, a load offered by the machine learning applications with network utilization. The load offered by the machine learning applications, sometimes referred to herein as the offered load, may be measured by instrumenting the machine learning application to share data at a time of requests. For example, the application may share data, such as an amount of information pending to be sent in association with the request, at a start time when the request starts and at an end time when the request ends.