Cluster Job Performance Analysis via Telemetry Correlation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing management systems for computing clusters lack the functionality to rapidly and efficiently detect performance degradation of individual computing jobs executed across multiple compute nodes and determine the root cause of such degradation.
Innovation Solution
The implementation involves storing telemetry data from multiple components of a computing cluster in a time-series database. A controller identifies impacted computing jobs, queries the database for relevant data, and uses diagnostic rules to determine if data values are correlated with trigger events, providing alerts and probable root cause analysis for performance degradation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If management systems monitor and analyze data from specific component types separately, then device complexity is reduced and ease of operation is improved, but the ability to detect performance degradation of individual computing jobs across multiple nodes is insufficient
Solution Approach 1:
The patent combines monitoring of multiple component types (compute nodes, storage nodes, network switches, cooling devices, power devices) into a unified management system that correlates telemetry data across all these components. This merging enables the system to detect performance degradation of individual computing jobs by analyzing correlated data from all component types simultaneously, rather than monitoring each component type separately.
Solution Approach 2:
The management system is designed with multi-functionality to handle diverse component types and telemetry data formats through a unified architecture. The system can identify computing jobs impacted by trigger events, query time-series databases for relevant data, correlate multiple types of telemetry, and determine root causes across different component types, making it universally applicable to various computing job scenarios.
2Measurement precision
If management systems correlate multiple types of telemetry across large numbers of nodes, then performance degradation detection accuracy is improved, but processing capacity requirements increase and system performance decreases
Solution Approach 1:
The system performs preliminary actions by pre-processing and storing telemetry data from multiple components in a time-series database with standardized formats and time synchronization. This preliminary organization of data enables efficient querying and correlation when performance degradation events occur, avoiding the need to process raw data from all nodes in real-time and thus maintaining system performance while achieving high measurement precision.
3Reliability
If management systems analyze telemetry data from all compute nodes executing a computing job, then detection completeness is improved, but the time and processing capacity required increase significantly
Solution Approach 1:
The system extracts only the relevant telemetry data needed for performance degradation detection by querying the time-series database for specific time windows associated with trigger events. Rather than analyzing all data from all compute nodes continuously, the system extracts and correlates only the necessary data points related to the specific computing job and time period, significantly reducing processing time while maintaining detection completeness.
Data Source
AI summary
Example implementations relate to performance analysis of jobs in computing clusters. In some examples, a processor detects a trigger event in a computing cluster, and identifies a computing job associated with the trigger event. The processor determines a time window associated with trigger event, and determines compute nodes executing the computing job during the time window. The processor determines database attributes associated with the compute nodes, and obtains data values for the determined database attributes in the determined time window. The processor determines whether the data values are correlated to the trigger event base according to a diagnostic rule. In response to a determination that the data values are correlated to the trigger event base according to the diagnostic rule, the processor determines provides an indication of a degraded performance for the computing job.