Cluster Aberrant System Detection via External Performance Metrics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for detecting failures in homogenous clusters of computing systems often lead to interruptions and data loss, as they require systems to fail before remedial action is taken, which can decrease efficiency and increase downtime.
Innovation Solution
A method that involves gathering performance metrics from each system in the cluster, calculating scores based on these metrics, comparing them to a threshold, and initiating remedial actions such as restarting or replacing aberrant systems to prevent failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If remedial action is taken only after system failure, then system complexity is reduced, but downtime and data loss increase
Solution Approach 1:
The system calculates anomaly scores and identifies aberrant systems before actual failures occur. By performing preliminary detection and alerting on systems showing early signs of failure through performance metric analysis, the system enables proactive remediation, reducing downtime and data loss while maintaining relatively simple architecture.
2Measurement precision
If continuous monitoring and scoring of all systems is implemented, then failure detection accuracy is improved, but computational resources and system complexity increase
Solution Approach 1:
The system uses performance metrics that are already being collected by existing monitoring infrastructure, copying this data for anomaly detection purposes. This approach achieves accurate failure detection by analyzing existing metric data through scoring algorithms without requiring additional sensors or complex monitoring hardware.
Solution Approach 2:
The anomaly scoring system is designed to work with multiple types of performance metrics from different system components (CPU, memory, storage, network). This universal approach allows a single scoring mechanism to detect various failure modes across diverse system types, reducing the need for specialized detection systems for each component.
3Reliability
If performance metrics are collected from all systems continuously, then system reliability monitoring is improved, but data processing load and energy consumption increase
Solution Approach 1:
The system leverages performance metrics that are already being collected and reported by the systems themselves as part of normal operation. By reusing this self-reported data for anomaly detection rather than implementing separate active monitoring queries, the system reduces additional energy consumption while maintaining reliable monitoring of cluster health.
Data Source
AI summary
Certain aspects of the present disclosure provide techniques for detecting and preventing failures in a cluster of systems. Embodiments include receiving a set of performance metrics from each respective system of the cluster of systems. Embodiments include determining a score for each respective system of the cluster of systems based on the set of performance metrics received from the respective system. Embodiments include comparing the score for each respective system of the cluster of systems to a threshold. Embodiments include identifying, based on the comparing, an aberrant system of the cluster of systems. Embodiments include performing a remedial action on the aberrant system.