Outlier Server Detection via Performance Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computer systems, detecting and remediating defective servers that behave uncharacteristically is challenging, especially when the changes are subtle or when the number of servers is large, leading to reduced system performance and user dissatisfaction.
Innovation Solution
A method that involves retrieving time-series data sets for performance metrics across servers, generating clusters based on threshold distances, and identifying outlier servers, which are then marked and either taken offline or terminated, allowing for automatic replacement with healthy servers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional monitoring methods are used to detect defective servers, then system operators can detect complete server failures, but they fail to detect servers behaving in an uncharacteristic manner with subtle changes
Solution Approach 1:
The patent segments the monitoring approach by dividing servers into clusters based on their performance characteristics. Instead of monitoring each server in isolation, the system groups servers with similar behaviors together and identifies outliers relative to their cluster, enabling detection of subtle deviations that traditional methods miss.
Solution Approach 2:
The patent introduces a new dimension for detection by comparing servers relative to their peer clusters rather than against fixed thresholds. This dimensional shift from absolute monitoring to relative comparison within groups enables detection of subtle behavioral changes that maintain normal system-level metrics.
2Productivity
If the number of deployed servers is increased to handle large demand, then system capacity improves, but detecting defective servers becomes more difficult
Solution Approach 1:
The patent applies segmentation by organizing large numbers of servers into manageable clusters based on performance characteristics. This grouping approach makes detection feasible even when tens of thousands of servers are deployed, as the system only needs to compare servers within their respective clusters rather than analyzing all servers individually.
Solution Approach 2:
The patent creates virtual representations (clusters) of server groups that capture the typical behavior patterns. By comparing individual servers against their cluster's characteristic behavior, the system can efficiently detect outliers in large deployments without examining every server in detail.
3Reliability
If manual monitoring of server performance is performed, then complete failures can be detected, but subtle behavioral changes in defective servers go undetected
Solution Approach 1:
The patent implements self-service monitoring where the system automatically performs clustering and outlier detection without requiring manual intervention. The automated system continuously monitors server behavior, groups servers into clusters, and identifies outliers, making reliability monitoring as easy as deploying the system.
Solution Approach 2:
The patent establishes a feedback loop where server performance data is continuously collected, clustered, and analyzed to identify outliers. The system automatically responds to detected outliers by taking corrective actions, creating a self-regulating monitoring system that improves reliability without increasing operational complexity.
4Duration of action of stationary object
If defective servers are not detected and remediated, then system operation continues without interruption, but client performance deteriorates due to sub-optimal server behavior
Solution Approach 1:
The patent applies preliminary action by detecting and remediating defective servers before they significantly impact client performance. By continuously monitoring for outliers and automatically remediating them, the system prevents sub-optimal server behavior from degrading overall system performance, maintaining both availability and productivity.
Solution Approach 2:
The patent uses feedback mechanisms to continuously monitor server performance and automatically respond to detected outliers. This closed-loop system ensures that defective servers are identified and remediated promptly, preventing performance degradation while maintaining system availability.
Data Source
AI summary
One embodiment of the invention disclosed herein provides techniques for detecting and remediating an outlier server in a distributed computer system. A control server retrieves a group of time-series data sets associated with a first time period, where each time-series data set represents a performance metric for a different server in a group of servers. The control server generates a cluster that includes two or more of the time-series data sets, where the performance metric for each server that is associated with one of the time-series data sets in the cluster is within a threshold distance from the performance metric for the servers that are associated with the other time-series data sets in the cluster. The control server determines that a particular time-series data set corresponds to a server included in the group of servers and is not included in the cluster, and marks the server as an outlier server.