Soft Failure Detection in Microservice Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In microservice-based systems, soft failures can cause insidious degradation that is difficult to detect and remediate, leading to performance issues until a hard failure occurs, and existing monitoring approaches are time-consuming and inefficient.
Innovation Solution
A statistical evaluation method is used to identify soft failures in cluster members by analyzing metric data, allowing for timely actions such as recycling, scaling, or removing affected members to maintain system performance with minimal disruption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional monitoring approaches are used to detect soft failures, then system reliability is maintained, but detection time and operational efficiency deteriorate
Solution Approach 1:
The patent replaces traditional mechanical monitoring approaches with a statistical evaluation system that automatically analyzes metric data from multiple members. The coordinator computes statistical values (mean, standard deviation, percentiles) to detect soft failures, substituting manual or traditional monitoring mechanisms with an automated statistical framework that reduces detection time while maintaining reliability.
Solution Approach 2:
The system enables self-service by having members automatically report their own metric data to the coordinator, which then performs statistical evaluations and identifies soft failures without external intervention. This self-monitoring and self-reporting mechanism reduces the need for external monitoring resources and accelerates detection response.
2Measurement precision
If statistical evaluation of all members is performed continuously, then soft failure detection precision is improved, but computational complexity and resource consumption increase
Solution Approach 1:
The patent applies partial action by computing statistical evaluations for only the necessary metrics and members required for soft failure detection, rather than analyzing all possible system parameters. The coordinator focuses on collecting and evaluating specific metric data from members that is relevant to identifying soft failures, reducing unnecessary computational overhead while maintaining detection precision.
Solution Approach 2:
The system segments the computational task by dividing the cluster into individual members, each reporting their own metric data independently to the coordinator. This segmentation allows parallel collection and evaluation of metrics from multiple members, reducing the computational burden on any single component and enabling scalable implementation.
3Device complexity
If soft failures are not detected early, then system simplicity is maintained, but performance degradation and hard failures increase
Solution Approach 1:
The patent implements feedback by continuously collecting metric data from members, computing statistical evaluations, and using these results to identify soft failures. The system feeds back this information to trigger appropriate actions (recycling, scaling, or removing affected members), creating a closed-loop control mechanism that maintains system performance without requiring complex manual intervention.
Solution Approach 2:
The system performs preliminary action by detecting soft failures before they progress to hard failures that would cause complete system downtime. By identifying degradation early through statistical evaluation of metric data, the system can take preventive actions (such as recycling affected members) to maintain performance and avoid catastrophic failures.
Data Source
AI summary
Examples herein relate to identifying a soft failure at member within a cluster dedicated to a microservice. Examples disclose identifying a statistical evaluation for each member within the cluster. Based on the statistical evaluation, identifying a soft failure among one of multiple members within the cluster. In response to the identification of the soft failure at one of the members within the cluster, performing an action.