Anomaly Evaluation System for Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face challenges in timely and accurate anomaly evaluation due to separate monitoring of metrics and log messages, leading to potential cascading failures and difficulty in identifying impending node failures.
Innovation Solution
An anomaly evaluation system that monitors system metrics and logs in real-time, creates a mapping between metric patterns and service-generated logs, detects potential anomalies, and provides a graphical representation with overlays indicating associated logs, while performing similarity searches to filter out non-critical patterns and exclude false positives.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If separate monitoring of metrics and log messages is performed, then monitoring coverage is comprehensive, but anomaly detection accuracy deteriorates due to inability to correlate metrics with corresponding logs
Solution Approach 1:
The patent combines separate metrics monitoring and log message monitoring into a unified anomaly evaluation system. The system ingests both metrics and logs simultaneously, creates a mapping between them, and evaluates anomalies by considering both data sources together, thereby improving detection accuracy while managing complexity through integration.
Solution Approach 2:
The patent introduces a mapping mechanism as an intermediary that connects metrics to corresponding log messages. This mapping allows the system to correlate numerical metric values with textual log entries, enabling accurate identification of which logs correspond to which metric anomalies without requiring direct complex analysis of all data.
2Reliability
If all detected potential anomalies are reported, then comprehensive anomaly detection is achieved, but false positives increase leading to incorrect identification of normal patterns as anomalies
Solution Approach 1:
The patent implements a feedback mechanism where the system learns from historical anomaly evaluations and log correlations. By analyzing patterns in previously evaluated anomalies and their corresponding logs, the system refines its detection criteria over time, reducing false positives while maintaining reliable anomaly identification through continuous improvement.
Solution Approach 2:
The patent performs preliminary evaluation of detected anomalies by checking them against known patterns and correlating with log messages before final reporting. This preliminary action filters out false positives by verifying whether detected anomalies are supported by corresponding log evidence, ensuring only reliable anomalies are reported.
3Loss of information
If detailed graphical representation with log overlays is provided, then anomaly analysis completeness is improved, but system complexity and processing overhead increase
Solution Approach 1:
The patent segments the graphical representation into distinct layers: a base layer showing metrics over time and an overlay layer showing correlated log messages. This segmentation allows the system to provide comprehensive anomaly analysis information while managing complexity by organizing data into manageable, visually distinct components that can be rendered efficiently.
Data Source
AI summary
A method for assisting evaluation of anomalies in a distributed storage system is disclosed. The method includes a step of monitoring at least one system metric of the distributed storage system. The method further includes steps of maintaining a listing of patterns of the monitored system metric comprising patterns which previously did not result in a failure within one or more nodes of the distributed storage system, and, based on the monitoring, identifying a pattern (i.e., a time series motif) of the monitored system metric as a potential anomaly in the distributed storage system. The method also includes steps of automatically (i.e. without user input) performing a similarity search to determine whether the identified pattern satisfies one or more predefined similarity criteria with at least one pattern of the listing, and, upon positive determination, excepting the identified pattern from being identified as the potential anomaly.