Storage Workload Anomaly Detection Using Two-Stage ML
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Univariate anomaly detection techniques in storage systems identify numerous anomalies, making it difficult to determine which are impactful, while multivariate approaches are complex and costly due to deep learning requirements.
Innovation Solution
A two-stage machine learning method involving univariate and multivariate stages to detect impactful performance anomalies, using unsupervised anomaly detection algorithms like the isolation forest algorithm to identify anomalies in a multi-dimensional search space based on series difference data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If univariate anomaly detection technique is used, then implementation complexity is reduced, but the ability to identify impactful anomalies deteriorates
Solution Approach 1:
The patent segments the anomaly detection process into two distinct stages: a univariate stage that processes each performance metric independently to generate difference data values, and a multivariate stage that combines these values to identify impactful anomalies. This segmentation allows the system to maintain computational simplicity while achieving multivariate anomaly detection capability, resolving the contradiction between implementation complexity and identification accuracy.
Solution Approach 2:
The patent introduces difference data values as an intermediary representation that captures the relationship between observed and normalized performance metric values. These difference values serve as a bridge that enables the unsupervised anomaly detection algorithm to identify impactful anomalies without requiring complex deep learning models, thus maintaining low implementation complexity while improving detection accuracy.
2Measurement precision
If multivariate anomaly detection technique is used, then impactful anomaly identification accuracy is improved, but implementation complexity and cost increase
Solution Approach 1:
The patent divides the multivariate anomaly detection task into two sequential stages: first processing each performance metric univariately to compute difference data values, then combining these values in a second stage for multivariate analysis. This segmentation enables the system to achieve multivariate detection accuracy while avoiding the computational burden of traditional deep learning-based multivariate methods.
Solution Approach 2:
The patent transforms the input data from raw performance metric values to difference data values that represent deviations from normalized patterns. This parameter transformation enables the use of simple unsupervised anomaly detection algorithms to achieve multivariate detection capability, significantly reducing implementation complexity and cost compared to deep learning approaches.
3Loss of energy
If univariate anomaly detection technique is used, then computational cost is reduced, but the number of anomalies to investigate increases
Solution Approach 1:
The patent extracts and isolates impactful anomalies from the larger set of detected anomalies by using the difference data values as input to the unsupervised anomaly detection algorithm. This extraction process filters out less significant anomalies, reducing the number of items requiring manual investigation while maintaining low computational cost through the use of simple algorithms.
Solution Approach 2:
The patent employs feedback mechanisms where the unsupervised anomaly detection algorithm uses the difference data values to identify and prioritize impactful anomalies. This feedback loop enables the system to focus computational resources on the most significant anomalies, reducing the overall number of anomalies requiring investigation while maintaining efficiency.
Data Source
AI summary
Techniques for detecting impactful performance anomalies in storage systems. The techniques include obtaining, for each performance metric of a storage system's workload, a training set of series diffs based on a threshold. Each diff represents a difference between an observed value from an observed set of time series values for the performance metric and a normalized value from a corresponding set of normalized time series values. The techniques include applying the training set of series diffs for each performance metric to an unsupervised anomaly detection algorithm and running the algorithm to identify potentially impactful anomalies in a multi-dimensional search space. The techniques include identifying impactful anomalies from among the potentially impactful anomalies that exceed an anomaly score. In this way, impactful anomalies having a causal effect on multiple performance metrics of the storage system's workload can be identified in a manner less complex and less costly than prior multivariate approaches.