Self Learning Signatures for Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional monitoring techniques for networked computing systems are impractical and often impossible to implement manually, especially in large data centers, due to the burden of analyzing numerous systems and the challenge of determining appropriate thresholds for abnormality detection, particularly when dealing with new systems lacking historical data and varying system configurations.
Innovation Solution
The development of automated techniques for creating baseline signatures using previously collected data, allowing for immediate abnormality detection by aggregating data from similar systems, and increasing data collection resolution upon detecting anomalies, such as sampling rate and monitored values, to characterize abnormalities along specified dimensions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual monitoring techniques are used for computing systems, then expert analysis can be performed, but the technique becomes impractical and impossible for large data centers due to the burden of analyzing numerous systems
Solution Approach 1:
The system performs self-monitoring by automatically collecting performance data, comparing it against baselines, and detecting abnormalities without requiring manual expert intervention. The automated anomaly detection system continuously monitors computing systems, eliminating the need for manual analysis while maintaining detection accuracy.
Solution Approach 2:
Manual expert monitoring is replaced with an automated computational system that uses algorithms to analyze performance data, compare against baselines, and detect abnormalities. This substitution of mechanical/manual processes with automated systems enables scalable monitoring across large data centers.
2Ease of operation
If fixed threshold techniques are used for abnormality detection, then simple monitoring can be implemented, but the technique obtains erroneous results due to difficulties in determining appropriate threshold levels and differences between system configurations
Solution Approach 1:
Instead of using fixed thresholds, the system dynamically adjusts monitoring parameters by creating system-specific baselines that adapt to individual system configurations, usage patterns, and performance characteristics. This allows the monitoring system to maintain simplicity while improving accuracy through customized parameter sets for each system.
Solution Approach 2:
The system applies localized monitoring strategies by creating unique baselines for each computing system based on its specific configuration and usage patterns, rather than applying a universal fixed threshold. This localized approach accounts for system-specific variations and improves detection accuracy.
3Measurement precision
If extensive data accumulation is required before monitoring can begin, then accurate baselines can be established, but the startup time becomes excessively long for new or modified systems
Solution Approach 1:
The system performs preliminary actions by pre-calculating baselines during system setup or modification before actual monitoring begins. This allows the monitoring system to be immediately operational with accurate baselines already in place, eliminating the need for extended data accumulation periods.
Solution Approach 2:
The system uses universal baseline calculation methods that can be applied across different system types and configurations, allowing baselines to be established quickly through standardized processes rather than requiring system-specific long-term data collection.
4Loss of information
If large volumes of data are collected for monitoring, then comprehensive analysis can be performed, but expert consultants have difficulty monitoring due to the large volumes of data generated
Solution Approach 1:
The system extracts only the essential and relevant information from large volumes of performance data by comparing against baselines and focusing on deviations that indicate abnormalities. This extraction of critical information reduces the data burden while maintaining monitoring completeness.
Solution Approach 2:
The system performs partial monitoring by focusing on key performance indicators and critical parameters rather than analyzing every piece of collected data in detail. This selective approach maintains information completeness for anomaly detection while reducing the overall data analysis burden.
Data Source
AI summary
A system and method for monitoring processes corresponding to measurable values based on signatures associated with the measurable values is provided. The signatures can be created based on data from auxiliary data sets or auxiliary data sources. Additional monitoring information can be obtained by collecting dimensional data for the measurable values.