Anomaly Detection State Vector for Heterogeneous Infrastructure Logs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current solutions for anomaly detection in complex computing infrastructures, such as High Performance Computing systems, fail to simultaneously process heterogeneous data like logs and metrics, leading to difficulties in identifying the root cause of incidents, as they either focus on monitoring performance or rely on limited data sources, making it challenging to detect systemic failures effectively.
Innovation Solution
A method that configures a state vector structure to distribute metric-like and log-like information, acquires timestamped operating data, constructs and normalizes state vectors, aggregates data, and applies anomaly detection methods to identify the machine involved in anomalies, using techniques like autoencoders and explanation methods to provide comprehensive anomaly detection across multiple machines without consuming computing resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple heterogeneous data sources (logs, metrics, network infrastructure) are simultaneously processed to detect systemic failures, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent segments the complex processing task by creating separate processing pipelines for different data types (logs, metrics, network infrastructure). Each pipeline processes its specific data type using appropriate techniques, and results are integrated at a higher level. This allows precise anomaly detection across multiple data sources while managing complexity through modular architecture.
Solution Approach 2:
The patent introduces intermediary components such as log parsers, metric collectors, and normalization layers that mediate between raw heterogeneous data and the anomaly detection engine. These intermediaries standardize data formats and extract relevant features, enabling precise detection without requiring the core system to directly handle raw complexity of multiple data sources.
2Reliability
If more information sources are used to maximize detection sensitivity, then reliability is improved, but loss of information increases due to data dilution
Solution Approach 1:
The patent applies local quality by treating different information sources with different processing strategies appropriate to their nature. Logs receive text analysis and pattern matching, metrics receive statistical processing, and network data receives topology-based analysis. This prevents data dilution by ensuring each source is processed with optimal techniques, maintaining both reliability and cause identification clarity.
Solution Approach 2:
The patent transforms heterogeneous data from different sources into a unified multi-dimensional feature space where each dimension represents a specific aspect (temporal patterns, statistical deviations, topological relationships). This dimensional transformation allows simultaneous processing of multiple information sources while preserving distinctive characteristics, preventing information loss and enabling reliable anomaly detection with clear cause identification.
3Ease of operation
If traditional monitoring solutions focus on system metrics only, then ease of operation is improved, but measurement precision deteriorates due to limited data sources
Solution Approach 1:
The patent creates a universal monitoring framework that can process multiple data types (logs, metrics, network infrastructure) through a single integrated system. This multi-functional approach maintains ease of operation by providing unified configuration and management while significantly improving measurement precision through diverse data sources. The system handles different data types through standardized interfaces and processing pipelines.
Data Source
AI summary
A method for determining at least one machine involved in an anomaly detected in a computing infrastructure comprising a plurality M of machines. This determination is performed by associating with each machine a state vector constructed using metrics and logs. Each vector is also associated with a time window. The monitoring of these vectors makes it possible to detect an anomaly that is then explained via an explanation method that makes it possible to assign an involvement indicator to each machine. The use of this indicator makes it possible to determine the machine most involved.