Neural Graph-Based Alert Representations for Related Event Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge lies in managing and analyzing vast amounts of diverse machine data from IT environments, where existing systems struggle to efficiently process and search unstructured data, leading to overwhelming alert volumes and a lack of insight into related events, hindering effective monitoring and analysis.
Innovation Solution
A data intake and query system utilizing a late-binding schema that processes and stores machine data with flexible field extraction rules, enabling real-time analysis and identification of related events through graph-based dense representations, allowing for user feedback to improve customization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If all machine data is stored and processed, then greater flexibility and insight opportunities are provided, but the number of alerts becomes overwhelming and analysis time increases
Solution Approach 1:
The system extracts only the most relevant events and alerts from the vast amount of machine data using graph-based representations. By identifying and extracting key events that are directly related to system health and performance, the system provides actionable insights without overwhelming the analyst with all possible alerts.
Solution Approach 2:
Graph-based dense representations serve as an intermediary between raw machine data and human analysis. These graphs transform complex multi-dimensional data relationships into visual, interpretable structures that highlight only the most significant patterns and anomalies, acting as a mediator that filters and presents information in a manageable form.
2Reliability
If traditional alerting systems process all data, then comprehensive monitoring is achieved, but related events are not connected and alert relevance decreases
Solution Approach 1:
The system merges multiple data sources and event types into a unified graph-based representation that preserves relationships between events. By combining structured and unstructured data into a cohesive graph model, the system maintains comprehensive monitoring coverage while explicitly representing event relationships that traditional separate processing would lose.
Solution Approach 2:
The patent transitions from traditional flat alert processing to multi-dimensional graph representations that capture relationships across different data dimensions. This dimensional transformation allows the system to maintain comprehensive monitoring while revealing hidden relationships between events that exist in different data layers and timeframes.
3Productivity
If pre-processing extracts specified data items, then retrieval efficiency is improved, but the remainder of generated data is discarded and flexibility is reduced
Solution Approach 1:
The system performs preliminary organization of data into graph-based structures during data ingestion, creating efficient indexes and relationships in advance. This preliminary action enables fast retrieval when needed while preserving all original data, allowing the system to balance retrieval efficiency with analytical flexibility without discarding any generated data.
Data Source
AI summary
A computerized method is disclosed that includes operations of receiving a plurality of alerts, generating a graph-based dense representation of each alert of the plurality of alerts including processing of each alert with a neural network, wherein a result of processing an individual alert by the neural network is a graph-based dense representation of the individual alert, computing relatedness scores between at least a subset of the plurality of alerts, and generating a graphical user interface illustrating a listing of at least a subset of the plurality of alerts, wherein the graphical user interface is configured to receive user input corresponding to selection of a first alert, wherein the graphical user interface is rendered on a display screen. Additionally, an additional operation may include training the neural network to produce graph-based dense representations, wherein the training is performed on a corpus of metapaths.