Streaming Data Processor Clustering for Incident Root Cause Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current tools lack the ability to efficiently search and analyze large sets of raw machine data, particularly in a visually intuitive and flexible manner, due to the complexity and diversity of data systems and formats.
Innovation Solution
A data intake and query system that processes and stores raw machine data as events with flexible schema, allowing for late-binding schema application during search time, enabling field-searchable queries and providing a graphical interface for designing data processing pipelines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If raw machine data is stored in massive quantities for later retrieval and analysis, then data flexibility and analysis completeness are improved, but data search and analysis efficiency deteriorate
Solution Approach 1:
The patent applies preliminary action by extracting and storing specific data fields (such as host, source, sourcetype, and custom fields) from raw machine data during the data intake phase. This pre-extraction of potentially useful information allows the system to maintain data flexibility while enabling efficient querying later, as analysts don't need to process all raw data but can directly query the extracted fields.
Solution Approach 2:
The patent implements extraction by pulling specific meaningful fields out of the massive raw data streams and storing them in a structured format. The system extracts key identifiers and attributes from unstructured or semi-structured machine data, separating the essential queryable information from the bulk raw data, thus enabling efficient search without sacrificing data completeness.
2Quantity of substance
If diverse data types from numerous devices are stored in a big data ecosystem, then data completeness and insight potential are improved, but data management complexity increases
Solution Approach 1:
The patent applies universality by creating a unified data model that can handle diverse data types from multiple sources through a common schema structure. The system uses a universal field naming convention and data representation that works across different device types and data formats, allowing diverse data to be managed through a single system rather than requiring separate management approaches for each data type.
Solution Approach 2:
The patent implements local quality by allowing flexible schema definition at the data source level while maintaining global consistency. Different data sources can have customized field extractions and transformations tailored to their specific formats, while all data converges on a common structured model that enables uniform querying and analysis across the entire ecosystem.
3Loss of time
If specified data items are extracted and stored to facilitate efficient retrieval, then data retrieval efficiency is improved, but data loss increases as remainder data is discarded
Solution Approach 1:
The patent applies copying by creating a structured representation or copy of the essential data elements from the raw machine data. Rather than storing all raw data or discarding most of it, the system extracts key fields and stores them in an optimized format that serves as a searchable copy, enabling fast retrieval of information without needing to process the complete raw data sets.
Data Source
AI summary
Systems and methods are described for using a streaming data processor to group notable events reflecting operation of a computing system into episodes of related events reflecting an incident on the computing system, such as to enable root cause analysis of the incident. Each notable event can be generated based on one or more events detected within raw machine data. The streaming data processor can ingest a data stream of notable events, and apply a clustering algorithm to the events to cluster those events into episodes. When the episodes satisfy an action rule, the streaming data processor can take an action appropriate to that rule, such as transmitting an alert or programmatically altering operation of the computing system. The streaming data processor can utilize feedback as to the grouping of events into episodes to modify the clustering algorithm and improve accuracy of clustering.