Event Clustering With Extraction Rules for Unstructured Log Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern data centers face challenges in processing and indexing large volumes of unstructured machine-generated data, leading to difficulties in retrieving relevant information due to the unstructured nature of the data and the complexity of applying semantic meaning, which hinders effective search and analysis.
Innovation Solution
The implementation of event-based data intake and query systems, such as the SPLUNK® ENTERPRISE system, utilizes a late-binding schema to apply extraction rules to event data during search time, allowing for flexible schema development and refinement based on user interaction, enabling clustering of events based on similarities identified by extraction rules.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is pre-processed and extracted before storage, then storage space is reduced, but data availability for later use is lost
Solution Approach 1:
The patent segments data into two parts: extracted structured data for immediate analysis and raw unstructured data for future use. The extracted data is stored in a structured format for quick access, while the raw data is preserved in its original form, allowing flexible retrieval later without loss of information.
Solution Approach 2:
The system discards only the redundant portions of data during pre-processing, retaining the essential extracted information for immediate use while recovering and preserving the raw data for future analysis. This allows the system to reduce storage space now while maintaining data availability for later inquiries.
2Loss of information
If data is maintained in unstructured form, then data loss is reduced, but indexing and searching operations become difficult
Solution Approach 1:
The patent segments the data handling process into two parallel paths: one that maintains raw unstructured data for future flexibility and another that extracts structured data for immediate indexing and searching. Both paths coexist, allowing the system to benefit from both unstructured data integrity and structured data searchability.
Solution Approach 2:
The system introduces an intermediary layer of extracted structured data that mediates between the raw unstructured data and the search/indexing operations. This intermediary structure enables efficient searching while preserving access to the original unstructured data through the extraction rules and mappings.
3Productivity
If extraction rules are applied to cluster events, then data analysis efficiency is improved, but system complexity increases
Solution Approach 1:
The patent applies extraction rules and clustering in advance during the data ingestion phase, before the data needs to be searched or analyzed. By pre-extracting structured data and pre-clustering events based on extraction rules, the system improves analysis efficiency during actual use without significantly increasing operational complexity.
Solution Approach 2:
The system uses self-service mechanisms where extraction rules automatically process incoming data streams, and clustering algorithms automatically organize events based on extracted features. This automation reduces manual intervention and simplifies system operation while maintaining high analysis efficiency through intelligent processing.
Data Source
AI summary
Systems and methods include causing presentation of a first cluster in association with an event of the first cluster, the first cluster from a first set of clusters of events. Each event includes a time stamp and event data. Based on the presentation of the first cluster, an extraction rule corresponding to the event of the first cluster is received from a user. Similarities in the event data between the events are determined based on the received extraction rule. The events are grouped into a second set of clusters based on the determined similarities. Presentation is caused of a second cluster in association with an event of the second cluster, where the second cluster is from the second set of clusters.