AI Log Parsing and Hierarchical Clustering for Root Cause Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log processing systems face challenges in handling vast volumes of unstructured log data, leading to inefficient and rigid regex-based pattern detection that limits intuitive sorting and analysis, and lack user-friendly interaction for log analytics.
Innovation Solution
An AI-driven pipeline that converts multi-line logs into single-line logs using hybrid parsing, augments them with metadata, and clusters them hierarchically, enabling interactive log analytics through a chatbot interface with a large language model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If regex-based pattern detection is used for log parsing, then data extraction can be performed, but the system becomes rigid and lacks adaptability to varying log structures
Solution Approach 1:
The patent replaces the mechanical regex-based pattern matching system with an AI-based system that uses natural language processing and machine learning models to understand and parse log structures. This substitution enables the system to adapt to varying log formats without requiring manual regex pattern creation, thereby resolving the contradiction between adaptability and complexity.
Solution Approach 2:
The system dynamically changes the parsing parameters and approaches based on the detected log structure type. By using AI models to identify log patterns and automatically adjust parsing parameters accordingly, the system achieves high adaptability to different log structures while maintaining manageable complexity through automated parameter adaptation.
2Adaptability or versatility
If handcrafted regular expressions are used for log parsing, then pattern detection can be performed, but malleability and breadth of pattern detection are restricted
Solution Approach 1:
The system performs self-service by automatically generating and optimizing regular expressions through AI-driven analysis of log data. Instead of requiring manual creation and maintenance of regex patterns, the system autonomously learns from log samples, generates appropriate parsing patterns, and updates them over time, thereby achieving broad pattern detection capability without manual intervention.
Solution Approach 2:
The system performs preliminary action by pre-processing and analyzing log data to automatically generate parsing patterns before actual log parsing operations. This preliminary AI-driven pattern generation enables the system to handle diverse log structures with broad malleability while eliminating the need for manual regex creation.
3Quantity of substance
If unstructured log data is processed, then large volumes of data can be handled, but intuitive sorting and analysis become difficult
Solution Approach 1:
The system segments unstructured log data into structured components by extracting key fields, timestamps, error codes, and contextual information using AI-based parsing. This segmentation transforms the unstructured data into organized, queryable formats that maintain the ability to handle large volumes while enabling intuitive sorting and analysis through structured representation.
Solution Approach 2:
The patent introduces an intermediary layer of AI-based processing that sits between the raw unstructured log data and the analysis layer. This intermediary systematically transforms and structures the data, making it suitable for sorting and analysis while preserving the ability to handle large volumes through automated processing.
4Ease of operation
If traditional log analysis methods are used, then basic parsing can be performed, but user-friendly interaction and intuitive analytics are lacking
Solution Approach 1:
The system replaces complex mechanical query interfaces with AI-based natural language processing that understands user queries in human language. This substitution provides user-friendly interaction by accepting natural language requests while the underlying AI models handle the complexity of data processing, analysis, and interpretation automatically.
Solution Approach 2:
The analysis system performs self-service by automatically executing complex queries, generating insights, and presenting results based on user-friendly natural language inputs. The AI models autonomously handle the complex analysis tasks, transforming simple user queries into comprehensive analytics without requiring users to interact with complex system interfaces.
Data Source
AI summary
A log system prompts a foundation model to identify regular expressions that delineate single line logs in log files and incorporates user feedback for the regular expressions in a feedback loop as part of a hybrid parsing approach. The log system splits the log files into single line logs according to the regular expressions and augments the single line logs with classifications and values of metadata fields. The augmented single line logs are then clustered into hierarchical clusters with corresponding cluster patterns and the log system stores the augmented single line logs in a database indexed by the classifications and values of metadata fields and associated with corresponding clusters/cluster patterns. A presentation module accesses the database when responding to user queries for filtered single line logs and log correlations/root cause analysis.