Knowledge Graph for Automated Log Parsing and Event Monitoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Staff at log-monitoring centers face a significant manual effort in accessing information from software logs, leading to underutilization and potential undetected attacks, data leaks, and service-agreement violations due to the lack of automated parsing and correlation of log entries.
Innovation Solution
A Knowledge Graph (KG) is used to automate event monitoring by parsing log entries into structured formats, grouping and consolidating them, and correlating with use cases to trigger alerts and actions through a Security Information and Event Management (SIEM) system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If manual effort is used to access information from software logs, then staff can understand and process log content, but the large volume of logs becomes underutilized and requires significant manual effort
Solution Approach 1:
The patent replaces manual mechanical processing of logs with an automated knowledge graph system that uses AI models and algorithms to parse, extract, and correlate log information automatically, eliminating the need for staff to manually create regular expressions and process logs
Solution Approach 2:
The knowledge graph system automatically processes logs without human intervention by self-learning from log data, automatically extracting fields, identifying patterns, and generating insights, allowing the system to serve itself rather than requiring staff to continuously manually process logs
2Reliability
If no automated parsing is implemented, then manual processing can be performed, but information remains invisible and unusable leading to undetected attacks and data leaks
Solution Approach 1:
The patent replaces manual log analysis with an automated knowledge graph system that uses AI-powered parsing, extraction, and correlation algorithms to automatically detect security threats, data leaks, and service agreement violations, significantly improving detection reliability through consistent automated analysis
Solution Approach 2:
The knowledge graph acts as an intermediary layer between raw logs and security analysts, automatically processing logs through multiple AI models and algorithms to extract meaningful information and present it in a usable format, making invisible log information visible and actionable
3Ease of manufacture
If manual effort is required to create and test regular expressions, then log parsing can be performed, but the process becomes complex and time-consuming
Solution Approach 1:
The patent replaces the manual creation and testing of regular expressions with an automated knowledge graph system that uses AI models to automatically parse logs, extract fields, and identify patterns, eliminating the complexity of manual regex development while improving parsing accuracy and consistency
Data Source
AI summary
Automated event monitoring is performed utilizing a Knowledge Graph (KG) constructed by grouping and consolidation of a variety of log Entry Types. A log entry is received by a knowledge graph parser (Kg parser). That parser finds contiguous sub-strings in a log entry that have a parameterized basic-format. The parser figures out which basic-formats are present, where, and with which parameters. Given a sub-string, its basic-format and its parameters, the parser can correctly parse the sub-string to components (e.g., keys and values if a key-value format; fields if a structured format). A result of the parsing is an entity type tree structure. Next, a grouping and consolidation capability functions to modify the KG to incorporate an incoming new entry type structure. The KG may be consumed by a user (e.g., visualization; querying), and may provide an artifact to an event monitoring system to automatically trigger certain actions (e.g., alerts).