Code-Derived Log Patterns for Fine-Grained Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing system monitoring tools lack the capability to detect finer-grained anomalies in log messages, making it difficult for administrators to identify and address performance issues before they affect user-facing functionality.
Innovation Solution
A system that uses code-derived message patterns to train a pattern matching model and anomaly detection model based on historical log messages and source code, allowing for efficient detection of anomalous log messages and triggering appropriate notifications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If system monitoring tools aggregate log messages into high-level metrics, then system performance is improved by reducing complexity and improving readability, but anomaly detection precision deteriorates because finer-grained problems are not visible
Solution Approach 1:
The system segments log messages into distinct patterns based on their structure and content. By dividing the complex log data into recognizable patterns (e.g., error patterns, warning patterns, info patterns), the system maintains both readability through pattern grouping and detection precision through pattern-specific analysis. The pattern matching model processes individual pattern instances while grouping them into meaningful categories.
Solution Approach 2:
The patent introduces pattern patterns as intermediaries between raw log messages and high-level metrics. These patterns serve as a middle layer that preserves fine-grained detail while enabling systematic analysis. The pattern matching model acts as an intermediary that translates detailed log patterns into actionable anomaly detections without losing precision.
2Measurement precision
If the system processes all log messages in detail, then anomaly detection precision is improved, but processing time and computational resources increase
Solution Approach 1:
The system performs preliminary action by pre-compiling and storing log patterns in a structured format before actual anomaly detection is needed. The pattern matching model is trained in advance on historical log data, creating a ready-to-query structure that enables fast matching during operation. This preliminary processing eliminates the need for real-time complex analysis of all log messages.
Solution Approach 2:
The patent extracts only the relevant pattern information from complete log messages. Instead of processing entire log messages in detail, the system extracts key pattern features (structure, keywords, format) and stores these as compact representations. During anomaly detection, only these extracted pattern features are matched against incoming logs, significantly reducing processing time while maintaining precision.
3Reliability
If the system monitors all software components and operations, then detection coverage is improved, but device complexity increases
Solution Approach 1:
The pattern matching model serves multiple functions simultaneously: it classifies log messages by pattern type, detects anomalies, identifies affected software components, and prioritizes notifications. This multi-functionality is achieved through a single unified model that processes all log patterns through consistent pattern matching, eliminating the need for separate specialized monitors for each component or operation type.
Solution Approach 2:
The patent merges the monitoring of multiple software components into a unified pattern-based approach. Instead of maintaining separate monitoring systems for each component, the system combines all component logs into a single pattern matching framework. This consolidation maintains comprehensive coverage while reducing complexity through shared pattern structures and unified processing logic.
4Loss of information
If the system generates detailed notifications for all anomalies, then information completeness is improved, but notification volume and false alarms increase
Solution Approach 1:
The system applies local quality by tailoring notification details to the specific pattern type and anomaly severity. Different notification levels and detail depths are applied to different pattern categories (e.g., critical errors receive detailed notifications with component information and suggested actions, while minor warnings receive summarized notifications). This selective detail level maintains information completeness for important anomalies while reducing notification volume for less critical issues.
Solution Approach 2:
The patent incorporates feedback mechanisms that allow the system to learn from historical anomaly resolutions. The anomaly detection model uses feedback about which patterns actually indicate problems and which are false alarms to refine its notifications. This feedback loop enables the system to reduce false alarm notifications while maintaining completeness for genuine anomalies, gradually optimizing notification quality over time.
Data Source
AI summary
Computer systems and processes are described herein for using code-derived message patterns to determine whether or not to trigger an anomaly notification. A system manager trains a pattern matching model and an anomaly detection model based on historical log messages, feedback about historical log messages, and message-generating portions of source code that generated the log messages. The message-generating portions of code may be processed to determine code-derived message patterns for a type of log messages. A log processor receives a message and determines the message is of the type for which code-derived message patterns are available. The message is matched to one of the available code-derived message patterns, and the log processor determines whether or not to trigger an anomaly notification based at least in part on which code-derived message pattern is matched to the message.