Multi-feature Log Anomaly Detection via BiGRU
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional log anomaly detection methods face challenges due to the continuous updating of software systems, inefficient log template extraction, and the inability of single features to cover all log information, leading to low accuracy in anomaly detection.
Innovation Solution
A multi-feature log anomaly detection method based on full log semantics, which involves preprocessing log data to extract semantic, type, time, and quantity features, and using an attention-mechanism-based BiGRU neural network model for prediction to determine normal or abnormal log sequences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If log template extraction is used for anomaly detection, then detection speed is improved, but accuracy deteriorates due to out-of-vocabulary words and template variations
Solution Approach 1:
The patent segments log entries into multiple features including semantic features (from log message), type features (log level, source), time features (timestamp, frequency), and quantity features (parameter counts). This segmentation allows each feature to be processed independently and combined for comprehensive anomaly detection, resolving the contradiction between speed and accuracy by avoiding reliance on a single template-matching approach.
Solution Approach 2:
The patent creates a composite feature representation by combining multiple different feature types (semantic, type, time, quantity) into a unified anomaly detection model. This composite approach is analogous to using composite materials - it leverages the strengths of each feature type while compensating for their individual weaknesses, particularly handling out-of-vocabulary words through the combination of semantic embedding with structural features.
2Device complexity
If single feature extraction is used, then algorithm complexity is reduced, but detection accuracy deteriorates due to inability to cover all log information
Solution Approach 1:
The patent divides log analysis into four distinct feature segments: semantic features (word embeddings from log messages), type features (categorical attributes like log level and source), time features (temporal information including timestamps and frequencies), and quantity features (numerical attributes like parameter counts). Each segment is processed with appropriate techniques and then integrated, allowing comprehensive coverage without overwhelming complexity.
Solution Approach 2:
The patent designs a multi-functional feature extraction system that simultaneously extracts multiple types of features from the same log data. The anomaly detection model serves multiple purposes by analyzing different feature dimensions, making the system universally applicable to various log formats and anomaly types while maintaining manageable complexity through modular architecture.
3Productivity
If conventional log parsing is used, then processing efficiency is improved, but adaptability deteriorates due to continuous log template updates
Solution Approach 1:
The patent employs dynamic feature extraction that adapts to varying log formats. The semantic feature extraction uses word embeddings that can handle unseen vocabulary, and the structural feature extraction dynamically identifies relevant attributes based on the actual log format. This dynamic approach maintains processing efficiency while adapting to continuous template updates without requiring complete reconfiguration.
Solution Approach 2:
The patent changes the parameters of feature extraction by using multiple feature types with different characteristics. When log templates update, the system can adjust the weighting and importance of different feature parameters (semantic, type, time, quantity) to maintain effective anomaly detection. This parameter flexibility allows the system to adapt to template changes while preserving processing efficiency.
Data Source
AI summary
A multi-feature log anomaly detection method includes steps of: preliminarily processing a log data set to obtain a log entry word group corresponding to all semantics of a log sequence in the log data set, and using the log entry word group as a semantic feature of the log sequence; extracting a type feature, a time feature and a quantity feature of the log sequence, and encoding the semantic feature, the type feature, the time feature and the quantity feature into a log feature vector set of the log sequence; training a BiGRU neural network model with all log feature vector sets to obtain a trained BiGRU neural network mode; and inputting the log data set to be detected into the trained BiGRU neural network model for prediction, and determining whether the log sequence is a normal or abnormal log sequence according to a prediction result.