Multi-stage Feature Extraction for ML Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current feature extraction methods for machine learning-based anomaly detection in system logs result in large feature vectors, leading to increased training time and decreased accuracy due to irrelevant features and feature explosion, causing high false positive rates and undetected anomalies.
Innovation Solution
A flexible automation system that dynamically selects preprocessing mechanisms to reduce the width of the feature vector by parsing log messages into key-value pairs, applying feature transformers and encoders to extract relevant security-related information, and automatically deciding on feature extraction techniques based on training data, thereby increasing semantic density and excluding irrelevant fields.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If original representations of feature values are used, then all available information is retained, but feature vector size increases leading to increased training time and decreased accuracy
Solution Approach 1:
The patent extracts only the most relevant features from log data using automated feature extraction techniques. The system identifies and extracts security-critical fields (e.g., authentication failures, privileged commands, network connections) while discarding irrelevant information, thereby reducing feature vector size and training time without significant information loss.
Solution Approach 2:
The patent transforms feature representations by applying encoding schemes (e.g., hashing, one-hot encoding, target encoding) and aggregation techniques that change the parameter space. This transformation reduces the dimensionality of feature vectors while preserving essential patterns, leading to faster training and improved accuracy.
2Reliability
If all fields in system logs are encoded as features, then comprehensive coverage is achieved, but feature explosion occurs causing very large feature vector size
Solution Approach 1:
The patent selectively extracts only security-relevant fields from comprehensive log data using automated feature extraction. The system applies domain-specific knowledge to identify critical fields (e.g., authentication events, process execution, network connections) and extracts only those, avoiding feature explosion while maintaining detection coverage.
Solution Approach 2:
The patent segments the feature extraction process into multiple stages: initial field identification, relevance filtering, encoding selection, and aggregation. This multi-stage segmentation allows the system to process comprehensive logs systematically while reducing the final feature set to only the most important elements.
3Measurement precision
If many features including irrelevant ones are included, then comprehensive analysis is possible, but false positive rate increases and true anomalies are undetected
Solution Approach 1:
The patent removes irrelevant and noisy features from the analysis using automated feature selection techniques. The system applies relevance filtering and importance ranking to eliminate features that do not contribute to anomaly detection, thereby reducing false positives and improving the detection of true anomalies.
Solution Approach 2:
The patent implements feedback mechanisms where the anomaly detection model continuously learns from labeled data and adjusts feature importance weights. The system uses performance metrics to identify and remove features that contribute to false positives, iteratively improving detection accuracy and reducing false alarm rates.
Data Source
AI summary
Herein are feature extraction mechanisms that receive parsed log messages as inputs and transform them into numerical feature vectors for machine learning models (MLMs). In an embodiment, a computer extracts fields from a log message. Each field specifies a name, a text value, and a type. For each field, a field transformer for the field is dynamically selected based the field's name and/or the field's type. The field transformer converts the field's text value into a value of the field's type. A feature encoder for the value of the field's type is dynamically selected based on the field's type and/or a range of the field's values that occur in a training corpus of an MLM. From the feature encoder, an encoding of the value of the field's typed is stored into a feature vector. Based on the MLM and the feature vector, the log message is detected as anomalous.