Autoencoder Anomaly Detection in Event Logs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing event logs is a laborious and time-consuming task due to their large volume and complexity, requiring deep domain knowledge and often involving manual pattern analysis, which is inefficient for timely detection of anomalies across widely deployed systems.
Innovation Solution
An unsupervised learning technique using autoencoders is applied to detect atypical events in event logs by training models on key value pairs within a threshold numeric range, allowing for automated anomaly detection without requiring labeled data or human intervention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual pattern analysis with deep domain knowledge is used to process event logs, then detection precision is improved, but productivity deteriorates due to laborious and time-consuming processing
Solution Approach 1:
The patent replaces manual mechanical analysis with an automated machine learning system. Autoencoders are trained on event log data to automatically detect anomalies, substituting the need for manual domain knowledge analysis while maintaining detection capability. The system processes event logs through neural network models that learn normal patterns and identify deviations automatically.
Solution Approach 2:
The system enables self-service anomaly detection by training autoencoders on historical event log data. The trained models then autonomously process new event logs and identify anomalies without requiring human intervention or domain knowledge. The system serves itself by using its own learned patterns to detect deviations in incoming data.
2Productivity
If automated processing techniques are applied to event logs, then productivity is improved, but detection precision deteriorates due to lack of domain knowledge integration
Solution Approach 1:
The system performs preliminary action by training autoencoder models on historical event log data before actual anomaly detection. This pre-training phase allows the system to learn normal event patterns and relationships in advance, so that when new event logs are processed, the models can immediately identify anomalies with high precision without requiring manual domain knowledge during the actual detection phase.
3Ease of operation
If traditional anomaly detection methods are used, then ease of operation is maintained, but loss of time increases due to inability to process massive event log data timely
Solution Approach 1:
The patent replaces manual operation with automated machine learning systems. The autoencoders automatically process event logs and identify anomalies without human intervention, dramatically reducing processing time while maintaining ease of operation through automated workflows. The system handles massive volumes of event log data that would be impossible for humans to process timely.
4Measurement precision
If manual processing of event logs is performed, then measurement precision is improved through human analysis, but device complexity increases due to requirement for deep domain knowledge
Solution Approach 1:
The system achieves self-service by automatically learning domain knowledge patterns from historical event log data through autoencoder training. The models internally capture complex relationships and patterns without requiring external domain knowledge input or manual configuration. This eliminates the need for human experts while maintaining high detection precision through learned patterns.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Examples described herein generally relate to processing event logs where, for each of multiple events in an event log of the one or more event logs, a table of logged event instances can be generated for the event. For each of the multiple events, the table can be processed using an autoencoder to identify one or more of the logged event instances as anomalies, and an indication of at least a portion of the anomalies can be output. In addition, the event logs and/or corresponding tables of events can be used to train models for the autoencoders.