Log Anomaly Detection via Autoencoder Count Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection methods in telecom networks rely on labeled data and clean training data, which are not available in the 5G/6G telecom domain, making it challenging to apply supervised or unsupervised machine learning for anomaly detection in unstructured log files.
Innovation Solution
The method converts each log line into a log identifier and generates count vectors based on the frequencies of these identifiers. These count vectors are then input into an autoencoder to detect anomalies by comparing the input and output vectors, marking the sequence as anomalous if the difference exceeds a threshold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If supervised or unsupervised machine learning is applied to anomaly detection in unstructured log files, then anomaly detection capability is improved, but the requirement for labeled data and clean training data makes the approach inapplicable in the 5G/6G telecom domain
Solution Approach 1:
The system performs self-supervised learning by using the log data itself to train the autoencoder model. The model learns normal log patterns automatically without external labeled data, enabling anomaly detection in the 5G/6G telecom domain where labeled data is unavailable. The autoencoder encodes normal log sequences and reconstructs them, using the reconstruction error to detect anomalies.
Solution Approach 2:
The patent introduces an intermediary representation layer (latent space) between the input log sequences and the anomaly detection decision. The autoencoder transforms log sequences into a compressed latent representation and back, with the reconstruction process serving as the mediator that identifies anomalies through reconstruction error analysis.
2Measurement precision
If complex machine learning models are used for anomaly detection, then detection accuracy is improved, but model complexity and number of parameters increase
Solution Approach 1:
The patent extracts only the essential information from log sequences by using a fixed-size window of N log lines and converting them into count vectors with K log identifiers. This extraction approach reduces the complex unstructured log data into a simplified representation that can be processed by a relatively simple autoencoder model, achieving good detection accuracy without requiring extremely complex models.
Solution Approach 2:
The patent changes the representation parameters by converting log sequences into count vectors with specific dimensions (K identifiers × 2 frequency components). This parameter transformation allows the use of a simpler autoencoder architecture with fewer parameters while maintaining effective anomaly detection capability through the structured count vector representation.
3Measurement precision
If traditional anomaly detection methods are used, then processing requirements are high, but the method provides high precision and recall values with lightweight operation
Solution Approach 1:
The patent segments the log sequence into a fixed window of N log lines and divides the frequency analysis into two parts (front n log lines and rear N-n log lines). This segmentation allows the model to process only the relevant local context around potential anomalies rather than the entire log stream, reducing computational requirements while maintaining high precision and recall through focused analysis.
Solution Approach 2:
The patent uses a partial action approach by analyzing only the frequency distribution of log identifiers within the window rather than processing every individual log detail. The count vector captures the essential patterns with minimal computation, and the autoencoder processes only this compressed representation, significantly reducing processing requirements while achieving high detection accuracy.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Method comprising: converting each log of a sequence of N logs into an identifier among K different identifiers to obtain a sequence of N identifiers; for each n between 0 and N: for each of the K identifiers: counting occurrences of the identifier among the first n identifiers of the sequence to obtain a front frequency of the identifier for the respective n; and for each of the K identifiers: counting occurrences of the identifier among the last N-n identifiers of the sequence to obtain a rear frequency of the identifier for the respective n; arranging the front frequencies and the rear frequencies of the identifiers in a count vector; inputting the count vector an autoencoder to obtain an output vector for the respective n; determining a difference between the output vector and the count vector; marking the sequence as anomalous if the difference between the output vector and the count vector is larger than a threshold; wherein each of the identifiers is an integer.