System Log Anomaly Detection Using Self-Supervised Token Sequences
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection methods in system logs face challenges in efficiency and reliability, particularly when dealing with continuous changes in log patterns, and there is a need for a more effective and fast methodology based on deep learning.
Innovation Solution
Applying a cut-and-paste technique for anomaly detection using self-supervised learning, which involves generating normal and abnormal token sequences from system logs, training a sentence classification model, and calculating an anomaly score for detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deep learning-based anomaly detection is used to increase coverage and reliability, then detection reliability is improved, but computational complexity and training time increase
Solution Approach 1:
The patent segments the log file into individual log lines and further into tokens, processing them in manageable chunks rather than the entire file at once. This segmentation allows the model to handle large datasets efficiently while maintaining high detection reliability through comprehensive analysis of individual components.
Solution Approach 2:
The patent performs preliminary tokenization and preprocessing of log files before feeding them to the language model. By preparing the data in advance and creating token sequences from preprocessed log lines, the system reduces computational complexity during actual anomaly detection while maintaining reliable detection performance.
2Reliability
If deep learning-based anomaly detection is used to increase coverage and reliability, then detection reliability is improved, but training time increases
Solution Approach 1:
The training process is segmented into multiple stages: data preprocessing, tokenization, creation of token sequences, and model training on segmented data. This segmentation enables efficient use of computational resources and reduces overall training time while maintaining high detection reliability through comprehensive data processing.
Solution Approach 2:
The patent transforms log data into token sequences with specific length parameters and uses these parameterized representations for training. By changing the data representation parameters through tokenization and sequence construction, the system achieves efficient training times without sacrificing detection reliability.
3Ease of operation
If pattern matching is used for log anomaly detection, then ease of operation is maintained, but detection reliability deteriorates when failure messages are continuously added or excluded
Solution Approach 1:
The patent replaces the mechanical pattern matching approach with a language model-based neural network system. This substitution enables the system to maintain ease of operation through automated learning while significantly improving detection reliability by understanding semantic meanings and contextual relationships in log messages, allowing it to adapt to continuously changing failure messages without manual pattern updates.
Data Source
AI summary
Disclosed is a method and apparatus for detecting anomalies in a system log on the basis of self-supervised learning, using a language model. The method comprises performing preprocessing on the system log, generating a normal token sequence having a preset length by concatenating tokenized log lines of the system log, generating an abnormal token sequence using the normal token sequence, calculating an anomaly score for a determination target token sequence using a sentence classification model, and determining the token sequence as an abnormal system log when the calculated anomaly score is greater than a threshold value.