Log Anomaly Detection via Token Pattern Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for analyzing log files to detect abnormal operating states of computer systems are complex, resource-intensive, and lack efficient methods to analyze all tokens of protocol lines for patterns simultaneously, leading to increased resource requirements and inefficiencies in anomaly detection.
Innovation Solution
A method that creates templates for log lines based on event types, with static and variable tokens, and uses data type and distribution type classification to identify patterns, allowing for rapid and reliable detection of anomalies by initializing data types and updating them periodically based on new log lines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If log files are analyzed by examining individual token values for patterns sequentially, then anomaly detection can be performed, but the resource requirements and complexity increase significantly
Solution Approach 1:
The patent segments log line analysis by creating templates that divide log lines into fixed (static) and variable (dynamic) tokens. This segmentation allows the system to focus pattern matching only on variable tokens while ignoring static portions, reducing computational complexity while maintaining detection reliability
Solution Approach 2:
The patent performs preliminary action by pre-processing log lines to create templates with identified static and variable tokens before actual anomaly detection. This pre-processing step establishes the structure once, allowing subsequent analyses to operate more efficiently on the prepared templates rather than parsing raw log lines repeatedly
2Productivity
If all tokens of protocol lines are analyzed for patterns simultaneously, then detection efficiency improves, but no concrete methods are known from prior art to achieve this
Solution Approach 1:
The patent segments tokens into static and variable categories, allowing simultaneous processing of all tokens while focusing computational effort only on variable tokens for pattern matching. This segmentation makes simultaneous analysis feasible by eliminating redundant processing of static content
Solution Approach 2:
The patent changes the parameter of token analysis by distinguishing between static and variable tokens, applying different processing strategies to each. Variable tokens undergo pattern matching analysis while static tokens are recognized and skipped, enabling efficient simultaneous processing of all tokens
3Reliability
If data types are re-initialized frequently to adapt to new log line patterns, then detection accuracy improves, but time and computing power are wasted
Solution Approach 1:
The patent introduces dynamics by making the template structure adaptable - while static tokens remain fixed, variable tokens can have their patterns updated when new log line patterns are detected. This dynamic adaptation allows the system to improve accuracy without complete re-initialization
Solution Approach 2:
The patent implements periodic action by updating variable token patterns at specific intervals or when triggered by sufficient new data, rather than continuously or at every log line. This periodic updating maintains detection accuracy while avoiding excessive re-initialization overhead
Data Source
AI summary
The invention relates to a method for detecting anomalous operating states of a computer system, wherein logs are created during operation by the computer or by processes running on the computer, wherein at least one event type is assigned to each log line based on its syntax, wherein a template is created for each event type comprising a number of sub-strings of the log lines representing the tokens of the respective event type, characterized in that the data type (DT) and/or distribution type of the individual tokens (T) of the respective template of the event type is learned for the templates of the individual event types by performing an initialization (IT) of the data types (DT) of the individual tokens (T) of the template of the event type based on a sample comprising a number of log lines of the respective event type.- where a value list is created for each individual token (T), - where the individual value lists are tested in a sequential test sequence for their belonging to predefined data types (DT) and/or distribution types, - that periodically, as soon as a sequence of newly created log lines of a respective event type has been created, it is checked whether the value lists of the individual tokens, to which the substrings of the sequence of newly created log lines belong, correspond to the data types (DT) and/or distribution types of the corresponding token of the respective event type, and - that an anomalous operating state of the computer system is detected if a value list of at least one token does not correspond to the properties of the data type (DT) and/or distribution type assigned to the respective token (T) of the event type.