Log Anomaly Detection via Time-Windowed Grouping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for detecting abnormal conditions in computer networks are complex and resource-intensive, struggling to efficiently identify anomalies from log files due to the difficulty in predicting changes in log groups over time and requiring significant resource allocation for pattern recognition.
Innovation Solution
A method that involves creating log data records, specifying seamless time windows, assigning protocol lines to these windows based on creation time, analyzing similarity, forming overlap indicators, and constructing directed paths to track group evolution, allowing for real-time or historical analysis of log data to detect anomalies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If pattern recognition methods combine several widely separated protocol lines to detect anomalies, then anomaly detection capability is improved, but resource requirements for protocol data analysis increase
Solution Approach 1:
The patent segments the analysis process by introducing time windows that divide log data into manageable temporal segments. Each time window contains a subset of protocol lines that can be analyzed independently, reducing the computational burden of analyzing all protocol lines simultaneously while maintaining anomaly detection capability through sequential analysis across multiple windows.
Solution Approach 2:
The patent applies preliminary action by pre-grouping protocol lines into time windows and pre-identifying anomalous groups within each window before performing the complex task of tracking group developments across time. This preliminary organization reduces the resource requirements for the subsequent anomaly detection process by having data ready in an structured format.
2Reliability
If pattern recognition methods are used to find anomalous states in computer networks, then anomaly detection is enabled, but the methods become relatively complex and require combining several widely separated protocol lines
Solution Approach 1:
The patent segments the complex protocol data into time-windowed groups, transforming the complex task of analyzing widely separated protocol lines into a simpler sequential process of analyzing grouped data across time windows. This segmentation reduces method complexity by organizing data into manageable units with clear temporal boundaries.
Solution Approach 2:
The patent introduces time windows as an intermediary structure between raw protocol lines and anomaly detection analysis. These time windows serve as a mediating layer that organizes protocol lines into coherent groups, simplifying the detection process by providing a structured intermediate representation that captures temporal relationships without requiring direct analysis of scattered protocol lines.
3Measurement precision
If groups of protocol lines are analyzed for similarity and grouped, then anomaly patterns can be identified, but it becomes difficult to predict the development of identified groups over time
Solution Approach 1:
The patent applies periodic action by systematically analyzing groups of protocol lines across sequentially arranged time windows. This periodic analysis across multiple time windows enables the detection of temporal developments and changes in group characteristics, preserving information about how groups evolve over time while maintaining precise anomaly pattern identification within each window.
Data Source
Figure 1
Figure 2
Figure 3~9
AI summary
The invention relates to a method for detecting abnormal operating states, in particular those caused by manipulation, in a computer network (1) comprising several computers (1a, 1b, 1c), wherein: - logs are created by the computers (1a, 1b, 1c) of the computer network (1) or by processes (2a, 2b, 2c) running on these computers (1a, 1b, 1c); - when predetermined events occur, a log data record in the form of a log line (s1, ..., sn) is created for each of these events by the computers (1a, 1b, 1c) or the processes (2a, 2b, 2c); and wherein each log line (s1, ..., sn) comprises a description data record (32a, 32b, 32c) for the respective logged event, wherein a) time windows (T1, T2, T3) are specified, wherein the time windows (T1, T2, T3) in particular, are seamlessly adjacent to each other and preferably of equal length, b) the protocol lines (s1, ...,a) are assigned to a predefined time window (T1, T2, T3) according to the time of their creation or processing, c) those log lines (s1, ..., sn) that are assigned to a given time window (T1, T2, T3) are analyzed for their similarity and, based on their similarity, are grouped into individual groups (C1a, C1b, C1c, C2a, C2b, C2c, C3a, C3b, C3c) according to a predefined metric and a similarity threshold, preferably each log line (s1, ..., sn) is assigned to only one group, d) log lines (s1, ..., sn) of a given time window (T1, T2, T3) are assigned to groups of the adjacent time windows (T1, T2, T3) using the metric used in step c), and optionally the similarity threshold used in step c), e) an overlap indicator is calculated,which, based on the assignments made in step d), indicates a measure of the degree of correspondence between the log lines of two or more groups in different time windows, f) assigns groups from temporally adjacent time windows (T1, T2, T3) to each other by evaluating the calculated overlap index, by - finding groups, in particular a predecessor group and a successor group, whose overlap index exceeds a predetermined first threshold (θ) and/or - finding a number of successor groups in a subsequent time window for a predecessor group, wherein the overlap index between each of the successor groups and the predecessor group exceeds a predetermined second threshold (θpart) and the sum of the overlap indexes thus determined exceeds a predetermined first threshold (θ), and/or - finding a number of predecessor groups in a preceding time window (T1,T2, T3) for a successor group, wherein the overlap measure between each of the predecessor groups with the successor group exceeds a predetermined second threshold (θpart) and the sum of the overlap measures thus determined exceeds a predetermined first threshold (θ), and/or - finding groups to which no predecessor group or successor group can be assigned, and g) a number of directed paths are created which comprise groups as nodes and assignments as edges and which describe the course of mutually assigned groups over a predetermined number of preceding time windows, wherein the individual paths indicate the course of mutually assigned groups in successive time windows in the graph, and h) wherein for each path a time course of an indicator for the respective group is formed over a predetermined number of time windows,and i) the trends of the indicators over the individual paths are used to determine whether an anomalous condition exists.