Log Message Correlation for Network Data Loss Prevention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In computer network systems, log messages generated by components within an IT infrastructure often face challenges during transmission to a central sink, leading to data loss and reduced detection accuracy of attacks due to queue capacity limitations and network unavailability, especially in low-bandwidth wireless networks.
Innovation Solution
A method where log messages are combined into a single 'combination log message' based on correlation conditions when the central sink is unavailable, allowing for local processing and storage, reducing data volume and minimizing data loss, and enabling efficient transmission when the sink becomes available again.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If log messages are transmitted continuously to the central sink, then complete log data is captured for analysis, but network bandwidth is consumed and queue capacity is exceeded during sink unavailability
Solution Approach 1:
The patent applies preliminary action by performing local correlation and filtering of log messages before transmission. The log processing component analyzes incoming log messages locally and pre-processes them by correlating related messages and filtering out redundant information, so that only essential correlated log data needs to be transmitted to the central sink when available.
Solution Approach 2:
The patent segments the log processing function into two parts: local preprocessing (correlation and filtering) performed at the log source, and centralized storage/analysis performed at the sink. This segmentation allows the system to reduce transmission data volume while maintaining complete correlated log information at the central repository.
2Reliability
If queue capacity is increased to store more log messages during sink unavailability, then more log data is retained, but local memory resources are consumed
Solution Approach 1:
The patent applies preliminary action by performing local correlation and filtering of log messages before transmission. The log processing component analyzes incoming log messages locally and pre-processes them by correlating related messages and filtering out redundant information, so that only essential correlated log data needs to be transmitted to the central sink when available.
Solution Approach 2:
The patent changes the parameter of log data representation by transforming multiple individual log messages into a compressed correlated log structure. Instead of storing each log message separately in the queue, the system represents correlated messages as a single correlated log entry with references to the original messages, significantly reducing the memory footprint of the queue.
3Productivity
If log messages are correlated locally before transmission, then data volume is reduced and bandwidth is saved, but local processing complexity increases
Solution Approach 1:
The patent applies preliminary action by performing local correlation and filtering of log messages before transmission. The log processing component analyzes incoming log messages locally and pre-processes them by correlating related messages and filtering out redundant information, so that only essential correlated log data needs to be transmitted to the central sink when available.
Solution Approach 2:
The patent introduces an intermediary log processing component that sits between the log generating applications and the central sink. This intermediary performs the correlation and filtering operations, acting as a mediator that simplifies the overall system architecture while enabling efficient local processing without requiring complex changes to the central sink or log generators.
4Reliability
If log sink becomes unavailable temporarily, then network issues occur, but log messages must be discarded when queue is full
Solution Approach 1:
The patent applies preliminary action by performing local correlation and filtering of log messages before transmission. The log processing component analyzes incoming log messages locally and pre-processes them by correlating related messages and filtering out redundant information, so that only essential correlated log data needs to be transmitted to the central sink when available.
Solution Approach 2:
The patent changes the parameter of log data representation by transforming multiple individual log messages into a compressed correlated log structure. Instead of storing each log message separately in the queue, the system represents correlated messages as a single correlated log entry with references to the original messages, significantly reducing the memory footprint of the queue and extending the period during which logs can be retained during sink unavailability.
Data Source
Figure 1
Figure 2
AI summary
The invention relates to a method for processing log messages (8) in a computer network system (1), wherein: - the log messages (8) are generated by a computing unit (2) of the computer network system (1), - the generated log messages (8) are provided by the computing unit (2) for transmission to a log message storage unit (9) of the computer network system (1), - the computing unit (2) checks whether the provided log messages (8) can be transmitted to the log message storage unit (9) under a predetermined condition, and - if not: the provided log messages (8) are combined with each other to form a combination log message (17) depending on a correlation condition (18), and - the combination log message (17) is transmitted to the log message storage unit (9).Furthermore, the invention relates to a computer network system (1) as well as a computer program (6) and an electronically readable data carrier (7).