Adaptive Log Deduplication via Dynamic Time Windows
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log deduplication techniques are ineffective in flood scenarios and idle scenarios, leading to wasteful resource consumption and delayed processing of unique log messages due to redundant message transmission.
Innovation Solution
Adaptive time window-based deduplication, where message values are counted and transmitted as marker messages with metrics, allowing the backend device to deduce underlying causes and take corrective actions, reducing redundant message transmission and resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If log messages are transmitted to backend device for processing, then information about computing device operation is provided, but redundant message transmission causes wasteful resource consumption and delayed processing of unique log messages
Solution Approach 1:
The patent applies preliminary action by performing deduplication of log messages at the source computing device before transmission to the backend device. The system counts log messages within a time window and identifies duplicates, then transmits only unique messages or aggregated summaries. This preliminary filtering action prevents redundant messages from consuming network bandwidth and backend processing resources, directly resolving the contradiction between ensuring reliable processing information delivery and reducing resource consumption.
2Reliability
If log messages are transmitted to backend device, then operational information is provided, but high-rate log generation overwhelms the backend device causing delays
Solution Approach 1:
The system performs preliminary aggregation and deduplication of log messages at the source device before transmission. By counting messages within time windows and identifying unique values, the system reduces the volume of transmitted data. This preliminary action prevents backend device overload while maintaining reliable delivery of unique operational information, resolving the contradiction between processing reliability and throughput.
Solution Approach 2:
The patent uses copying by transmitting a representative copy or summary of duplicate log messages rather than each individual message. Instead of sending multiple identical copies of the same log entry, the system sends one representative message with aggregated count information. This copying approach maintains the informational value for backend processing while dramatically reducing message volume and preventing backend overload.
3Loss of energy
If deduplication is performed to reduce redundant messages, then resource consumption is reduced, but effectiveness decreases in flood scenarios and idle scenarios
Solution Approach 1:
The patent applies dynamics by making the deduplication time window adaptive rather than fixed. The system adjusts the time window duration based on the observed log message generation rate and patterns. During flood scenarios with high message rates, the window expands to capture and aggregate more duplicates effectively. During idle scenarios with low rates, the window contracts to ensure timely transmission of unique messages. This dynamic adjustment maintains deduplication effectiveness across varying operational conditions while optimizing resource consumption.
Data Source
AI summary
Example techniques for adaptive time window-based log message deduplication are described. In an example, message values are obtained from received log messages. Further, the number of log messages received in a time window having a message value is counted. A log message from which the message value is obtained and the counted number are transmitted upon expiry of the time window. A length of a time window in which a subsequent counting of log messages is to be performed is determined based on various parameters.