Log Aggregation Rules Using Entropy and Repeating Sequences
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log collection systems face inefficiencies in maintaining aggregation rules due to unknown behavior patterns from new log sources, requiring manual updates and time-consuming decision-making for execution windows.
Innovation Solution
A system and method for generating aggregation rules using entropy scores and repeating sequence scores to automatically identify low-entropy features and repeating sequences, enabling efficient aggregation and dynamic time-window management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If manual updating of aggregation rules is performed to address unknown behavior patterns from new log sources, then the system can adapt to new sources, but the maintenance time and operational effort increase significantly
Solution Approach 1:
The system performs self-service by automatically analyzing new log sources, identifying repeating sequences through entropy calculation, and generating aggregation rules without human intervention. The log collector autonomously adapts to new sources by computing entropy scores, detecting repeating patterns, and updating aggregation rules based on discovered behavior patterns.
Solution Approach 2:
The system performs preliminary analysis of log sources by calculating entropy scores and identifying repeating sequences before finalizing aggregation rules. This advance preparation allows the system to pre-process and understand new log sources, enabling faster adaptation when new sources are added without requiring manual rule creation.
2Use of energy by moving object
If manual decision-making is used to determine execution time windows for aggregation rules, then the system can control resource usage, but the process becomes inefficient and time-consuming
Solution Approach 1:
The system uses feedback by continuously monitoring log data patterns and adjusting aggregation rule execution time windows based on observed repeating sequences. Entropy calculations provide feedback on data patterns, enabling the system to dynamically optimize when aggregation rules should execute to balance resource usage with effectiveness.
Solution Approach 2:
The system implements dynamic time window selection for aggregation rule execution based on detected repeating patterns. Instead of static manually-configured schedules, the system adapts execution timing dynamically according to the actual behavior patterns discovered through entropy analysis of the log data.
3Loss of energy
If aggregation rules are manually configured to save storage and processing resources, then resource efficiency improves, but the complexity of system configuration and maintenance increases
Solution Approach 1:
The system automatically performs the complex task of analyzing log patterns, calculating entropy, identifying repeating sequences, and generating optimized aggregation rules. This self-service capability eliminates the need for manual configuration while achieving resource efficiency through data-driven rule generation.
Solution Approach 2:
The system changes the approach from manual parameter specification to automatic parameter derivation through entropy calculation. By computing entropy scores and analyzing repeating sequence parameters automatically, the system determines optimal aggregation rules without requiring manual configuration of complex parameters.
4Device complexity
If existing aggregation logic is used without automated analysis, then the system structure remains simple, but it cannot effectively adapt to unknown behavior patterns from new log sources
Solution Approach 1:
The system replaces manual mechanical configuration processes with automated computational analysis. Instead of manually configuring aggregation rules based on human knowledge, the system uses entropy calculation algorithms and automated pattern recognition to analyze and adapt to new log sources.
Solution Approach 2:
The system introduces entropy calculation and repeating sequence analysis as intermediary processes between raw log data and aggregation rules. These intermediary analytical steps enable the system to bridge the gap between simple structure and high adaptability by automatically extracting behavioral patterns from diverse log sources.
Data Source
AI summary
The presently disclosed subject matter aims to a system and method for generating one or more aggregation rules configured to be utilized by a log collector. The system and method involve an aggregation rules generator directed to allow a log collector to handle the aggregation of gathered log information, even for event logs with previously unobserved behavior patterns.