Compressing Event Trace Logs via Merging Repeated Traces
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Raw event logs are inefficient to store and process due to their large size, making it difficult to analyze and query for information, as existing compression techniques do not significantly reduce the number of entries and require decompression for analysis.
Innovation Solution
Compress event logs by replacing repeated event traces with single entries and sequences of identical events with substitute records, reducing the log size and improving searchability by concatenating events into individual records that identify entire sets of actions in a computing environment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If raw event logs are stored in their original format, then complete event information is preserved, but storage space requirements increase and processing efficiency decreases
Solution Approach 1:
The patent merges multiple identical or similar event entries into a single consolidated entry. When consecutive events are detected to be identical or similar, they are combined into one representative entry that preserves the essential information while eliminating redundancy, thereby reducing storage requirements without significant loss of analytical value
Solution Approach 2:
Instead of storing each individual event entry separately, the patent inverts the approach by storing a single representative entry that stands for multiple identical events. This inversion transforms the data structure from many individual records to fewer aggregated records, achieving space efficiency while maintaining the ability to reconstruct original event sequences when needed
2Loss of information
If raw event logs are stored in their original format, then all event details are available, but the number of entries remains large making analysis difficult
Solution Approach 1:
The patent combines multiple event entries into consolidated representations that reduce the total number of entries. By merging identical or similar events into single representative entries, the analysis workload is significantly reduced while the essential event details and patterns remain preserved for security analysis and troubleshooting
Solution Approach 2:
The patent extracts and retains only the essential distinguishing features of event sequences while eliminating redundant information. By taking out only the critical event characteristics needed for analysis and discarding repetitive details, the system improves analysis efficiency while maintaining the availability of key event information
3Quantity of substance
If existing compression techniques are applied to event logs, then file size is reduced, but decompression is required for analysis and the number of entries remains unchanged
Solution Approach 1:
The patent merges events at the data structure level rather than applying bit-level compression. This structural merging creates fewer entries that inherently require less storage space and can be analyzed directly without decompression, as the consolidated entries are already in an analysis-ready format that preserves essential event information
Solution Approach 2:
The patent performs event consolidation in advance, before analysis is needed. By pre-processing the event logs to merge identical or similar events into representative entries, the system eliminates the need for decompression operations during analysis, as the compressed structure is already optimized for direct querying and examination
Data Source
AI summary
Method and apparatus for compressing raw event logs into smaller readable formats are described. An example includes receiving an uncompressed log file including traces of events executed on a computing system. In the uncompressed log file, a number of consecutive events are identified referencing an action performed with different parameters, and the uncompressed log file is modified by replacing the identified consecutive events with a record indicating that an event has been repeated the number of times. In the modified log file, repeated sequences of events are identified, a compressed log file is generated by replacing, in the modified log file, repeated sequences of events with a record referencing an initial repetition of events and a difference between parameters included in the initial repetition of events and a respective repeated sequence, and the generated compressed log file is output.