System Log Compression via Metadata Grouping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression techniques for system logs do not provide sufficient storage savings due to the lack of repeating elements in individual logs, leading to inefficient use of storage resources.
Innovation Solution
Grouping and compressing system logs based on metadata that describes their similarity, such as generation time and prominent events, allows for higher compression ratios by identifying and grouping logs with similar data patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If individual system logs are compressed using conventional techniques, then storage space is reduced, but compression ratio is insufficient due to lack of repeating elements
Solution Approach 1:
The patent combines multiple individual system logs into a single aggregated log file containing logs from multiple sources and time periods. This merging creates sufficient repeating elements and patterns within the compressed data structure, enabling conventional compression algorithms to achieve significantly higher compression ratios while maintaining the ability to retrieve and analyze individual log entries when needed.
2Duration of action of stationary object
If system logs are retained for longer periods to improve support capabilities, then more storage space is consumed
Solution Approach 1:
The patent changes the physical state of system logs by compressing them into a compact binary format with optimized data structures. This parameter change in storage density allows logs to be retained for extended periods (months or years) without requiring proportional increases in storage capacity, as the compressed format reduces space requirements by significant factors.
3Quantity of substance
If compression is applied to reduce storage requirements, then access time increases due to decompression requirements
Solution Approach 1:
The patent performs preliminary organization and indexing of log data during the compression phase, creating an optimized data structure that facilitates rapid random access. By pre-organizing the compressed log file with efficient indexing and metadata structures, the system enables quick location and extraction of specific log entries without requiring full decompression of the entire file, thus minimizing access time overhead.
Data Source
AI summary
A method and system for grouping and compressing system logs are described herein. A generation module residing and executing on a computer system may be configured to generate system logs containing system data important for support purposes describing the computer system, and system log metadata for describing the system logs. A grouping module residing and executing on a computer system may be configured to group system logs based on the system log metadata. Each group of system logs may comprise two or more system logs having the same or similar system log metadata. A data container is produced for each group of system logs and the data container is compressed. System logs having the same or similar system log metadata are more likely to have similar data, thus resulting in a higher degree of compression when grouped and compressed together.