Log Compression System Using Reference Table Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large log files consume storage space and increase network bandwidth and transmission time, making them less valuable for monitoring system performance and troubleshooting due to frequent overwriting and difficulty in transferring.
Innovation Solution
A log compression system that maintains a log reference table to identify repetitive log message patterns, allowing for the creation of compressed log messages with timestamps and variable portions, reducing the size of log files by referencing pre-defined log entries instead of storing entire message strings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If log files are stored in full format to maintain complete information for troubleshooting, then information completeness is improved, but storage space consumption increases
Solution Approach 1:
The patent segments log files into two distinct parts: a log reference table containing repetitive message patterns and a log data file containing only variable portions and references. This segmentation allows the system to store minimal information in the main log file while preserving complete information by referencing the external reference table, thereby resolving the contradiction between information completeness and storage space consumption.
Solution Approach 2:
The patent creates a separate copy of repetitive log message patterns in the form of a log reference table. Instead of storing full message strings repeatedly in the log file, the system stores references to these copied patterns, significantly reducing storage space while maintaining the ability to retrieve complete information when needed.
2Measurement precision
If log files are stored in full format to ensure accurate troubleshooting data, then measurement precision is improved, but file size increases making transfer difficult
Solution Approach 1:
By segmenting the log storage into a reference table and a compact data file with references, the patent reduces the file size stored locally while preserving troubleshooting accuracy. The segmented structure allows the system to maintain precise troubleshooting data through references to the external reference table, resolving the contradiction between measurement precision and file size.
3Loss of information
If log files are maintained at large size to capture detailed system performance data, then data detail is improved, but frequency of overwriting increases reducing log value
Solution Approach 1:
The patent segments log information into a persistent reference table containing detailed system performance data patterns and a compact log data file that references these patterns. This segmentation allows the system to maintain detailed data in the reference table while the compact log file can grow larger without consuming proportional storage space, reducing the frequency of overwriting and increasing log file utility.
Data Source
AI summary
A computer-implemented method to compress log messages is disclosed. The method starts with maintaining a log reference table having a plurality of log reference entries, each containing a string of characters to be recited repeatedly during logging of messages. In response to a request for logging a log message initiated from an application, the method creates a compressed log message entry in a compressed log file. The compressed log message entry includes: a time stamp indicating the time the log message being created; a log reference identifier identifying one of the plurality of log reference entries without reciting a corresponding string of characters of the log reference entry, where the one of the plurality of log reference entries contains one or more variable indications; and a log variable portion associated with the log message, where the log variable portion contains a part of the log message.