Log Data Compression via Template Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional log file systems store redundant information, consuming storage resources and complicating data analysis due to the inclusion of uninformative text in log messages, which reappears frequently.
Innovation Solution
A method for compressing and decompressing log data by identifying and representing recurring message templates with short signatures, using a compression index to map message templates to signatures and a decompression index to restore original messages, thereby reducing storage needs and simplifying analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all log data is stored in traditional log file systems, then complete log information is preserved, but storage resources are consumed and data analysis becomes complicated
Solution Approach 1:
The log message is segmented into two parts: a message template (which is compressed and stored in the compression index) and message variables (which are stored in the log file). This segmentation allows the system to store only the variable parts in the log file while keeping the reusable template parts in the compression index, thereby reducing storage resources while preserving complete log information.
Solution Approach 2:
The message template is copied into the compression index and replaced by a reference (message signature) in the log file. This copying mechanism allows the system to store the template once in the compression index and reference it multiple times in different log messages, significantly reducing the storage space required for recurring log patterns.
2Loss of information
If all log data is stored in traditional log file systems, then complete log information is preserved, but data analysis tasks are complicated
Solution Approach 1:
By segmenting log messages into templates and variables, the system simplifies data analysis. The compression index stores the mapping between message signatures and full templates, allowing analysis tools to efficiently retrieve and compare log patterns without processing redundant text, thereby reducing analysis complexity while maintaining information completeness.
Solution Approach 2:
The compression index acts as a copied reference system that maps message signatures to full templates. This copying mechanism enables efficient data analysis by allowing quick lookup of template meanings without storing redundant text in the log files, simplifying the analysis process while preserving complete log information.
3Quantity of substance
If message templates are compressed using short signatures, then storage resources are reduced, but compression and decompression processes are required
Solution Approach 1:
The message templates are pre-processed and stored in the compression index before log messages are written to the log file. This preliminary action of creating the compression index allows the system to quickly replace templates with signatures during log writing, reducing storage resources while managing compression complexity through advance preparation.
Data Source
AI summary
Techniques are provided for compression and decompression of log data. An exemplary method comprises: obtaining a log message, wherein the log message comprises a message template and one or more message variables; obtaining a compression index that maps a plurality of message templates to a corresponding message signature; and writing the one or more message variables and a message signature corresponding to the message template of the log message to a log file. A counter may be maintained for each of a plurality of distinct message templates, and a given message signature may be assigned to a particular message template based on a length of the given message signature and a frequency of occurrence of the particular message template. The compression index comprises, for example, a key/value database where the message templates are keys and the corresponding message signatures are values of the key/value database. A decompression index maps message signatures to corresponding message templates.