Log Message Template Compression Using Field-Type Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional log file systems store repetitive and uninformative information, leading to increased storage resource consumption and challenges in data analysis, as they store entire log messages regardless of repetitive text, which is not informative for machine learning or natural language processing.
Innovation Solution
The method involves compressing log data using pattern recognition and field-based techniques, where repetitive message templates are represented by short signatures, and message variables are compressed based on predefined field types, reducing storage needs and simplifying analysis tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If traditional log file systems store all log data generated by monitored systems, then complete log information is preserved, but storage resources are consumed and data analysis becomes challenging
Solution Approach 1:
The patent extracts and removes repetitive uninformative information from log messages by identifying common message templates and representing them once with signatures. Only the variable portions (message variables) are stored in full, while repetitive template text is eliminated, thereby reducing storage consumption while preserving essential log information.
Solution Approach 2:
The patent segments log messages into two distinct components: message templates (repetitive portions) and message variables (variable portions). This segmentation allows the system to handle each component differently - templates are compressed into signatures while variables are stored completely, optimizing both storage efficiency and information retention.
2Manufacturing precision
If traditional log file systems store all log data, then full log detail is available, but the size of log file systems increases
Solution Approach 1:
The patent changes the representation parameter of repetitive log text from full-text storage to signature-based storage. By transforming message templates into compact signatures and storing only variable portions in detail, the system maintains log data detail where needed while dramatically reducing overall file system volume.
3Reliability
If repetitive uninformative information is stored in log files, then complete log records are maintained, but challenges for data analysis tasks arise
Solution Approach 1:
The patent extracts repetitive uninformative template text from log messages and replaces it with compact signatures. This extraction eliminates the complexity of analyzing repetitive patterns while maintaining the ability to reconstruct complete log records when needed, thereby simplifying data analysis tasks without sacrificing record completeness.
4Quantity of substance
If message variables are compressed using reduced granularity and precision, then compression ratio improves, but measurement precision of log data decreases
Solution Approach 1:
The patent applies different quality levels to different parts of the log message. Message templates are compressed into signatures (lower precision representation), while message variables are stored with their original precision and detail. This local differentiation allows compression without sacrificing the precision of critical variable data.
Data Source
AI summary
Techniques are provided for compression of log data using field types. An exemplary method comprises: obtaining at least one log message, wherein the at least one log message comprises a message template and at least one message variable, wherein the message template comprises at least one field corresponding to the at least one message variable; obtaining a compression index that maps a plurality of message templates to a corresponding message signature; identifying a predefined field type of the at least one field; selecting a compression technique to apply to the at least one message variable based on the predefined field type to obtain a compressed message variable; and writing the compressed message variable and a message signature corresponding to the message template of the at least one log message to a log file. The at least one log message can be a historical log message or part of a real-time stream of log messages.