Context-Based Log Compression With On-Demand Regeneration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing logging systems fail to optimize log files based on context, leading to increased storage and transmission costs without preserving critical information, known as the 'Fear of Missing Out' (FOMO), especially in modern workloads like container applications and serverless functions.
Innovation Solution
A context-based sustainable logging framework that uses a log optimizer to analyze log entries, identifying non-critical portions for reduction and a dynamic log regenerator to restore them on demand, based on application operations and source code repositories.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If log files are reduced in size by removing non-critical information, then storage and transmission costs are reduced, but critical information may be lost
Solution Approach 1:
The log entry is segmented into two parts: a compact reference identifier and full log content stored in a database. The reference identifier points to the full content, allowing selective retrieval of only critical information when needed, thus reducing storage space while preserving access to complete logs.
Solution Approach 2:
Non-critical portions of log entries are extracted and separated from critical information. The system identifies and removes redundant or less important log data while retaining essential information, thereby reducing overall log file size without compromising the utility of critical logs.
2Loss of energy
If log optimization is applied to reduce storage costs, then storage and transmission expenses decrease, but application development and compliance requirements may be compromised
Solution Approach 1:
The system dynamically adjusts log retention policies based on compliance requirements and operational context. It can selectively retain or remove log entries based on real-time needs, ensuring that compliance and development requirements are met without unnecessarily increasing storage costs.
Solution Approach 2:
The system incorporates feedback mechanisms that monitor compliance status and development needs. Based on this feedback, it automatically adjusts log optimization strategies to ensure that critical information is preserved while still achieving storage cost reductions.
3Quantity of substance
If log entries are compressed or archived to reduce storage space, then storage requirements decrease, but access speed and search efficiency may be reduced
Solution Approach 1:
The system extracts and stores only the essential reference information in the log file, while full log content is stored in a database. This allows fast access to log references without the overhead of searching through compressed or archived full logs, thereby maintaining access speed while reducing storage requirements.
Data Source
AI summary
Mechanisms are provided for optimizing application log messages. A first log message is received and parsed to extract features. One or more second log messages are parsed to extract context information for determining a context of the first log message. The first log message is classified as to whether the first log message is a candidate for optimization based on the extracted features and context of the first log message. In response to the first log message being classified as a candidate for optimization, a template is generated for the first log message, wherein the template stores parameters of the first log message for later regeneration of the first log message. The first log message is replaced with a reference to the template to generate a reduced size log entry, and the reduced size log entry is stored in the log file.