Differential Logging for Log Data Retention Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing logging practices face challenges in efficiently managing log data, particularly in maintaining detailed logs for long periods, which can lead to storage and computational resource issues, and often result in either discarding logs or maintaining unnecessary data.
Innovation Solution
Implementing differential logging techniques, where a subset of log entries is stored in a secondary log for longer retention, allowing for selective storage and retrieval of relevant information, reducing storage and processing burdens.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all log entries are maintained for long periods, then log data availability is improved, but storage resources and computing resources are excessively consumed
Solution Approach 1:
The patent segments log entries into two categories: those requiring long-term retention and those suitable for short-term storage. This is achieved by implementing differential logging that separates log entries based on their retention requirements, storing only essential entries long-term while discarding others after a shorter period.
Solution Approach 2:
The patent extracts and identifies essential log entries that need to be retained long-term from the complete set of log entries. By scanning and evaluating log entries to determine their retention value, the system extracts only the necessary portions for long-term storage, leaving the rest for short-term or temporary storage.
2Quantity of substance
If a subset of log entries is stored for longer periods, then storage resources are optimized, but log data completeness is reduced
Solution Approach 1:
The patent implements a feedback mechanism where log entries are scanned and evaluated to determine their retention value. This feedback loop ensures that only log entries meeting specific retention criteria are stored long-term, while others are appropriately discarded or archived, maintaining an optimal balance between storage efficiency and data completeness.
Solution Approach 2:
The patent changes the retention parameter for different log entries based on their evaluated importance. By dynamically adjusting retention periods for different log entries rather than applying a uniform retention policy, the system optimizes storage resource utilization while maintaining necessary log data completeness.
3Difficulty of detecting and measuring
If detailed logs are maintained for troubleshooting, then diagnostic capability is improved, but processing overhead increases
Solution Approach 1:
The patent performs preliminary scanning and evaluation of log entries to identify which ones require long-term retention for diagnostic purposes. By pre-identifying and separating these essential entries before storage, the system reduces subsequent processing overhead when accessing logs for troubleshooting, as only relevant entries need to be maintained in accessible storage.
Data Source
AI summary
Techniques and solutions are provided to facilitate logging of computing processes. A computing process generates multiple log entries. A first portion of the multiple log entries are written to a first log. A second portion of the multiple log entries are written to the first log and to a second log, where the second portion is less than the first portion, such as being a proper subset of the first portion. Log entries can be determined to be written to the second log by scanning all or a portion of a log, by evaluating log entries as they are generated, or through computer code that causes log entries to be written to the second log. Typically, the second log is maintained for a longer period of time than the first log.