Log Entry Deduplication for Lower Storage and Easier Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log recording systems suffer from redundant and duplicate log data, which occupy excessive storage space and complicate analysis, especially during abnormal system operations, making it difficult to identify critical messages.
Innovation Solution
A method to determine a target log recording component, group similar log entries, identify duplicate content, and de-duplicate it, retaining distinctive content, thereby reducing storage needs and improving resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If log recording component records all operations, then complete system operation tracking is achieved, but storage space is excessively consumed and analysis becomes complicated
Solution Approach 1:
The patent extracts and removes duplicate content from log entries while retaining distinctive information. The de-duplication process identifies and eliminates redundant portions of log data, keeping only the unique and necessary information for analysis, thus reducing storage consumption while preserving essential log data completeness
Solution Approach 2:
The patent discards duplicate log content that consumes excessive storage space while recovering and retaining the distinctive valuable information. By identifying duplicate patterns and removing them, the system recovers storage resources while maintaining the essential diagnostic value of log entries
2Loss of information
If log recording component records all operations, then complete system operation tracking is achieved, but log analysis becomes difficult due to duplicate contents
Solution Approach 1:
The patent extracts and removes duplicate content from log entries, isolating only the distinctive valuable information. This extraction process eliminates the noise of redundant data, making log analysis significantly easier while preserving the complete set of essential operational information needed for system troubleshooting
Solution Approach 2:
Instead of trying to analyze logs with all their duplicate content, the patent inverts the approach by removing the duplicates first. This inversion simplifies the analysis process by presenting only the unique, non-redundant information that matters for diagnosing system issues
3Quantity of substance
If de-duplication is performed on all log entries, then storage space is reduced, but processing time increases
Solution Approach 1:
The patent applies partial de-duplication by focusing on removing duplicate content within log entries rather than processing entire log files extensively. By targeting only the redundant portions of log data for de-duplication, the system achieves significant storage reduction while minimizing the processing time required
Solution Approach 2:
The patent extracts only the duplicate portions of log entries for removal, rather than processing and analyzing entire log entries. This selective extraction approach reduces storage space effectively while keeping the processing time limited to identifying and removing only the redundant content
Data Source
AI summary
Embodiments of the present disclosure relate to a method, a device, and a computer program product for processing logs. The method includes determining, based on a duration of a predetermined number of log entries recently recorded by a log recording component, a target log recording component from a group of candidate log recording components. The method further includes determining, based on a predetermined number of log entries recently recorded by the target log recording component, a log group including a plurality of similar log entries. The method further includes determining a duplicate content among the plurality of log entries in the log group and a distinctive content for each of the plurality of log entries. The method further includes updating the plurality of log entries by de-duplicating the duplicate content and retaining the distinctive content for each of the plurality of log entries.