Log File Preprocessing via Event Template Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing complex log files generated during software application execution is time-consuming and error-prone, leading to prolonged downtime and increased repair costs due to their massive and intricate nature.
Innovation Solution
A method and system for automated log file processing that involves generating templates to map log events to candidate values, aggregating and ranking these events based on candidate values, and partitioning the log file to identify chains of related events for further analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automated analytics are used to analyze log files, then analysis speed is improved, but analysis time remains significant due to the massive size and complexity of log files
Solution Approach 1:
The patent divides the massive log file into multiple partitions based on log event patterns and candidate values. Each partition contains a subset of log events that can be analyzed independently and in parallel, reducing the overall analysis time while maintaining comprehensive coverage of the log data.
Solution Approach 2:
The patent performs preliminary processing by generating templates from log events, extracting candidate values, and creating partitions before the actual analysis begins. This preprocessing step organizes the data structure in advance, enabling faster analysis execution and reducing the time required for the main analysis task.
2Measurement precision
If detailed log analysis is performed to improve troubleshooting accuracy, then measurement precision is improved, but analysis complexity increases leading to more errors
Solution Approach 1:
The patent introduces templates as an intermediary structure between raw log events and analysis results. Templates capture common log event patterns and associate them with candidate values, simplifying the analysis process while maintaining the ability to perform detailed troubleshooting by linking specific log events to their corresponding templates and candidate values.
3Reliability
If comprehensive log file analysis is performed to reduce errors, then reliability is improved, but analysis time increases prolonging downtime
Solution Approach 1:
By partitioning the log file into multiple independent segments, the patent enables parallel analysis of different partitions. This maintains comprehensive coverage and reliability of the analysis while significantly reducing the total time required, as multiple partitions can be processed simultaneously rather than sequentially.
Solution Approach 2:
The patent analyzes log events in organized partitions with specific criteria rather than attempting to analyze every single log event in detail. This partial action approach focuses computational resources on the most relevant log events and patterns, achieving sufficient reliability for troubleshooting while reducing overall analysis time and minimizing downtime.
Data Source
AI summary
Embodiments of the disclosure provide systems and methods for analyzing log files. Automated processing of log files can comprise reading a log file generated during execution of an application and comprising a plurality of log events and generating a plurality of templates based on the plurality of log events in the log file. Each template can map a log event to a candidate value for the log event. The plurality of log events can be aggregated into a plurality of groups based on the candidate value mapped to each log event in the plurality of templates and the plurality of groups of log events can be ranked. The log file can be partitioned based on the ranking of the plurality of groups of log events and one or more groups of log events can be provided to an analysis process based on the partitioning of the log file.