Log Summarization and Diff for DevOps Triage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
IT professionals and DevOps teams face difficulties in effectively using log data for operational analytics due to the complexity and volume of log information, with existing tools returning too many raw results and lacking in facilitating quick triaging of live site issues.
Innovation Solution
A method for analyzing log data by identifying and summarizing similar log lines, providing user-selectable indicators to reveal differences within sets, and comparing buckets of log data to highlight changes and unique sets, employing techniques like vectorization, clustering, and auto-faceting to facilitate visualization and troubleshooting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If text search tools are used to enable search over logs, then search capability is provided, but too many raw results are returned for a user to handle
Solution Approach 1:
The patent extracts and displays only the most relevant and distinct log patterns from the search results, rather than showing all raw results. It identifies and presents unique log line patterns with their frequencies, filtering out redundant information while maintaining search functionality.
Solution Approach 2:
The patent segments the large set of search results into grouped patterns based on similarity analysis. Log lines are clustered into representative patterns, each showing a template form with variable substitutions, making the results manageable and interpretable.
2Loss of information
If all log lines are displayed to provide complete information, then information completeness is maintained, but it becomes difficult for users to quickly triage live site issues
Solution Approach 1:
The patent performs preliminary analysis of log data to identify and group similar patterns before presentation to the user. Log lines are pre-processed to extract key patterns, calculate frequencies, and organize them into hierarchical groups, so that when users query the logs, the results are already optimized for quick triage.
Solution Approach 2:
The patent applies different levels of detail and aggregation to different portions of the log data based on their relevance and frequency. Common patterns are summarized with high-level statistics, while less frequent patterns are shown in more detail, allowing users to quickly identify issues without being overwhelmed by all raw data.
3Measurement precision
If log data is analyzed in detail to identify specific differences, then diagnostic precision is improved, but the complexity of log analysis increases
Solution Approach 1:
The patent adds a new dimension of analysis by comparing log patterns across different time windows or environments. It presents differences in terms of pattern frequency changes, new patterns introduced, or patterns that disappeared, providing diagnostic precision through temporal or contextual comparison rather than raw text analysis.
Solution Approach 2:
The patent changes the parameters of log analysis from raw text matching to pattern frequency and similarity metrics. By analyzing logs based on configurable parameters such as similarity thresholds, time windows, and pattern templates, the system provides detailed diagnostic capability while keeping the analysis approach flexible and manageable.
Data Source
AI summary
Analyzing log data. The method includes obtaining a first bucket of a log data. The first bucket of log data includes a plurality of log lines. The method further includes analyzing the first bucket of log data to identify different sets of similar log lines. The method further includes providing to a user in a user interface one or more summaries of the different sets of similar lines. The summary comprises at least one user selectable indicator representing differences in log lines in a set of similar log lines that when selected by a user in the user interface reveals specific differences in the log lines in the set of similar log lines.