Log Templatization Using Representative Sampling for Edge Sites
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In complex computing environments, manual debugging and remediation of faults are infeasible within a reasonable timeframe, and existing log templatization methods are resource-intensive and impractical for edge environments, necessitating a more efficient and resource-conserving approach.
Innovation Solution
A system identifies a representative set of log lines using incremental sampling and clustering techniques to generate log templates, reducing the dataset size while maintaining template accuracy, and applies these templates across edge sites without requiring full dataset processing at each site.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full dataset processing is used for log templatization, then template accuracy is improved, but computing resource consumption increases
Solution Approach 1:
The patent divides the log processing task into two stages: (1) template generation phase where a representative subset of log lines is processed to create templates, and (2) template matching phase where the generated templates are applied to the full log dataset. This segmentation allows accurate templates to be created with minimal resource consumption during the generation phase, while the resource-intensive matching phase uses pre-generated templates rather than processing raw logs from scratch.
Solution Approach 2:
The patent applies partial action by processing only a representative subset of log lines during the template generation phase rather than the entire dataset. The system selects and processes a carefully chosen sample that captures the essential log patterns, generating accurate templates with fraction of the computational resources that would be required to process the complete log dataset.
2Reliability
If log templatization is applied across multiple edge sites, then fault detection capability is improved, but computational overhead increases
Solution Approach 1:
The patent performs preliminary action by pre-generating log templates during an offline phase using a representative subset of logs. These pre-generated templates are then distributed to edge sites for deployment. This preliminary template generation eliminates the need for each edge site to independently process full log datasets, significantly reducing the computational overhead at deployed sites while maintaining accurate fault detection capability.
Solution Approach 2:
The patent creates copies of the generated log templates and distributes them across multiple edge sites. Instead of each site independently generating templates from their local logs, the system creates a master set of templates from a representative sample and replicates these templates to all edge sites, reducing computational overhead at each site while maintaining consistent fault detection capability across the distributed system.
Data Source
AI summary
In some implementations, a computing device may obtain a set of log lines associated with operations performed at a computing device. The computing device may select a first sample of the set of log lines. The computing device may identify a first set of log templates from the first sample, the first set of log templates having a first quantity of log templates. The computing device may select a second sample of the set of log lines. The computing device may identify a second set of log templates from the second sample, the second set of log templates having a second quantity of log templates. The computing device may apply the first set of log templates to the set of log lines for log templatization based at least in part on the first quantity of log templates being equal to the second quantity of log templates.