Log Sampling for Microservice Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In microservice systems, the massive and disorganized logs generated by distributed microservices pose challenges for log parsing and anomaly detection, requiring excessive processing resources and time due to their chaotic nature, with issues like missing, duplicated, or disordered logs, and noise introduced during collection and preprocessing.
Innovation Solution
A method and system for log data sampling that uses multiple sampling methods to determine a diverse, representative, and unbiased sample of logs, with feedback-based adjustments to improve the quality of training data for log parsing and anomaly detection models, reducing the need for extensive processing resources and time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all logs are processed for training, then model quality is improved, but processing resources and time increase excessively
Solution Approach 1:
The patent extracts only the most informative and representative log entries from the massive log corpus using multiple sampling methods (random sampling, stratified sampling, uncertainty sampling). This extraction process selects a small subset of high-quality training samples that capture the essential patterns and anomalies, eliminating the need to process all logs while maintaining model training effectiveness
Solution Approach 2:
The patent changes the parameter of sample selection from exhaustive inclusion to selective sampling by introducing multiple sampling strategies with different selection criteria. By adjusting sampling parameters (sample size, selection probability, diversity metrics), the system optimizes the balance between training data quality and processing efficiency, achieving high model quality with reduced processing time
2Measurement precision
If all logs are processed for training, then model quality is improved, but processing resources increase excessively
Solution Approach 1:
The system extracts a curated subset of training logs using multiple sampling methods that prioritize information density and representativeness. By selecting only the most valuable log entries through techniques like uncertainty sampling (which identifies logs that maximize model uncertainty reduction) and stratified sampling (which ensures coverage across different log types), the system achieves effective model training with minimal processing resources
Solution Approach 2:
The patent applies partial action by processing only a carefully selected portion of logs rather than the complete set. The sampling mechanisms determine an optimal subset size that provides sufficient training signal without the excessive resource consumption of full-log processing, achieving diminishing returns optimization where additional logs beyond the sampled subset provide minimal marginal improvement
3Loss of time
If sampling is applied to reduce processing, then processing time is reduced, but log quality and representativeness may deteriorate
Solution Approach 1:
The patent segments the log sampling process into multiple distinct sampling methods (random sampling, stratified sampling, uncertainty sampling) that each target different aspects of log diversity and representativeness. By combining these segmented sampling approaches, the system ensures comprehensive coverage of log patterns while maintaining processing efficiency, preventing any single sampling method from introducing quality degradation
Solution Approach 2:
The system incorporates feedback mechanisms where sampling results are evaluated for representativeness and quality metrics. The sampling process adjusts based on feedback about log distribution coverage, anomaly detection performance, and model training effectiveness, ensuring that the reduced sample set maintains high quality and reliability comparable to or better than exhaustive processing
Data Source
AI summary
A method for log data sampling is disclosed. The method includes receiving logs of a computer system. A log comprises information regarding an operation of the computer system. The method also includes determining a sample of the logs by applying a set of sampling methods to the logs. The method further includes providing the sample of the logs as an input to an anomaly detection model for the computer system. The anomaly detection model identifies a fault in the operation of the computer system based on the input.