Log Alerting Rules via Pattern-Based Offline Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing complexity and volume of cloud platform logs pose challenges for effective problem diagnosis, as traditional statistical or machine learning techniques struggle to analyze unstructured textual log messages and detect anomalies in cross-component interactions.
Innovation Solution
A method and system that utilize offline analytics based on machine learning, statistical analysis, and deep learning to identify abnormal and normal patterns from history logs, automatically generate alerting rules, and augment online log messages to enforce these rules through an alerting engine.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If traditional statistical or machine learning techniques are used to analyze unstructured textual log messages, then the analysis process becomes simpler, but the accuracy of anomaly detection deteriorates due to the complexity of cross-component interactions
Solution Approach 1:
The patent introduces log templates as an intermediary layer between raw unstructured log messages and analysis algorithms. These templates serve as mediators that structure the unstructured data, enabling traditional machine learning techniques to work effectively while maintaining high detection accuracy for complex cross-component interactions
Solution Approach 2:
The patent segments the analysis process into distinct stages: log message parsing, template matching, pattern identification, and anomaly detection. This segmentation allows each stage to specialize in specific tasks, improving overall accuracy while keeping individual processing steps manageable and simple
2Measurement precision
If manual creation of alerting rules is performed, then the rules can be highly accurate and specific, but the time and resources required to maintain and update rules increase significantly
Solution Approach 1:
The system implements self-service by automatically generating alerting rules through machine learning models that analyze historical log data and identified patterns. The system autonomously creates, updates, and refines rules without requiring manual intervention, thereby maintaining high accuracy while eliminating the time cost of manual rule management
Solution Approach 2:
The patent incorporates feedback mechanisms where the system continuously monitors alert performance and uses this information to automatically refine and update alerting rules. This closed-loop feedback ensures rules remain accurate and adaptive to changing system conditions without manual maintenance
3Reliability
If comprehensive log analysis is performed on all log messages, then the detection coverage is improved, but the processing time and computational resources increase dramatically
Solution Approach 1:
The patent applies partial action by focusing computational resources on analyzing only those log messages that match identified abnormal patterns or deviate from normal behavior. Instead of comprehensively analyzing every log message, the system selectively processes suspicious or anomalous entries, maintaining high detection coverage while significantly reducing processing time and resource consumption
Data Source
AI summary
A computer-implemented method is presented for automatically generating alerting rules. The method includes identifying, via offline analytics, abnormal patterns and normal patterns from history logs based on machine learning, statistical analysis and deep learning, the history logs stored in a history log database, automatically generating the alerting rules based on the identified abnormal and normal patterns, and transmitting the alerting rules to an alerting engine for evaluation. The method further includes receiving a plurality of online log messages from a plurality of computing devices connected to a network, augmenting the plurality of online log messages, and extracting information from the plurality of augmented online log messages to be provided to the alerting engine, the alerting engine configured to approve and enforce the alerting rules automatically generated by the offline analytics processing.