Mainframe Log Anomaly Detection Using Seasonal Autoregressive Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current anomaly detection methods for mainframe systems face challenges in modeling complex patterns and evolving threats, leading to inefficiencies in data correlation, alert fatigue, and integration complexity, while existing techniques struggle to predict future anomalies effectively.
Innovation Solution
A system and method utilizing a model that receives real-time log data, parses metrics, determines seasonality, and employs an autoregressive model trained with a grid search for parameter optimization, enabling proactive anomaly detection and response, including visualization and classification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If statistical methods are used for anomaly detection, then computational simplicity is maintained, but modeling capability for complex patterns and evolving threats is insufficient
Solution Approach 1:
The patent implements a hybrid anomaly detection system that dynamically selects between statistical methods and machine learning models based on the complexity and nature of the data being analyzed. This allows the system to adapt its computational approach to match the specific requirements of different anomaly detection scenarios, achieving both simplicity when appropriate and advanced modeling capability when needed.
Solution Approach 2:
The system applies different detection methodologies to different data sources and anomaly types locally. Statistical methods are used for straightforward metrics where they suffice, while machine learning models are deployed for complex patterns and evolving threats, optimizing computational resources while maintaining high detection accuracy across diverse scenarios.
2Adaptability or versatility
If machine learning methods are used for anomaly detection, then modeling capability is improved, but computational weight increases
Solution Approach 1:
The patent implements a tiered anomaly detection approach where machine learning models are applied selectively to specific data sources and anomaly types that require advanced modeling, rather than applying them universally. This partial application reduces overall computational weight while maintaining enhanced modeling capability where it provides the most value.
Solution Approach 2:
The system segments the anomaly detection process into multiple stages: initial filtering using lightweight statistical methods, followed by more intensive machine learning analysis only for suspicious or complex patterns. This segmentation reduces the total computational burden by limiting heavy ML processing to cases where it is most beneficial.
3Ease of manufacture
If rule-based methods are used for anomaly detection, then ease of implementation is maintained, but adaptability to complex patterns and evolving threats is limited
Solution Approach 1:
The patent introduces machine learning models as an intermediary layer between simple rule-based detection and complex anomaly patterns. The ML models learn evolving threat patterns from data and generate adaptive rules that enhance the basic rule-based system, maintaining ease of implementation while significantly improving adaptability to complex and evolving threats.
Solution Approach 2:
The system creates a composite anomaly detection framework that combines rule-based methods with machine learning capabilities. This hybrid approach integrates the simplicity and interpretability of rules with the adaptive pattern recognition of ML, achieving both ease of implementation and high adaptability to evolving threats.
4Measurement precision
If deep learning techniques are used for log-based anomaly detection, then detection accuracy for unexpected system behavior is improved, but integration complexity with legacy mainframe tools increases
Solution Approach 1:
The patent extracts and isolates the deep learning anomaly detection component as a separate modular service that interfaces with legacy mainframe tools through standardized APIs. This extraction allows high detection accuracy to be achieved while minimizing integration complexity, as the complex ML model remains encapsulated and interacts with legacy systems only through well-defined interfaces.
Solution Approach 2:
The system introduces an intermediary layer that translates between legacy mainframe data formats and the deep learning model's requirements. This intermediary handles data preprocessing, format conversion, and result interpretation, enabling accurate deep learning-based anomaly detection without requiring direct complex integration between modern ML tools and legacy mainframe infrastructure.
Data Source
AI summary
Disclosed are methods and techniques of detecting network anomalies and responding to the anomalies once detected. The methods, for example, include receiving, by a model executed by a processor, real-time log data of an operating network; parsing, by the model executed by the processor, the log data to identify one or more metrics; determining, by the model executed by the processor, a seasonality of the one or more metrics; determining whether the model should use an autoregressive model if seasonality is detected; and on determining that an autoregressive model should be used, training a model based on determining a grid search for a parameter based on an Akaike information criterion.