Network Anomaly Detection via Micro-Batch Hint Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Continuous monitoring of key performance indicators in computing systems for anomaly detection adds stress and increases costs, leading to delayed detection of performance issues which can cause damage to the system.
Innovation Solution
A notification system that generates alerts based on anomalies in key performance indicators exceeding threshold parameters for a predetermined duration, using a micro-batch processing approach to filter and process hints, reducing unnecessary queries and improving efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If continuous monitoring of key performance indicators is implemented, then anomaly detection capability is improved, but system stress and operational costs increase
Solution Approach 1:
The system implements periodic monitoring at scheduled intervals rather than continuous monitoring. The notification system evaluates performance indicators at discrete time points, reducing the constant computational overhead and system stress while still maintaining effective anomaly detection capability through regular periodic assessments.
Solution Approach 2:
The system pre-configures threshold parameters and evaluation criteria before monitoring begins. By establishing notification rules and performance thresholds in advance, the system avoids real-time complex decision-making, reducing computational stress during operation while maintaining reliable anomaly detection through pre-planned evaluation logic.
2Loss of time
If continuous monitoring is performed, then detection timeliness is improved, but resource consumption and costs increase
Solution Approach 1:
The notification system performs periodic evaluations at optimized intervals that balance detection timeliness with resource conservation. By monitoring at strategically chosen time points rather than continuously, the system achieves timely anomaly detection while significantly reducing computational resource consumption and operational costs.
Solution Approach 2:
The system dynamically adjusts monitoring parameters such as evaluation intervals and threshold sensitivity based on system conditions. This allows the system to maintain timely detection capability while optimizing resource usage by adapting monitoring intensity to actual system needs rather than maintaining constant high-resource monitoring.
3Productivity
If performance monitoring adds stress to core infrastructure, then monitoring coverage is improved, but system reliability deteriorates
Solution Approach 1:
The notification system uses periodic monitoring that reduces the continuous computational load on core infrastructure. By evaluating performance indicators at discrete intervals rather than continuously, the system maintains comprehensive monitoring coverage while minimizing the stress imposed on system resources, thereby preserving overall system stability and reliability.
Solution Approach 2:
The system creates a notification instance that handles monitoring and evaluation separately from core system operations. This notification copy processes performance data independently, allowing comprehensive monitoring coverage without adding significant stress to the core infrastructure, thus maintaining system reliability.
Data Source
AI summary
Systems and methods are provided for detecting an anomaly in a performance indicator associated with a network device. One method may include: generating a plurality of hints; identifying a subset of hints from the plurality of hints, where each hint in the subset is associated with an open alert and has a same severity level as the associated open alert; querying in bulk from a time series data store a set of historical data; processing the set of historical data and the remaining hints that are not in the subset of hints to form processed data; determining whether the processed data exceeds a threshold parameter associated with the performance indicator; and performing an alert action in bulk based on the processed data exceeding the threshold parameter of the performance indicator.