Network Assurance Anomaly Ranker Clustering and Reliability Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Network assurance systems face challenges in accurately identifying and filtering feedback from anomaly rankers, leading to potential biases in anomaly detection models due to 'bad' rankers providing inappropriate or inaccurate feedback, which can impinge on the effectiveness of anomaly detection.
Innovation Solution
A network assurance service clusters anomaly rankers based on their feedback to assign reliability scores, selectively providing anomaly detection alerts to reliable rankers, and dynamically recruiting additional rankers to ensure accurate anomaly classification and filtering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If feedback from all anomaly rankers is used for model training, then the quantity of feedback data increases, but the reliability of anomaly detection decreases due to bad rankers providing inaccurate feedback
Solution Approach 1:
The system segments the feedback data by clustering rankers into groups based on the similarity of their feedback patterns. This segmentation allows the system to identify and separate reliable rankers from bad rankers, processing feedback from each segment differently to maintain overall system reliability while utilizing available data quantity.
Solution Approach 2:
The system applies local quality by assigning different reliability scores to different rankers based on their individual feedback quality. Instead of treating all feedback uniformly, the system evaluates and weights feedback from each ranker locally, ensuring that high-quality feedback has greater influence on model training while filtering out problematic feedback.
2Reliability
If clustering and reliability scoring of rankers is implemented, then the reliability of anomaly detection improves, but the device complexity increases
Solution Approach 1:
The system implements self-service by enabling rankers to automatically evaluate each other through their feedback patterns. The clustering and reliability scoring are performed automatically based on feedback similarity, without requiring manual intervention to identify or manage bad rankers. This automated self-evaluation reduces operational complexity while maintaining reliability.
Solution Approach 2:
The system uses feedback from rankers not only for anomaly detection but also for evaluating the rankers themselves. This feedback-on-feedback mechanism creates a self-regulating system where the quality of feedback determines the weight given to each ranker, automatically adjusting system behavior based on observed performance without complex external management.
3Reliability
If a subset of reliable rankers is selected for feedback, then the reliability of anomaly detection improves, but the quantity of feedback data decreases
Solution Approach 1:
The system applies partial action by selectively incorporating feedback from only the most reliable rankers for each anomaly detection task. Rather than requiring feedback from all rankers, the system identifies and utilizes the sufficient subset of high-quality feedback needed to maintain detection reliability, avoiding the diminishing returns of incorporating additional low-quality feedback.
Solution Approach 2:
The system dynamically changes parameters by adjusting which rankers are selected for feedback based on current reliability scores and clustering results. The composition of the reliable ranker subset is not fixed but adapts based on observed feedback quality, allowing the system to optimize the balance between feedback quantity and quality for different anomaly detection scenarios.
Data Source
AI summary
In one embodiment, a network assurance service that monitors a network detects anomalies in the network by applying one or more machine learning models to telemetry data from the network. The network assurance service ranks feedback from a plurality of anomaly rankers regarding relevancy or criticality of the detected anomalies. The network assurance service clusters the plurality of anomaly rankers into clusters of similar rankers, based on the received ranking feedback. The network assurance service uses the clusters of similar rankers to assign reliability scores to each of the anomaly rankers. The network assurance service selects, based on the reliability scores, a subset of the plurality of anomaly rankers to receive an anomaly detection alert regarding a particular detected anomaly to be ranked. The network assurance service provides the anomaly detection alert to the selected subset of the plurality of anomaly rankers for ranking.