Ensemble Residual Clustering for Anomalous QoE Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models for quality-of-experience (QoE) assessment in applications like voice and video are hindered by noisy user feedback, which can lead to incorrect performance estimates due to adverse or unreliable user behaviors, especially when small datasets with inconsistent or adversarial feedback are involved.
Innovation Solution
An ensemble-based detection method that computes residual error vectors from multiple models, clusters these vectors to identify outlier feedback, and performs mitigation actions to address anomalous feedback, thereby improving the reliability of QoE modeling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If user feedback is directly collected from users for QoE assessment, then the dataset size increases and more user experiences are captured, but the reliability of the feedback decreases due to potentially adverse or unreliable user behaviors
Solution Approach 1:
The patent introduces an intermediary system that acts as a mediator between users and the QoE assessment model. This system collects user feedback through chatbots or feedback mechanisms, processes it through anomaly detection algorithms, and filters out unreliable feedback before it reaches the training data. The intermediary layer enables the system to maintain both large dataset size and high reliability by filtering adversarial or erroneous feedback while preserving legitimate user experiences.
2Adaptability or versatility
If machine learning models process noisy user feedback, then the model can handle some level of noise, but too much label noise impacts model performance and leads to incorrect QoE figures
Solution Approach 1:
The patent applies preliminary action by performing anomaly detection and filtering of noisy feedback before the data reaches the machine learning training process. The system pre-processes user feedback through residual error analysis and outlier detection algorithms, removing adversarial or erroneous labels in advance. This preliminary cleaning action enables the model to train on high-quality data, maintaining both adaptability to handle various feedback types and precision in QoE assessment.
3Reliability
If multiple machine learning models are used to compute predictions, then the system can detect outliers more effectively, but the computational complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the anomaly detection task into distinct stages: multiple machine learning models generate individual predictions, each model's residual errors are computed separately, and then these residuals are aggregated and analyzed to identify outliers. This segmented approach allows the system to achieve high reliability through ensemble methods while managing computational complexity through structured processing and efficient residual aggregation techniques.
Data Source
AI summary
In one embodiment, a method herein comprises: computing a plurality of predictions based on session telemetry data for an application, each of the plurality of predictions computed by inputting a particular input of the session telemetry data into a respective machine learning model of a plurality of machine learning models; computing, for each respective feedback of a plurality of collected feedbacks correlated with the session telemetry data, a residual error vector that quantifies a level of mis-prediction of each respective machine learning model as compared to each respective feedback; clustering each residual error vector into one or more clusters based on similarity to thereby identify one or more outlier residual error vectors that are not within any of the one or more clusters; and performing one or more mitigation actions responsive to the one or more outlier residual error vectors.