QoE Feedback Suitability Screening for ML Training Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for estimating user quality of experience (QoE) in online applications rely on network service level agreement (SLA) thresholds, which fail to account for complex impairments and user behaviors, leading to inaccurate QoE models due to unsuitable user feedback collection.
Innovation Solution
A system that collects telemetry data and user satisfaction ratings to assess suitability for training a machine learning model, preventing unsuitable interactions from training, and using cross-layer telemetry and user feedback to predict QoE metrics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If user feedback is collected without suitability assessment, then the quantity of training data increases, but the accuracy of QoE model deteriorates due to inclusion of unsuitable interactions
Solution Approach 1:
The patent segments the user feedback collection process by introducing a suitability assessment mechanism that divides feedback into suitable and unsuitable categories. The suitability estimator evaluates each interaction based on multiple criteria (user reliability, interaction quality, telemetry completeness) and only selects suitable interactions for model training, thereby resolving the contradiction between quantity and accuracy of training data.
2Device complexity
If SLA thresholds are used as proxy for QoE, then the complexity of QoE measurement is reduced, but the accuracy of QoE estimation deteriorates due to inability to capture complex impairments and user behaviors
Solution Approach 1:
The patent introduces an intermediary machine learning model that bridges the gap between simple SLA thresholds and accurate QoE estimation. The model takes multiple inputs including SLA metrics, telemetry data, and user feedback, and produces QoE predictions that capture complex impairments and user behaviors while maintaining measurement feasibility through automated suitability assessment.
3Adaptability or versatility
If all user interactions are used for model training, then the diversity of training scenarios increases, but the reliability of training data deteriorates due to inclusion of noisy or biased feedback
Solution Approach 1:
The patent applies local quality by evaluating each user interaction individually through the suitability estimator, which assigns different reliability weights to different interactions based on user-specific factors (feedback history, device characteristics) and interaction-specific factors (telemetry completeness, impairment severity). This allows the system to maintain diversity in training scenarios while filtering out unreliable data points through localized quality assessment.
Data Source
AI summary
In one embodiment, a device obtains telemetry data regarding an interaction between a user and an online application. The device also obtains a satisfaction rating provided by the user regarding the interaction. The device makes, based on the telemetry data and the satisfaction rating, a suitability assessment as to how suitable the interaction is for training a machine learning model to predict a quality of experience metric for the online application. The device prevents the telemetry data and satisfaction rating from being used to train the machine learning model, when the suitability assessment indicates that the interaction is unsuitable for training the machine learning model.