Subjective Evaluation Qualification via Self-Comparison Score Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Subjective evaluation methods for media quality, such as audio and video, face challenges in ensuring reliable and consistent scoring from subjects, as existing techniques fail to effectively filter out unreliable or inconsistent human opinions, leading to inaccurate media quality assessments.
Innovation Solution
The implementation of a qualification test using self-comparison scores, where processing circuitry applies rules to determine if scores fall within specific ranges, ensuring that only qualified subjects' scores are used for further data analysis, thereby enhancing score reliability and excluding outliers and inconsistent responders.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If subjective evaluation scores from multiple subjects are collected for media quality assessment, then the quantity of evaluation data increases, but the reliability and consistency of scores deteriorate due to inclusion of unreliable or inconsistent human opinions
Solution Approach 1:
The patent extracts and removes unreliable scores from the evaluation dataset by applying qualification rules based on self-comparison tests. Subjects who fail to meet the predetermined criteria (e.g., inconsistent self-comparison scores) have their evaluation data extracted and excluded from further analysis, thereby maintaining high reliability while preserving sufficient quantity of valid evaluation data
Solution Approach 2:
The patent implements a feedback mechanism where subjects perform self-comparison tests and their results are used to determine qualification status. The system provides feedback by comparing self-comparison scores against predetermined ranges, and this feedback loop ensures that only subjects demonstrating consistent and reliable evaluation patterns contribute to the final media quality assessment
2Loss of information
If all subject scores are included in media quality assessment, then the completeness of evaluation data is maintained, but the accuracy of assessment deteriorates due to inclusion of outlier scores and inconsistent responders
Solution Approach 1:
The patent applies preliminary qualification tests to subjects before their evaluation data is fully utilized. By conducting self-comparison tests in advance and establishing predetermined acceptance criteria, the system performs preliminary filtering to ensure that only qualified subjects' scores are included in the final assessment, thereby maintaining accuracy without sacrificing the completeness of valid evaluation data
Solution Approach 2:
The patent changes the parameters used for evaluating subject qualification by introducing self-comparison score ranges as additional criteria. By modifying the qualification parameters to include consistency checks across multiple tests, the system ensures that only subjects meeting elevated precision standards contribute to the assessment, thereby improving measurement precision while maintaining completeness of qualified data
Data Source
AI summary
Aspects of the disclosure provide methods and apparatuses for subjective evaluation. In some examples, processing circuitry receives scores graded by a subject to a media presentation. The scores by the subject includes a plurality of self comparison scores that are graded to self comparison tests in the media presentation. The processing circuitry applies a first rule and a second rule to the plurality of self comparison scores. The first rule requires a first subset of the plurality of self comparison scores in a first range. The second rule requires a second subset of the plurality of self comparison scores in a second range to limit at least an outlier to the first rule according to the second range. The processing circuitry determines that the scores by the subject are qualified for the subjective evaluation in response to the first rule and the second rule being satisfied.