Comparison Pair Generation for Sensory Rating Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge in developing AI models for regression tasks, such as assessing the pleasantness of sounds, lies in the need for a large number of human-rated samples due to subjective ratings and the inefficiency of data collection methods, which limits the scalability and accuracy of these models.
Innovation Solution
A method for generating and distributing comparison pairs to experiment participants based on a metric that estimates the expected rating, ensuring each sample is equally compared and rated across subgroups, reducing the effort and cost of data collection while increasing accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all possible comparison pairs are presented to experiment participants for mutual comparison, then the completeness of rating data is improved, but the number of comparisons increases quadratically making the process extremely time-consuming and impractical
Solution Approach 1:
The patent segments the complete set of comparison pairs into multiple subsets, where each subset contains a manageable number of comparisons. Experiment participants are assigned different subsets, dividing the overall task into smaller, parallelizable units that can be completed in reasonable time while collectively covering the necessary comparison space.
Solution Approach 2:
The patent performs preliminary selection of comparison pairs using a metric that estimates expected rating based on quantitative sample characteristics. This pre-filtering identifies the most informative comparisons before presenting them to participants, avoiding the need to present all possible pairs while ensuring data quality.
2Reliability
If each comparison is rated by many participants to obtain stable results, then the reliability of ratings is improved, but the total number of comparisons required increases making the process unsustainable
Solution Approach 1:
The patent applies a metric based on quantitative sample characteristics to pre-estimate expected ratings and identify promising comparison pairs. This preliminary action ensures that each participant's time is used efficiently on comparisons most likely to yield informative results, reducing the total number of comparisons needed while maintaining reliability.
Solution Approach 2:
The patent changes the approach from increasing the number of participants per comparison to optimizing the selection of comparison pairs themselves. By adjusting the metric parameters and selection criteria, the system achieves reliable results with fewer total comparisons, improving scalability without sacrificing data quality.
3Measurement precision
If a high number of experiment participants are surveyed to obtain statistically stable results, then the accuracy of overall ratings is improved, but the cost and complexity of organizing and managing the experiment increases
Solution Approach 1:
The patent segments both the comparison pairs and participant assignments into organized subsets. This structured segmentation simplifies experiment management by creating manageable units that can be independently assigned, tracked, and processed, reducing the administrative burden while maintaining statistical accuracy.
Solution Approach 2:
The patent performs preliminary metric-based selection and organization of comparison pairs before participant assignment. This pre-processing reduces the complexity of experiment management by establishing a clear, data-driven framework for participant tasks, making coordination and quality control more straightforward.
Data Source
AI summary
A method for generating comparison pairs and distributing them to individual comparison requests for experiment participants for subjective individual ratings of pair comparisons. The method includes: providing a data set comprising sensory samples; identifying potentially promising comparison samples for each sample based on a metric, wherein the metric is based on a relationship between a possible estimation of the rating by experiment participants and at least one quantitative characteristic of the sample; distributing a predetermined number of overall comparisons to the potentially promising comparison pairs; generating the comparison requests for experiment participants by assigning the comparison pairs to the comparison requests, taking into account the following conditions to the effect that each sample is not provided more frequently than once in a comparison request, each sample is equally often provided for predetermined subgroups of the entirety of all experiment participants.