Online Evaluation Server Algorithm for Scoring Bias Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing online evaluation methods impose a heavy burden on evaluators by requiring them to assess both evaluation targets and each other's connoisseurship, leading to inaccuracies in measuring connoisseurship independently of idea creation ability and potential biases in scoring.
Innovation Solution
A method and server system that analyze the degree of strictness of each evaluator's evaluations, calculate an evaluation ability score based on the closeness between provisional and corrected evaluations, and weight evaluations accordingly to provide a reliable and independent assessment of both evaluation targets and evaluators' connoisseurship without overburdening them.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If evaluators are required to assess both evaluation targets and each other's connoisseurship, then evaluation objectivity is improved, but evaluator burden increases
Solution Approach 1:
The patent extracts the connoisseurship assessment function from the evaluator's manual evaluation task and implements it through automated algorithmic analysis. The system automatically calculates evaluation ability scores by analyzing evaluation patterns, thereby removing the burden of manual mutual assessment while preserving the objective evaluation function.
Solution Approach 2:
The evaluation system performs self-assessment of evaluator connoisseurship through automated analysis of their evaluation behaviors. The system uses the evaluators' own evaluation data to calculate their evaluation ability scores, allowing the system to serve itself in assessing evaluator quality without requiring additional human input.
2Productivity
If connoisseurship is ranked based on idea creation ability, then evaluation efficiency is improved, but measurement precision deteriorates
Solution Approach 1:
The patent segments the evaluation ability into separate dimensions: idea creation ability and connoisseurship as evaluator. By analyzing evaluation data independently from creation data, the system can measure connoisseurship precision without being confounded by creation ability variations, thereby improving measurement accuracy while maintaining evaluation efficiency.
3Reliability
If evaluators with high connoisseurship are weighted more heavily, then evaluation reliability is improved, but bias in scoring increases
Solution Approach 1:
The patent changes the parameter for evaluator weighting from binary (high/low connoisseurship) to continuous (evaluation ability score based on statistical analysis of evaluation patterns). This allows for more nuanced weighting that reflects actual evaluation quality while reducing the impact of arbitrary thresholds and associated biases.
Data Source
AI summary
A method for online evaluation includes analyzing, by a server, a degree of strictness of each evaluator, calculating, by the server, an evaluation ability score of each evaluator based on closeness between a provisional score of an evaluation target by all the evaluators and an evaluation of the evaluation target by each evaluator, and calculating, by the server, a final score of the evaluation target in consideration of the evaluation ability score of each evaluator.