Exam Evaluation Using Answer Time Guess Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional multiple choice exams cannot determine if a test taker is guessing or fully understanding the material, as they lack the ability to analyze answer selection reasoning and provide feedback on incorrect answers.
Innovation Solution
A system that uses a processor to analyze answer time and confidence scores to identify potential guessing, providing additional questions related to the original question to evaluate the test taker's understanding, combining objective and subjective criteria for evaluation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple choice exams are used, then grading speed and objectivity are improved, but the ability to analyze test taker understanding and detect guessing deteriorates
Solution Approach 1:
The evaluation process is segmented into multiple stages: initial multiple choice answer collection, answer time analysis, confidence score assessment, and conditional follow-up questioning. This segmentation allows the system to maintain the efficiency of multiple choice grading while selectively adding depth of analysis only where needed, thus resolving the contradiction between speed and understanding analysis.
Solution Approach 2:
Instead of requiring detailed responses to all questions (excessive action), the system applies partial action by only requesting additional questions for specific items where guessing is suspected. This is determined through answer time comparison against thresholds and confidence score analysis, thereby maintaining overall grading efficiency while obtaining deeper understanding data when necessary.
2Loss of information
If detailed response questions are used, then analysis of test taker understanding is improved, but grading time and grader burden increase
Solution Approach 1:
The system implements partial action by administering detailed response questions only to a subset of test items where guessing is detected through answer time analysis and confidence scoring. This selective approach ensures that understanding analysis is obtained where needed without imposing the time burden of detailed responses across the entire exam.
Solution Approach 2:
The system performs automated analysis of answer times and confidence scores to identify potential guessing, eliminating the need for graders to manually determine point values for each essay component. This self-service approach to detecting guessing reduces grader burden while maintaining the ability to analyze understanding.
3Measurement precision
If answer time analysis is performed to detect guessing, then accuracy in identifying guessing behavior is improved, but system complexity increases
Solution Approach 1:
The system changes parameters by analyzing answer time duration and confidence score values to detect guessing behavior. By establishing threshold values for answer times and confidence scores, the system achieves accurate guessing detection through relatively simple parameter comparisons rather than complex analysis, thus improving measurement precision while limiting complexity growth.
Solution Approach 2:
The system uses feedback from answer time measurements and confidence score inputs to dynamically adjust evaluation decisions. When answer time falls outside expected ranges or confidence scores indicate uncertainty, the system triggers additional evaluation steps, creating a feedback loop that improves guessing detection accuracy without requiring complex predetermined rules for all scenarios.
Data Source
AI summary
One embodiment provides a method, including: obtaining a first question on a multiple choice exam comprising a plurality of possible answers; receiving his or her answer selection of one of the plurality of possible answers; identifying an answer time corresponding to the amount of time between presentation of the question and receiving the answer selection; determining if the test taker is guessing the answer to the question, wherein the determining comprises (i) comparing the answer time to a predetermined threshold answer time and (ii) ascertaining that the test taker is guessing when the answer time is outside the threshold answer time; providing to the test taker one or more additional questions, wherein the one or more additional questions are related to the first question; and evaluating the test taker using at least one of: the answer selection and any test taker response to the one or more additional questions.