Response Evaluation System Using Categorized Similarity Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional computer-assisted assessment (CAA) techniques for evaluating multiple choice questions (MCQs) and open-ended responses are unreliable due to guesswork and variability in responses, while CAA for free-text responses lacks accuracy in considering linguistic, subjective, and topical variations.
Innovation Solution
A method and system that extracts questions and responses from electronic documents, determines scores based on similarity measures categorized by question type, and renders evaluations on a user interface, using a document processor and processor to handle metadata and similarity calculations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If multiple choice questions are used for assessment, then response evaluation becomes easier and more standardized, but reliability decreases due to guesswork and limited ability to measure acquired knowledge
Solution Approach 1:
The patent segments the assessment into multiple dimensions: factual knowledge (MCQs), conceptual understanding (open-ended), and applied knowledge (scenario-based questions). This segmentation allows each question type to evaluate different aspects of student knowledge, thereby improving overall assessment reliability while maintaining ease of evaluation through structured categories.
Solution Approach 2:
The system changes the parameters of evaluation by introducing multiple scoring dimensions (factual accuracy, conceptual understanding, reasoning quality) rather than relying on a single correct answer parameter. This allows for more nuanced and reliable assessment while maintaining systematic evaluation processes.
2Reliability
If open-ended questions are used for assessment, then ability to measure integrated knowledge and reasoning improves, but response evaluation becomes more difficult due to linguistic variations and subjectivity
Solution Approach 1:
The patent applies different evaluation qualities to different parts of the response: factual accuracy is evaluated with strict criteria, while conceptual understanding and reasoning are evaluated with more flexible, context-dependent criteria. This local differentiation allows for reliable assessment of knowledge integration while accommodating linguistic variations in open-ended responses.
Solution Approach 2:
The system introduces an intermediary evaluation layer that translates diverse open-ended responses into standardized scoring categories. This intermediary process handles linguistic variations and subjectivity by mapping different表达方式 to common evaluation dimensions, making the assessment process more manageable while preserving the ability to measure integrated knowledge.
3Ease of operation
If conventional CAA techniques are used for free-text responses, then evaluation process is simplified, but accuracy decreases due to failure to consider linguistic, subjective, and topical variations
Solution Approach 1:
The patent implements a dynamic evaluation system that adapts the scoring criteria based on the question type, topic domain, and response characteristics. Rather than applying a static scoring rule to all free-text responses, the system dynamically adjusts evaluation parameters to account for linguistic variations, subjectivity, and topical differences, thereby improving accuracy while maintaining process simplicity through automated adaptation.
Data Source
AI summary
A method and a system for response evaluation of users from electronic documents are disclosed. In an embodiment, one or more questions and a first response pertaining to each of the one or more questions are extracted from one or more first electronic documents. Further, a second response pertaining to each of the one or more extracted questions and metadata are extracted from one or more second electronic documents. For the second response pertaining to each of the one or more extracted questions, a score is determined based on one or more similarity measures that correspond to a category of each of the one or more extracted questions. Thereafter, the response evaluation is rendered on a user interface displayed on a display screen. The response evaluation comprises at least the determined score for the second response pertaining to each of the one or more extracted questions.