AI Exam Item Generation With Expert Validation Workflow
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The process of manually or semi-automated generation of exam items is time-consuming, labor-intensive, and prone to errors, requiring significant human effort and attention to detail.
Innovation Solution
Utilizing artificial intelligence and machine learning (AI/ML) models to automate the generation of exam items, which are validated and stored in an item bank, with human intervention to ensure quality and relevance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual or semi-automated methods are used to generate exam items, then quality control and expert validation can be maintained, but the process becomes time-consuming and labor-intensive
Solution Approach 1:
The exam item generation process is segmented into distinct phases: AI/ML models handle the generation of multiple exam items from content, while subject matter experts focus on validating and approving the generated items. This segmentation allows parallel processing where the AI generates multiple items simultaneously while experts review them, significantly reducing the overall time required while maintaining quality control through expert validation.
Solution Approach 2:
The AI/ML model acts as an intermediary between the content material and the subject matter experts. Instead of experts manually creating exam items directly from content (a time-consuming process), the AI/ML model first processes the content and generates draft exam items, which are then presented to experts for validation. This intermediary step accelerates the process while preserving expert oversight.
2Reliability
If manual methods are used to generate exam items, then expert judgment can be applied throughout the process, but significant human effort and attention to detail are required
Solution Approach 1:
The AI/ML model performs the self-service of generating exam items from content material, including creating multiple items of different types (multiple choice, true/false, matching, sequencing) without requiring expert intervention at each step. This automates the labor-intensive generation process while the system maintains reliability through subsequent expert validation of the generated items.
Solution Approach 2:
The AI/ML model performs preliminary action by generating draft exam items, selecting appropriate distractors, and organizing items by topic and difficulty level before presenting them to subject matter experts. This preliminary generation reduces the effort required by experts, who only need to review and validate rather than create items from scratch, thereby improving productivity while maintaining expert judgment.
3Productivity
If semi-automated methods are used to generate exam items, then some efficiency can be improved, but the process remains error-prone and requires multiple approval steps
Solution Approach 1:
The system implements feedback loops where subject matter experts validate and provide feedback on the AI-generated exam items. The AI/ML model can incorporate this feedback to refine and improve subsequent item generation, reducing errors over time. This feedback mechanism ensures that while automation improves efficiency, reliability is maintained through continuous validation and improvement based on expert input.
Data Source
AI summary
Embodiments of the present disclosure relate to a method, a system, and a computer program product for generating knowledge assessment items for an assessment of candidates in an examination and populating the generated knowledge assessment items in an item bank, the knowledge assessment items including different item types, and the knowledge assessment items being generated using artificial intelligence and machine learning, and further the knowledge assessment items generated (created) by the AI/ML module are authenticated and vetted by a subject matter expert before storing or updating the knowledge assessment item(s) in the item bank.