LLM Question Generation With Cognitive Difficulty Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multiple-choice question generation systems, such as those using GPT-4, struggle to control cognitive difficulty levels, leading to low difficulty questions that fail to accurately evaluate learner understanding and model performance.
Innovation Solution
A question generation apparatus and system that includes an input unit, question generation unit, evaluation unit, and filtering unit to generate and select multiple-choice questions based on a user-defined cognitive difficulty level, using a large language model to process context data and summary information to create questions with desired difficulty levels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single-step procedure is used to generate multiple-choice questions, then the generation process is simple and fast, but the cognitive difficulty level cannot be controlled and the questions tend to be low difficulty
Solution Approach 1:
The single-step question generation process is divided into multiple steps: first generating candidate questions, then evaluating their cognitive difficulty levels, and finally selecting questions that meet the target difficulty threshold. This segmentation allows the system to maintain high generation speed while achieving precise control over cognitive difficulty levels through the intermediate evaluation and selection stages.
2Ease of manufacture
If low difficulty level questions are generated, then the generation process is easier and requires less computational resources, but the questions cannot accurately evaluate learner understanding
Solution Approach 1:
The system incorporates a feedback mechanism where generated questions are evaluated against predetermined cognitive difficulty level criteria, and the evaluation results feed back into the selection process. This feedback loop ensures that only questions meeting the target cognitive difficulty threshold are selected, thereby maintaining both generation ease and evaluation accuracy.
3Device complexity
If multiple-choice questions are generated without difficulty filtering, then the generation process is simple, but the distribution of difficulty levels is unbalanced and lacks high difficulty questions
Solution Approach 1:
The manual process of curating and filtering questions by difficulty level is replaced with an automated evaluation system that uses predetermined cognitive difficulty level criteria to objectively assess and filter generated questions. This substitution maintains process simplicity while achieving precise control over difficulty level distribution through automated mechanical evaluation rather than manual curation.
Data Source
AI summary
A question generation apparatus includes: an input unit that acquires context data including text information; a question generation unit that generates a first set of multiple-choice questions for the context data by processing the context data using a first large language model; an evaluation unit that determines difficulty level evaluation values indicating cognitive difficulty levels for the first set of multiple-choice questions by evaluating the first set of multiple-choice questions based on a predetermined cognitive difficulty level evaluation criterion; and a filtering unit that selects a first subset of multiple-choice questions of which the difficulty level evaluation values satisfy a predetermined cognitive difficulty level threshold from among the first set of multiple-choice questions, and outputs the selected first subset of multiple-choice questions.