Automated Distractor Generation via QA Semantic Criteria

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current methods for generating distractors for text-based multiple-choice test items lack efficiency in producing plausible, incorrect alternatives that match the semantic context of the test stem, often relying on manual effort or simplistic algorithms.

Innovation Solution

A system utilizing a question answering (QA) system to generate candidate answers, followed by feature extraction and semantic criteria analysis to select efficient distractors through machine learning techniques, ensuring the selected distractors are semantically relevant and contextually plausible.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of manufacture

If manual methods or simplistic algorithms are used to generate distractors, then the process is simple to implement, but the quality and semantic relevance of the generated distractors deteriorates

Engineering Contradiction:

Improveease of implementationVSAvoidquality of distractors

Core Design Contradiction:

Ease of manufactureVSManufacturing precision

Solution Approach 1:

The patent introduces a Question Answering (QA) system as an intermediary component between the test stem and the distractor generation process. The QA system retrieves relevant context and generates candidate answers based on semantic understanding, which then feed into the distractor selection process. This intermediary enables sophisticated semantic analysis without requiring the entire system to be manually designed, thus maintaining ease of implementation while improving distractor quality.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces manual distractor creation (mechanical human effort) with an automated system combining QA technology and machine learning. The system automatically extracts textual features, generates semantic criteria, and selects distractors based on learned patterns from training data, substituting the manual mechanical process with an intelligent automated system that achieves both quality and efficiency.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Manufacturing precision

If sophisticated semantic analysis is performed to generate high-quality distractors, then the quality of distractors improves, but the computational complexity and processing time increases

Engineering Contradiction:

Improvequality of distractorsVSAvoidsystem complexity

Core Design Contradiction:

Manufacturing precisionVSDevice complexity

Solution Approach 1:

The patent segments the distractor generation process into distinct modular components: (1) QA system for retrieving context and generating candidates, (2) textual feature extraction module, (3) semantic criteria generation, and (4) distractor selection based on learned patterns. Each module performs a specific function, making the complex system manageable and maintainable while achieving high-quality results through coordinated operation of specialized components.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs preliminary actions by pre-processing the test stem through the QA system to generate candidate answers and extract textual features before the actual distractor selection occurs. The machine learning model is trained in advance on relevant data to learn semantic criteria, so that during runtime, the system can quickly apply these pre-learned patterns to select distractors without performing complex analysis from scratch.

Inventive Principle:

Principle #10Preliminary action

3Productivity

If automated methods are used to generate distractors, then productivity and efficiency improve, but the semantic relevance and contextual plausibility of distractors may deteriorate

Engineering Contradiction:

Improvegeneration efficiencyVSAvoidsemantic relevance

Core Design Contradiction:

ProductivityVSManufacturing precision

Solution Approach 1:

The system incorporates feedback mechanisms where the machine learning model learns from training data consisting of test items and their appropriate distractors. The model receives feedback in the form of labeled examples showing which distractors are semantically relevant and contextually plausible, and adjusts its parameters to maximize semantic accuracy. This feedback loop enables the automated system to improve its semantic understanding over time while maintaining high productivity.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS10789552B2Question answering system-based generation of distractors using machine learning

Publication Date: 2020.09.29 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Generating distractors for text-based MCT items. An MCT item stem is received. The stem is transmitted to a QA system and a plurality of candidate answers related to the stem is received from the QA system. Incorrect answers in the plurality of candidate answers are identified. Textual features are extracted from the stem. A set of semantic criteria associated with the extracted textual features is generated. Based on the generated semantic criteria, a subset of the incorrect candidate answers is selected.