NLP-Based Scoring Quality Assurance for Constructed-Response Tests
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Scoring constructed-response tests, such as situational judgement tests (SJTs), relies heavily on human judgment, which can be influenced by factors like writing ability and interpretation, making it challenging to ensure the validity and reliability of scores, especially since conventional psychometric methods are less suitable for open responses.
Innovation Solution
A computer system employing natural language processing (NLP) with engines like LLMs (GPT-4, PaLM, BLOOM, LLAMA, BERT) to automatically evaluate written responses, assess sentiment and subjectivity, and perform unsupervised text classification, ensuring accurate alignment with intended non-cognitive skill aspects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If human judgment is used to score constructed-response tests, then the assessment can capture nuanced non-cognitive skills, but the scoring validity and reliability are compromised due to influences like writing ability and interpretation
Solution Approach 1:
The patent introduces an AI-based natural language processing system as an intermediary between the constructed-response test and the scoring process. This intermediary automatically analyzes written responses using NLP techniques, generating objective scores that mediate between the raw response and the final score, thereby eliminating human bias while maintaining the ability to assess non-cognitive skills.
Solution Approach 2:
The patent replaces the mechanical human judgment process with an automated AI-based scoring system. The AI system uses natural language processing, machine learning models, and computational algorithms to automatically evaluate written responses, substituting the mechanical process of human reading and scoring with an automated digital system that provides consistent, reliable results.
2Measurement precision
If conventional psychometric methods are used for open responses, then the assessment process remains simple, but the measurement precision is insufficient for constructed-response tests
Solution Approach 1:
The patent changes the fundamental parameters of the scoring system by transitioning from conventional psychometric methods to AI-based natural language processing. The system uses advanced NLP techniques including sentiment analysis, topic modeling, and machine learning classification to automatically extract meaningful scores from open responses, dramatically improving measurement precision while managing system complexity through automated processing.
3Reliability
If human raters are used for scoring, then the system remains simple to operate, but variability in scoring and human dependence increase
Solution Approach 1:
The patent implements a self-service scoring system where the AI-based natural language processing system automatically evaluates written responses without requiring human raters. The system performs self-scoring using automated algorithms, eliminating human variability and dependence while maintaining operational simplicity through automated processing that requires minimal human intervention.
Data Source
AI summary
Embodiments described herein provide systems and processes for natural language processing for quality assurance of a situational judgement test. For example, system can use natural language processing engine for sentiment analysis and unsupervised text classification to automatically score or rate response data. The system can provide quality assurance for rating data by generating predicted scorings or ratings that can be compared to human rating data.