Spoken Language Grading via Crowdsourced Transcription
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current automated systems for assessing constructed responses, particularly in spoken language evaluation, face challenges in accuracy and reliability, as they struggle to derive precise features and are susceptible to falsification, and crowd-based approaches are limited in high-stake assessments and expert tasks.
Innovation Solution
A method and system that integrates crowdsourcing to derive speech and language features, combining crowd grades, force alignment features, and natural language processing features to generate individual and composite scores for spoken language evaluation, using a computer processor and automated speech assessment tools.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If automated machine learning approaches are used for constructed response grading, then scalability and automation extent are improved, but measurement precision and reliability deteriorate due to inability to accurately derive features and susceptibility to falsification
Solution Approach 1:
The patent introduces crowdsourced human transcriptions as an intermediary between the automated speech recognition system and the machine learning grading model. The process uses crowd-sourced volunteers to transcribe speech responses, and these human transcriptions serve as a mediator to train and validate the automated speech recognition system, thereby improving its accuracy without sacrificing automation scalability
Solution Approach 2:
The patent combines multiple assessment approaches by merging automated speech recognition with crowdsourced human evaluation. The system integrates both automated feature extraction and human-validated transcriptions to create a hybrid grading system that maintains automation benefits while achieving human-level accuracy through the combination of machine efficiency and human intelligence
2Measurement precision
If crowd-based approaches are used for response evaluation, then measurement precision is improved through human intelligence, but device complexity and reliability deteriorate due to crowd drift and unsuitability for expert tasks
Solution Approach 1:
The patent performs preliminary crowdsourced transcription and validation before the actual automated grading process. By having crowd members transcribe speech responses in advance and use these transcriptions to train the automated speech recognition system, the system establishes accurate reference data beforehand, simplifying the subsequent automated evaluation process while maintaining high precision
Solution Approach 2:
The system uses crowdsourced transcriptions to automatically train and validate the speech recognition model without requiring continuous expert intervention. The crowd-generated data serves the dual purpose of both training material and validation benchmark, enabling the system to self-improve and maintain accuracy autonomously
Data Source
AI summary
A method and system for constructive response grading for spoken language is disclosed. The method and system are computer implemented and involve a crowdsourcing step to derive evaluation features. The method includes steps for posting a speech test through an automated speech assessment tool, receiving candidate responses from candidates for the speech test; delivering the candidate responses to crowdsource volunteers; receiving crowdsourced responses from crowdsource volunteers, where the crowdsourced responses comprise a transcription of the speech test; deriving features from the transcription; and deriving a individual scores based on the features, where the individual scores are representative of pronunciation score, fluency score, content organization score and grammar score of the spoken language for each candidate.