Non-scorable Speech Filters for Scoring Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automated speech assessment systems face challenges in providing valid assessments due to issues like poor audio quality, off-topic responses, insufficient speech, incorrect language, and plagiarism, which can lead to erroneous scores in non-native speaker evaluations.
Innovation Solution
A non-scorable speech detection system that uses automatic speech recognition and various filters to determine the scorability of speech samples based on audio quality, speech quantity, relevance, language, and plagiarism, preventing unsuitable responses from being scored by filtering them out before reaching the scoring system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automated speech assessment systems evaluate all speech samples, then productivity is improved, but measurement precision deteriorates due to non-scorable responses
Solution Approach 1:
The system performs preliminary filtering of speech samples before they reach the scoring model. Multiple filters (audio quality filter, insufficient speech filter, off-topic filter, incorrect language filter, plagiarism filter) are applied in advance to identify and exclude non-scorable responses, ensuring only valid samples are scored.
Solution Approach 2:
The patent introduces an intermediary filtering layer between the speech sample input and the scoring model. This intermediary system includes various specialized filters that mediate by examining speech samples and determining their scorability before they reach the final scoring stage.
2Measurement precision
If multiple filters are applied to detect non-scorable speech, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The filtering system is segmented into multiple specialized filters, each responsible for detecting specific types of non-scorable responses. The filters include: audio quality filter (detects poor audio), insufficient speech filter (detects short responses), off-topic filter (detects irrelevant content), incorrect language filter (detects wrong language), and plagiarism filter (detects copied content). Each filter operates independently to examine specific aspects of speech samples.
3Measurement precision
If speech samples are filtered before scoring, then measurement precision is improved, but loss of information increases due to rejected samples
Solution Approach 1:
The system provides feedback by generating detailed indications of why each speech sample was rejected as non-scorable. The filters produce specific feedback information including: audio quality metrics, speech duration measurements, topic relevance analysis, language identification results, and plagiarism detection outcomes. This feedback mechanism ensures that rejected samples are not simply discarded but are accompanied by diagnostic information about their non-scorability.
Data Source
AI summary
A method for scoring non-native speech includes receiving a speech sample spoken by a non-native speaker and performing automatic speech recognition and metric extraction on the speech sample to generate a transcript of the speech sample and a speech metric associated with the speech sample. The method further includes determining whether the speech sample is scorable or non-scorable based upon the transcript and speech metric, where the determination is based on an audio quality of the speech sample, an amount of speech of the speech sample, a degree to which the speech sample is off-topic, whether the speech sample includes speech from an incorrect language, or whether the speech sample includes plagiarized material. When the sample is determined to be non-scorable, an indication of non-scorability is associated with the speech sample. When the sample is determined to be scorable, the sample is provided to a scoring model for scoring.