End-to-End Neural Speech Scoring via Dual Encoder Architecture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated speech scoring systems rely on handcrafted features, which are difficult to develop and may not effectively capture the nuances of speech proficiency, limiting their accuracy in assessing speaking skills.
Innovation Solution
An end-to-end neural network-based solution that uses deep learning models to encode both lexical and acoustic cues, incorporating 1D Convolutional Neural Networks, Bi-Directional Recurrent Neural Networks with Long Short Time Memory cells, and attention weighting schemes to automatically learn predictive features for speech scoring.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If handcrafted speech features are used in automated speech scoring systems, then the system can be built with conventional methods, but the accuracy in assessing speaking skills is limited
Solution Approach 1:
The patent replaces the mechanical system of handcrafted feature extraction with a deep learning-based automatic feature learning system. Specifically, it uses a bi-directional RNN to automatically learn lexical and acoustic cues from raw speech data, eliminating the need for manual feature engineering while significantly improving scoring accuracy through data-driven feature representation
Solution Approach 2:
The patent changes the parameter representation from fixed handcrafted features to dynamic learned embeddings. The system uses word embeddings for lexical cues and learned acoustic representations, allowing the features to adapt and optimize themselves during training based on the scoring task, thereby improving measurement precision
2Productivity
If deep learning models are used to automatically learn features, then the model-building efficiency improves, but the computational complexity increases
Solution Approach 1:
The patent segments the feature learning process into two independent but complementary streams: a lexical cue processing stream using a bi-directional RNN and an acoustic cue processing stream using separate feature extractors. This segmentation allows each stream to be optimized independently and processed in parallel, improving model-building efficiency while managing computational complexity through modular architecture
Solution Approach 2:
The patent creates a universal bi-directional RNN architecture that can process both lexical and acoustic cues through the same computational framework. This multi-functional design allows the system to handle different types of input data uniformly, improving productivity by reusing the same model structure for multiple feature types while reducing overall system complexity
Data Source
AI summary
A system for end-to-end automated scoring is disclosed. The system includes a word embedding layer for converting a plurality of ASR outputs into input tensors; a neural network lexical model encoder receiving the input tensors; a neural network acoustic model encoder implementing AM posterior probability, word duration, mean value of pitch and mean value of intensity based on a plurality of cues; and a linear regression module, for receiving concatenated encoded features from the neural network lexical model encoder and the neural network acoustic model encoder.