Time-Shifted Deep Neural Networks for ASR Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automatic speech recognition (ASR) technology remains computationally intensive, requiring efficient training of neural networks that can effectively utilize available resources.
Innovation Solution
Deep Neural Networks (DNNs) are time-shifted relative to each other and combined to improve recognition accuracy, with fine-tuning of the top and last hidden layers using left- and right-shifted features, and the resulting networks are used for initial recognition and rescoring to enhance performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple time-shifted DNNs are trained and combined to improve recognition accuracy, then recognition accuracy improves, but computational intensity and training complexity increase
Solution Approach 1:
The training process is segmented into distinct phases: initial centered DNN training, followed by separate left-shifted and right-shifted fine-tuning stages. Each phase focuses on specific network components (bottom layers shared, top layers retrained), dividing the complex task of training multiple DNNs into manageable segments that can be executed sequentially with different feature alignments.
Solution Approach 2:
Multiple time-shifted DNNs (centered, left-shifted, right-shifted) are merged into a unified recognition system. The networks are combined through lattice integration and ROVER (Recognizer Output Voting Error Reduction) algorithms, where recognition results from individual networks are aggregated to produce a final output that leverages the complementary strengths of each time-shifted model.
2Measurement precision
If multiple time-shifted DNNs are trained and combined to improve recognition accuracy, then recognition accuracy improves, but computational resources and processing time increase
Solution Approach 1:
The bottom layers of the DNN are pre-trained on centered features and then frozen during subsequent left-shifted and right-shifted fine-tuning phases. This preliminary training establishes a robust feature extraction foundation that can be reused across different time-shifted variants, avoiding redundant computation and reducing overall computational resource requirements.
Solution Approach 2:
Different parts of the DNN architecture are treated differently during training: bottom layers are trained once on centered data and shared across all variants, while top layers are retrained separately for each time-shifted condition. This localized training approach optimizes computational resources by applying intensive computation only where necessary (top layers) while reusing pre-computed features from shared bottom layers.
Data Source
AI summary
Deep Neural Networks (DNN) are time shifted relative to one another and trained. The time-shifted networks may then be combined to improve recognition accuracy. The approach is based on an automatic speech recognition (ASR) system using DNN and using time shifted features. Initially, a regular ASR model is trained to produce a first trained DNN. Then a top layer (e.g., SoftMax layer) and the last hidden layer (e.g., Sigmoid) are fine-tuned with same data set but with a feature window left- and right-shifted to create respective second and third left-shifted and right-shifted DNNs. From these three DNN networks, four combination networks may be generated: left- and right-shifted, left-shifted and centered, centered and right-shifted, and left-shifted, centered, and right-shifted. The centered networks are used to perform the initial (first-pass) ASR. Then the other six networks are used to perform rescoring. The resulting are combined using ROVER (recognizer output voting error reduction) or another technique to improve recognition performance as compared to the centered DNN by itself.