Conditional Multipass ASR for Low-Confidence Audio Segments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automatic speech recognition (ASR) systems are limited by their vocabularies and grammars, leading to poor recognition of naturally spoken language, accents, and varied command phrases, restricting their ability to interpret multiple applications effectively.
Innovation Solution
A conditional multipass ASR system that employs multiple ASR modules with different recognition methods, allowing for high-confidence recognition of specific words and re-processing of low-confidence or unrecognized audio frames using more powerful ASR systems, enabling broader transcription capabilities and handling multiple applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a limited vocabulary or grammar-based ASR is used, then device resources are conserved and the system remains manageable, but the recognition capability is restricted and cannot handle naturally spoken language, accents, or varied commands
Solution Approach 1:
The ASR system is segmented into multiple passes with different capabilities. The first pass uses a limited grammar-based ASR for common commands, while subsequent passes use more powerful ASR systems with larger vocabularies for unrecognized speech. This segmentation allows the system to achieve high adaptability for diverse speech patterns while keeping the base system simple and resource-efficient.
Solution Approach 2:
The patent implements a nested ASR architecture where simpler ASR systems are embedded within a larger multipass framework. The first-pass ASR is nested within the overall system, and when it fails to recognize speech, the same audio is passed to more powerful nested ASR systems in subsequent passes. This nested structure enables progressive enhancement of recognition capability without requiring the full-power ASR to run continuously, thus managing complexity.
2Adaptability or versatility
If a more powerful ASR with extensive vocabulary is used, then naturally spoken language and multiple applications can be interpreted, but device resources are consumed more rapidly
Solution Approach 1:
Instead of always using the full-power ASR system, the patent applies partial action by using the limited grammar-based ASR first for common recognition tasks. Only when necessary (when speech is unrecognized or confidence is low) does the system activate the more powerful ASR in subsequent passes. This partial application of computational resources significantly reduces energy consumption while maintaining the capability to handle complex speech when needed.
Solution Approach 2:
The multipass ASR system implements periodic action by processing the same audio through multiple passes at different levels of complexity. The first pass uses a lightweight ASR, and only if recognition fails does the system periodically activate the more resource-intensive second pass with the powerful ASR. This periodic activation pattern reduces overall energy consumption compared to continuous use of the powerful ASR, while still achieving high transcription capability when required.
3Measurement precision
If the same audio is re-processed by a more powerful ASR system, then recognition accuracy improves for low-confidence results, but processing time increases
Solution Approach 1:
The system applies partial re-processing by selectively sending only the audio segments with low confidence scores or unrecognized speech to the second-pass ASR, rather than re-processing the entire audio stream. This selective partial action maintains high recognition accuracy for problematic segments while minimizing the time penalty associated with multipass processing.
Solution Approach 2:
The audio processing is segmented into different confidence levels. High-confidence recognized speech is accepted immediately without re-processing, while low-confidence or unrecognized segments are segmented out and sent to subsequent passes for enhanced processing. This segmentation approach optimizes processing time by avoiding redundant re-processing of already-accurately-recognized portions while still achieving high overall accuracy through targeted re-processing of uncertain segments.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In a conditional multipass automatic speech recognition system, one or more intent templates may be received from an application. A spoken utterance is received and audio frames are generated from the utterance. The audio frames are compared to a first grammar. Recognized speech results are generated and unrecognized audio frames or low confidence frames are collected. One of one or more intent templates and one or more corresponding intent parameters may be determined based on the recognized speech results. The unrecognized audio frames may be conditionally compared to a second grammar in instances when additional information is requested, relative to the determined intent template or the corresponding intent parameters.