Visual Context Aware ASR for Robotic Speech Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Automatic Speech Recognition (ASR) systems face challenges in accuracy due to factors like dialect, distance from the microphone, ambient noise, and novel or low-frequency vocabularies, particularly in robotic applications, leading to poor translation accuracy and impaired task execution.
Innovation Solution
A visual context-aware ASR method and system that utilizes a robotic agent to capture ego-view images, detect objects and their text descriptions using a dense image captioning network, and generate a dynamic word vocabulary through a bias prediction network. This dynamic vocabulary is then compiled into a trie, providing visual context for speech recognition, and is used to bias the acoustic model's probability distribution sequence during decoding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional ASR systems are used in robotic applications, then the system is simple and easy to implement, but the transcription accuracy deteriorates significantly due to dialect, distance, ambient noise, and novel vocabularies
Solution Approach 1:
The patent merges visual processing capabilities with speech recognition by integrating an image captioning network that generates dynamic vocabulary from visual context. This combination allows the ASR system to leverage both audio and visual information, improving transcription accuracy in challenging robotic application scenarios while managing complexity through unified processing architecture.
Solution Approach 2:
The system performs preliminary visual scene understanding by capturing images and generating dynamic vocabulary before speech recognition occurs. This pre-processing of visual context creates a biased vocabulary that guides the subsequent ASR decoding process, allowing the system to anticipate likely words based on visual evidence before hearing the speech input.
2Measurement precision
If visual context processing is added to improve transcription accuracy, then the measurement precision improves, but the loss of time increases due to additional image capture and processing steps
Solution Approach 1:
The system implements periodic visual sampling triggered by speech detection events rather than continuous visual processing. When speech is detected, the system captures an image and processes it to generate dynamic vocabulary, then resets. This periodic approach reduces overall processing time while maintaining accuracy improvements during speech recognition events.
Solution Approach 2:
Visual processing is performed preliminarily and in parallel with speech signal processing. The image captioning network generates dynamic vocabulary concurrently with the acoustic model processing speech, allowing both processes to overlap in time and reduce total system latency while maintaining improved transcription accuracy.
3Measurement precision
If a dynamic word vocabulary is generated from visual context, then the transcription accuracy for novel and low-frequency vocabularies improves, but the device complexity increases due to the bias prediction network and modified beam search decoding
Solution Approach 1:
The bias prediction network applies localized quality adjustments to the vocabulary by generating context-specific probability biases for words relevant to the current visual scene. Rather than uniformly processing all vocabulary, the system focuses computational resources on words visually present in the scene, improving accuracy for novel vocabularies while managing complexity through selective processing.
Solution Approach 2:
The system implements dynamic vocabulary adjustment where the bias vocabulary changes adaptively based on the current visual context captured in each scene. The beam search decoding dynamically incorporates visual context information to bias word selection, allowing the system to adapt to different environments and objects while maintaining manageable complexity through context-dependent processing.
4Measurement precision
If dense image captioning network is used to detect objects and text descriptions, then the visual context quality improves, but the loss of energy increases due to complex image processing requirements
Solution Approach 1:
The dense image captioning network operates periodically only when speech input is detected, rather than continuously processing images. This event-driven approach reduces overall energy consumption by activating complex visual processing only when needed for speech recognition, while still maintaining high visual context quality during those critical moments.
Data Source
AI summary
Accuracy of transcript is of foremost importance in Automatic Speech Recognition (ASR). State of the art system mostly rely on spelling correction based contextual improvement in ASR, which is generally a static vocabulary based biasing approach. Embodiments of the present disclosure provide a method and system for visual context aware ASR. The method provides biasing using shallow fusion biasing approach with a modified beam search decoding technique, which introduces a non-greedy pruning strategy to allow biasing at the sub-word level. The biasing algorithm brings in the visual context of the robot to the speech recognizer based on a dynamic biasing vocabulary, improving the transcription accuracy. The dynamic biasing vocabulary, comprising objects in a current environment accompanied by their self and relational attributes, is generated using a bias prediction network that explicitly adds label to objects, which are detected and captioned via a state of the art dense image captioning network.