Speech Segmentation Using Multi-Scale Metadata for Endpointing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional speech processing systems face bottlenecks in speech segmentation due to differing time scales and conflicting requirements between speech enhancement and automated speech recognition, leading to high error rates and false triggering, as they lack the ability to update metadata in real-time and incorporate the internal state of the speech enhancement module.
Innovation Solution
The implementation of multiple speech detectors that generate metadata on different time scales, allowing for parallel processing and adjustment of parameters to balance latency and confidence, enabling robust speech recognition in adverse acoustic environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If speech enhancement is made more aggressive to improve speech recognition accuracy, then speech recognition accuracy is improved, but false triggering increases leading to high error rates
Solution Approach 1:
The speech detection process is segmented into multiple independent detectors operating at different time scales. A first detector operates at frame-by-frame level for fast detection, while a second detector operates at sequence level for high confidence verification. This segmentation allows the system to balance between fast response and accurate verification, reducing false triggering while maintaining speech recognition accuracy.
Solution Approach 2:
The system performs preliminary detection using the first speech detector at frame-by-frame time scale to identify potential speech onsets. This preliminary action triggers buffering of audio data before final confirmation by the second detector, ensuring that speech is not missed while allowing time for verification to reduce false positives.
2Speed
If fast detection is implemented to start buffering audio stream timely, then speech onsets are not missed, but false alarms increase due to limited history and confidence
Solution Approach 1:
The detection system is divided into two stages with different time scales. The first detector operates at frame-by-frame level (fast time scale) to quickly identify potential speech onsets and trigger buffering. The second detector operates at sequence level (slower time scale) to verify speech activity with higher confidence. This segmentation enables fast detection without excessive false alarms.
Solution Approach 2:
The first detector uses a lower threshold for speech detection to ensure fast response and capture all potential speech onsets, accepting some false positives. The second detector then applies stricter criteria to filter out false alarms. This partial action approach ensures no speech is missed while ultimately reducing false alarms through verification.
3Device complexity
If single metadata from one detector is used to simplify processing, then device complexity is reduced, but the ability to balance latency and confidence is limited
Solution Approach 1:
Instead of using a single metadata stream, the system generates multiple metadata streams from different detectors operating at different time scales. The first metadata is generated at frame-by-frame level for fast latency response, while the second metadata is generated at sequence level for high confidence. The endpointing module processes both metadata streams independently, maintaining low complexity while achieving versatile latency and confidence control.
Solution Approach 2:
The multiple metadata streams serve different functions: the first metadata provides fast response for timely buffering initiation, while the second metadata provides high confidence for final speech recognition triggering. This multi-functionality allows the system to address both latency and confidence requirements simultaneously without significantly increasing processing complexity.
4Device complexity
If metadata is not updated during audio stream buffering to reduce processing load, then computational complexity is reduced, but speech segmentation accuracy deteriorates
Solution Approach 1:
The metadata update process is segmented into two phases: during the buffering phase (triggered by first detector), metadata updates are reduced or paused to lower processing load; during the verification and recognition phase (triggered by second detector), metadata updates resume to ensure segmentation accuracy. This temporal segmentation of processing intensity balances computational load with accuracy requirements.
Data Source
AI summary
Methods and apparatus to process microphone signals by a speech enhancement module to generate an audio stream signal including first and second metadata for use by a speech recognition module. In an embodiment, speech recognition is performed using endpointing information including transitioning from a silence state to a maybe speech state, in which data is buffered, based on the first metadata and transitioning to a speech state, in which speech recognition is performed, based upon the second metadata.