Multi-Speaker Transcript Generation Using Channel Change Symbols
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generating accurate and efficient transcripts from multi-speaker audio streams is challenging due to overlapping speech, and real-time solutions are lacking in existing technologies.
Innovation Solution
A computerized method using a trained model that generates channel change (CC) symbols to separate overlapping speech, allowing for real-time transcript generation by sorting words into virtual channels based on these symbols, and optionally incorporating d-vector analysis for speaker identification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional automatic speech recognition is used on multi-speaker audio streams, then transcript generation can be achieved, but computational cost increases significantly and accuracy decreases when speech overlaps
Solution Approach 1:
The audio stream is segmented into multiple virtual channels based on speaker separation. The system divides overlapping speech into distinct channels (e.g., channel 1, channel 2) using speaker embedding comparison, allowing each channel to be processed independently by the ASR model. This segmentation resolves the contradiction by reducing computational complexity through divide-and-conquer while maintaining transcript accuracy for each speaker's speech.
Solution Approach 2:
Speaker embeddings serve as an intermediary mechanism to identify and separate different speakers in the audio stream. By comparing speaker embeddings against reference embeddings, the system creates virtual channels that organize overlapping speech by speaker identity. This intermediary approach enables accurate transcript generation without requiring computationally expensive multi-speaker ASR models.
2Measurement precision
If post-conversation analysis is used to generate transcripts, then accuracy can be improved, but real-time transcript generation is not achieved
Solution Approach 1:
The system performs preliminary speaker identification and channel assignment during the audio processing stage, creating virtual channels organized by speaker identity before transcript generation. By pre-organizing the audio data into speaker-specific channels using embedding comparison, the ASR model can generate accurate transcripts in real-time without requiring post-conversation analysis, thus resolving the time-accuracy tradeoff.
Data Source
AI summary
The disclosure herein describes using a transcript generation model for generating a transcript from a multi-speaker audio stream. Audio data including overlapping speech of a plurality of speakers is obtained and a set of frame embeddings are generated from audio data frames of obtained audio data using an audio data encoder. A set of words and channel change (CC) symbols are generated from the set of frame embeddings using a transcript generation model. The CC symbols are included between pairs of adjacent words that are spoken by different people at the same time. The set of words and CC symbols are transformed into a plurality of transcript lines, wherein words of the set of words are sorted into transcript lines based on CC symbols, and a multi-speaker transcript is generated based on the plurality of transcript lines. The inclusion of CC symbols by the model enables efficient, accurate multi-speaker transcription.