Streaming Speech-to-Speech Conversion Model for Mobile Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deploying an end-to-end Speech to Speech (STS) conversion model on mobile devices poses challenges due to memory footprint and computation requirements, particularly in achieving acceptable delay with minimal loss in conversion quality, especially when compared to non-streaming server-based models.
Innovation Solution
A streaming-based approach is adopted using a hybrid model with look-ahead stacking and self-attention, combined with int4 quantization aware training and int8 post training quantization, allowing the model to run in real-time with a streaming conformer encoder and a modified decoder, and a streaming vocoder to minimize quality loss and reduce model size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a non-streaming server-based STS conversion model is used, then conversion quality is maintained, but device memory footprint and computation requirements increase significantly
Solution Approach 1:
The STS conversion model is divided into separate encoder and decoder components that can be processed independently. The encoder processes input speech and generates intermediate representations, while the decoder generates output speech, allowing for optimized resource allocation and reduced memory footprint on mobile devices.
Solution Approach 2:
The model employs quantization techniques to change the precision parameters of model weights and activations, reducing memory requirements while maintaining conversion quality. This allows the complex model to be deployed on mobile devices with limited memory resources.
2Productivity
If a streaming-based approach with look-ahead stacking and self-attention is used, then processing speed increases to approximately 2× faster than real-time, but model complexity increases
Solution Approach 1:
The look-ahead stacking mechanism pre-processes and stacks future acoustic frames before they are needed for decoding. This preliminary action allows the decoder to access prepared data structures, enabling faster processing while managing complexity through organized data flow.
Solution Approach 2:
The stacked acoustic frames serve as an intermediary data structure between the input audio and the decoder. This intermediate representation facilitates efficient processing by organizing acoustic information in a format optimized for the decoder, improving processing speed while managing computational complexity.
3Productivity
If quantization techniques (int4 and int8) are applied, then model size is reduced and real-time processing is enabled, but conversion quality may be affected
Solution Approach 1:
The model applies quantization to change the precision parameters of model weights and activations from high precision (float32) to lower precision (int8 and int4). This parameter change reduces memory footprint and enables real-time processing on mobile devices while maintaining acceptable conversion quality through optimized quantization schemes.
Data Source
AI summary
The present disclosure relates to a streaming speech-to-speech conversion model, where an encoder runs in real time while a user is speaking, then after the speaking stops, a decoder generates output audio in real time. A streaming-based approach produces an acceptable delay with minimal loss in conversion quality when compared to other non-streaming server-based models. A hybrid model approach for combines look-ahead in the encoder and a non-causal stacker with non-causal self-attention.