Streaming Speech Conversion Using Overlapping Frame Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech conversion methods struggle to achieve low-latency and high-quality conversion of infinite streaming speech data, often resulting in semantic loss and unnatural prosody due to the need for complete data input and processing.
Innovation Solution
A method involving feature extraction on overlapping speech frames, attention-based fusion of features, and conversion using a preset speech attribute to process speech streams in real-time, enabling low-latency and coherent speech conversion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If complete speech data is input and processed for speech conversion, then conversion quality is improved, but processing latency increases and real-time performance deteriorates
Solution Approach 1:
The speech stream is divided into overlapping frame sequences where each frame sequence shares frames with adjacent sequences. This segmentation allows incremental processing of speech data without requiring complete data input, reducing latency while maintaining conversion quality through progressive feature extraction and fusion on each segment.
2Productivity
If speech frames are processed independently, then processing speed increases, but semantic coherence and prosody naturalness deteriorate
Solution Approach 1:
Speech frames are pre-processed and stored in overlapping sequences before final conversion. The overlapping frame sequences prepare contextual information in advance, allowing fast processing while maintaining semantic coherence through attention-based feature fusion that leverages pre-computed contextual relationships between adjacent frames.
Solution Approach 2:
The use of overlapping frame sequences ensures continuous processing where each frame is processed in multiple contexts. This continuity maintains semantic coherence and prosody naturalness by preserving contextual relationships across frame boundaries, while the pipeline architecture enables sustained high processing speed without interruption.
3Loss of information
If overlapping frame sequences are used, then semantic integrity is maintained, but computational complexity increases
Solution Approach 1:
Adjacent frame sequences are merged through overlapping frames, allowing the model to process multiple contextual representations simultaneously. This merging maintains semantic integrity by capturing relationships across frame boundaries, while the shared computational structure of overlapping sequences reduces redundant calculations compared to processing completely separate sequences.
Data Source
AI summary
A method of processing a speech stream, which is related to the field of artificial intelligence technology, and more particularly to the fields of deep learning, speech processing, and voice conversion technologies, and includes: performing a feature extraction on a first speech frame sequence in a speech stream to be processed to obtain a first speech feature, where the first speech frame sequence overlaps with at least one second speech frame in a second speech frame sequence, and the second speech frame sequence precedes the first speech frame sequence in the speech stream; fusing, based on an attention mechanism, the first speech feature and a second speech feature determined based on the second speech frame sequence to obtain a speech fusion feature; and converting the speech fusion feature based on a preset speech attribute to obtain converted speech data corresponding to the first speech frame sequence.