Real-Time Voice Conversion Model for Portable Applications
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current voice conversion systems operate with high latency, making them unsuitable for real-time applications such as two-way communication, where delays between user speech are unacceptable.
Innovation Solution
A computer-implemented method and system that uses a voice conversion model to receive a sequence of acoustic frames from a source speech utterance, generate a soft speech representation using a content encoder, determine an acoustic estimation, and produce a synthetic speech representation with target speech characteristics, all while maintaining semantic information from the source speech.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If traditional voice conversion systems are used, then voice conversion functionality is achieved, but latency is high making them unsuitable for real-time applications
Solution Approach 1:
The voice conversion system segments the speech signal into individual acoustic frames and processes them sequentially through the neural network model. This segmentation allows the system to generate conversions in real-time as each frame is processed, rather than waiting for the complete speech utterance, thereby reducing latency and enabling real-time applications.
Solution Approach 2:
The system performs preliminary processing by extracting acoustic frames and generating soft speech representations before final synthesis. This preliminary action of frame extraction and intermediate representation generation enables the system to prepare conversion data in advance, reducing overall latency when the complete speech input is received.
2Ease of operation
If voice conversion is applied for real-time communication, then communication effectiveness is improved, but system complexity increases
Solution Approach 1:
The neural network model acts as an intermediary between the input acoustic frames and the output synthetic speech. This intermediary component automatically handles the complex voice conversion transformations, allowing the system to maintain communication effectiveness while managing complexity through a dedicated processing layer that abstracts the conversion logic.
Solution Approach 2:
The system replaces traditional mechanical or rule-based speech processing with a neural network-based approach. This substitution enables more sophisticated voice conversion capabilities that maintain natural speech characteristics while operating in real-time, improving communication effectiveness without requiring complex manual configuration or post-processing.
3Measurement precision
If semantic information is preserved during voice conversion, then speech content accuracy is maintained, but processing time increases
Solution Approach 1:
The neural network model continuously processes acoustic frames through multiple convolutional and recurrent layers, maintaining semantic information throughout the transformation pipeline. This continuous processing ensures that semantic content is preserved at each stage of conversion rather than being added in separate post-processing steps, thereby maintaining accuracy without significant time penalty.
Solution Approach 2:
The system transforms acoustic parameters through the neural network while preserving semantic information. By changing the representation parameters from raw acoustic frames to soft speech representations and then to synthetic speech, the system maintains semantic content accuracy through parameter transformation rather than through separate semantic analysis steps, reducing processing time.
Data Source
AI summary
A method includes receiving a sequence of acoustic frames characterizing a source speech utterance including semantic information and source speech characteristics, obtaining a latent speaker embedding representing target speech characteristics, and generating, at each of a plurality of output steps, using a content encoder of a voice conversion model, a soft speech representation for a corresponding acoustic frame. The method also includes determining, at each of the plurality of output steps, an acoustic estimation for the corresponding acoustic frame, and generating, at each of the plurality of output steps, using a decoder of the voice conversion model, a synthetic speech representation for the corresponding acoustic frame based on the soft speech representation and the acoustic estimation. The synthetic speech representation includes the semantic information of the source speech utterance and the target speech characteristics of the latent speaker embedding. The decoder is conditioned on the latent speaker embedding.