Prompt-Based Timbre Conversion Using Self-Attention Diffusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing timbre conversion technologies using deep neural networks or generative adversarial networks achieve low timbre similarity and unsatisfactory audio quality due to insufficient expressive capability and inclusion of timbre information in semantic features, necessitating model pre-training for specific timbres.
Innovation Solution
Employ a self-attention-based diffusion model to generate converted acoustic features using semantic features of the original and prompt audio, allowing timbre conversion without pre-training, and utilize a vocoder to produce high-quality audio with improved timbre similarity and pronounce accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deep neural networks or generative adversarial networks are used for timbre conversion, then the conversion process can be implemented, but the timbre similarity and audio quality remain low
Solution Approach 1:
The patent changes the fundamental parameters of the conversion model by switching from traditional deep neural networks or generative adversarial networks to a diffusion model architecture. This parameter change enables the system to achieve high timbre similarity and audio quality without requiring complex pre-training for specific timbres, thereby resolving the contradiction between reliability and device complexity
Solution Approach 2:
The patent extracts and removes the timbre information from the semantic features of the original audio. By separating the semantic content from the timbre characteristics, the system can convert timbre more effectively using the diffusion model, improving timbre similarity while avoiding the complexity of handling intertwined semantic and timbre information
2Reliability
If model pre-training is performed for specific timbres, then timbre conversion capability is improved, but the conversion process time increases
Solution Approach 1:
The patent performs preliminary extraction of timbre information from the prompt audio and stores it as a separate embedding. This preliminary action enables the diffusion model to directly use the extracted timbre characteristics during conversion without requiring time-consuming pre-training for each specific timbre, thus improving conversion capability while reducing process time
Solution Approach 2:
The patent creates a timbre embedding copy from the prompt audio that can be reused across multiple conversions. Instead of re-training the model for each timbre conversion task, the system copies and reuses the extracted timbre characteristics, maintaining high conversion capability while significantly reducing the time required for each conversion process
3Loss of information
If semantic features containing timbre information are used, then the original audio semantics are preserved, but the target timbre conversion accuracy decreases
Solution Approach 1:
The patent segments the audio features into distinct components: semantic features that preserve original meaning and timbre embedding that captures target timbre characteristics. By segmenting these features and processing them through separate pathways in the diffusion model, the system maintains semantic information while achieving accurate timbre conversion, resolving the contradiction between information preservation and conversion accuracy
Data Source
AI summary
Embodiments of the present disclosure relate to a method and apparatus for timbre conversion, an electronic device, and a product. The method includes determining a semantic feature of an audio to be converted, where the audio to be converted has an original timbre. The method further includes acquiring a prompt audio, where the prompt audio has a target timbre different from the original timbre. The method further includes generating, based on the semantic feature of the audio to be converted and the prompt audio, a converted acoustic feature using a self-attention-based diffusion model. Additionally, the method further includes generating a converted audio based on the converted acoustic feature, where the converted audio is an audio in which a timbre of the audio to be converted is converted into the target timbre.