Neural Network Audio Conversion Timbre Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks used for generating audio signals are insufficient for certain tasks, as they fail to effectively convert the timbre of one audio signal to match another while maintaining the linguistic content and prosody of the original signal.
Innovation Solution
A voice conversion system and audio conversion system utilizing neural networks with components like F0 extractors, spectrogram generators, energy extractors, phoneme posteriorgram encoders, and generators, along with discriminators, to convert input speech to output speech with the timbre of a reference speech, and similarly for audio signals, by processing audio features and using residual blocks for improved timbre transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If neural networks are used to generate audio signals, then audio signal generation capability is provided, but the ability to effectively convert timbre while maintaining linguistic content and prosody is insufficient
Solution Approach 1:
The audio signal processing is divided into separate functional components: F0 extractor for pitch, spectrogram generator for frequency representation, energy extractor for amplitude, phoneme posteriorgram encoder for linguistic content, and generator for timbre conversion. This segmentation allows each component to specialize in maintaining specific audio characteristics while converting timbre.
Solution Approach 2:
The patent introduces intermediate representations (spectrograms, phoneme posteriorgrams, F0 contours) that mediate between the input audio signal and the output signal. These intermediaries preserve linguistic content and prosody information while enabling timbre transformation through the generator network.
2Reliability
If residual blocks are added to improve timbre transfer, then timbre conversion quality is improved, but device complexity increases
Solution Approach 1:
Residual blocks are applied locally within the generator network at specific layers where timbre transformation benefits most. This targeted application improves timbre transfer quality without adding residual complexity throughout the entire network, optimizing the balance between performance and computational cost.
Data Source
AI summary
Apparatuses, systems, and techniques to generate audio signals. In at least one embodiment, features are identified in input audio signals using one or more neural networks which generate an output audio signal based on the identified features.