Voice quality conversion learning device and voice quality conversion device

Abstract

This voice quality conversion learning device comprises a learning unit configured to learn a first machine learning model and a second machine learning model. The first machining learning model takes an input of: a speaker vector calculated on the basis of first audio; and second audio obtained by mixing noise with the first audio. The second machine learning model takes an input of: a first error relating to a speech-content predicted value outputted by the first machine learning model, relative to a correct-interpretation value of speech content in the first audio; the predicted value; and the speaker vector. The learning of the machine learnings models is carried out on the basis of a second error relating to predicted audio outputted by the second machine learning model, relative to the first audio. This enables robust voice quality conversion in noisy environments.

Description

Voice quality conversion learning device and voice quality conversion device

[0001] This invention relates to a voice quality conversion learning device and a voice quality conversion device.

[0002] References 1, 2, and 3 disclose methods for training a voice conversion (non-parallel voice conversion) model using non-parallel data. This method involves separating the utterance content and speaker identity from the input speech using a content encoder and speaker encoder, respectively, and then feeding these back into the decoder to reconstruct the input speech.

[0003] In this framework, the key is how to separate (1) the content of the spoken utterance from (2) the speaker identity. References 4 and 5 propose methods using VAE and normalizing flow, and vector quantization, respectively.

[0004] However, these methods focus solely on how to accurately convert clean speech, and in environments with noise such as background noise or other people's voices, the conversion performance deteriorates significantly due to a mismatch between the conditions during training and inference.

[0005] One possible method for robustly operating voice conversion in noisy environments is to apply speech enhancement to the input audio for voice conversion. By using speech enhancement to make noisy audio closer to clean audio before inputting it to voice conversion, the mismatch in conditions during voice conversion training and inference can be reduced (Reference 6).

[0006] Y. Saito, Y. Ijima, K. Nishida and S. Takamichi、"Non-parallel voice conversion using variational autoencoders conditioned by phonetic posteriorgrams and D-Vectors"、Proc. ICASSP, pp. 5274-5278, 2018.S. Liu, Y. Cao, D. Wang, X. Wu, X. Liu and H. Meng、"Any-to-many voice conversion with location-relative sequence-to-sequence modeling"、IEEE. Trans. on Audio, Speech, and Language Processing, vol. 29, pp. 1717-1728, 2021.H. Kameoka, T. Kaneko, K. Tanaka and N. Hojo、"StarGAN-VC: non-parallel many-to-many voice conversion using star generative adversarial networks"、Proc. SLT, pp. 266-273, 2018.K. Qian, Y. Zhang, S. Chang, X. Yang, M. Hasegawa-Johnson、"AutoVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss"、Proc. ICML, pp. 5210--5219, 2019.B. van Niekerk, M. -A. Carbonneau, J. Zaidi, M. Baas, H. Seute and H. Kamper、"A comparison of discrete and soft speech units for improved voice conversion"、Proc. ICASSP, 2022, pp. 6562-6566, 2022.Y. -J. Chan, C. -J. Peng, S. -S. Wang, H. -M. Wang, Y.Tsao, T.-S. Chi, "Speech enhancement-assisted voice conversion in noisy environments", Proc. APSIPA, pp. 1533-1538, 2022.

[0007] While voice enhancement can make speech sound cleaner, it can sometimes distort the enhanced audio, meaning it doesn't always provide a good input for voice conversion. Furthermore, both voice enhancement and voice conversion need to be running simultaneously, and the processing time required for both is a trade-off for real-time performance.

[0008] The present invention has been made in view of the above points, and aims to enable robust voice quality conversion in noisy environments.

[0009] To solve the above problems, the voice quality conversion learning device includes a learning unit configured to learn the first machine learning model and the second machine learning model based on a first machine learning model that takes a speaker vector calculated based on the first speech and a second speech in which noise is mixed with the first speech as input, a first error between the predicted value of the speech content output by the first machine learning model and the correct value of the speech content of the first speech, and a second error between the predicted speech output by the second machine learning model that takes the predicted value and the speaker vector as input and the first speech and the first machine learning model.

[0010] This enables robust voice quality conversion even in noisy environments.

[0011] This figure shows an example of the hardware configuration of the voice conversion device 10 in an embodiment of the present invention. This figure shows an example of the functional configuration of the voice conversion device 10 during learning of voice conversion of normal speech in the first embodiment. This figure shows an example of the functional configuration of the voice conversion device 10 during inference of voice conversion of normal speech in the first embodiment. This figure shows an example of the functional configuration of the voice conversion device 10 during learning of voice conversion of whispered speech in the second embodiment. This figure shows an example of the functional configuration of the voice conversion device 10 during inference of voice conversion of whispered speech in the second embodiment.

[0012] Reference 1 discloses a method for speech recognition that limits the input speaker. In this method, in order to shut out noise other than the input speaker's voice (noise and other people's voices), the input speaker information is explicitly provided to the speech recognition system along with the voice. Among the modules that constitute the parallel voice conversion model, the content prediction model, which obtains content from the input voice, operates in a manner similar to the speech recognition system. Therefore, in this embodiment, similar to non-reference 1, the content prediction model is conditioned with the speaker information of the input voice. The content obtained from the content prediction model has information of other noises excluded. By providing this and the target speaker's speaker information to the speech waveform prediction model, the target speaker's speech waveform is generated, and voice conversion is achieved.

[0013] Embodiments of the present invention will be described below with reference to the drawings. Figure 1 is a diagram showing an example of the hardware configuration of a voice conversion device 10 in an embodiment of the present invention. The voice conversion device 10 in Figure 1 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, etc., which are all interconnected by bus B.

[0014] The program that enables processing in the voice conversion device 10 is provided on a recording medium 101 such as a CD-ROM. When the recording medium 101 containing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101; it may also be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files and data.

[0015] The memory device 103 reads and stores a program from the auxiliary storage device 102 when a program startup command is received. The processor 104 is either a CPU or a GPU (Graphics Processing Unit), or both a CPU and a GPU, and executes the functions related to the voice conversion device 10 according to the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0016] First, the first embodiment will be described. In the first embodiment, the learning and inference of voice quality conversion of normal speech will be described. First, the learning of voice quality conversion of normal speech will be described. Normal speech refers to speech other than whispering speech, which is the target of the second embodiment. Whispering speech refers to, for example, speech that is devoiced without vibrating the vocal cords.

[0017] Figure 2 shows an example of the functional configuration of the voice conversion device 10 during the learning phase of normal speech voice conversion in the first embodiment. In Figure 2, the voice conversion device 10 includes a noisy speech generation unit 11, a voice conversion model 12, a voice conversion model learning unit 13, and a speaker vector calculation unit 14. Each of these units is realized by processing that one or more programs installed in the voice conversion device 10 cause the processor 104 to execute.

[0018] The noisy speech generation unit 11 includes a speech mixing unit 111. The voice quality conversion model 12 is a machine learning model (e.g., a neural network) that includes a content prediction model 121 and a speech waveform prediction model 122. The voice quality conversion model learning unit 13 includes a content prediction error calculation unit 131, a speech prediction error calculation unit 132, and an error minimization unit 133.

[0019] The learning process of the voice conversion device 10 will be explained in accordance with Figure 2.

[0020] [Noisy Speech Generation Unit 11] First, with the aim of generating training data to enable robust content acquisition in noisy environments, the speech mixing unit 111 receives a clean speech x from a certain speaker, noise n, and correct content c as input, and outputs a noisy speech y in which noise n is randomly mixed (superimposed) on the speech x. Clean means that there is no noise or reverberation at all, and that it was recorded in a quiet environment such as a studio or booth. Speech mixing can be performed using publicly known techniques. Here, the correct content c is information corresponding to the content of the speech utterance (information indicating the content of the speech), and if a correct transcript or its time information is available, it is used as a phoneme alignment with time information. If there is no information corresponding to a transcript, a hidden state obtained by inputting speech x into a self-supervised learning model (SSL model) such as HuBERT or wav2vec2, or a token sequence obtained by discretizing the said hidden state, can be used instead as the correct content c.

[0021] Separately, the speaker vector calculation unit 14 receives speech x as input and calculates the speaker vector s. Here, the speaker vector is a fixed-length vector that represents the speaker's characteristics, and can be an i-vector, an x-vector, or an aggregate of the hidden states of an SSL model. For example, the speaker encoders in Non-Patent Documents 1 to 3 can be used as the speaker vector calculation unit 14. The speaker vector calculation unit 14 may be embedded in the voice quality conversion model 12, or a model obtained using a different learning criterion such as an i-vector or an x-vector may be reused.

[0022] [Voice quality conversion model 12] The content prediction model 121 takes noisy speech y and speaker vector s as input and outputs predicted content c' (an example of a predicted value of the utterance) based on the learning parameters.

[0023] Next, the speech waveform prediction model 122 takes the predicted content c' and speaker vector s as input and outputs the predicted speech x' (an example of predicted speech) based on the learning parameters.

[0024] [Voice Conversion Model Learning Unit 13] The voice conversion model learning unit 13 calculates errors from multiple perspectives so that the voice conversion model 12 can correctly predict the content and the audio, respectively.

[0025] First, the content prediction error calculation unit 131 takes the correct content c (an example of the correct value of the spoken content of voice x) and the predicted content c' as input and calculates the content prediction error, which is the error of the predicted content c' compared to the correct content c. Here, the error function can be a cross-entropy loss if the correct content c is a discrete series, or an L1 loss or least squares loss if it is a continuous series.

[0026] Next, the speech prediction error calculation unit 132 receives speech x and the predicted speech x' as input and calculates the speech prediction error, which is the error of the predicted speech x' with respect to speech x. Here, the speech prediction error is a weighted sum of the reconstruction error at the speech waveform level as proposed in Reference 4, the spectral reconstruction error between spectra obtained from the speech waveform, or the adversarial loss obtained within the framework of a generative adversarial network.

[0027] Next, the error minimization unit 133 receives the voice conversion model 12, the content prediction error, and the speech prediction error as input, and updates the learning parameters of the voice conversion model 12 (the learning parameters of the content prediction model 121 and the speech waveform prediction model 122, respectively) based on the sum of the content prediction error and the speech prediction error, similar to Reference 5.

[0028] Next, we will explain the reasoning process for voice quality conversion in normal speech.

[0029] Figure 3 shows an example of the functional configuration of the voice conversion device 10 during voice conversion inference in the first embodiment of normal speech. In Figure 3, the same parts as in Figure 2 are denoted by the same reference numerals, and their descriptions are omitted. In Figure 3, the voice conversion device 10 has a voice conversion model 12 and a speaker vector calculation unit 14. The voice conversion model 12 in Figure 3 is a machine learning model that has been trained by the learning process described in Figure 2. In Figure 3, two speaker vector calculation units 14 are shown for the convenience of explaining the processing procedure, but only one speaker vector calculation unit 14 is needed.

[0030] The flow during inference of the voice conversion apparatus 10 will be described with reference to FIG. 3.

[0031] [Voice Conversion Model 12] The voice conversion model 12 receives a noisy source speaker speech y ｓｏｕｒｃｅ as an input to perform voice conversion. Specifically, the speaker vector calculation unit 14 receives a clean source speaker speech y ｓｏｕｒｃｅ as an input, and calculates a source speaker vector s ｓｏｕｒｃｅ . Here, the source speaker speech y ｓｏｕｒｃｅ refers to speech uttered by a source speaker. The utterance content of the clean source speaker speech y ｓｏｕｒｃｅ does not have to be the same as the utterance content of the noisy source speaker speech y ｓｏｕｒｃｅ . A content prediction model 121 receives the source speaker vector s ｓｏｕｒｃｅ and the noisy source speaker speech y ｓｏｕｒｃｅ as inputs, and outputs predicted content c' based on learning parameters.

[0032] The speaker vector calculation unit 14 also receives a clean target speaker speech x ｔａｒｇｅｔ as an input, and calculates a target speaker vector s ｔａｒｇｅｔ . Here, the target speaker speech x ｔａｒｇｅｔ refers to speech uttered by a target speaker, and the target speaker and the source speaker are different persons. The reason why the target speaker speech x ｔａｒｇｅｔ is clean is to avoid a situation where the speaker characteristics of the target cannot be sufficiently expressed due to interference from noise. A speech waveform prediction model 122 receives the speaker vector s ｔａｒｇｅｔ and the predicted content c' as inputs, and outputs the predicted speech x ｔａｒｇｅｔ of the target speaker.

[0033] As described above, according to the first embodiment, the speaker vector of the speaker source speaker calculated from clean speech x is input to the content prediction model 121 as a key for identifying the speaker of noisy speech y. This allows for robust content acquisition by focusing on the speech of the speaker corresponding to the key (i.e., the speaker corresponding to the speaker vector) from noisy speech y (speech containing noise) (shutting out ambient noise and the speech of others). Therefore, when the speech of multiple people is input, it is possible to avoid converting all incoming speech into speaker vectors. As a result, robust voice quality conversion can be achieved in noisy environments. Furthermore, since there is no need to add speech enhancement before voice quality conversion, a decrease in operating speed can be avoided.

[0034] Next, a second embodiment will be described. The differences between the second embodiment and the first embodiment will be described. Points not specifically mentioned in the second embodiment may be the same as in the first embodiment.

[0035] In the second embodiment, the learning and inference of voice quality conversion of whispered speech will be described. First, the learning of voice quality conversion of whispered speech will be described.

[0036] Figure 4 shows an example of the functional configuration of the voice conversion device 10 during the learning phase of voice conversion of whispered speech in the second embodiment. In Figure 4, the same reference numerals are used for parts that are the same as or corresponding to parts in Figure 2, and their descriptions are omitted as appropriate.

[0037] In Figure 4, the voice conversion device 10 includes a noisy whispering voice generation unit 11a, a voice conversion model 12, a voice conversion model learning unit 13, and a speaker vector calculation unit 14. Each of these units is realized by processing that one or more programs installed in the voice conversion device 10 cause the processor 104 to execute. In Figure 4, two speaker vector calculation units 14 are shown for the convenience of explaining the processing procedure, but only one speaker vector calculation unit 14 is needed.

[0038] The difference from the first embodiment is that the voice quality conversion device 10 includes a noisy whisper voice generation unit 11a instead of a noisy voice generation unit 11. The noisy whisper voice generation unit 11a has a whisper voice generation unit 112 in addition to the voice mixing unit 111.

[0039] The learning process of the voice conversion device 10 will be explained in accordance with Figure 4.

[0040] [Noisy whispering voice generation unit 11a] The whispering voice generation unit 112 generates a clean normal voice x ｎｏｒｍａｌ The input is used to generate a simulated whispering voice through signal processing, and the whispering voice x ｗｈｉｓｐｅｒ The whispering voice generation unit 112 can be implemented using simple signal processing that does not rely on machine learning, as disclosed in Reference 2 and Reference 3. Note that normal voice x ｎｏｒｍａｌ This can be the same as audio x in the first embodiment (Figure 2).

[0041] The audio mixing unit 111 mixes whispered audio x ｗｈｉｓｐｅｒ Input noise n and correct content c, and a noisy whispering voice y ｗｈｉｓｐｅｒ Outputs.

[0042] Separately, the speaker vector calculation unit 14 calculates the whispered voice x ｗｈｉｓｐｅｒ Enter the speaker vector s of the whispered voice. ｗｈｉｓｐｅｒ Calculate.

[0043] Furthermore, the speaker vector calculation unit 14 calculates the normal speech x ｎｏｒｍａｌ Input the speaker vector s for normal speech. ｎｏｒｍａｌ Calculate.

[0044] [Voice quality conversion model 12] The content prediction model 121 is a noisy whispering voice y ｗｈｉｓｐｅｒ Speaker vector s of whispered voice ｗｈｉｓｐｅｒ The program takes the input and outputs the predicted content c' based on the learning parameters.

[0045] Next, the audio waveform prediction model 122 uses the predicted content c' and normal audio x ｎｏｒｍａｌ The input is a normal speech x' predicted based on the learning parameters. ｎｏｒｍａｌOutputs.

[0046] [Voice quality conversion model learning unit 13] Predicted speech x' is the predicted normal speech x' ｎｏｒｍａｌ Except for the fact that it is replaced by the first embodiment, it is the same as the first embodiment.

[0047] Next, we will explain the reasoning behind voice quality conversion in whispered speech.

[0048] Figure 5 shows an example of the functional configuration of the voice conversion device 10 during voice conversion inference for whispered speech in the second embodiment. In Figure 5, the same parts as in Figure 4 are denoted by the same reference numerals, and their descriptions are omitted as appropriate. In Figure 5, the voice conversion device 10 includes a voice conversion model 12 and a speaker vector calculation unit 14. The voice conversion model 12 in Figure 5 is a voice conversion model that has been learned by the learning process described in Figure 4.

[0049] The flow of the voice conversion device 10 during inference will be explained with reference to Figure 5. The difference from the inference in the first embodiment (Figure 3) is that the input to the voice conversion model 12 is a noisy source speaker voice y ｓｏｕｒｃｅ The difference is that instead, it uses noisy source speaker whispers ysource,whisper. Also, the speaker vector calculation unit 14 takes clean source speaker whispers xsource,whisper as input and outputs source speaker vectors ssource,whisper. The content prediction model 121 takes source speaker vectors ssource,whisper based on the clean source speaker whispers xsource,whisper as input. Even for the same speaker, the speaker vector differs greatly between normal speech and whispering (whisper almost completely eliminates the speaker's identity). This is because the content prediction model 121 uses the speaker vector of the person you want to listen to as a key, so when the input is whispering, the speaker vector also needs to be generated based on whispering.

[0050] Other processing may be the same as in the first embodiment.

[0051] As described above, according to the second embodiment, robust voice quality conversion is possible even for whispered speech containing noise.

[0052] Reference 2 discloses a technology for converting whispered speech to normal speech, but it focuses only on operation in quiet environments. Whispered speech is more difficult to speak louder than normal speech, resulting in a poor signal-to-noise ratio (SNR) and significantly degrading conversion performance. Furthermore, because the SSL model is used as the content prediction model 121 for inference, streaming operation is not possible, and the audio cannot be converted until the end of the input audio is reached, which is another challenge.

[0053] According to the second embodiment, it operates robustly even in environments with a poor signal-to-noise ratio, and since it does not use an SSL model for inference, streaming operation is possible.

[0054] In each of the above embodiments, voice x or normal voice x ｎｏｒｍａｌ This is an example of the first type of sound: a noisy voice y or a noisy whispering voice y. ｗｈｉｓｐｅｒ This is an example of a second voice. The content prediction model 121 is an example of a first machine learning model. The voice waveform prediction model 122 is an example of a second machine learning model. The content prediction error is an example of a first error. The voice prediction error is an example of a second error. The source speaker is an example of a first speaker. The target speaker is an example of a second speaker. The voice quality conversion device 10 during learning is an example of a voice quality conversion learning device.

[0055] [References] [Reference 1] T. Moriya, H. Sato, T. Ochiai, M. Delcroix, T. Shinozaki, "Streaming target-speaker ASR with neural transducer", Proc. Interspeech, pp. 2673-2677, 2022. [Reference 2] J. Rekimoto, "WESPER: Zero-shot and Realtime Whisper to Normal Voice Conversion for Whisper-based Speech Interactions", Proc. CHI, pp. 1-12, [Reference 3] toWhisper, Internet, <URL: https: / / github.com / zeta-chicken / toWhisper.git> [Reference 4] R. Yamamoto, et. al., "Parallel Wavegan: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram", Proc. ICASSP, pp. 6199-6203, 2020. [Reference 5] H. Kanagawa, et. al., "Knowledge distillation from self-supervised representation learning model with discrete speech units for any-to-any streaming voice conversion", Proc. Interspeech, pp. 4393-4397, 2024. Although embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention as described in the claims.

[0056] 10 Voice quality conversion device 11 Noisy speech generation unit 11a Noisy whispering speech generation unit 12 Voice quality conversion model 13 Voice quality conversion model learning unit 14 Speaker vector calculation unit 100 Drive device 101 Recording medium 102 Auxiliary storage device 103 Memory device 104 Processor 105 Interface device 111 Speech mixing unit 112 Whispering speech generation unit 121 Content prediction model 122 Speech waveform prediction model 131 Content prediction error calculation unit 132 Speech prediction error calculation unit 133 Error minimization unit B Bus

Claims

1. A voice quality conversion learning device comprising: a learning unit configured to learn the first machine learning model and the second machine learning model based on a first machine learning model that takes a speaker vector calculated based on a first voice and a second voice mixed with noise from the first voice as input, a first error between the predicted value of the speech content output by the first machine learning model and the correct value of the speech content of the first voice, and a second error between the predicted voice output by the second machine learning model that takes the predicted value and the speaker vector as input and the first voice and the first voice.

2. A voice conversion device comprising: a first machine learning model that inputs a speaker vector calculated based on noise-free speech uttered by a first speaker and noise-inclusive speech uttered by the first speaker, and outputs a predicted value of the speech content; and a second machine learning model that inputs a speaker vector calculated based on noise-free speech uttered by a second speaker and the predicted value, and outputs a predicted speech, wherein the first and second machine learning models are trained based on a first error with respect to the correct value of the speech content of the first speech for the predicted value output by the first machine learning model, which is input to the speaker vector calculated based on the first speech and the second speech in which noise is mixed with the first speech, and a second error with respect to the first speech for the predicted speech output by the second machine learning model, which is input to the predicted value and the speaker vector.

3. A voice quality conversion learning device comprising: a learning unit configured to learn the first machine learning model and the second machine learning model based on a first machine learning model that takes input of a speaker vector calculated based on a first voice converted into a whisper and a second voice in which noise is mixed with the first voice, and a first error between the predicted value of the speech content output by the first machine learning model and the correct value of the speech content of the first voice, and a second error between the predicted voice output by the second machine learning model that takes input of the predicted value and a speaker vector calculated based on the first voice and the first voice, and the first machine learning model.

4. A voice conversion device comprising: a first machine learning model that inputs a speaker vector calculated based on a noise-free whispered voice uttered by a first speaker and a noise-inclusive whispered voice uttered by the first speaker, and outputs a predicted value of the utterance content; and a second machine learning model that inputs a speaker vector calculated based on a noise-free voice uttered by a second speaker and the predicted value, and outputs a predicted voice, wherein the first and second machine learning models are trained based on a first error with respect to the correct value of the utterance content of the first voice for the predicted value output by the first machine learning model, which is input to a speaker vector calculated based on a voice converted from the first voice into a whisper and a second voice in which noise is mixed with the first voice; and a second error with respect to the first voice for the predicted voice output by the second machine learning model, which is input to the predicted value and a speaker vector calculated based on the first voice.

Images