Learning device, learning method, and program

Abstract

Provided is a learning device comprising a control unit that performs a first learning process for training a first mathematical model as a training target, wherein: the first mathematical model performs a speech feature sequence calculation process, an encoding process, a bottleneck feature sequence conversion process, a decoding process, and a speech waveform generation process so as to generate a speech waveform on the basis of input speech; the speech feature sequence calculation process calculates a speech feature sequence from the input speech; the encoding process uses an encoder to encode the speech feature sequence; the bottleneck feature sequence conversion process converts an output of the encoder by a reverse diffusion process of a probabilistic diffusion model; the decoding process uses a decoder to decode the result of conversion by the bottleneck feature sequence conversion process; and the speech waveform generation process uses a vocoder to generate the speech waveform from the result of the decoding process.

Description

Learning device, learning method, and program

[0001] This invention relates to a learning device, a learning method, and a program.

[0002] Many conventional voice conversion technologies have limitations, namely the need to prepare pairs of voice samples (parallel data) of the source speaker and target speaker who uttered the same sentence as training data, and the need for the source speaker's voice data to be included in the training data.

[0003] In contrast, VoiceGrad, a voice conversion technology based on score-based generative models or diffusion probability models, is characterized by its ability to learn from non-parallel data and to perform voice conversion with high accuracy even for the voices of arbitrary speakers.

[0004] VoiceGrad is an approach that views the process of converting the feature sequence of source speech into a feature sequence that has a voice quality and speech style similar to the target speech as a despreading process in a diffusion stochastic model, and is a voice quality conversion technology based on the following ideas 1 and 2.

[0005] Idea 1: Train a neural network to estimate the score function (gradient of a logarithmic distribution) of the feature sequence of the target audio, or the quantity for which it can be calculated.

[0006] Idea 2: Using the feature sequence of the source audio as the initial point, the feature sequence of the target audio is iteratively changed based on the estimated score function.

[0007] One advantage of employing score-based generative models or spreading probability models for voice conversion is the flexibility to adjust the conversion process to meet the diverse needs of users. This advantage arises because, when there are other features that the converted speech should satisfy, the final result can be adjusted by adjusting the update direction during each time step of the despreading process using some function (e.g., an independently pre-trained classifier or other spreading probability model) that gives a higher score when the converted speech matches those features.

[0008] Therefore, using VoiceGrad eliminates the need to retrain the model each time to meet user needs, and allows for customization of converted speech by combining it with a pre-trained model.

[0009] In addition to VoiceGrad, several diffusion-stochastic model-based voice conversion methods, such as Diff-VC, have been proposed in recent years. Diff-VC takes an approach where a pre-trained speech encoder first converts the input mel spectrogram (a type of speech feature sequence) into a "average voice" mel spectrogram, and then uses this as an initial point to convert it into the target speaker's mel spectrogram through the despreading process of a diffusion-stochastic model.

[0010] VoiceGrad differs from Diff-VC in that it directly uses the sequence of speech features, such as the Mel spectrogram of the input speech, as the initial point of the despreading process. The speech feature sequence is a sequence of speech features.

[0011] International Publication No. 2022 / 101967

[0012] Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, Nobukatsu Hojo, Shogo Seki, "VoiceGrad: Non-Parallel Any-to-Many Voice Conversion with Annealed Langevin Dynamics," arXiv:2010.02977 [cs.SD], Oct. 2020.

[0013] Thus, VoiceGrad offers superior performance compared to conventional technologies, and its processing time is also shorter. However, further improvements in processing time are desired. In other words, even faster voice conversion is needed.

[0014] In view of the above circumstances, the present invention aims to provide a technology for accelerating voice quality conversion.

[0015] One aspect of the present invention is a learning device comprising a control unit that performs a first learning process, which is a process of learning with a first mathematical model as the learning target, wherein the first mathematical model is a mathematical model that generates an audio waveform based on input audio by performing an audio feature sequence calculation process, an encoding process, a bottleneck feature sequence transformation process, a decoding process, and an audio waveform generation process, wherein the audio feature sequence calculation process is a process of calculating an audio feature sequence from the input audio, the encoding process is a process of encoding the audio feature sequence using an encoder, the bottleneck feature sequence transformation process is a process of transforming the output of the encoder by the inverse diffusion process of a probability diffusion model, the decoding process is a process of decoding the result of the transformation by the bottleneck feature sequence transformation process using a decoder, and the audio waveform generation process is a process of generating an audio waveform using a vocoder from the result of the decoding process.

[0016] One aspect of the present invention is a learning method comprising a control step in which a computer performs a first learning process, which is a process of learning with a first mathematical model as the learning target, wherein the first mathematical model is a mathematical model that generates an audio waveform based on input audio by performing an audio feature sequence calculation process, an encoding process, a bottleneck feature sequence transformation process, a decoding process, and an audio waveform generation process, wherein the audio feature sequence calculation process is a process of calculating an audio feature sequence from the input audio, the encoding process is a process of encoding the audio feature sequence using an encoder, the bottleneck feature sequence transformation process is a process of transforming the output of the encoder by the inverse diffusion process of a probability diffusion model, the decoding process is a process of decoding the result of the transformation by the bottleneck feature sequence transformation process using a decoder, and the audio waveform generation process is a process of generating an audio waveform using a vocoder from the result of the decoding process.

[0017] One aspect of the present invention is a program for causing a computer to function as the learning device described above.

[0018] This invention makes it possible to speed up voice quality conversion.

[0019] An explanatory diagram illustrating the conversion system of the embodiment. A diagram showing algorithm 1 in the embodiment. A diagram showing algorithm 2 in the embodiment. A diagram showing algorithm 3 in the embodiment. A diagram showing algorithm 4 in the embodiment. A first diagram showing an example of experimental results in the embodiment. A second diagram showing an example of experimental results in the embodiment. A third diagram showing an example of experimental results in the embodiment. A fourth diagram showing an example of experimental results in the embodiment. A diagram showing an example of the hardware configuration of the learning device in the embodiment. A flowchart showing a first example of the processing flow executed by the learning device in the embodiment. A flowchart showing a second example of the processing flow executed by the learning device in the embodiment. A flowchart showing a third example of the processing flow executed by the learning device in the embodiment. A diagram showing an example of the hardware configuration of the conversion device in the embodiment. A flowchart showing a first example of the processing flow executed by the conversion device in the embodiment. A flowchart showing a second example of the processing flow executed by the conversion device in the embodiment. A flowchart showing a third example of the processing flow executed by the conversion device in the embodiment.

[0020] (Embodiment) Figure 1 is an explanatory diagram illustrating the conversion system 100 in an embodiment. However, before describing the conversion system 100 in this embodiment, we will first explain the diffusion probability model and VoiceGrad. The diffusion probability model and VoiceGrad are well-known technologies. The explanation of the diffusion probability model and VoiceGrad here is intended to facilitate understanding of the conversion system 100.

[0021] <Diffusion Probabilistic Model> This section outlines the principle of the Diffusion Probabilistic Model (DPM). A data sample normalized to a mean of 0 and a variance of 1 is given x ０ Therefore, the diffusion process of DPM is x ０ Gradually add Gaussian noise to (x ０ It is defined as a Markov chain (which gradually spreads).

[0022] Based on this diffusion process assumption, from noise to x ０Finding the parameters θ of the reverse diffusion process model that gradually reconstructs is the learning objective of the DPM. Usually, DPM refers to a latent variable model represented by the following equation (1).

[0023]

[0024] However, x ｉ：ｊ represents the set {x ｉ , …, x ｊ}, and x １ , …, x Ｌ represent latent variables corresponding to the diffused version of x ０ . Here, l = 1, …, L represents the time steps of the diffusion process. Here, when x ０ is given, the joint distribution q(x １：Ｌ |x ０ ) is called the diffusion process. And. This joint distribution is represented by the Markov chain of the following equation (2).

[0025]

[0026] Also, each conditional distribution q(x ｌ [ |x ｌ－１ ) is given by the following equation (3).

[0027]

[0028] From this, the conditional distribution q(x ｌ |x ｌ－１ ) is interpreted as a process of multiplying x ｌ－１ by the square root of (1 - β １ ) and then adding Gaussian noise with variance β ｌ to obtain x_l. This is why the joint distribution q(x １：Ｌ |x ０ ) is called the diffusion process.

[0029] The scaling by the square root of (1 - β １ ) serves to keep the variance of x ｌ at 1 at each time step. From the fact that the sum of Gaussian noises is also Gaussian noise, according to the definition of equation (3), the conditional distribution q(x ｌ [ |x ０ ) at any time step 1 is represented by the following equation (4). Note that αｌ = 1 - β ｌ That is the case.

[0030]

[0031]

[0032] Hereafter, the symbol on the left side of equation (5) will be denoted as \bar{α_l}.

[0033] Here, if we denote the Gaussian noise following a standard normal distribution as ε, equation (4) can be rewritten as equation (6) below. Note that the standard normal distribution is denoted by the symbol N(0,I). N(0,I) represents a normal distribution with mean 0 and variance I.

[0034]

[0035] joint distribution p θ (x ０：Ｌ This is called the backdiffusion process. The simultaneous distribution p θ (x ０：Ｌ ) is x Ｌ Assuming that it follows a standard normal distribution, it can be represented by the Markov chain in equation (7) below.

[0036]

[0037] Here, each conditional distribution p θ (x ｌ－１ :x ｌ ) is expressed by the following equation (8), where the mean is μ θ (x ｌ ,l), variance is v ｌ ２ Assume a Gaussian distribution, where μ θ (x ｌ ,l) is x ｌ This represents a model of a predetermined form that takes l as input, and θ represents its parameter.

[0038]

[0039] Based on equation (8), each step of the backdiffusion process is expressed by the following equation (9). Since ε is Gaussian noise following a standard normal distribution as described above, in other words, it is a random variable following a standard normal distribution.

[0040]

[0041] In DPM, the above μ θ (x ｌ The following variable transformation (10) is performed on l) and then ε θ (x ｌ The process involves modeling the function (i) using a neural network with parameter θ.

[0042]

[0043] The learning objective of DPM is to determine the parameter θ under the above assumptions and model.

[0044] By the way, the learning criteria for this type of learning is -log(p θ (x ０ The variational upper bound of the expected value of )) may be used. -log(p θ (x ０ The variational upper bound of the expected value of )) is given by the following equation (11).

[0045]

[0046] This variational upper bound L ＤＰＭ (θ) is p θ (x １：Ｌ | x ０ ) and q(x １：Ｌ | x ０ -log(p) when the Kullback-Leibler (KL) divergence between ) and ) is zero. θ (x ０ This matches the expected value of ). Therefore, L ＤＰＭ The process of minimizing (θ) with respect to θ is p θ This is a process of fitting the data distribution, and p θ (x １：Ｌ | x ０ ) and q(x １：Ｌ | x ０ This is also a process to bring it as close as possible to ). The derivation is omitted, but using equation (6) and equation (10), equation (11) can be rewritten as equation (12) below.

[0047]

[0048] However, c ｌ is, αｌ , \bar{α_l} and v ｌ It is a constant determined by, but for simplicity, c ｌ Let be 1. If θ is learned according to the above criteria, p θ Samples following (x) are generated by a Markov chain in the backdiffusion process (sequential sampling according to equation (9)).

[0049] <VoiceGrad> This section provides an overview of VoiceGrad. In VoiceGrad, the speech feature sequence of the source speech is considered to be a spread version of the speech feature sequence of the target speech. In VoiceGrad, speech feature sequence transformation is performed by executing a despreading process from an appropriate step L', using the speech feature sequence of the source speech as the initial point.

[0050] The configuration of VoiceGrad may be, for example, one in which a score network is prepared for each target speaker. The configuration of VoiceGrad may not be one in which a score network is prepared for each target speaker. Such a configuration may be one in which, in addition to the noise level l, the speaker embeddings s of the target speaker are also input, so that a single score network can be used to convert to any speaker. As speaker embeddings s, one-hot vectors corresponding to speaker numbers, or output vectors of speaker encoders trained for speaker authentication, for example, can be used.

[0051] The VoiceGrad architecture may, for example, include a configuration in which a phoneme embedding sequence p is input, in order to avoid losing linguistic information of the input speech during the feature transformation process. As the phoneme embedding sequence p, for example, an intermediate feature sequence from an end-to-end speech recognition system may be used. Hereinafter, the model for extracting the phoneme embedding sequence p will be referred to as a phoneme encoder.

[0052] These conditioned s θIf (x, l, s, p) is trained using multi-speaker speech data, it becomes possible to transform the speech feature sequence of the input speech into a speech feature sequence that has a voice quality or speech style similar to that of a reference speaker, through a despreading process. This transformation is performed using the phoneme embedding sequence p of the input speech and the speaker embedding s of the reference speaker's speech, which is the target of the transformation, as clues.

[0053] s θ As a learning criterion for (x, l, s, p), for example, the criterion expressed by equation (13) below may be used. Equation (13) is a criterion devised based on equation (12).

[0054]

[0055] Also, s θ As a learning criterion for (x, l, s, p), for example, the criterion expressed by equation (14) below may be used. Equation (14) is obtained by replacing the squared error with the absolute error in equation (13).

[0056]

[0057] However, x ｏ ~p(x|k) represents the speech feature sequence of a training sample randomly selected from speaker k. Also, s and p are x ０ These are speaker embeddings and phoneme embedding sequences obtained from [source]. Also, the symbols in equation (15) below represent the acquisition of expected values, so they may be replaced with symbols in equation (16) below, for example, or they may represent the operation of sampling l and k appropriately and then taking the average.

[0058]

[0059]

[0060] The speaker encoder and phoneme encoder used to extract s and p during training are denoted as SpeakerEncoder(•) and PhonemeEncoder(•), respectively, and the respective speech feature sequences of the input speech and the target speaker's reference speech are denoted as x and x. ref Therefore, the VoiceGrad speech conversion process is, for example, the process of executing algorithm 1 shown in Figure 2.

[0061] Figure 2 shows algorithm 1 in an embodiment. L' represents the noise level at which the despreading process begins, and h represents the vocoder that generates the audio waveform from the audio feature sequence.

[0062] <Regarding the speech features to be converted by VoiceGrad> Any speech features can be used as the speech features to be converted by VoiceGrad, as long as they are sufficient to ultimately construct a speech waveform, and several options are possible. In recent years, many high-quality neural vocoders such as WaveNet and HiFi-GAN have been proposed, and one option is to use a speech feature sequence that is intended for use with these. Since the neural vocoders proposed so far often use mel spectra for short intervals as features, it is conceivable to use a sequence of mel spectra (mel spectrograms) as the speech feature sequence. Alternatively, it is also possible to construct a waveform from a sequence of amplitude spectra (amplitude spectrograms) obtained by the Short-Time Fourier Transform (STFT), and the amplitude spectrograms can also be used as the speech feature sequence.

[0063] <β in DPM ｌ Regarding the decision: Noise-dispersive scheduling > DPM in {β l} l is a hyperparameter that can be set arbitrarily. It can be determined arbitrarily, but it may be determined by a method called noise dispersion scheduling, for example. In noise dispersion scheduling, \bar{α_l} is first determined based on the scheduling represented by the following equation (17), and then β ｌ This is determined according to the relationship in equation (18) below.

[0064]

[0065]

[0066] In noise-dispersive scheduling, β should not be set too close to 1. ｌ This may be clipped. This is, for example, βｌ β is set so that it is below an appropriate threshold (e.g., 0.999). ｌ This is achieved by clipping. Note that η is β ｌ This is an offset value to prevent it from becoming too small when l = 0; for example, in the experiment described later, η = 0.008.

[0067] <Network Structure of the Score Network> The architecture of the score network is not restricted to any particular form other than the input and output being isomorphic, and can be designed arbitrarily. For example, a U-Net architecture with a full-layer convolutional network may be adopted. The noise level l can be incorporated into the network by, for example, representing l as a one-hot vector or a vector representation using positional encoding, repeating this vector in the time direction until it is long enough to fit the input of the convolutional layer, and then connecting the repeated vector sequence to the input along the channel direction.

[0068] Speaker embeddings s may be incorporated into the network in a similar manner (s is repeated in the time direction until it is long enough to fit the input to the convolutional layer, and the repeated vector sequence is coupled to the input along the channel direction). Phoneme embeddings p are assumed to be vector sequences. For example, the sequence length may be linearly stretched to fit the input to the convolutional layer, and the stretched vector sequence may be coupled to the input along the channel direction.

[0069] This concludes the overview of the well-known diffusion probability model and VoiceGrad. Now, let's return to the explanation of Figure 1 and describe the conversion system 100.

[0070] <Conversion System 100> The conversion system 100 comprises a learning apparatus 1 and a conversion apparatus 2. The learning apparatus 1 includes a control unit 11 which comprises a processor 91 such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or NPU (Neural Network Processing Unit) connected by a bus, and a memory 92.

[0071] <<First Learning Process>> The control unit 11 executes, for example, the first learning process. The first learning process is a process that performs learning with the first mathematical model as the learning target. The first mathematical model is a mathematical model that generates an audio waveform based on the input audio, which is the audio input to the model. The first mathematical model generates an audio waveform based on the input audio by executing an audio feature sequence calculation process, an encoding process, a bottleneck feature sequence transformation process, a decoding process, and an audio waveform generation process.

[0072] The audio feature sequence calculation process is the process of calculating an audio feature sequence from the input audio. Image G1 in Figure 1 represents an example of an audio feature sequence, specifically an example of a Mel spectrogram.

[0073] The encoding process is the process of encoding the audio feature sequence obtained by the audio feature sequence calculation process using an encoder. Hereafter, the output of this encoder (i.e., the result of encoding) will be called the bottleneck feature sequence. In Figure 1, "Encode" represents the encoding process. In Figure 1, "f φ The symbol ' represents a mapping that expresses the encoding process. φ is a parameter whose value is determined through learning. Image G2 in Figure 1 represents an example of a bottleneck feature sequence.

[0074] The bottleneck feature sequence transformation process is a process that transforms the bottleneck feature sequence using the inverse diffusion process of a stochastic diffusion model. The process represented by the blue frame W1 in Figure 1 represents the bottleneck feature sequence transformation process. "Eq.(3)" in Figure 1 indicates that the transformation is performed by the above equation (3). Image G5 in Figure 1 shows an example of the result of the transformation by the bottleneck feature sequence transformation process. Images G3 and G4 in Figure 1 show an example of a customized process for the bottleneck feature sequence transformation process.

[0075] The decoding process is the process of decoding the result of the transformation performed by the bottleneck feature sequence transformation process using a decoder. In Figure 1, “Decode” represents the decoding process. φThe symbol " represents a mapping that expresses the decoding process. Image G6 in Figure 1 shows an example of the result of the decoding process.

[0076] Audio waveform generation is the process of generating an audio waveform using a vocoder from the results of the decoding process. The results of the decoding process are the output of the decoder used in the decoding process (i.e., the result of decoding by the decoder used in the decoding process).

[0077] <<<Autoencoder Learning Method>>> In the first learning process, an autoencoder is trained in order to obtain an encoder that obtains a bottleneck feature sequence from an audio feature sequence and a decoder used in the decoding process. Therefore, the method for training this autoencoder is described below. Note that the autoencoder here is an autoencoder that includes the above encoder and decoder. This autoencoder is, for example, an autoencoder that performs the encoding process executed by the first mathematical model and a decoder that performs the decoding process executed by the first mathematical model.

[0078] The encoder and decoder of the trained autoencoder obtained through the training of this autoencoder are, respectively, the encoder and decoder in the trained first mathematical model.

[0079] It is desirable that the autoencoder outputs a sequence of audio features that can ultimately be converted into a high-quality audio waveform. Therefore, in the first training process, the autoencoder is trained using a loss that evaluates the "realism" of the signal generated from the autoencoder's output by a pre-trained vocoder (a differentiable computational process such as a neural network), along with a well-known reconstruction loss.

[0080] This is achieved by adversarial training between an autoencoder and a waveform classifier (waveform classifier). In other words, the first training process involves adversarial training between the autoencoder and the classifier.

[0081] Specifically, in the above-described adversarial learning, the learning of the autoencoder is performed so as to output a voice feature amount series in which the vocoder finally generates a waveform as realistic as possible to be identified as true data (referred to as class 1) by the discriminator. That is, in the above-described adversarial learning, the learning of the autoencoder is performed so as to output a voice feature amount series that causes the vocoder to generate a waveform with a higher probability of being identified as true data by the discriminator.

[0082] In the above-described adversarial learning, the discriminator is learned so as to improve its ability to identify the generated waveform as fake data (referred to as class 0).

[0083] Thus, in the first learning process, the autoencoder and the waveform discriminator are adversarially learned so that the autoencoder outputs a voice feature amount series that causes the vocoder to generate a waveform with a higher probability of being identified as true data by the waveform discriminator that discriminates the waveform.

[0084] Note that the learning loss of the above-described adversarial learning includes the distance between the distribution of the bottleneck feature amounts and the standard normal distribution. In DPM, it is assumed that each element value of the data sample has an average of 0 and a variance of 1. By including the distance between the distribution of the bottleneck feature amounts and the standard normal distribution in the learning loss, this condition is also satisfied for the bottleneck feature amount series of the autoencoder.

[0085] Let the voice waveform and its voice feature amount series of the learning sample be y and x, respectively. Let the encoder and the decoder of the autoencoder with the parameter φ be g φ (·) and f φ (·), respectively. At this time, the reconstruction loss L ｒｅｃ (φ) may be the expected value of the distance between the input x to the autoencoder and the output g φ (f φ (x)) of the autoencoder. Therefore, such a loss may be, for example, the L １ loss defined by the following equation (19).

[0086]

[0087] The waveform discriminator determines whether the input waveform is that of true speech or synthesized speech. Hereinafter, the waveform discriminator with parameter ψ is denoted as d ψ (·). Note that the value of d ψ (·) is 0 or 1. Also, hereinafter, the vocoder that generates a waveform from the speech feature quantity series is denoted as h(·).

[0088] In the adversarial learning between the autoencoder and the waveform discriminator, ψ and φ are learned according to their respective criteria. As the learning criterion for ψ, any loss may be used as long as it returns a low value when the discriminator d ψ (·) can identify the waveform generated from the autoencoder output as fake data (class 0), or when it can identify the real speech sample y as real data (class 1).

[0089] On the other hand, as the learning criterion for φ, any loss may be used as long as it returns a low value when the waveform generated from the autoencoder output is identified as real data (class 1) by the discriminator d ψ (·).

[0090] As described above, the learning objectives of ψ and φ are contrary to each other. Therefore, the loss with such an intention is called the adversarial loss.

[0091] Hereinafter, the adversarial losses of φ and ψ are denoted as J ａｄｖ （ａ） (φ) and J ａｄｖ （ｄ） (ψ), respectively. These may use, for example, the learning criterion of Least Squares Generative Adversarial Networks (LSGAN). The learning criterion of LSGAN is represented by, for example, the following equations (20) and (21).

[0092]

[0093]

[0094] Note that hereinafter, the symbol represented by equation (22) described in equation (20) or equation (21) is denoted as \^{y}. \^{y} represents the waveform generated from the autoencoder output.

[0095]

[0096] Incidentally, in HiFi-GAN, which was proposed as a neural vocoder, it is known that the quality of the generated waveform can be improved by including the first vocoder loss and the second vocoder loss in the learning criteria.

[0097] The first vocoder loss is the loss that returns a lower value when the vocoder's output (generated waveform) is closer to the vocoder's input when it is re-featured. The second vocoder loss is the loss that returns a lower value when the outputs of each hidden layer in the classifier are closer to the generated waveform and the actual audio waveform as inputs.

[0098] In the first learning process, the first or second vocoder may be included in the learning criteria. Hereafter, the process of re-featuring the audio waveform (into a Mel spectrogram, etc.) is represented by m(•). Also, the classifier d ψ The output value of the ith layer of (・) is d ψ （ｉ） This is represented by (•). In this case, the first vocoder is the loss represented by equation (23), for example, and the second vocoder is the loss represented by equation (24) below.

[0099]

[0100]

[0101] However, T represents the number of layers in the classifier, and N ｉ This represents the number of output values ​​in the i-th layer (the dimension of the vector or the number of elements in the array).

[0102] The distance J between the distribution of the bottleneck feature and the standard normal distribution. ＫＬ For (φ), for example, the loss represented by the following equation (25) may be used.

[0103]

[0104] However, μ(x) is z = g φ (x) represents the average of the elements, σ ２ (x) is z = g φ This represents the variance of the elements of (x).

[0105] Thus, the learning criteria for φ are, for example, J ａｅ(φ) = λ(J) ｒｅｃ (φ) + J ｍｅｌ (φ) + J ＫＬ (φ) + J ａｄｖ （ａ） (φ) + J ｆｅａｔ (ψ) may be the case. In this case, the learning criterion for ψ is J ｄｉｓ (ψ) = J ａｄｖ （ｄ） (ψ) is the case. Note that λ is J ｒｅｃ (φ), J ｍｅｌ (φ), J ＫＬ This is the regularization weight that determines the specific gravity of (φ). In the experiment described later, λ = 45.

[0106] <<

[0107] Therefore, x ０ f φ The bottleneck feature sequence obtained by z ０ Expressed as such, the learning criteria for DPM in the first mathematical model is, for example, x in equation (13) or equation (14). ０ to z ０ This is a replacement.

[0108] Thus, the first learning process also involves learning the DPM (Probability Spread Model).

[0109] If the score network can be trained according to the above criteria, speech conversion can be performed using the flow of algorithm 2 shown in Figure 3.

[0110] Figure 3 shows algorithm 2 in the embodiment. Algorithm 2 illustrates an example of the specific processing flow involved in executing a first mathematical model that has been trained or is in the process of being trained.

[0111] Returning to the explanation of Figure 1, the conversion device 2 includes a control unit 21 (conversion unit) which comprises a processor 93 such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or NPU (Neural Network Processing Unit) connected by a bus, and a memory 94. The control unit 21 executes the learned mathematical model obtained by the learning device 1.

[0112] The term "trained mathematical model" is a commonly used term in the field of machine learning, and refers to the mathematical model at the point when a predetermined condition for the end of training (hereinafter referred to as the "training termination condition") is met. The training termination condition can be any condition relating to the end of training, for example, the condition that the number of updates to the mathematical model due to training has reached a predetermined number. The training termination condition may also be, for example, the condition that the change in the mathematical model due to updates is less than a predetermined change.

[0113] The control unit 21 performs voice quality conversion on the target voice quality by executing the learned mathematical model obtained by the learning device 1. Specifically, the target voice quality is speech. Therefore, voice quality conversion on the target voice quality means converting the voice quality of the speech set as the target voice quality conversion.

[0114] Therefore, when the learning device 1 obtains a first mathematical model that has been learned, the control unit 21 performs the voice quality conversion of the voice quality to be converted by executing the learned first mathematical model.

[0115] <<Other Learning Examples>> The control unit 11 may execute the second or third learning processes described later instead of the first learning process. To facilitate understanding of the second and third learning processes, the FM (Flow Matching) model will be explained first.

[0116] <<<<Effects of using the FM model>>> In DPM, as the number of time steps L approaches infinity, the diffusion process can be interpreted as a stochastic differential equation (SDE). Under this interpretation, data sampling by the inverse diffusion process corresponds to the process of numerically solving the inverse time SDE or the corresponding stochastic flow ordinary differential equation (ODE).

[0117] In recent years, a new class of generative models called Flow Matching (FM) models has been proposed as an advanced version of deep generative models known as Continuous Normalizing Flow (CNF) models. Unlike DPM models, which aim to learn the score function of SDEs, FM models aim to directly learn the vector field of an arbitrary ODE.

[0118] If a neural network can be trained to infer a properly assumed vector field of the ODE, it is possible to generate data samples by numerically solving that ODE. FM models often pre-plan linear sampling trajectories and design the ODE and vector field to minimize transportation costs, which allows for more efficient sampling of high-quality data compared to DPM.

[0119] Due to these advantages of the FM model, when the FM model is used instead of DPM, the voice quality conversion performed by its mathematical model is faster than when using DPM, without compromising voice quality.

[0120] <<<Basic Principles of FM Models>>> The basic principles of FM models are explained. An unknown data distribution q １ The data sample generated from (x) is x １ Let's assume that in the FM model, as in the CNF model, x １ The ODE is expressed by the following equation (26), and x from time t=0 to t=1 ｔ The initial point x ０It is assumed that this point was obtained through time evolution.

[0121]

[0122] Let me briefly explain the notation here. In the above explanation of DPM, the data sample is x ０ , point x in step l of the diffusion process ｌ (where l is an integer) was used as notation. In the explanation of the FM model, following the notation of many literatures related to the FM model, the data sample was x １ , the point at time t is x ｔ (where t is a continuous value) is the notation. ｔ This is the vector field of this ODE.

[0123] The CNF model is u ｔ (x) is a neural network v with parameter θ θ This is based on the idea of ​​expressing it in terms of (x, t). The FM model is also based on this. If the solution to equation (26) is unique, then the point x at any time ｔ The initial point is x ０ Mapping φ from ｔ It is expressed as follows: More specifically, point x ｔ It is expressed by the following equation (27). At this time, φ ｔ This is called a flow.

[0124]

[0125] This means that v is learned by learning θ. θ If we can obtain (x, t), then u in equation (26) ｔ (x) to v θ After substituting (x, t), an appropriate predetermined initial point x ０ Using optimization techniques such as Euler's method, x １ = φ １ (x ０ This means that data sampling is possible by numerically solving the equation.

[0126] Next, we have the learning sample x １ ~q １ This explains how to learn θ given (x).

[0127] x ０ Assume that the sample is generated from a suitable known distribution such as a standard normal distribution. In this case, point x at an arbitrary time t ｔ The probability density p ｔ (Hereafter referred to as "stochastic path") is based on the formula for variable transformation of the probability density function, q ０ and φ ｔ It can be described using a suitable vector field v θ Assuming that v θ From the probability path p ｔ Whether it can be described appropriately is determined by the continuity equation expressed by the following equation (28) v θ It depends on whether the condition is met or not.

[0128]

[0129] Under this constraint, p １ = q １ The learning objective in CNF and FM models is to determine θ such that this occurs.

[0130] In the CNF model, θ is learned to directly maximize the mean log-likelihood derived from equation (28). This mean log-likelihood is expressed by the following equation (29). For each sample, p １ (x １ When calculating the derivative of the logarithm of ) with respect to θ, integration with respect to t is required each time. Therefore, this learning process is computationally expensive.

[0131]

[0132] Here, if the true vector field u ｔ If (x) is known, then v θ (x, t) and u ｔ The distance between (x) and (x) may be used as the learning criterion. Such a criterion may be, for example, the criterion expressed by the following equation (30).

[0133]

[0134] This standard L ＦＭ(θ) does not require integral calculation with respect to t when differentiating with respect to θ, and can be calculated independently for each value of t. Therefore, it is possible to learn θ more efficiently than the log-likelihood mentioned above. However, u ｔ and p ｔ It is not easy to directly design or assume this. Therefore, this standard is difficult to use directly for learning.

[0135] However, the probability path p ｔ A conditional stochastic path p is given by (x) being conditioned by an arbitrary variable c. ｔ When expressed as a mixture of (x | c), the loss function is L except for the constant term. ＦＭ This is equal to (θ). Here, the conditional stochastic path p is conditioned on an arbitrary variable c. ｔ A combination of (x | c) is a stochastic path represented by, for example, the following equation (31). Also, stochastic path p ｔ The loss function when (x) is expressed by equation (31) can be expressed, for example, by the following equation (32).

[0136]

[0137]

[0138] Note that π represents the distribution of c, and u ｔ (x | c) is p ｔ This is the vector field corresponding to (x | c). This is a fact that holds for any c. However, for example, c = (x ０ , x １ ) and p ｔ (x | x ０ , x １ ) and π(x ０ , x １ Assuming that ) is the following equations (33) and (34), then L will be explained below. ＣＦＭ (θ) becomes the evaluable learning criterion. Furthermore, in this case, the learned v θ This enables efficient data sampling using (x, t).

[0139]

[0140]

[0141] First, under the above assumptions, from equation (31), p １ (x) is expressed by the following equation (35).

[0142]

[0143] This is for each training sample x １ This is a mixture of Gaussian distributions centered around the value of σ. Therefore, the variance of the Gaussian distribution is σ. ２ When p is sufficiently small, １ and q １ This is almost identical. From equation (33), p ｔ (x | x ０ , x １ ) is when the center is x as time t progresses from 0 to 1. ０ from x １ This represents a Gaussian distribution that moves in a straight line.

[0144] If ε is a random variable following a standard normal distribution, then p ｔ (x | x ０ , x １ ) is x ｔ This means that it can be expressed by the following equation (36).

[0145]

[0146] Therefore, from equation (26), p ｔ (x | x ０ , x １ ) The corresponding vector field u ｔ (x | x ０ , x １ Specifically, this can be expressed by the following equation (37).

[0147]

[0148] Substituting equations (33), (34), and (37) into equation (32), L ＣＦＭ This can be expressed by the following equation (38).

[0149]

[0150] q ０ Any distribution can be assumed for q. Therefore, for simplicity, q ０We will explain the case where is a standard normal distribution. Unlike equation (30), equation (38) is easy to evaluate and differentiate with respect to θ.

[0151] <<

[0152] In the FM model, data sampling is performed by the ODE solution process, and the vector field network v θ The data samples are iteratively updated based on the vector field predicted. In this way, the two are conceptually similar.

[0153] The difference is that, according to equation (38), the initial point x ０ From actual data sample x １ v at all points in the neighborhood of the trajectory up to that point θ The output is x １ -x ０ The key point is that it is oriented in a certain direction. As a result, in the FM model, the trajectory of ODE becomes linear, and the ODE solution process is constructed via a more efficient path than the DPM backdiffusion process.

[0154] <<Second Mathematical Model and Third Mathematical Model>> Now, let's describe the other learning examples mentioned above. The control unit 11 may execute the second learning process instead of the first learning process. The learning target of the second learning process is the second mathematical model, not the first mathematical model. The second mathematical model differs from VoiceGrad in that it executes the FM model instead of the DPM. In other words, the second mathematical model is a mathematical model in which the DPM in VoiceGrad is replaced with the FM model.

[0155] Just as VoiceGrad allows feature sequence transformation through the despreading process of a probabilistic diffusion model using a score network conditioned on target speaker embeddings, the second mathematical model allows for more efficient feature sequence transformation through an OED solution process using a vector field network conditioned on target speaker embeddings. This can be seen by considering the similarities between the FM model and DPM described above, as well as the advantages of the FM model.

[0156] In the second mathematical model, the vector field network v θ It is conceivable to configure it so that, in addition to time t, the target speaker embedding s and the phoneme embedding sequence p are conditional inputs. Under that configuration, v θ (x ｔ , t, s, p) and (x １ -x ０ The parameter θ is learned using the expected value of the distance to ) as the learning criterion. For example, such a learning criterion may be the learning criterion of equation (39) below.

[0157]

[0158] However, k represents the speaker number. １ ~q １ (x | k) represents the speech feature sequence of an utterance randomly selected from speaker k's training samples. s is x １ This represents the speaker embedding obtained from x. １ This represents the phoneme embedding sequence obtained from [0, 1]. Furthermore, t represents a continuous value uniformly sampled from the interval [0, 1]. k is a discrete value uniformly sampled from {1, ..., K} (where K is the number of speakers included in the training data).

[0159] Furthermore, the control unit 11 may perform a third learning process instead of the first or second learning process. The learning target of the third learning process is the third mathematical model, not the first mathematical model. The third mathematical model differs from the first mathematical model in that it executes an FM model instead of a DPM model.

[0160] Therefore, the third mathematical model performs the following processes: audio feature sequence calculation, encoding, FM conversion, decoding, and audio waveform generation. The FM conversion process is the process of converting the output of the encoder using the FM model.

[0161] In the third mathematical model, instead of the audio feature sequence, the encoder output of the autoencoder is treated as data to be transformed by the FM model. In this case, x １ f φ The bottleneck feature sequence obtained by z １ Therefore, the learning criterion in the third mathematical model of the FM approach is x in equation (39). １ to z １ This is a replacement.

[0162] Thus, the second and third learning models include the FM model. As mentioned above, learning criteria can be defined for the FM model as well, such as the criterion in equation (30). Therefore, the FM model is a mathematical model that can be learned. Accordingly, the FM model is also learned in the second and third learning processes.

[0163] To facilitate understanding, the diagram shows examples of the speech conversion process for the second and third mathematical models when Euler's method is used as the ODE solution method.

[0164] Figure 4 shows algorithm 3 in the embodiment. Algorithm 3 illustrates an example of the specific processing flow involved in executing a trained or partially trained second mathematical model.

[0165] Figure 5 shows algorithm 4 in the embodiment. Algorithm 4 illustrates an example of the specific processing flow involved in executing a trained or partially trained third mathematical model.

[0166] Furthermore, experiments have confirmed that in voice quality conversion using the FM model, mixing in noise at a predetermined rate at the initial point is effective in converting the voice quality. The process of mixing in noise at a predetermined rate at the initial point can be expressed mathematically as, for example, x ← (1-γ)x + γε or z ← (1-r)z + γε (where ε represents Gaussian noise following a standard normal distribution, and γ is a predetermined value between 0 and 1). For this reason, this noise mixing is also performed in algorithms 3 and 4.

[0167] <Effects of the First Learning Process> The effects of the first learning process are explained below. VoiceGrad has so far treated Mel spectrograms as speech feature sequences, but Mel spectrograms are a highly dimensional quantity, although they are a considerably compressed representation compared to waveforms. On the other hand, the first mathematical model performs a despreading process of a stochastic diffusion model on an autoencoder bottleneck feature sequence, which is a further compressed version of the speech feature sequence. Because the dimensionality is compressed, the difficulty of predicting the score function becomes lower than in VoiceGrad, and the processing time is shortened. Therefore, the first learning process, which trains the first mathematical model, can speed up voice quality conversion.

[0168] <Regarding the effects of the second and third learning processes> When the FM model is used instead of the DPM, as described in <<<Regarding the relationship between the FM model and DPM>>>, the trajectory of the ODE becomes linear, and the ODE solution process is constructed through a more efficient path than the dediffusion process of the DPM. Therefore, the second learning process, which trains the second mathematical model using the FM model instead of the DPM, can speed up voice quality conversion. And the third learning process, which trains the third mathematical model using the FM model instead of the DPM, can also speed up voice quality conversion.

[0169] <About the Experiment> We conducted an experiment to evaluate the voice quality conversion performance of the first to third mathematical models, and will now explain the results. Hereafter, the first mathematical model will be referred to as LatentVoiceGrad-DPM, the second mathematical model as VoiceGrad-FM, and the third mathematical model as LatentVoiceGrad-FM.

[0170] <<Experimental Conditions>> The experiment involved zero-shot voice conversion between arbitrary speakers. The CSTR VCTK Corpus (version 0.92) was used as experimental data. This corpus contains voice data from 110 English speakers with various accents, who read approximately 400 sentences.

[0171] To simulate a zero-shot arbitrary speaker voice conversion scenario, utterances from 10 speakers (p238, p241, p243, p252, p261, p294, p334, p343, p360, p362) were used as test audio (source and reference audio), and utterances from the remaining 100 speakers were used as training audio. Therefore, the test had 90 combinations of source and target speakers, and for each combination, a task was performed to convert the input audio to a voice similar to the reference audio.

[0172] Furthermore, to ensure that the reference audio and source audio of the target speaker were different sentences, utterance number 003 of the test audio was used as the reference audio, and the remaining utterances were used as the source audio to be converted. The sampling frequency of all audio signals was 22.05 kHz, and an 80-dimensional Mel spectrogram was used as the audio feature sequence. A speaker-independent HiFi-GAN was used as the vocoder to generate waveforms from the Mel spectrogram.

[0173] <<Baseline Methods>> For comparison, VoiceGrad and Diff-VC, a DPM-based voice conversion method similar to VoiceGrad, were selected. These methods have been confirmed to perform at or above the level of many state-of-the-art methods in zero-shot arbitrary-speaker voice conversion using non-parallel speech corpora.

[0174] <<Model Settings>> Score network ε in Diff-VC, LatentVoiceGrad-DPM, VoiceGrad-FM and LatentVoiceGrad-FM θ and vector field network v θAll of these were implemented to have the same architecture as the VoiceGrad score network. More specifically, the Diff-VC, LatentVoiceGrad-DPM, VoiceGrad-FM, and LatentVoiceGrad-FM score network ε are all convolutional networks with a U-Net structure consisting of 1D stride (transposed) convolutional layers with gated linear units. θ and vector field network v θ It has been implemented.

[0175] However, the vector field network was designed to incorporate a vectorized version of the time parameter t obtained by sinusoidal position encoding. Specifically, the vector field network was configured to pass the encoded vector through three fully connected layers flanked by two Mish activation functions, repeat this process along the time axis, and then add it to the input of each convolutional layer.

[0176] For mathematical models using DPM instead of FM, the time step (L) was set to 20, and the start time step (L') in the despreading process was set to 18. Also, the noise dispersion {σ ｌ} and {β ｌ Regarding}, see the above-mentioned <β in DPM ｌ Regarding the determination of: The value was set to the value determined by the determination method described in "Noise-Dispersive Scheduling". Note that l is an integer between 1 and L, inclusive. Under these settings, the parameters of these networks were trained using the Adam method with a learning rate of 0.0002 and a mini-batch size of 8.

[0177] <<Objective Evaluation Indicators>> To evaluate the sound quality and clarity of the converted speech samples, the following objective indicators were used: Predicted Mean Opinion Score (pMOS) and Character Error Rate (CER) [%]. pMOS is the average of the MOS values ​​predicted for each sample by the Mean Opinion Score (MOS) prediction system. CER was calculated using the wav2vec 2.0 model. In addition, Speaker Verification Accuracy (SVA) [%] was used as an indicator to evaluate the speaker similarity between the converted speech and the target speech. This is a probability value that represents how likely each converted speech sample was spoken by the target speaker, calculated using the speaker verification system. Higher pMOS and SVA values, and lower CER values, indicate higher performance.

[0178] <<Effects of Adversarial Learning of Autoencoders>> To evaluate the effect of including adversarial loss in autoencoder learning in LatentVoiceGrad, the autoencoder was reconstructed using loss J. ｒｅｃ and KL Los Angeles ＫＬ A comparison was made between the quality of converted speech samples when trained using only the LatentVoiceGrad model and the quality of converted speech samples when trained using the full loss model, including adversarial loss.

[0179] Figure 6 is the first figure showing an example of experimental results in the embodiment. Figure 6 shows the autoencoder reconstructing loss J ｒｅｃ and KL Los Angeles ＫＬ This table shows a comparison of the quality of converted speech samples when trained using only the adversarial loss and the quality of converted speech samples of LatentVoiceGrad when trained using the total loss including adversarial loss. In Table 1, “Regular” is J ｒｅｃ＋ J ＫＬ This means the result when using the learning method, while "Adversarial" means the result when using the total loss method.

[0180] In this comparison, LatentVoiceGrad-DPM was used. Figure 6 shows that "Adversarial" performs better than "Regular" in terms of sound quality and speaker similarity.

[0181] <<Comparison with Baseline>> Figure 7 is the second figure showing an example of experimental results in the embodiment. Figure 7 shows the results of various evaluation metrics for each speech conversion by Diff-VC, VoiceGrad, LatentVoiceGrad-DPM, VoiceGrad-FM and LatentVoiceGrad-FM, and the results of various evaluation metrics for the actual speech (Ground Truth) by the target speaker. Note that "VoiceGrad-DPM" means VoiceGrad.

[0182] Figure 7 shows that VoiceGrad, LatentVoiceGrad-DPM, VoiceGrad-FM, and LatentVoiceGrad-FM all performed better than Diff-VC. Figure 7 also shows that LatentVoiceGrad-DPM and LatentVoiceGrad-FM showed high performance in pMOS, while VoiceGrad and VoiceGrad-FM showed high performance in CER. Furthermore, Figure 7 shows that LatentVoiceGrad-DPM and VoiceGrad achieved superior results in SVA compared to the others. These results indicate that LatentVoiceGrad-DPM and LatentVoiceGrad-FM excel in sound quality, while VoiceGrad and VoiceGrad-FM excel in clarity and speaker similarity.

[0183] <<Real-time factor of Mel spectrogram transformation>> Figure 8 is the third figure showing an example of experimental results in an embodiment. Figure 8 shows the real-time factor (RTF) of the computation time required for Mel spectrogram transformation in each method, where L represents the number of iterations in the dediffusion process or ODE solution process.

[0184] In LatentVoiceGrad, the processing time includes the time required to encode and decode the Mel spectrogram using an autoencoder. Similarly, in Diff-VC, the processing time includes the time required to encode the Mel spectrogram of the source audio into the Mel spectrogram of the average voice. Since the Diff-VC paper examines the cases of L=6 and L=30, the experiment to obtain the results in Figure 8 measured RTF in both cases (abbreviated as Diff-VC-6 and Diff-VC-30, respectively).

[0185] Note that in the experiment that yielded the results shown in Figure 8, the processing time does not include the time required for waveform generation by the vocoder (HiFi-GAN). Both algorithms were implemented in PyTorch, and the processing time was measured on a 2.30 GHz CPU with 32.0 GB of memory.

[0186] As shown in Figure 8, VoiceGrad (L=20) was approximately 12 times faster than Diff-VC-30 and approximately 3 times faster than Diff-VC-6. Comparing VoiceGrad and LatentVoiceGrad-DPM, Figure 8 shows that LatentVoiceGrad-DPM is slightly faster despite requiring additional processing by the autoencoder. This is because the dimensionality of the autoencoder's bottleneck feature is lower than that of the Mel spectrum, allowing for a slightly smaller score network size.

[0187] Comparing VoiceGrad and VoiceGrad-FM, the latter allows for flexible setting of the number of iterations during testing, so the processing time varies depending on the number of iterations. Figure 8 shows that when the number of iterations is the same for both VoiceGrad and VoiceGrad-FM, the processing times are almost identical. This is because the architectures of the score network and the vector field network are common.

[0188] Furthermore, when comparing VoiceGrad-FM and LatentVoiceGrad-FM, the latter was more efficient when the number of iterations was high, but its efficiency relatively decreased when the number of iterations was low. This is because LatentVoiceGrad-FM requires additional processing by an autoencoder, and this processing does not depend on the number of iterations. When L=10, the processing time of both methods was almost the same.

[0189] Figure 8 shows that LatentVoiceGrad-DPM and LatentVoiceGrad-FM (L=10) exhibit conversion performance equivalent to or better than VoiceGrad, and that their processing times are approximately 70% and 50% of those of VoiceGrad, respectively. Thus, Figure 8 shows that LatentVoiceGrad-DPM and LatentVoiceGrad-FM primarily contribute to improved conversion performance, while VoiceGrad-FM and LatentVoiceGrad-FM primarily contribute to improved processing speed.

[0190] <<Subjective Evaluation Experiment Results>> In addition to objective evaluation experiments, subjective evaluation experiments were also conducted on sound quality and speaker similarity. In the sound quality evaluation, the actual voice of the target speaker was included as a sample, and listeners were asked to evaluate the naturalness of each sample on a five-point scale: "5: Excellent", "4: Good", "3: Fair", "2: Poor", and "1: Bad".

[0191] Hereafter, this evaluation score will be referred to as qMOS. In the speaker similarity evaluation, listeners were presented with pairs of audio converted using each method and the actual audio of the target speaker, and were asked to evaluate on a four-point scale of "4: Same (sure)", "3: Same (not sure)", "2: Different (not sure)", and "1: Different (sure)". Hereafter, this evaluation score will be referred to as sMOS.

[0192] Figure 9 is the fourth figure showing an example of experimental results from the experiment. More specifically, Figure 9 shows an example of the results of a subjective evaluation experiment comparing sound quality and speaker similarity. Therefore, Figure 9 shows examples of results for qMOS and sMOS.

[0193] The results in Figure 9 show that LatentVoiceGrad-DPM and LatentVoiceGrad-FM exhibit higher performance in both qMOS and sMOS than VoiceGrad and VoiceGrad-FM. Furthermore, the results in Figure 9 show that when the substrate generation model is the FM model, qMOS and sMOS are higher than when the DPM model is used. In addition, the results in Figure 9 show that the converted samples generated by LatentVoiceGrad-FM exhibit sound quality very close to that of actual voice samples.

[0194] <Example of Hardware Configuration of Learning Device 1> Figure 10 shows an example of the hardware configuration of learning device 1 in an embodiment. Learning device 1 includes a control unit 11 and executes a program. By executing the program, learning device 1 functions as a device comprising a control unit 11, an interface unit 12, and a storage unit 13.

[0195] More specifically, the processor 91 reads the program stored in the storage unit 13 and stores the read program in the memory 92. By executing the program stored in the memory 92, the processor 91 functions as a device comprising a control unit 11, an interface unit 12, and a storage unit 13.

[0196] The control unit 11 controls the operation of each functional unit of the learning device 1. The control unit 11 executes, for example, a first learning process. The control unit 11 may, for example, execute a second learning process instead of the first learning process. The control unit 11 may, for example, execute a third learning process instead of the first and second learning processes. The control unit 11 acquires, for example, information stored in the memory unit 13. Specifically, the process of acquiring information stored in the memory unit 13 is a read operation.

[0197] The interface unit 12 includes a communication interface for connecting the learning device 1 to an external device. The interface unit 12 communicates with the external device via wired or wireless connection.

[0198] When the control unit 11 executes the first learning process, the external device is, for example, the device that transmits the learning data used in the first learning process. In such a case, the interface unit 12 obtains the learning data used in the first learning process by communicating with the device that transmits the learning data used in the first learning process. The learning data used in the first learning process is, for example, a pair of audio input to the first mathematical model (i.e., input audio) and data indicating the correct audio waveform obtained by the first mathematical model.

[0199] When the control unit 11 executes the second learning process, the external device is, for example, the device that transmits the learning data used in the second learning process. In such a case, the interface unit 12 obtains the learning data used in the second learning process by communicating with the device that transmits the learning data used in the second learning process. The learning data used in the second learning process is, for example, a pair of audio input to the second mathematical model (i.e., input audio) and data indicating the correct audio waveform obtained by the second mathematical model.

[0200] When the control unit 11 executes the third learning process, the external device is, for example, the device that transmits the learning data used in the third learning process. In such a case, the interface unit 12 obtains the learning data used in the third learning process by communicating with the device that transmits the learning data used in the third learning process. The learning data used in the third learning process is, for example, a pair of audio input to the third mathematical model (i.e., input audio) and data indicating the correct audio waveform obtained by the third mathematical model.

[0201] The external device is, for example, a converter 2. In such a case, when the control unit 11 executes the first learning process, the converter 2 can execute the first mathematical model that has been learned by the execution of the first learning process by communication via the interface unit 12.

[0202] When the control unit 11 executes the second learning process, the conversion device 2 can execute the second mathematical model obtained by the execution of the second learning process via communication through the interface unit 12.

[0203] When the control unit 11 executes the third learning process, the conversion device 2 can execute the learned third mathematical model obtained by the execution of the third learning process via communication through the interface unit 12.

[0204] The interface unit 12 may include input devices such as a mouse, keyboard, touch panel, and microphone. The interface unit 12 may also be configured as an interface for connecting these input devices to the learning device 1. In this way, the input devices of the interface unit 12 receive various information or signals to the learning device 1 via wired or wireless connections. Note that the information or signals do not necessarily have to be input to the communication interface of the interface unit 12, but may also be input to the input devices of the interface unit 12.

[0205] The interface unit 12 outputs various types of information, for example. The interface unit 12 includes, for example, a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display, as well as a speaker. The interface unit 12 may be configured as an interface for connecting these display devices or speakers to the learning device 1. Therefore, the interface unit 12 may output information indicated by information or signals input to the input device of the interface unit 12 as an image or sound.

[0206] The storage unit 13 is configured using a computer-readable recording medium such as a magnetic hard disk drive or a semiconductor memory device. The storage unit 13 stores various information related to the learning device 1. The storage unit 13 stores various information generated by the operation of the control unit 11, for example. The storage unit 13 may reside, for example, on the cloud.

[0207] Figure 11 is a flowchart showing a first example of the processing flow performed by the learning device 1 in the embodiment. The control unit 11 acquires learning data (step S101). The control unit 11 executes the first learning process (step S102). Since the first mathematical model includes an encoder and a decoder, the first learning process involves adversarial learning of the autoencoder and classifier, which include the encoder and its decoder. The first learning process also includes learning of a DPM (Probability Spread Model).

[0208] Figure 12 is a flowchart showing a second example of the processing flow performed by the learning device 1 in the embodiment. The control unit 11 acquires learning data (step S201). The control unit 11 executes the second learning process (step S202). The FM model is also trained in the second learning process.

[0209] Figure 13 is a flowchart showing a third example of the processing flow performed by the learning device 1 in the embodiment. The control unit 11 acquires learning data (step S301). The control unit 11 executes the third learning process (step S302). Since the third mathematical model includes an encoder and a decoder, the autoencoder and classifier, which include the encoder and its decoder, are also learned adversarially in the third learning process. The FM model is also learned in the third learning process.

[0210] <Example of Hardware Configuration of Conversion Device 2> Figure 14 shows an example of the hardware configuration of the conversion device 2 in the embodiment. The conversion device 2 includes a control unit 21 and executes a program. The conversion device 2 functions as a device comprising a control unit 21, an interface unit 22, and a storage unit 23 through the execution of the program.

[0211] More specifically, the processor 93 reads the program stored in the storage unit 23 and stores the read program in the memory 94. By executing the program stored in the memory 94, the processor 93 functions as a device comprising a control unit 21, an interface unit 22, and a storage unit 23.

[0212] The control unit 21 controls the operation of each functional unit of the conversion device 2. For example, the control unit 21 performs voice quality conversion of the voice quality target using the mathematical model learned by the learning device 1. Therefore, when the learning device 1 executes the first learning process, the control unit 21 performs voice quality conversion of the voice quality target using the first mathematical model that has been learned. When the learning device 1 executes the second learning process, the control unit 21 performs voice quality conversion of the voice quality target using the second mathematical model that has been learned. When the learning device 1 executes the third learning process, the control unit 21 performs voice quality conversion of the voice quality target using the third mathematical model that has been learned.

[0213] The control unit 21 retrieves information stored in the memory unit 23, for example. Specifically, the process of retrieving information stored in the memory unit 23 is a read operation.

[0214] The interface unit 22 is configured to include a communication interface for connecting the conversion device 2 to an external device. The interface unit 22 communicates with the external device via wired or wireless connection. The external device is, for example, the source device of the voice quality conversion target performed by the control unit 21. As described above, the conversion target is voice. Therefore, acquiring the conversion target means acquiring the waveform signal of the conversion target, or information indicating the waveform of the conversion target. The interface unit 22 obtains the voice quality conversion target through communication with such an external device.

[0215] The external device may be, for example, a learning device 1. In such a case, the conversion device 2 can execute the learned mathematical model obtained by the learning device 1 through communication via the interface unit 22.

[0216] The interface unit 22 may include input devices such as a mouse, keyboard, touch panel, and microphone. The interface unit 22 may also be configured as an interface for connecting these input devices to the conversion device 2. In this way, the input devices of the interface unit 22 receive various information or signals to the conversion device 2 via wired or wireless connections. Note that the various information or signals that can be input to the communication interface of the interface unit 22 do not necessarily have to be input to the communication interface of the interface unit 22, but may also be input to the input devices of the interface unit 22.

[0217] The interface unit 22 outputs various types of information, for example. The interface unit 22 is comprised of a display device such as a CRT display, liquid crystal display, or organic EL display, and a speaker. The interface unit 22 may be configured as an interface for connecting these display devices or speakers to the conversion device 2. Therefore, the interface unit 22 may output information indicated by information or signals input to the input device of the interface unit 22 as an image or sound.

[0218] The storage unit 23 is configured using a computer-readable recording medium such as a magnetic hard disk drive or a semiconductor memory device. The storage unit 23 stores various information related to the conversion device 2. The storage unit 23 stores various information generated by the operation of the control unit 21, for example. The storage unit 23 may reside, for example, on the cloud.

[0219] Figure 15 is a flowchart showing a first example of the processing flow performed by the conversion device 2 of the embodiment. The control unit 21 of the conversion device 2 acquires the object to be converted (step S401). Next, the control unit 21 executes the first mathematical model learned by the learning device 1 to convert the voice quality of the object to be converted (step S402).

[0220] Figure 16 is a flowchart showing a second example of the processing flow performed by the conversion device 2 of the embodiment. The control unit 21 of the conversion device 2 acquires the object to be converted (step S501). Next, the control unit 21 executes the second mathematical model that has been learned by the learning device 1 to convert the voice quality of the object to be converted (step S502).

[0221] Figure 17 is a flowchart showing a third example of the processing flow performed by the conversion device 2 of the embodiment. The control unit 21 of the conversion device 2 acquires the object to be converted (step S601). Next, the control unit 21 executes the third mathematical model that has been learned by the learning device 1 to convert the voice quality of the object to be converted (step S602).

[0222] In this way, the learning device 1 executes the first learning process. Therefore, as described above in <About the effects of the first learning process>, the conversion speed can be increased.

[0223] The conversion device 2, configured in this way, performs voice quality conversion by executing the first mathematical model learned by the learning device 1, thereby enabling faster conversion speeds.

[0224] The conversion system 100 configured in this way includes a learning device 1 that performs a first learning process. Therefore, the conversion speed can be increased.

[0225] In this way, the learning device 1 executes the second learning process. Therefore, as described above in "<Effects of the second and third learning processes>", the conversion speed can be increased.

[0226] The conversion device 2, configured in this way, performs voice quality conversion by executing the second mathematical model learned by the learning device 1, thereby enabling faster conversion speeds.

[0227] The conversion system 100 configured in this way includes a learning device 1 that performs a second learning process. Therefore, the conversion speed can be increased.

[0228] In this way, the learning device 1 executes the third learning process. As a result, as described above in "<Effects of the second and third learning processes>", the conversion speed can be increased.

[0229] The conversion device 2, configured in this way, performs voice quality conversion by executing the third mathematical model learned by the learning device 1, thereby enabling faster conversion speeds.

[0230] The conversion system 100 configured in this way includes a learning device 1 that performs a third learning process. Therefore, the conversion speed can be increased.

[0231] (Variation) The learning device 1 may be implemented using multiple information processing devices connected to each other via a network. In this case, each process executed by the control unit 11 may be performed by multiple information processing devices in a distributed manner.

[0232] The conversion device 2 may be implemented using multiple information processing devices connected to each other via a network. In this case, each process executed by the control unit 21 may be performed by multiple information processing devices in a distributed manner.

[0233] Furthermore, all or part of the functions of the learning device 1 and the conversion device 2 may be implemented using hardware such as ASICs (Application Specific Integrated Circuits), PLDs (Programmable Logic Devices), and FPGAs (Field Programmable Gate Arrays). The program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, magnetic disks, magneto-optical disks, optical disks (CD-ROMs, DVD-ROMs, etc.), portable media such as semiconductor memory (volatile memory, non-volatile memory, etc.) (ROM, RAM, etc.), and storage devices such as hard disks built into computer systems. The program may also be transmitted via a telecommunications line.

[0234] Although embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention.

[0235] 100...Conversion system, 1...Learning device, 2...Conversion device, 11...Control unit, 12...Interface unit, 13...Storage unit, 21...Control unit, 22...Interface unit, 23...Storage unit, 91...Processor, 92...Memory, 93...Processor, 94...Memory

Claims

1. A learning device comprising: a control unit that performs a first learning process, which is a process of performing learning with a first mathematical model as the target of learning; the first mathematical model is a mathematical model that generates an audio waveform based on input audio by performing an audio feature sequence calculation process, an encoding process, a bottleneck feature sequence transformation process, a decoding process, and an audio waveform generation process; the audio feature sequence calculation process is a process of calculating an audio feature sequence from the input audio; the encoding process is a process of encoding the audio feature sequence using an encoder; the bottleneck feature sequence transformation process is a process of transforming the output of the encoder using the inverse diffusion process of a probability diffusion model; the decoding process is a process of decoding the result of the transformation by the bottleneck feature sequence transformation process using a decoder; and the audio waveform generation process is a process of generating an audio waveform using a vocoder from the result of the decoding process.

2. The learning apparatus according to claim 1, wherein the learning process involves adversarial training of the autoencoder and the waveform classifier so that the autoencoder, which includes the encoder and the decoder, outputs a sequence of audio features that causes the vocoder to generate a waveform that has a higher probability of being identified as true data by a waveform classifier that identifies waveforms.

3. A learning method comprising: a control step in which a computer performs a first learning process, which is a process of learning with a first mathematical model as the learning target; the first mathematical model is a mathematical model that generates an audio waveform based on input audio by performing an audio feature sequence calculation process, an encoding process, a bottleneck feature sequence transformation process, a decoding process, and an audio waveform generation process; the audio feature sequence calculation process is a process of calculating an audio feature sequence from the input audio; the encoding process is a process of encoding the audio feature sequence using an encoder; the bottleneck feature sequence transformation process is a process of transforming the output of the encoder using the inverse diffusion process of a probabilistic diffusion model; the decoding process is a process of decoding the result of the transformation by the bottleneck feature sequence transformation process using a decoder; and the audio waveform generation process is a process of generating an audio waveform using a vocoder from the result of the decoding process.

4. A program for causing a computer to function as a learning device according to claim 1.

Images