Information processing method, information processing device, and program

Abstract

The present disclosure relates to an information processing method, an information processing device, and a program that make it possible to improve the performance of a downstream task. Provided is an information processing method comprising inputting music data to a music-based model trained using a data set and extracting a plurality of intermediate feature quantities having different information granularities from an intermediate layer of the music-based model, and training a model of a downstream task of the music-based model using the extracted intermediate feature quantities. The present disclosure can be applied, for example, when training a model of a downstream task including music tagging, music transcription, music source separation, and music mixing.

Description

Information Processing Method, Information Processing Apparatus, and Program

[0001] The present disclosure relates to an information processing method, an information processing apparatus, and a program, and particularly relates to an information processing method, an information processing apparatus, and a program that can improve the performance of downstream tasks.

[0002] In recent years, research on foundation models that are pre-trained using large-scale and extensive datasets and can be applied to various downstream tasks has been actively conducted (see, for example, Patent Document 1). In addition, music foundation models applicable to various tasks related to music have been proposed.

[0003] Japanese Patent Application Laid-Open No. 2025-079533

[0004] When applying a music foundation model to a downstream task, it is required to improve the performance of the downstream task.

[0005] The present disclosure has been made in view of such a situation and aims to improve the performance of downstream tasks.

[0006] An information processing method according to one aspect of the present disclosure includes: an information processing apparatus inputting music data into a music foundation model learned using a dataset, and extracting a plurality of intermediate feature amounts having different information granularities from an intermediate layer of the music foundation model; and learning a model of a downstream task of the music foundation model using the extracted intermediate feature amounts.

[0007] An information processing apparatus according to one aspect of the present disclosure includes: a first processing unit that inputs music data into a music foundation model learned using a dataset, and extracts a plurality of intermediate feature amounts having different information granularities from an intermediate layer of the music foundation model; and a second processing unit that learns a model of a downstream task of the music foundation model using the extracted intermediate feature amounts.

[0008] One aspect of this disclosure is a program that causes a computer to function as a first processing unit that inputs music data into a music-based model trained using a dataset and extracts a plurality of intermediate features having different information granularities from the intermediate layer of the music-based model, and a second processing unit that uses the extracted intermediate features to train a model for a downstream task of the music-based model.

[0009] In one aspect of the information processing method, information processing device, and program of this disclosure, music data is input to a music-based model trained using a dataset, multiple intermediate features with different information granularities are extracted from the intermediate layer of the music-based model, and a model for a downstream task of the music-based model is trained using the extracted intermediate features.

[0010] Furthermore, the information processing device, which is one aspect of this disclosure, may be an independent device or an internal block constituting a single device.

[0011] A diagram showing a configuration example of an embodiment of an information processing apparatus to which the present disclosure is applied. A diagram showing an example of a method of utilizing intermediate feature amounts extracted from a music base model of the present disclosure. A diagram showing an example of the architecture of a first stage model. A diagram showing an example of the architecture of a second stage model. A diagram showing an example of attention-based feature aggregation and token out data augmentation. A diagram showing an example of a benchmark result of intermediate feature amounts in a music tagging task. A diagram showing an example of an evaluation result of feature probing using a shallow backend network in MAPS. A diagram showing an example of an F1 score result in MAPS. A diagram showing an example of the architecture of a linear music transcription model for a piano. A diagram showing a first example of a feature alignment scheme of various models in music transcription. A diagram showing a second example of a feature alignment scheme of various models in music transcription. A diagram showing an example of a modified hFT transformer that accepts intermediate feature amounts of a music base model or MusicGen feature amounts as additional inputs. A diagram showing an example of an evaluation result regarding MUSDB18 and MDXDB21. A diagram showing an example of the original architecture of UMX. A diagram showing an example of the architecture of a music base model using UMX. A diagram showing an example of an evaluation result regarding the MDXDB21-dry test set and the MUSDB18 test set. A flowchart explaining the flow of downstream task learning processing using intermediate feature amounts executed by the information processing apparatus of FIG. 1. A diagram showing a configuration example of the hardware of the information processing apparatus 1 of FIG. 1.

[0012] <Device Configuration> FIG. 1 is a diagram showing a configuration example of an embodiment of an information processing apparatus to which the present disclosure is applied.

[0013] The information processing device 1 comprises a foundation model processing unit 11 and a downstream task processing unit 12. The foundation model processing unit 11 performs processing related to the music foundation model (MFM). The music foundation model is a large-scale AI model that has been pre-trained (machine learning) using a large-scale dataset and can be applied to various downstream tasks. The music foundation model can estimate output data such as generated music, information about the input data, and edited samples of the input data from input data such as music data.

[0014] The base model processing unit 11 includes an intermediate feature extraction unit 21. The intermediate feature extraction unit 21 extracts multiple intermediate features from the music base model and outputs them to the downstream task processing unit 12. Intermediate features are features extracted from each intermediate layer of the music base model (intermediate layers excluding the first and last layers). Multiple intermediate features contain information of different granularities. Intermediate features are included in internal features.

[0015] The downstream task processing unit 12 performs processing related to downstream tasks. Downstream tasks include understanding downstream tasks and generative downstream tasks. For example, downstream tasks include tasks such as music tagging, music transcription, music source separation, and music mixing. The downstream task processing unit 12 uses the intermediate features output from the intermediate feature extraction unit 21 to perform processing such as training the downstream task model.

[0016] The information processing device 1 is composed of, for example, a PC (personal computer), a server, or other equipment. The base model processing unit 11 and the downstream task processing unit 12 may be provided by the same information processing device or by different information processing devices. For example, it may be configured as an information processing system consisting of a first information processing device equipped with the base model processing unit 11 and a second information processing device equipped with the downstream task processing unit 12.

[0017] <Relationship between the music infrastructure model and downstream task models> Figure 2 shows an example of how to utilize intermediate features extracted from the music infrastructure model proposed in this disclosure.

[0018] In Figure 2, the music base model 31 is an example of a music base model processed by the base model processing unit 11 in Figure 1. The music base model 31 is a generative model composed of a multi-level transformer equipped with a multi-level hierarchical encoder. For example, the music base model 31 can use a two-stage model in which the first stage consists of a hierarchical autoencoder and the second stage consists of sparse transformers.

[0019] In other words, the music-based model 31 can employ a two-stage generative modeling approach consisting of a first-stage model and a second-stage model. In the first-stage model, hierarchically quantized VAEs (HQ-VAEs) are used for hierarchical representation learning, which divides information into different levels based on granularity. In the second-stage model, multi-level token streams extracted from the first-stage model are learned using auto-regressive modeling.

[0020] The music-based model 31 is designed to extract intermediate features from the input music data (audio input). Specifically, the music-based model 31 converts the input music data into tokens using the autoencoder in the first-stage model, and inputs these tokens into the sparse transformer in the second-stage model, thereby extracting intermediate features as intermediate outputs (intermediate representations) from each intermediate layer.

[0021] Intermediate features can be hierarchical features corresponding to different information granularities (e.g., coarse-grained, medium-grained, fine-grained). For example, in the music base model 31, the autoencoder of the first-stage model is constructed based on a hierarchical representation such as the top layer, middle layer, and bottom layer, and the sparse transformer of the second-stage model is constructed to handle different levels of the hierarchical representation, thereby enabling the extraction of hierarchical intermediate features with different information granularities (e.g., intermediate features corresponding to the top layer, middle layer, and bottom layer). However, the intermediate features extracted from the music base model 31 are intermediate outputs of the music base model 31 and only need to have different information granularities; they do not need to form a hierarchical relationship with each other.

[0022] In Figure 2, downstream task models 41 to 44 are examples of downstream task models (task-specific models) processed by the downstream task processing unit 12 in Figure 1. Downstream task model 41 is a model that corresponds to the task of music tagging, which involves embedding information such as music genre, tempo (BPM: beats per minute), and mood. Downstream task model 42 is a model that corresponds to the task of music transcription, which converts music into musical notation. Downstream task model 43 is a model that corresponds to the task of music source separation, which involves separating sound sources such as vocals and accompaniment. Downstream task model 44 is a model that corresponds to the task of music mixing, which involves combining and adjusting multiple tracks to create a single song. For example, music tagging and music transcription are classified as understanding downstream tasks, while music transcription and music source separation are classified as generative downstream tasks.

[0023] For downstream tasks, the music-based model 31 is applied, and the downstream task models 41-44 are trained (retrained) for each task. In training the downstream task models 41-44, data such as music data (audio input) and intermediate features are used as training data. The music data is the music data (original music data) that is input when the music-based model 31 extracts intermediate features.

[0024] Here, by using intermediate features extracted from the music-based model 31 to train the downstream task models 41-44, the granularity of information can be restricted by multiple intermediate features with different information granularities, thereby improving the performance of the downstream task models through this training. In other words, a representation that divides information of various granularities into embeddings at different levels can provide an efficient information hierarchy for all downstream tasks, including both comprehension and generation tasks. Therefore, the intermediate features extracted from the music-based model 31 can be used as boosters for the downstream task models 41-44.

[0025] For example, in a music tagging task, intermediate features extracted from the music base model 31 are directly input to the downstream task model 41 along with the original music data and used for training. In this case, if the intermediate features are hierarchical intermediate features (hierarchical intermediate features corresponding to coarse, medium, and fine granularity), the downstream task model 41 can use the hierarchical intermediate features. Specifically, the intermediate features of the top layer are intermediate features corresponding to the coarsest information granularity, the intermediate features of the bottom layer are intermediate features corresponding to the finest information granularity, and the intermediate features of the middle layer are intermediate features corresponding to the medium granularity, which is between coarse and fine granularity. The downstream task model 41 can train using intermediate features corresponding to coarse, medium, and fine granularity.

[0026] Furthermore, in the music transcription task, the intermediate features extracted from the music base model 31 are directly input to the downstream task model 42 along with the original music data and used for training. In this case, if the intermediate features are hierarchical, the downstream task model 42 can train using at least the intermediate features corresponding to coarse and medium granularity. In the music source separation task, the intermediate features extracted from the music base model 31 are directly input to the downstream task model 43 along with the original music data and used for training. In this case, if the intermediate features are hierarchical, the downstream task model 43 can train using at least the intermediate features corresponding to coarse granularity.

[0027] Furthermore, in the music mixing task, the intermediate features extracted from the music base model 31 are used as conditional data for training the downstream task model 44. However, the original music data is directly input to the downstream task model 44 as is. In this case, if the intermediate features are hierarchical, the downstream task model 44 can be trained using at least intermediate features corresponding to the coarseness of the granularity. It has been confirmed that using intermediate features corresponding to coarse and medium granularity in the downstream task model 42, or using intermediate features corresponding to coarse granularity in the downstream task models 43 and 44, improves the performance of the downstream task model, but this does not prevent the use of intermediate features of other granularities. That is, the downstream task model 42 may use intermediate features corresponding to fine granularity, and the downstream task models 43 and 44 may use intermediate features corresponding to medium and fine granularity. Furthermore, the downstream task model 41 does not need to use all of the intermediate features corresponding to coarse, medium, and fine granularity; it may use at least intermediate features corresponding to any of the granularities.

[0028] <First Stage Model> Details of the first stage model included in the music infrastructure model 31 will be explained. Figure 3 shows an example of the architecture of the first stage model. As shown in Figure 3, the first stage model 51 has an architecture based on the SQ-VAE-2 structure (see reference R1).

[0029] Reference R1: Yuhta Takida et al., Hq-vae: Hierarchical discrete representation learning with variational bayes. Transactions on Machine Learning Research (TMLR), 2024.

[0030] The first stage model 51 consists of a 3-level SQ-VAE-2 (L=3) autoencoder. bottom Encoder middle Encoder topIt has three encoders 61 to 63 called, and the bottom layer processes input audio sampled at 44.1 kHz. The encoders 61 to 63 are composed of one-dimensional convolutional layers (1-D convolutional layers) with strided convolution for downsampling. The downsampling ratios are set to 8, 4, and 4 respectively for Encoder bottom , Encoder middle , Encoder top .

[0031] The first-stage model 51 has three decoders 81 to 83 called Decoder bottom , Decoder middle , Decoder top . The decoders 81 to 83 are mirror versions of the encoders 61 to 63 with the same resolution respectively. For example, the first-stage model 51 can be trained using 4000 hours of 44.1 kHz studio-quality library music on the HQ-VAE framework.

[0032] Here, to learn the hierarchical representation of the target dataset, the architecture of the music base model 31 is constructed. For example, assume a music sample x of length T (x ∈ X ⊂ R T ). However, ∈ represents that an element belongs to an element, and ⊂ represents a subset. The set of codebooks {B1, B2, B3} is used to learn the three-layer hierarchical representation of x. For l ∈ {1, 2, 3}, the l-th codebook is represented by the following formula (1) and consists of K l trainable vectors b l, k ∈ R dl .

[0033]

[0034] This architecture is designed to extract the hierarchical latent representation of a music sample, and is expressed by equation (3) below, where tl is the latent sequence layer at the l-th layer. In equation (3), the symbols representing operations represent concatenation operators.

[0035]

[0036]

[0037] The discrete tensors Z1, Z2, and Z3 are expected to transmit information about coarse, medium, and fine grain sizes. Their reconstruction can be performed using the optimized neural function shown in equation (4), i.e., equation (5) below.

[0038]

[0039]

[0040] As shown in Figure 3, this architecture consists of a bottom-up path and a top-down path, and its inference process is as follows: That is, a series of encoders 61 to 63 constituting the bottom-up path process three different information resolutions (H) from sample x. l Extract the feature tensor of (x) (let's denote it as l = 1, 2, 3). l (x) is used for a top-down path to process data hierarchically. The top-down path has three top-down blocks (top, middle, and bottom) to model the hierarchical discrete latent representation.

[0041] The top-down block 73 first quantizes the most global (coarse) information among the encoded features, 'Z1 := H1(x),' into a discrete tensor Z1 using the nearest neighbor search of code block B1. In the next step, the intermediate latent tensor is conditioned on Z1 to emphasize local details by injecting H2(x). Thus, the middle-down block 72 uses both tensors processed in the top-up pass, namely Z1 and H2(x), to generate a raw continuous feature, 'Z2 := G2(H2(x), Z1).' This raw feature is quantized to Z2 in the same way as code block B2. The bottom-down block 71 repeats the same process for Z2 and H3(x), further refining the representation with an additional discrete feature Z3. Finally, decoders 81-83 decode the set of Z1, Z2, and Z3 into the data space to reconstruct x. Note that in Figure 3, Z with a tilde added to the top is denoted as 'Z'.

[0042] Here, we train an architecture containing a codebook within a variational Bayes framework, as an instance of HQ-VAE, a stochastic quantization VAE-2 (SQ-VAE-2) (see reference R1). To establish the generative process, we first use Z 1,2,3 The above prior probability distribution is P(Z 1,2,3 ) = P1(Z1)P2(Z2|Z1)P3(Z3|Z 1,2 ) is defined as follows. Given chunks of latent functions Z1, Z2, Z3, the conditional probability distribution p(x|Z 1,2,3 Data samples can be generated under the following conditions.

[0043] Specifically, the conditional distribution is a function f and a trainable isotropic covariance matrix p(x|Z1,2,3 ) = N(f(Z 1,2,3 ), σ 2 Parameterize as a normal distribution with I). In summary, the generation process consists of two steps: from the prior distribution, Z 1,2,3 It consists of the steps of sampling and decoding it using a conditional distribution. Note that, although details will be explained later, in practice, Z 1,2,3 It is important to note that this is sampled from the estimated posterior distribution, not the prior distribution. Next, p(Z 1,2,3 The approximate posterior distribution of |x) is Q(Z 1,2,3 |x) = Q1(Z1|x)Q2(Z2|Z1,x)Q3(Z3|Z 1,2 It is set as ,x). Q1, Q2, and Q3 are each connected to the components shown in Figure 3.

[0044] Specifically, the category distribution Q of the first layer. l (Z l |Z <l , x) is defined as a stochastic quantization corresponding to the relationship expressed in equation (6) below, and is a learnable positive coefficient s l It has Z. l,n and 'Z l,n These are Z l and 'Z l This represents the nth vector. Note that here, Z with a tilde added to the top is denoted as 'Z'. Finally, the resulting learning objective consists of reconstruction and latent regularization terms, as shown in equation (7) below.

[0045]

[0046]

[0047] In equation (7), H(•) is the entropy of the probability mass function. Furthermore, progressive coding (see reference R1) is applied to maintain the balance of information among the three layers.

[0048] <Second Stage Model> Details of the second stage model included in the music infrastructure model 31 will be explained. Figure 4 shows an example of the architecture of the second stage model. In Figure 4, the first stage encoder 51A corresponds to the dashed line 51A in Figure 3, and the first stage decoder 51B corresponds to the dashed line 51B in Figure 3. In Figure 4, the solid arrows represent the default dataflow, the dashed arrows represent feature extraction, and the dashed arrows represent sample generation.

[0049] In the second-stage model 52, the posterior distribution is directly learned, thereby learning the pre-set prior distribution (i.e., P(Z)). 1,2,3 )) and the marginalized posterior distribution (i.e., Q(Z 1,2,3 ) : = E p(x) [Q(Z 1,2,3 The gap between |x))) is closed. In the second stage model 52, input x is y through the contrastive language-audio pretraining (CLAP) model proposed by LAION (see reference R2). audio It is being converted and input. Here, in order to include CLAP conditions, Q φ (Z 1,2,3 ) is approximated by conditional decomposition as shown in equation (8) below.

[0050] Reference R2: Yusong Wu et al., Largescale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2023.

[0051]

[0052] In equation (8), π := {π1, π2, π3} is the set of neural networks of the second-stage model 52, and y audio ∈ R 512 {n1, n2, n3} represents the features generated from the CLAP encoder. {n1, n2, n3} refers to the three sparse transformers (models 91-93 in Figure 4) within the second-stage model 52. Due to the alignment between audio and text embeddings in CLAP, audio data can be input during the training phase even if the audio dataset lacks text captions, but either audio or text can be input during the inference phase. The use of pre-trained encoders is common in generative models. For example, MusicGen (see reference R3) models text conditions using a pre-trained T5 encoder (see reference R4). The training objective is the negative log-likelihood, as shown in equation (9) below.

[0053] Document R3: Jade Copet et al., Simple and controllable music generation, 2023. Document R4: Colin Raffel et al., Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67, 2019. URL https: / / api.semanticscholar.org / CorpusID:204838007.

[0054]

[0055] Here, similar to Jukebox (see reference R5), a sparse transformer (see references R6, R7) is used to construct the network π. That is, as shown in Figure 4, the second-stage model 52 employs a method of generating hierarchical tokens at the same downsampling rate as the first-stage model 51. The tokens at the top, middle, and bottom levels are compressed from 128, 32, and 8 music samples, respectively. Specifically, three transformers (autoregressive sparse transformers) are trained autoregressively to obtain P(Z1|y audio ), P(Z2|Z1, y audio ), P(Z3|Z 1,2 ,y audio ) will be modeled.

[0056] Document R5: Prafulla Dhariwal et al., Jukebox:A generative model for music, 2020. Document R6: Ashish Vaswani et al., 2017; Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: / / proceedings.neurips.cc / paper_files / paper / 2017 / file / 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. Reference R7: Rewon Child et al., Generating long sequences with sparse transformers. CoRR, abs / 1904.10509, 2019. URL http: / / arxiv.org / abs / 1904.10509.

[0057] These are called the top prior model 91, the middle conditional prior model 92, and the bottom conditional prior model 93, respectively. The middle conditional prior model 92 and the bottom conditional prior model 93 use higher-level token sequences, and upsampling is implemented by the upsampling module corresponding to the conditioner in Jukebox. Furthermore, each prior is y audio It is conditioned by this.

[0058] In the music-based model 31, first, a sparse transformer is trained to determine the probability distribution of the top-level token sequence Z1, y audio It learns autoregressively under the condition (i.e., P π1 (Z1|y audio )) That is, next, train other sparse transformers, P πi (Z l |Z <l , y audio This model is created. These transformers are conditioned by higher-level token sequences. Here, we adopted a configuration similar to that of Jukebox transformers, but with modified hyperparameters such as batch size, training step, and learning rate.

[0059] The main difference between these transformers lies in their respective preconditioning mechanisms. That is, instead of the original conditioned inputs used in Jukebox, such as artist, genre, lyrics, etc., y audio The conditional module has been modified to use . Here, as mentioned above, by performing the conditional through the CLAP encoder, y audio In addition, CLAP embedding in text can also be used as a condition. This means that the bottom conditional transformer of this model is all top token sequences Z <lIt is conditioned by the fact that the l-th transformer is adjacent to the higher token sequence Z l-1 This differs from the Jukebox approach, which is conditioned by [unspecified criteria]. This difference stems from the hierarchical structure obtained with SQ-VAE-2, where there are strict interrelationships between different levels, unlike the independently trained multilevel token sequences in Jukebox.

[0060] Here, in Figure 4, the overall conditioning mechanism is explained using the bottom conditional pre-sample. First, the pre-trained first-stage model 51 is applied to the input audio (audio x) to obtain hierarchical discrete representations Z1, Z2, and Z3. The training goal is to obtain Z1, Z2, and CLAP embeddings y audio This means that the model can perform the next token prediction at Z3, conditional on CLAP.audio_embedding(x). Here, the transformer is adjusted frame by frame.

[0061] Since each level has a different time resolution, each token sequence at a higher level is first transformed into an embedding sequence, and then upsampled to the next time resolution by transposed convolution. Two upsampling modules are used here: one for Z1 → Z2 and another for Z1, Z2 → Z3. The Z1 → Z2 upsampling module upsamples the embedding sequence Z1 to the resolution of Z2. Similarly, the Z1, Z2 → Z3 upsampling module takes Z2 as input and generates upsampled embeddings for Z3 along with the upsampled embeddings of Z1. The resulting frame-level embedding is used to adjust the transformer of Z3.

[0062] In the actual next-token prediction task, the token sequence Z3 is shifted one position to the right, and each token is embedded into a continuous vector using an embedding layer. The first empty token is optionally y so that the system adjusts to CLAP embedding. audio This is embedded. Frame-level aggregated conditioning vectors, i.e., up-sampled embeddings from upper layers, y audio Using time positional embeddings, the transformer is trained to estimate Z3 using an autoregressive method.

[0063] <Music-based model for downstream tasks> First, intermediate features are obtained from the input audio using a feature extraction process that is independent of the downstream task. Next, different preprocessing is applied depending on whether the downstream task is time-invariant or time-varying. Finally, the preprocessed intermediate features are inserted into the appropriate positions in the model specific to the target task.

[0064] This section describes a feature extraction process that is independent of downstream tasks. Here, we assume that the feature extraction pipeline (see reference R8, etc.) is followed based on a pre-trained music-based model 31.

[0065] Reference R8: Rodrigo Castellon et al., Codified audio language modeling learns useful representations for music information retrieval. In Jin Ha Lee, Alexander Lerch, Zhiyao Duan, Juhan Nam, Preeti Rao, Peter van Kranenburg, and Ajay Srinivasamurthy (eds.), Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021, pp. 88-96, 2021. URL https: / / archives.ismir.net / ismir2021 / paper / 000010.pdf.

[0066] The music waveform of the input audio is first converted into multi-level token sequences via the first-stage encoder 51A of the music base model 31. The tokens are then sent to the top prior distribution model 91, middle conditional prior distribution model 92, and bottom conditional prior distribution model 93 of the second-stage model 52 without auto-regressive iteration.

[0067] The middle-conditional prior distribution model 92 and the bottom-conditional prior distribution model 93 are conditioned by ground-truth tokens generated by the first-stage encoder 51A. The output of the Nth (N = 36, as in reference R8) transformer layer of these prior distribution models is then extracted as intermediate features. Note that if CLAP audio embedding is not used as a condition in the prior distribution, feature extraction is called unconditional extraction. Otherwise, it is called CLAP-conditional extraction.

[0068] The maximum sequence length of the prior distribution is 8192. Since the downsampling rates of the first-stage model 51 are 128× (top), 32× (middle), and 8× (bottom), the same amount of 8192 tokens from different prior distributions correspond to 24 seconds, 6 seconds, and 1.5 seconds in the time domain, forming a set of hierarchical multi-rate features.

[0069] Here, to conserve computational resources, intermediate features are pre-computed for most downstream tasks, except for HTDemucs (where music clips are randomly selected on the fly during training), which will be discussed later. To compute features for long audio inputs, the input is divided into a certain ratio N. ovlp They are treated as multiple overlapping segments. ovlp If the overlap is sufficiently large and longer than the perception field of the second-stage model 52, it is guaranteed that the feature extraction results will not be affected by segmentation.

[0070] <Feature Preprocessing for Time-Invariant Downstream Tasks> In this disclosure, if the downstream task is time-invariant, the input audio is first divided into non-overlapping segments of 24 seconds (top), 6 seconds (middle), and 1.5 seconds (bottom). For each prior distribution, the intermediate features of the segment are reduced to one token by average pooling, ultimately forming a sequence of three tokens.

[0071] Previous studies (see reference R8, etc.) have suggested using a multi-layer perceptron (MLP) with a single 512-dimensional hidden layer to investigate features. However, the token sequences in this disclosure are generated (originated) from prior distributions with different temporal resolutions, which differs from previous studies. Sequence aggregation is necessary to effectively utilize these hierarchical features.

[0072] Therefore, this disclosure proposes aggregating sequences via a standard attention block, namely an attention layer followed by a feed-forward layer. This is inspired by attention-based feature aggregation in instrument classification tasks (see reference R9, etc.).

[0073] Reference R9: Siddharth Gururani et al., An attention mechanism for musical instrument recognition. In ISMIR 2019, 2019.

[0074] First, hierarchical intermediate features are concatenated into a single token sequence, and then a learnable class token is attached to the beginning. The attention block is trained to aggregate all features into class tokens and is then converted into music tags or emotion scores by the multilayer perceptron (MLP) described above.

[0075] Furthermore, this disclosure proposes an on-the-fly data augmentation technique called token-out to prevent overfitting when training attention blocks using concatenated token sequences. This technique references SpecAugment (see reference R10), PaSST (see reference R11), and Masked Autoencoder (MAE) (see reference R12), and involves masking a portion of the input before feeding it into a deep neural network (DNN).

[0076] Document R10: Daniel S Park et al., Specaugment: A simple data augmentation method for automatic speech recognition. In Proc. of Interspeech 2019, 2019. Document R11: Khaled Koutini et al., Efficient training of audio transformers with patchout. In Proc. of Interspeech 2022, 2022. Document R12: Kaiming He et al., Masked autoencoders are scalable vision learners. In CVPR 2022, pp. 16000-16009, 2022.

[0077] The masking rate here is sampled uniformly between 0 and 100%. The token-out extension of this disclosure follows the principles of PaSST and MAE, but is applied to tokens extracted by the music-based model 31, as shown in Figure 5. Figure 5 shows an example of attention-based feature aggregation and token-out data augmentation. In Figure 5, "T", "M", and "B" are marks indicating which model (top-level prior distribution model 91, middle-level conditional prior distribution model 92, bottom-level conditional prior distribution model 93) the features were extracted from. The token-out extension removes masked tokens from the input sequence. The attention block aggregates the sequence into a single vector, and then a multilayer perceptron (MLP) predicts the tag. Here, we confirmed that performance improves when the intermediate features of the music-based model 31 are aggregated with a shallow attention layer and token-out extension.

[0078] <Feature Preprocessing for Time-Varying Downstream Tasks> When applying the intermediate features of the music-based model 31 to a model for a downstream task that exhibits time variation (task-specific model), several issues must be considered, such as temporal alignment before injecting features into the task-specific model, appropriate amount of information compression, and sufficient feature adaptation. Temporal alignment between the intermediate features of the music-based model 31 and the target task-specific model can be achieved by pooling the intermediate features or by using a linear layer.

[0079] Details of these two cases will be discussed later, but we confirmed that performance is improved when using linear layers compared to average pooling and maximum pooling. Here, both information compression and feature adaptation can be performed with linear layers. The output dimension is simply set to match the feature dimension of the target task-specific model. Empirically, one layer is sufficient for most of the models tested, but this excludes music transcription using an hFT transformer (hFT-Transformer) which requires four layers (see reference R13), as will be discussed later.

[0080] Reference R13: Keisuke Toyama et al., Automatic music transcription with hierarchical frequency-time transformer. In Proceedings of the 24th International Society for Music Information Retrieval Conference, pp. 215-222, 2023.

[0081] <Experiment> To investigate the usefulness of features extracted from the music-based model for downstream tasks (comprehension tasks and generation tasks), we conducted experiments answering the following two questions: The first question was "Do the extracted features contain information useful for understanding music?" and the second question was "Can the extracted features enhance the current task-specific model in both comprehension and generation tasks?" To verify the generalization of the results, we included not only intermediate features from the music-based model 31, but also features extracted from two publicly available versions of MusicGen (see reference R3), MusicGen Small and MusicGen Large.

[0082] To answer the first question, we selected music tagging and music transcription as representative of understanding tasks. We confirmed that the extracted features include not only time-invariant information about the overall music characteristics but also time-varying information about specific music events. To answer the second question, we tested injecting the extracted features into multiple task-specific models that cover both understanding and generation tasks. These models consist of one music transcription model (see reference R13), two music source separation models (see reference R38 below), and two music mixing models (see reference R46 below).

[0083] The feature extraction described above is used in all experiments. Task-dependent preprocessing is applied before applying the features to downstream tasks. Details of the experimental setup and results will be described later. Among the intermediate features of the music-based model 31, intermediate features extracted from the top and middle layers are used in particular. In preliminary experiments, it was confirmed that including intermediate features extracted from the bottom layer of the music-based model 31 does not necessarily improve the performance of the comprehension task. This is presumably because the bottom layer mainly contains detailed information irrelevant to the task, which degrades performance.

[0084] Through the following experiments, we will demonstrate that useful information is contained in both the intermediate features of the music-based model 31 and the features of MusicGen. Below, we will describe the downstream tasks of music tagging, music transcription, music source separation, and music mixing in that order.

[0085] <Music Tagging> We broadly test music tagging tasks and emotion regression tasks. For automatic tagging, we can use MTAT (MagnaTagATune) (see reference R14), for pitch and instrument recognition, Nsynth (see reference R15), for emotion regression, EmoMusic (see reference R16), for genre classification, GTZAN (see reference R17), for music key estimation, GiantSteps (see reference R18, etc.), and for singer and vocal technique identification, VocalSet (see reference R19). Here, following the preprocessing of previous studies (see reference R20, etc.), metric computation is performed using scikit-learn (see reference R21) and mir_eval (see reference R22). In EmoMusic, the average R of arousal and valence axis 2 This has been reported.

[0086] Document R14: Edith Law et al., Evaluation of algorithms using games: The case of music tagging. In Keiji Hirata et al., Proceedings of the 10th International Society for Music Information Retrieval Conference, ISMIR 2009, Kobe International Conference Center, Kobe, Japan, October 26 - 30, 2009, pp. 387 - 392. International Society for Music Information Retrieval, 2009. URL http: / / ismir2009.ismir.net / proceedings / OS5-5.pdf. Document R15: Jesse Engel et al., Neural audio synthesis of musical notes with wavenet autoencoders, 2017. Document R16: Mohammad Soleymani et al., 1000 songs for emotional analysis of music. In Proceedings of the 2nd ACM international workshop on Crowdsourcing for multimedia, pp. 1 - 6, 2013. Document R17: George Tzanetakis and Perry Cook, Musical genre classification of audio signals. IEEE Transactions on speech and audio processing, 10(5):293 - 302, 2002. Document R18: Peter Knees et al., Two data sets for tempo estimation and key detection in electronic dance music annotated from user corrections.In Proceedings of the 16th International Society for Music Information Retrieval Conference (ISMIR); 2015 Oct 26-30; Malaga, Spain.[Malaga]: International Society for Music Information Retrieval, 2015. p. 364-70. International Society for Music Information Retrieval (ISMIR), 2015. Document R19: Julia Wilkins et al., A singing voice dataset. InISMIR, pp. 468-474, 2018. Reference R20: Yizhi Li et al., Acoustic music understanding model with large-scale self-supervised training, 2023b. Reference R21: F. Pedregosa et al., Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011. Reference R22: Colin Raffel et al., Mir_eval: A transparent implementation of common mir metrics. In ISMIR, volume 10,pp. 2014, 2014.

[0087] The feature preprocessing (including feature aggregation and tokenization augmentation) for time-invariant downstream tasks described above is applicable to all tasks. Then, following previous studies (see references R8 and R20), the aggregated features are probed using a multilayer perceptron (MLP).

[0088] To compare CLAP conditional extraction, unconditional extraction, and features from the CLAP encoder, a preliminary study was conducted using top-performing pre-features on the music-based model 31. Here, MTAT was used for the coarse-grained concepts tagging task, and Nsynth was used for the fine-grained concepts (pitch) classification task. The results showed that CLAP performed well for coarse-grained concepts, while unconditional extraction improved the accuracy of pitch estimation. CLAP conditional extraction achieved excellent scores in both tasks. This disclosure presents the scores of the music-based model 31 using CLAP conditional extraction in a time-independent comprehension task.

[0089] Figure 6 shows an example of benchmark results for intermediate features in a music tagging task. Figure 6 shows test results using various datasets and benchmarks from previous studies. MFM31 in Figure 6 shows the test results when using intermediate features from the music-based model 31. In investigations using intermediate features from the music-based model 31 and features from MusicGen, both models showed competitive scores in most tasks.

[0090] As shown in Figure 6, the intermediate features of the music-based model 31 achieved top-two performance in auto tagging, pitch estimation, instrument classification, and genre classification. Furthermore, the intermediate features demonstrated excellent performance in emotion regression and singer identification. These are prompt-conditioned generative models, but feature analysis using these models achieved performance equivalent to that of a model using only a state-of-the-art encoder specialized for comprehension tasks.

[0091] In this disclosure, the attention-based aggregator described above is implemented as a one-layer standard transformer. The attention layer has only one attention head. Features from different prior distributions are processed in the input normalization layer and transformed into 768 dimensions in the linear layer. Here, as in reference R8, a grid search is performed on the input normalization method (BatchNorm or LayerNorm) and the dropout rate in MLP (0.10, 0.25, 0.50, 0.75). Label smoothing is set to 0.1 for all single-label tasks. The batch size is set to 256 for MTAT and Nsynth with large datasets, and to 64 for others. The learning rate is 5 × 10⁻⁶. -5 That is what they say.

[0092] Furthermore, the music-based model 31 was trained on an upstream autoregressive modeling task, which introduced both conditional and unconditional cases of CLAP, making it unclear which setting should be used for the downstream task. Therefore, in this disclosure, preliminary experiments were conducted using MTAT, a dataset designed for coarse-grained tagging tasks, and Nsynth-pitch, a dataset designed for fine-grained classification tasks. In these experiments, the features of the top prior distributions were aggregated by mean pooling, and the learning rate was 1 × 10⁻⁶. -4 It was set to [value]. As described above, the CLAP audio encoder receives the same music clip as the music-based model 31 and encodes it into a single feature vector.

[0093] Next, an ablation study was conducted to verify the effectiveness of the feature aggregation pipeline. The learning rates for attention-based and mean pooling were 5 × 10⁻⁶, respectively. -5 and 1 x 10 -4 Here, we confirmed the following: Firstly, we confirmed that the feature aggregation pipeline outperformed the baseline average pooling. Secondly, applying the same pipeline to non-hierarchical features did not yield the same performance improvement as the hierarchical case, confirming that hierarchical features contain complementary information for music tagging. We confirmed that the bottom prior did not contribute to performance improvement. This is presumably because the bottom prior processes high-frequency components that are not very important to the task tested.

[0094] <Music Transcription> In addition to time-invariant understanding tasks, we continue testing music transcription, which is a time-varying understanding task. The intermediate features of the music-based model 31 and the features of MusicGen are obtained by the unconditional extraction described above. The dimensions and time resolution of the extracted features are matched to the dimensions and time resolution of the spectrogram by linear layers. According to reference R8, these matched features are concatenated with the spectrogram, and these features are explored using a single-layer shallow back-end network.

[0095] Figure 7 shows an example of the evaluation results of feature probing using a shallow backend network on the MAPS dataset (see reference R23). In Figure 7, the intermediate features of the music-based model 31 are denoted as MFM31.

[0096] Document R23: Valentin Emiya et al., Multipitch estimation of piano sounds using a new probabilistic spectral smoothness principle. IEEE Transactions on Audio, Speech, and Language Processing, 18(6):1643-1654, 2010.

[0097] As shown in Figure 7, the F1 score per note has increased significantly for all features compared to when only the spectrogram was used. This suggests that the intermediate features of the music-based model 31 and the features of MusicGen contain information useful for the time-varying understanding task. In other words, the extracted features contain useful knowledge for understanding music.

[0098] Next, we tested both the intermediate features and MusicGen features of the music-based model 31 in state-of-the-art task-specific models, including music transcription, music source separation, and music mixing. These tasks cover both comprehension and generation tasks. The test results show that the extracted features consistently improve the performance of the task-specific models. We also confirmed that using the intermediate features of the music-based model 31 reduces the training loss in the early epochs.

[0099] Furthermore, the extracted features were applied to the hFT transformer (see reference R13), a state-of-the-art music transcription model for piano on MAPS (see reference R23), and evaluated whether it outperformed existing models that rely solely on spectrograms. The hFT transformer estimates frame-based note activation, note onset, offset, and velocity (frame, onset, offset, velocity) based on the input spectrogram.

[0100] This is a transformer-based model consisting of two transformer encoders operating on different input axes and a transformer decoder located at the center of these two encoders. When we tried injecting the intermediate features of the music-based model 31 and the features of MusicGen before the first encoder, the second encoder, and the decoder, following the processing pipeline, we found that injecting the features before the decoder yielded the best results, so we will adopt this method in subsequent experiments.

[0101] Following previous research (see references R13 and R24), four F1 scores—frame-based, note-based, offset-based note-based, and offset-based note-based note-based—are presented using the checkpoint for the best-validated F1 score.

[0102] Document R24: Josh Gardner et al., Mt3: Multi-task multitrack music transcription. In Proceedings of the 10th International Conference on Learning Representations,2022.

[0103] Figure 8 shows an example of the F1 score results in MAPS. In Figure 8, the intermediate features of the music-based model 31 are denoted as MFM31. As shown in Figure 8, the performance of the hFT transformer improves when either the intermediate features of the music-based model 31 or the features of MusicGen Small are inserted. The performance gap is particularly large when the model is trained on a small subset of MAPS. This demonstrates the usefulness of inserting the intermediate features of the music-based model 31 into downstream task models when training data is insufficient. Furthermore, as shown in Figure 8, it has been confirmed that the reduction in training loss is faster when either the intermediate features of the music-based model 31 or the features of MusicGen are inserted.

[0104] In experiments targeting the entire MAPS model, injecting intermediate features from the top and middle layers improved performance. However, injecting features from all three layers did not improve performance, and a similar trend was observed in the MusicGen Large results. This suggests that the network capacity required to interpret all the information contained in the features may exceed the capacity of the hFT transformer, potentially negatively impacting the model. This indicates that separating features based on information granularity and excluding irrelevant information is crucial for such injections.

[0105] Incidentally, in music transcription, both pitch and temporal information need to be accurately predicted to achieve optimal performance. Current models (see references R13 and R24) rely solely on spectrograms as input. Attempts to apply additional features to music transcription have been made before. For example, reference R25 uses Jukebox for melody transcription. However, the HookTheory dataset used in reference R25 does not include octave information, and therefore differs from typical music transcription.

[0106] Document R25: Chris Donahue et al., Melody transcription via generative pre-training. In Proceedings of the 23rd International Society for Music Information Retrieval Conference, 2022.

[0107] There are two main methods for modeling a typical music transcription task: piano-roll-based music transcription (see references R13, R24, R26, etc.) and token-based music transcription (see reference R24). The former converts the spectrogram into a piano roll with the same temporal resolution. The latter converts the spectrogram into a series of tokens indicating the pitch of a note and whether the note is on or off. In this disclosure, we selected the relatively well-known piano-roll-based music transcription and verified the effectiveness of the intermediate representation (intermediate features) extracted from the music-based model 31.

[0108] Document R26: Curtis Hawthorne et al., Dual-objective piano transcription. In Proceedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018, Paris, France, 2018, 2018. URL https: / / arxiv.org / abs / 1710.11153

[0109] <<Piano Transcription using Linear Layers>> Here, in order to understand the usefulness of the extracted intermediate features, we started with piano transcription and examined their features using linear layers. The architecture used in this experiment is shown in Figure 9. Figure 9 is a diagram showing an example architecture of a linear music transcription model for piano. In Figure 9, the inputs are a log mel spectrogram and feature tokens of the music-based model 31. The input audio signal is downsampled to 22.05 kHz and converted into a 256-bin mel spectrogram using a 512-hop, 2048-point Hann window. The arrangement of the mel spectrogram and extracted features is shown in Figure 10. Figure 10 is a diagram showing a first example of a feature alignment scheme for various models in music transcription. Figure 10A shows the feature and spectrogram alignment scheme of a linear transcription model using the music-based model 31. Figure 10B shows the alignment scheme of the features and spectrogram of a linear transcription model using MusicGen.

[0110] The Mel spectrogram is a three-dimensional tensor (B, N, F), where B = 8 is the batch size, N = 256 is the number of frames, and F = 256 is the number of frequency bins. The linear layer transforms the tensor along the last axis to (B, N, Z), where Z = 512 is the typical size of the latent embedding. The intermediate features of the music-based model 31 and the features of MusicGen are formed as a three-dimensional tensor (B, V, G), where V is the number of tokens per N frames. This number of tokens is 2,048 for the top layer of the music-based model 31, 8,192 for the middle layer of the music-based model 31, and 299 for MusicGen. G is the feature embedding size. These sizes are 4,800 for the top tier of MusicBase Model 31, 3,200 for the middle tier of MusicBase Model 31, 2,048 for MusicGen Large, and 1,024 for MusicGen Small.

[0111] The first linear layer transforms G to Z, and then the second linear layer transforms V to N, so that the tensor takes the shape of (B, N, Z). Next, the tensor is concatenated along the last axis. When the top layer of MusicBasedModel 31, the middle layer of MusicBasedModel 31, or MusicGen is used, the dimension of the last axis is 2Z, but when both layers of MusicBasedModel 31 are used, the dimension is 3Z. The following linear layers are used to transform 2Z or 3Z to 2P, where P = 88 is the pitch number. Finally, the tensor is split into two (B, N, P) tensors, showing the frame and onset information estimated by sigmoid activation. Here, the precise timing of onsets is extracted from the dataset according to reference R27.

[0112] Reference R27: Qiuqiang Kong et al., High-resolution piano transcription with pedals by regressing onset and offset times. IEEE / ACM Transactions on Audio, Speech, and Language Processing, 29:3707-3717, 2021.

[0113] Furthermore, the hyperparameter controlling the target sharpness J is set to 3. The loss function for frames and onsets is binary cross entropy. In addition, multiple models were trained using the following inputs: (1) spectrogram only, (2) top layer of the spectrogram and intermediate features of the music-based model 31, (3) middle layer of the spectrogram and intermediate features of the music-based model 31, (4) both the top and middle layers of the spectrogram and intermediate features of the music-based model 31, (5) spectrogram and MusicGen Large features, and (6) spectrogram and MusicGen Small features.

[0114] Here, for each model, the learning rate is 1 × 10⁻⁶. -4 The model is trained for 50 epochs using the Adam optimizer (see reference R28) on a single NVIDIA A100 GPU (graphics processing unit). For learning rate scheduling, PyTorch ReduceLROnPlateu is used with default parameters.

[0115] Reference R28: Diederik P Kingma and Jimmy Ba, A method for stochastic optimization. In Proc. International Conference on Learning Representation (ICLR), 2015

[0116] Each model selects the optimal checkpoint based on its F1 score in the validation set. As shown in Figure 7, models using intermediate features from the music-based model 31 or features from MusicGen show better performance than the baseline using only spectrograms. In particular, the contribution of the middle layer of the music-based model 31 is higher than that of the top layer. These results suggest that the intermediate features of the music-based model 31 are promising for music transcription tasks.

[0117] <<Instrument-Independent Linear Music Transcription>> Following the setup for piano transcription using the linear layer described above, we investigated the effectiveness of extracted features in instrument-agnostic music transcription. This was to determine whether the intermediate features of the music-based model 31 could be applied to transcription scenarios involving multiple instruments. In this experiment, we reused the model shown in Figure 9 and trained it using the URMP dataset (see reference R29) which includes string, woodwind, and brass instruments. Unlike piano transcription, instrument-agnostic transcription does not require splitting the final output into frames and onset predictions. This is because some instruments, such as violins and flutes, can generate ambiguous onset attacks. Applying onset predictions to these instruments may negatively impact the F1 score (see reference R30).

[0118] Reference R29: Bochen Li et al., Creating a multitrack classical music performance dataset for multimodal music analysis: Challenges, insights, and applications. IEEE Transactions on Multimedia, 21(2):522-535, 2019. doi: 10.1109 / TMM.2018.2856090. Reference R30: Kin Wai Cheuk et al., A semi-supervised automatic music transcription framework for low-resource real-world data. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 3918-3926, 2021a.

[0119] Here, we validated the model's performance using the Bach10 dataset (see reference R31) and selected the optimal checkpoint based on the validation performance. Next, we evaluated the best checkpoint using the GuitarSet (see reference R32), Su (see reference R33), and TRIOS (see reference R34) datasets.

[0120] Reference R31: Zhiyao Duan et al., Multiple fundamental frequency estimation by modeling spectral peaks and non-peak regions. IEEE Transactions on Audio, Speech, and Language Processing, 18 (8):2121-2133, 2010. doi: 10.1109 / TASL.2010.2042119. Reference R32: Qingyang Xi et al., Guitarset: A dataset for guitar transcription. In ISMIR, pp. 453-460, 2018. Reference R33: Li Su and Yi-Hsuan Yang, Escaping from the abyss of manual annotation: New methodology of building polyphonic datasets for automatic music transcription. In Music, Mind, and Embodiment: 11th International Symposium, CMMR 2015, Plymouth, UK, June 16-19, 2015, Revised Selected Papers 11, pp. 309-321. Springer, 2016. Reference R34: Joachim Fritsch, High quality musical audio source separation. Master's thesis, 2012.

[0121] The evaluation results confirmed that, regardless of whether it was the music-based model 31 or MusicGen, the features of the music-based model improved instrument-independent transcription performance compared to the baseline model which used only spectrograms as input features. This indicates that models trained using these features have stronger generalizability across different datasets. The music-based model 31 outperformed MusicGen's F1 score when using Bach10, GuitarSet, and TRIOS, while MusicGen performed better when using Su. This difference may be due to the different datasets on which the music-based model 31 and MusicGen were trained.

[0122] <<hFT Transformer>> The detailed model architecture of the hFT-Transformer is described in reference R13. Figure 11 shows a second example of feature alignment schemes for various models in music transcription. Figure 11A shows the feature and spectrogram alignment scheme of the hFT transformer using the music-based model 31. Figure 11B shows the feature and spectrogram alignment scheme of the hFT transformer using MusicGen. The hFT transformer (16kHz), music-based model 31 (44.1kHz), and MusicGen (32kHz) cannot perfectly align feature tokens and spectrograms due to their different sampling frequencies. N frames of the spectrogram correspond to approximately 707 tokens in the top layer of intermediate features of the music-based model 31, 2,828 tokens in the middle layer of intermediate features, and 104 tokens of MusicGen features.

[0123] Figure 12 shows an example of a modified hFT transformer that accepts intermediate features of the music base model 31 or MusicGen features as additional input. Following the setup of the linear layer piano transcription described above, the feature tokens of the music base model 31 and MusicGen have three dimensions (B, V, G), where B = 8 is the batch size, V is the number of tokens per frame (N = 128), and G is the embedding size of the feature tokens. In Figure 12, the first linear layer of the feature token reduces G to Z' = 256. The second linear layer reduces V to N and then reshapes the tensor to (B, N, F'' = 16, Z'' = 16). The third linear layer changes Z'' to Z = 256, and the last linear layer changes F'' to F' = 128. Therefore, a tensor of shape (B, N, F', Z) is obtained. When using both the top and middle layers of the intermediate features of the music-based model 31, such a tensor is formed in each layer according to the pipeline described above. Finally, a linear layer reduces the size of the connected tensor to 256 (F in reference R13). Note that F', F'', Z', Z'' are determined in a preliminary experiment.

[0124] Similar to the linear layer piano transcription experiment described above, the following models with different inputs were trained: (1) spectrogram only, (2) top layer of spectrogram and intermediate features of MusicBasedModel 31, (3) middle layer of spectrogram and intermediate features of MusicBasedModel 31, (4) both top and middle layers of spectrogram and intermediate features of MusicBasedModel 31, (5) spectrogram and MusicGen Large features, and (6) spectrogram and MusicGen Small features. Here again, the models were trained over 50 epochs on a single A100 GPU. Other conditions followed reference R13. To verify the effectiveness of intermediate features of MusicBasedModel 31 and MusicGen features when training data is limited, training was performed using 100%, 50%, 25%, and 10% of the training data. For further evaluation, the optimal checkpoint was selected based on the F1 score in the validation set.

[0125] The training and validation loss curves confirmed the following: Models using intermediate features from MusicBased Model 31 or MusicGen features reached lower loss values ​​in earlier epochs compared to the baseline model using only spectrograms as input. When all training data was available, models using intermediate features from MusicBased Model 31 and MusicGen features outperformed the baseline model, except for the model using MusicGen Large features. When the models were trained with less data (scarce data), the performance of models using intermediate features from MusicBased Model 31 and MusicGen features was superior to that of models using only spectrograms. When the training data size was 50% and 25%, the performance of models using intermediate features from MusicBased Model 31 was comparable to the baseline model trained with 100% of the data.

[0126] <Music Source Separation> Here, Open-Unmix (UMX) (see reference R35) and Democs (HTDemucs) (see reference R36) are targeted for feature injection. UMX estimates the time-frequency mask of the target sound source based on a recurrent neural network (RNN) block.

[0127] Document R35: F.-R. Stoter et al., Open-unmix - a reference implementation for music source separation. Journal of Open Source Software, 2019. doi: 10.21105 / joss.01667. URL https: / / doi.org / 10.21105 / joss.01667. Document R36: Simon Rouard et al., Hybrid transformers for music source separation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), pp. 1-5. IEEE, 2023.

[0128] HTDemucs is a hybrid model consisting of a waveform U-Net branch and a spectral U-Net branch. Extracted features are injected into the UMX encoder block using a downsampling block and into each HTDemucs branch using a cross-domain transformer. For simplicity, only the internal features of the top layer from the music-based model 31 are injected here.

[0129] Figure 13 shows examples of evaluation results for MUSDB18 and MDXDB21. Figure 13 shows the SDR scores for the test set (test split) of MUSDB18 (see reference R37) and the hidden test set (hidden split) of MDXDB21 (see references R38 and R39). In Figure 13, the intermediate features of the music-based model 31 are denoted as MFM31.

[0130] Document R37: Zafar Rafii et al., Musdb18-a corpus for music separation. 2017. Document R38: Yuki Mitsufuji et al., Music Demixing Challenge 2021. Frontiers in Signal Processing, 1:808395, 2022. Document R39: Giorgio Fabbro et al., The Sound Demixing Challenge 2023 demixing track. arXiv preprint arXiv:2308.06979, 2023.

[0131] Similar to the experiments described in the music transcription section above, we confirmed a rapid decrease in training loss. When intermediate features from the music-based model 31 were injected into both UMX and HTDemucs, the separation performance of both models improved significantly, even on the unknown dataset MDXDB21. Furthermore, the separation performance of HTDemucs also improved when training on data corrupted by bleeding errors (SDXDB23_Bleeding) (see reference R39). However, the results are not always improved by MusicGen features.

[0132] In other words, while UMX improved when MusicGen Small features were injected, it did not improve in other cases. According to the results of the HTDemucs ablation study, injecting short-term Fourier transform (STFT) signals, CLAP features, or MusicGen features causes training instability. However, when intermediate features of the music-based model 31 were injected into HTDemucs, such behavior was not observed. It is expected that performance will improve if training instability can be avoided. As mentioned in the explanation of music transcription above, the capacity of downstream task models may be excessively consumed in order to interpret the information contained in MusicGen Large, potentially degrading performance. In summary, it was confirmed that injecting intermediate features of the music-based model 31 into the separation model not only speeds up training and improves performance, but also improves robustness against dataset corruption.

[0133] <<UMX Details>> Figure 14 shows an example of the original architecture of UMX. Figure 15 shows an example of the architecture of a music-based model 31 using UMX. As shown in Figure 14, UMX starts with an input audio waveform and converts it into an STFT spectrogram. The amplitude portion of the spectrogram is normalized to mean 0 and standard deviation 1 based on statistics collected from the entire pre-training dataset. The normalized amplitude spectrograms are passed to an encoder block that includes a linear layer, a batch normalization (BN) layer, and a hyperbolic tangent function.

[0134] Next, a sequence modeling block consisting of three RNN layers and a skip connection structure is used. This block processes the encoded features and combines their input and output features. The resulting features are then converted into a time-frequency mask by the decoder block. The decoder block consists of multiple linear layers, a BN layer, and a ReLU (rerected linear unit) activation. These time-frequency masks are used to modify the amplitude spectrogram, and a multi-channel wiener filter (MWF) (see reference R40, etc.) is applied to the spectrogram as needed. Finally, the separated audio waveforms are generated by an ISTFT (inverse short-term Fourier transform).

[0135] Reference R40: Aditya Arie Nugraha etal., Multichannel music separation with deep neural networks. In Proc. of 24th European Signal Processing Conference (EUSIPCO), pp. 1748-1752, 2016.

[0136] All learnable parameters are optimized using the mean-squared error of the amplitude spectrogram. This disclosure proposes using intermediate features of the music-based model 31 without modifying the UMX architecture. As shown in Figure 15, the music-based model 31 (MFM31) is introduced to extract features from the input audio waveform. These features are adjusted to have a standard statistical distribution calculated across the entire training dataset, similar to the normalization used for the amplitude spectrogram of UMX. They are then processed in a downsampling block, a linear layer. The downsampling block is introduced due to the difference in time resolution between the music-based model 31 and UMX. The music-based model 31 generates features every 128 waveform samples, while the original branch of the UMX model processes every 1024 waveform samples, determined by the STFT hop size.

[0137] The downsampled features are transformed from 4800 dimensions to 512 dimensions by a linear layer and summed with the original features from the UMX branch. In the original UMX architecture, the input is an STFT spectrogram. Since all modules of UMX operate at the time resolution of the STFT, the downsampling block needs to match the time resolution of the intermediate features of the music-based model 31 with that of the STFT. In the ablation study, three different downsampling operations were considered: max pooling (MP), average pooling (AP), and unfolding (UF). In this disclosure, average pooling (AP) averages multiple intermediate features by a hop size and kernel size corresponding to the hop size and window size of the STFT, respectively. Max pooling (MP) operates similarly to average pooling, but replaces the average pooling with a max pooling operation. In unfolding (UF), a linear layer and a ReLU function are first used to transform the dimension of the intermediate feature sequence from 4800 to 150. Next, the feature sequences corresponding to the STFT window size are concatenated for each STFT hop size, and after concatenation, the feature dimension is returned to 4800.

[0138] Furthermore, as shown in Figure 15, two methods for injecting downsampled features into UMX were evaluated. In the early injection (EI) method, intermediate features of the music-based model 31 are injected into the encoder block of UMX. In the late injection (LI) method, intermediate features of the music-based model 31 are injected into the decoder block. The optimal downsampling and injection method for UMX was investigated in a vocal extraction task using MUSDB18. Here, all experiments followed the original training configuration of UMX, except that data augmentation was disabled for simplification. MWF was skipped in all experiments to evaluate only the trained components.

[0139] As a result, the following was confirmed: In the downsampling block, maximum pooling (MP) and average pooling (AP) did not improve the SDR score. Expansion (UF) achieved an improvement of 0.34 dB for vocals and 0.36 dB for accompaniment. This indicates that expansion (UF) is the appropriate choice for the downsampling block. This is presumably because, with maximum pooling (MP) and average pooling (AP), low-level information is lost during the pooling operation over time, whereas expansion (UF) preserves such information by stacking multiple tokens into a single frame in an expansion manner. Also, early injection (EI) outperformed late injection (LI). This suggests that the sequence modeling block of UMX effectively utilized the intermediate features of the music-based model 31 to improve separation performance. The loss curve of UMX using the music-based model 31 (UF, EI) tended to be lower than the loss curve of UMX in both training and validation. In other words, when UMX is trained using the music-based model 31, the number of epochs required to reach the target loss is reduced. Furthermore, the convergence loss of UMX using the music-based model 31 was lower than that of the original UMX. Thus, the benefits of using the intermediate features of the music-based model 31 can be consistently observed throughout the training phase, demonstrating a significant improvement in performance.

[0140] <<Details of HTDemucs>> To investigate the effect of intermediate features in the music-based model 31, multiple models were trained using MUSDB18 (see reference R37). The models compared are as follows:

[0141] ・HTDemucs (default): This is the default model (Dora6 signature '955717e8'). The default batch size is 32, which corresponds to 4 samples per GPU because parallel training was performed on 8 GPUs. The default number of training epochs is 360. ・HTDemucs + MFM31: This is a combination of HTDemucs and the music-based model 31. HTDemucs introduces two new cross-domain transformer encoders that are inserted immediately after the encoder of the original cross-domain transformer. The first encoder facilitates information exchange between spectral feature sequences and intermediate feature sequences, and the second encoder enables interaction between waveform feature sequences and intermediate feature sequences. The intermediate features of the music-based model 31 are computed from a monaural downmix of the mixed sound using the top-prior model 91. Thus, the dimension of the intermediate feature sequence is (T = 2688) × (C = 4800), reflecting the characteristics of 7.8 seconds of training samples. These features are then normalized through LayerNorm and projected back to 2048 from the original 4800 dimensions using a linear layer, resulting in the same size as the sequence features from both the spectral and waveform branches of HTDemucs. The encoder depth for each additional cross-domain transformer is set to 3, and cross_first = True. The batch size needs to be reduced to 16 because music-based model 31 has been added. The number of training epochs was increased to 720 to keep the number of samples the model checks during training the same. Additionally, 860 random mixes generated from the 86 training tracks of MUSDB18 were added to match the random remix extension of the default HTDemucs model. It should be noted that this was done to ensure the same extension and does not necessarily mean that new tracks are introduced into the training set. Training was then performed using only the training split of MUSDB18. HTDemucs (ablation 1): Trained with default settings (955717e8).The batch size was reduced to 16, and the number of training epochs was increased to 720. Additionally, 860 random mixes used in HTDemucs + MFM31 were added. ・HTDemucs (ablation 2): The same training as HTDemucs (ablation 1) was performed, but the number of cross-domain transformer layers was increased from 5 to 11. This corresponds to 2 x 3 = 6 additional transformer layers in HTDemucs + MFM31. ・HTDemucs + STFT-2048: The same training settings as HTDemucs + MFM31 were used, but STFT features were used instead of intermediate features. STFT was calculated from a monaural downmix with a Hann window of 2048 samples and a hop size of 512. ・HTDemucs + STFT-4096: The same training settings as HTDemucs + MFM31 were used, but STFT features were used instead of intermediate features. STFT was calculated from a monaural downmix with a Han window of 4096 samples and a hop size of 256. • HTDemucs + CLAP: The same training settings as HTDemucs + MFM31 were used, but instead of intermediate features, CLAP embeddings (see reference R2) were used. Specifically, fine-grained embeddings were used before the final AP of CLAP. This allowed for a comparison of features obtained from the two trained models (instead of STFT). • HTDemucs + MusicGen Small: The same training settings as HTDemucs + MFM31 were used, but instead of intermediate features, MusicGen intermediate features (see reference R3) were used. Specifically, the 'musicgen-small' activation was used in the output of the 12th layer. This allowed for a comparison of features obtained from the two music-based models.・HTDemucs + MusicGen Large: The training settings are the same as HTDemucs + MusicGen, but the 'musicgen-large' activation was used for the output of the 24th layer.This allowed us to compare the features obtained from two music-based models with a similar number of learnable parameters. To reduce instability during training, AdamW (see reference R41) was used as the optimizer instead of Adam in this experiment.

[0142] Reference R41: Ilya Loshchilov and Frank Hutter., Decoupled weight decay regularization. In International Conference on Learning Representations, 2017. URL https: / / api.semanticscholar.org / CorpusID:53592270.

[0143] Here, we confirmed the following: When four different feature vectors (STFT, CLAP, MusicGen, and intermediate features of the music-based model 31) were injected into HTDemucs in the music source separation task, the best separation score was achieved when the intermediate features of the music-based model 31 were injected. In this case, instability was observed when the STFT and CLAP feature vectors were injected, but no instability was observed when the intermediate features of the music-based model 31 were injected.

[0144] <Music Mixing> Music mixing is a crucial task in music production and is typically performed using audio processors or audio effects. These are signal processing systems that alter specific characteristics of the input signal. Several signal processing and machine learning techniques have been studied to automate this task (see reference R42). The goal is to simplify the process for less experienced content creators and improve the workflow capabilities of professionals (see reference R43).

[0145] Document R42: Christian J. Steinmetz et al., Deep Learning for Automatic Mixing. ISMIR, December 2022. URL https: / / dl4am.github.io / tutorial. Document R43: David Moffat and Mark B Sandler., Approaches in intelligent music production. In Arts, volume 8, pp. 125.MDPI, 2019.

[0146] Data-driven deep learning approaches for automated music mixing have focused on two fundamental frameworks: direct transformation networks, where the model performs mixing in a black-box manner, and parameter estimation networks, where mixing is performed via a differentiable audio processor. In reference R44, we proposed Wave-U-Net, an improved version for drum mixing, as a direct transformation system. Meanwhile, in reference R45, we presented a differentiable mixing console where a neural proxy functions as the parameter estimation network.

[0147] Reference R44: Marco A Martinez-Ramirez et al., A deep learning approach to intelligent drum mixing with the Wave-U-Net. Journal of the Audio Engineering Society, 2021. Reference R45: Christian J Steinmetz et al., Automatic multitrack mixing with a differentiable mixing console of neural audio effects. In ICASSP. IEEE, 2021.

[0148] Both systems are recognized as having limited performance due to insufficient available training data and failing to meet professional audio engineering standards. Such systems require raw or dry multi-track recordings and corresponding mixes, and large datasets are not readily accessible. To address this limitation, reference R46 proposes an Fx normalization preprocessing method that allows direct-to-transform automated mixing systems to be trained using processed or wet multi-track audio datasets similar to those used in source separation.

[0149] Reference R46: Marco A Martinez-Ramirez et al., Automatic music mixing with deep learning and out-of-domain data. In ISMIR, 2022.

[0150] Building on this approach, reference R47 introduced a contrast learning approach that enables direct transformation networks to perform mixing style transfer. Reference R48 further evolved both the parameter estimation and style transfer approaches by proposing a differentiable mixing style transfer system that predicts console parameters from raw tracks and a reference mix. Reference R47 used SSL embedding of the reference mix to guide the mixing style transfer task, an approach that is a data-driven automated mixing approach that improves the performance of automated mixing by incorporating SSL features from the input stem, as well as high-level information about genre, instruments, and mood.

[0151] Reference R47: Junghyun Koo et al., Music mixing style transfer: A contrastive learning approach to disentangle audio effects. 2023. Reference R48: Soumya Sai Vanka et al., Diff-MST: Differentiable mixing style transfer. In ISMIR, 2024.

[0152] Here, we use Mix-Wave-U-Net (see reference R42) and the modified CRAFx (see reference R49) (hereinafter referred to as CRAFx2) as baselines.

[0153] Document R49: Marco A Martinez-Ramirez, Emmanouil Benetos, and Joshua D Reiss. Deep learning for black-box modeling of audio effects. Applied Sciences, 10(2):638, 2020.

[0154] The inputs to both the Mix-Wave-U-Net and CRAFx2 networks are stereo stems preprocessed by Fx-normalization (see reference R46), and the output is a stereo mix. These models do not process high-abstract level information related to mixing, such as genre, instrumentation, and atmosphere. Therefore, it is expected that mixing performance will be improved by tuning these models with extracted features that implicitly contain such information. The features are computed from the monaural downmix of the mix, corresponding to the sum of the Fx-normalized input stems. To incorporate these features, both networks are tuned using a FiLM (feature-wise linear modulation) layer (see reference R50). In Mix-Wave-U-Net, the features are injected into upsampling and bottleneck 1D convolutional blocks. CRAFx2 uses a FiLM layer to tune both the latent-space mixer and the synthesis back-end.

[0155] Reference R50: Ethan Perezet al., Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.

[0156] All models were trained on MUSDB18, and the stereo-invariant loss and all training hyperparameters were the same as in previous studies (see reference R46). Because of inherent subjectivity in the task, identifying a single optimal model is difficult; therefore, objective evaluation is performed by measuring the proximity of the output mix to the target mix using the same test set as in previous studies (see reference R46). Proximity measurement is based on objective metrics (see references R42, R46). These metrics consist of low-level audio characteristics such as spectrum, panning, dynamics, and loudness, and are related to key audio characteristics frequently manipulated in the mixing process.

[0157] Figure 16 shows examples of evaluation results for the MDXDB21-dry test set and the MUSDB18 test set. In Figure 16, the intermediate features of the music-based model 31 are denoted as MFM31. As shown in Figure 16, adjusting both the Mix-Wave-U-Net and CRAFx2 architectures with the intermediate features of the music-based model 31 improves objective performance. In this case, the training curve and validation curve show a rapid decrease in loss in the initial epoch and improved generalization, respectively. Here, due to the subjective nature of the task (see reference R42), there is no standardized objective evaluation, but the presented metrics suggest that the best-performing model closely matches the target mix and resembles professional human-made mixes.

[0158] Using the intermediate features of the music-based model 31 yields the best model performance. Using MusicGen features also provides improvement, but not as effectively as the intermediate features of the music-based model 31. In particular, in CRAFx2, the default model outperforms MusicGen across various metrics. Therefore, the performance gap of MusicGen is thought to stem from the fact that it is trained on 32kHz audio compared to the 44.1kHz audio used in the music-based model 31. This may limit its effectiveness for full-band tasks such as music mixing, which require higher sampling rates. There is no data-driven approach to improving music mixing that utilizes task-agnostic features of the input stem. Therefore, it can be concluded that incorporating the intermediate features of the music-based model 31 is beneficial for both the training and performance of the automated music mixing model. This is consistent with recent design studies (see references R51, R52) that recommend incorporating contextual input.

[0159] Reference R51: M Nyssim Lefford et al., Context aware intelligent mixing systems. Journal of the Audio Engineering Society, 2021. Reference R52: Soumya Sai Vanka et al., Adoption of ai technology in the music mixing workflow: An investigation. 2023.

[0160] <<Details of Mix-Wave-U-Net, CRAFx2>> Mix-Wave-U-Net extends the U-Net architecture for audio signal processing tasks. Using a FiLM layer (see reference R50), intermediate features of the music-based model 31 are incorporated into each upsampling 1D convolutional block and bottleneck 1D convolutional block of Mix-Wave-U-Net. Following reference R53, the FiLM layer is placed after the normalization layer and before the LeakyReLU activation function.

[0161] Document R53: Gabriel Meseguer-Brocal and Geoffroy Peeters., Conditioned-u-net: Introducing a control mechanism in theu-net for multiple source separations. 2019.

[0162] CRAFx2 (see reference R46) consists of an adaptive front-end that learns the filter bank, a latent-space mixer that learns a mixing mask that functions as an equalizer, dynamic range compression, and reverberation transformer, and a synthesis backend that implements loudness and panning transforms for each filter bank channel via adaptive gain. To improve the network's mixing performance, intermediate features of the music-based model 31 are incorporated into the network's relevant layers. In CRAFx2, following an approach similar to Mix-Wave-U-Net, FiLM layers are used to tune both the latent-space mixer and the synthesis backend. The latent-space mixer is first built on a TCN (temporal dilated convolutional network), followed by stacked BLSTM (bidirectional long short-term memory) layers, with the FiLM layer inserted within the TCN block, before the BLSTM layer. In the TCN portion, the FiLM layer is placed after the depth convolution operation and before the second nonlinear activation function. The FiLM layer is introduced before the input of the BLSTM layer. Finally, the synthetic backend incorporates a squeeze-and-excitation (SE) block (see reference R54) that applies adaptive gains to scale the information per channel. The GroupNorm and FiLM layers are added after the second linear layer of the SE and before the sigmoid function.

[0163] Reference R54: Jie Hu et al., Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.

[0164] The influence of intermediate features of the music-based model 31 on a music mixing task was investigated using Mix-Wave-U-Net and CRAFx2. All network inputs were Fx-normalized (see reference R46) stereo stems (vocals, bass, drums, etc.), and the output was a stereo mix. Each input stem consisted of two channels of 10-second audio frames at 44.1 kHz. Intermediate features of the music-based model 31 were calculated from the sum of the Fx-normalized input stems using a top prior distribution and unconditional sampling. Therefore, the dimensionality of the intermediate feature sequence was (T = 3446) × (C = 4800).

[0165] Music mixing is a time-varying task, but we hypothesize that high-level abstract concepts such as genre, instrumentation, and atmosphere can improve mixing performance. Therefore, following the time-invariant retrieval preprocessing described above, AP is performed on the time dimension, averaging tokens over time to a dimension of (T = 1) × (C = 4800), then normalizing with a non-trainable LayerNorm layer (which does not include training parameters), and finally reducing the dimension to 512 with a linear layer. Here, we observed that reducing the dimension too much (e.g., to 128) degrades performance. After the linear layer, a dropout layer is added with a probability of 0.25 to avoid overfitting. The same token processing procedure is performed on all networks.

[0166] Here, 86 songs from MUSDB18 were used for training, 14 for validation, and 50 for testing. Because this dataset is significantly smaller than the one used in reference R46, the batch size per epoch was reduced from 1600 to 320, and all models were trained for 520 epochs. The default network used a recommended initial learning rate of 1 × 10⁻¹⁶. -3 This was used. In the model including the music-based model 31, the initial learning rate was set to 1 × 10 in order to ensure stability. -4The settings were configured as follows: The loss function corresponds to the stereo invariant loss (referred to as Lb), which was reported to have the best performance in reference R46, and consists of A-weighting pre-emphasis, low-pass finite impulse response filters, L2-norm on spectral magnitude, and L1-norm on spectral log-magnitude.

[0167] To investigate the effect of intermediate features in the Music-Based Model 31 (MFM31), various models were trained using features from both MusicGen Small and MusicGen Large. The training and feature injection settings were the same as for Mix-Wave-U-Net + MFM31 and CRAFx2 + MFM31, respectively, and the MusicGen features were extracted in the same way as described in the music source separation section above. To objectively evaluate the performance of all mixing systems, audio features related to the key audio characteristics manipulated by audio engineers during the mixing process were used, as shown in several studies (see references R42, A46, A48, etc.).

[0168] The calculated audio features are as follows: Spectrum features include centroid, bandwidth, contrast, roll-off, and flatness (see reference R55). Panning features include total panning, low frequency, mid frequency, and high frequency panning root mean square (PRMS) (see reference R56). Dynamic features include RMS level, dynamic spread, crest factor (see reference R57), loudness units full scale (LUFS) level, and peak loudness (see reference R58). All features were calculated using a 0.5-second running mean (see reference R56). The objective evaluation test set is the same as that in reference R46. This includes MDXDB21-dry, a dry test set of 18 tracks extracted from MDXDB21, and the MUSDB18 test set, consisting of 50 wet multi-track songs.

[0169] Document R55: Geoffroy Peeters., A large set of audio features for sound description (similarity and classification) in the CUIDADO project. Analysis / Synthesis Team. IRCAM, Paris, France, 54(0):1-25, 2004. Document R56: George Tzanetakis et al., Stereo panning features for classifying recording production style. In ISMIR, pp. 441-444, 2007. Document R57: Zheng Ma et al., Intelligent multitrack dynamic range compression. Journal of the Audio Engineering Society, 63(6):412-426, 2015. Document R58: Rec ITU-R. Itu-r bs. 1770-2, algorithms to measure audio programme loudness and true-peak audio level.International Telecommunications Union, Geneva, 2011.

[0170] The training and validation curves confirmed the following: both Mix-Wave-U-Net + MFM31 and CRAFx2 + MFM31 showed significant improvement during training and enhanced generalization during validation. These results are reflected in Figure 16, where the model including tokens from the music-based model 31 (MFM31) consistently showed improved stereo-invariant loss performance. Regarding audio effect-related features, Mix-Wave-U-Net + MFM31 performed better than the default and Mix-Wave-U-Net for most feature categories. Incorporating MusicGen features generally improves the default network, but not as much as the intermediate features of the music-based model 31.

[0171] MusicGen Small performed slightly better than the Large model, confirming the trend observed in previous downstream tasks. However, in CRAFx2, improvement over the default network was not always achieved. On the MDXDB21-dry test set, the default model performed better than MusicGen in terms of spectral and pan features and stereo invariance loss. However, MusicGen enhanced dynamics and loudness characteristics. The difference in MusicGen's performance in spectral-related metrics and generally in comparison to the MusicBased Model 31 may be due to the fact that MusicGen is trained on 32kHz audio, while both the evaluation dataset and the MusicBased Model 31's training dataset support 44.1kHz audio.

[0172] The relatively low performance of MusicGen is presumed to be due to the difference in sampling rates of the training data. Further investigation is needed to confirm this hypothesis and examine the impact of sampling rate on music-based models as a booster for music mixing tasks. It is noteworthy that the inherent differences between the MDXDB21-dry and MUSDB18 test sets are reflected in these scores. This is expected, given that the MDXDB21-dry data consists of dry multitracks, which represent a more realistic mixing scenario than the wet multitracks of MUSDB18.

[0173] Therefore, the large discrepancies in stereo invariance shown in Figure 16 are thought to be due to the fact that the target mixes are processed by different audio processors, and the timbral characteristics do not necessarily match across the trained networks. The discrepancies in the performance of low-level features related to stereo invariance loss and acoustic effects reflect the difficulty of objectively evaluating automated mixing systems, which remains an ongoing and unresolved area of ​​research (see reference R42, etc.). Therefore, a listening test is necessary for a more detailed analysis of such systems. However, the objective results reported in this disclosure support the hypothesis that incorporating intermediate features of the music-based model 31, i.e., high-level knowledge about the input stems to be mixed, improves training and generalization overall when these features are used appropriately.

[0174] As described above, this disclosure extends the use of the music-based model from MIR (music information retrieval) to general downstream music tasks, allowing task-independent intermediate representations (multiple intermediate features with different information granularities) extracted using the music-based model 31 to be applied to task-specific models for downstream music tasks (music tagging, music transcription, music source separation, music mixing, etc.). By leveraging the multiple intermediate features extracted from the music-based model 31, the granularity of information can be restricted, improving the performance of various downstream tasks, including both comprehension and generation tasks. As mentioned above, this approach was evaluated in representative tasks such as music tagging, and the results showed that the multiple intermediate features extracted from the music-based model 31 enhance the functionality of downstream task models during training.

[0175] This demonstrates that intermediate features extracted from the music-based model 31 can be used as boosters for downstream task models. Therefore, by using intermediate features extracted from the music-based model 31, the performance of various downstream tasks (downstream music tasks) can be improved. In addition to benefiting existing task-specific models, it can also support downstream music tasks that are constrained by difficulties in obtaining sufficient datasets or by computational resources that do not allow for large-scale training. This makes it possible to provide more effective and accessible music processing solutions.

[0176] <Processing Flow> Next, referring to the flowchart in Figure 17, we will explain the flow of the downstream task learning process using intermediate features, which is executed by the information processing device 1 in Figure 1.

[0177] In step S11, the intermediate feature extraction unit 21 extracts intermediate features from the music base model 31. Specifically, as explained with reference to Figures 2 to 4, by inputting music data into the music base model 31, which includes the first stage model 51 and the second stage model 52, multiple intermediate features (e.g., hierarchical features) with different information granularities (e.g., coarse granularity, medium granularity, fine granularity, etc.) are extracted.

[0178] In step S12, the downstream task processing unit 12 learns a downstream task model using the intermediate features extracted by the intermediate feature extraction unit 21. Specifically, as explained with reference to Figures 6 to 16, the downstream task model 41 for the music tagging task, the downstream task model 42 for the music transcription task, the downstream task model 43 for the music source separation task, and the downstream task model 44 for the music mixing task can be learned (retrained) using multiple intermediate features with different information granularities extracted from the music base model 31.

[0179] Furthermore, the intermediate feature extraction process in step S11 and the downstream task model training process in step S12 may be performed on the same information processing device (processor) or on different information processing devices (processors). That is, the processor of information processing device 1 may perform all processing related to the extraction process and the training process, or it may perform some processing and then have the remaining processing performed on the processor of another information processing device. In addition, information processing device 1 may have a common processor for performing each function of AI processing such as the extraction process and the training process, or it may have separate processors for each function.

[0180] <Hardware Configuration> Figure 18 shows an example of the hardware configuration of the information processing device 1 shown in Figure 1. The information processing device 1 includes a CPU (central processing unit) 211 that controls the operation of each part and performs various processing, a GPU (graphics processing unit) 212 specialized for parallel processing including AI processing such as learning and inference, a main memory 213 such as DRAM (dynamic random access memory), and an auxiliary memory 214 such as flash memory.

[0181] Auxiliary memory 214 stores data such as programs and various parameters for AI processing. The CPU 211 loads the programs and parameters stored in auxiliary memory 214 into main memory 213 and executes the programs. Alternatively, the CPU 211 and GPU 212 load the programs and parameters stored in auxiliary memory 214 into main memory 213 and execute the programs. This allows the GPU 212 to be used as GPGPU (general-purpose computing on graphics processing units). Note that if the CPU 211 executes the AI ​​processing program, the GPU 212 does not need to be provided.

[0182] The information processing device 1 also includes an input unit 215 including a keyboard and mouse, an output unit 216 including a display for displaying various information and a speaker for outputting sound, a communication unit 217 including a communication module compatible with a predetermined communication method, a drive 218 for driving removable media 219 such as semiconductor memory and optical discs, and a bus 220 for connecting them.

[0183] In the information processing device 1, AI processing can be performed by processors such as the CPU 211 and GPU 212. For example, the series of processes shown in the flowchart of Figure 17 can be executed by processors such as the CPU 211 and GPU 212. The program executed by the processor such as the CPU 211 can be recorded and provided on removable media 219, such as package media. The program can also be provided via wired or wireless transmission media such as a local area network, the internet, or digital satellite broadcasting.

[0184] In the information processing device 1, a program can be installed in the auxiliary memory 214 by inserting a removable media 219 into the drive 218. Alternatively, a program can be received by the communication unit 217 via a wired or wireless transmission medium and installed in the auxiliary memory 214. In addition, the program may be pre-installed in the auxiliary memory 214. The processing performed by the information processing device 1 according to the program includes processing that is executed in parallel or individually (for example, parallel processing or object-based processing). Furthermore, the program may be processed by one computer (processor) or processed in a distributed manner by multiple computers.

[0185] The embodiments of this disclosure are not limited to those described above, and various modifications are possible without departing from the gist of this disclosure. For example, this technology can be configured as cloud computing, in which a single function is shared and processed jointly by multiple devices via a network. In addition, each step described in the flowchart above can be executed by one device or shared and executed by multiple devices. Furthermore, if a single step includes multiple processes, those processes included in that single step can be executed by one device or shared and executed by multiple devices.

[0186] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur. This disclosure can take the following form:

[0187] (1) An information processing method comprising: inputting music data into a music base model trained using a dataset, extracting a plurality of intermediate features having different information granularities from the intermediate layer of the music base model, and training a model for a downstream task of the music base model using the extracted intermediate features. (2) The information processing method according to (1), wherein the music base model is composed of a multilevel transformer equipped with a multi-layer encoder. (3) The information processing method according to (2), wherein the music base model is composed of a first stage model that performs hierarchical representation learning that divides information into different levels based on granularity, and a second stage model that learns a multilevel token stream extracted from the first stage model using autoregressive modeling. (4) The information processing method according to any one of (1) to (3), wherein the intermediate features are hierarchical features corresponding to different information granularities. (5) The information processing method according to any one of (1) to (4), wherein the intermediate features include a first intermediate feature corresponding to a first granularity which is the coarsest information granularity, a second intermediate feature corresponding to a second granularity which is the finest information granularity, and a third intermediate feature corresponding to a third granularity which is an information granularity between the first and second granularities. (6) The information processing method according to (5), wherein the downstream task includes an understanding task and a generation task. (7) The information processing method according to (6), wherein the downstream task includes a first task corresponding to music tagging, and the model for the first task is learned using the first intermediate features, the second intermediate features, and the third intermediate features. (8) The information processing method according to (6) or (7), wherein the downstream task includes a second task corresponding to music transcription, and the model for the second task is learned using at least the first intermediate features and the third intermediate features. (9) The information processing method according to any one of (6) to (8), wherein the downstream task includes a third task corresponding to music source separation, and the model for the third task is learned using at least the first intermediate features.(10) The downstream task includes a fourth task corresponding to music mixing, and the model for the fourth task is learned using at least the first intermediate features, as described in any of (6) to (9). (11) An information processing device comprising: a first processing unit that inputs music data into a music base model learned using a dataset and extracts a plurality of intermediate features having different information granularities from the intermediate layer of the music base model; and a second processing unit that learns a model for a downstream task of the music base model using the extracted intermediate features. (12) A program that causes a computer to function as a first processing unit that inputs music data into a music base model learned using a dataset and extracts a plurality of intermediate features having different information granularities from the intermediate layer of the music base model; and a second processing unit that learns a model for a downstream task of the music base model using the extracted intermediate features.

[0188] 1 Information processing device, 11 Base model processing unit, 12 Downstream task processing unit, 21 Intermediate feature extraction unit, 31 Music base model, 41-44 Downstream task models, 51 First stage model, 51A First stage encoder, 51B First stage decoder, 52 Second stage model, 61-63 Encoders, 71 Bottom block, 72 Middle block, 73 Top block, 81-83 Decoders, 91 Top prior distribution model, 92 Middle conditional prior distribution model, 93 Bottom conditional prior distribution model, 211 CPU, 212 GPU, 213 Main memory, 214 Auxiliary memory, 215 Input unit, 216 Output unit, 217 Communication unit, 218 Drive, 219 Removable media, 220 Bus

Claims

1. An information processing method comprising: inputting music data into a music-based model trained using a dataset, extracting a plurality of intermediate features having different information granularities from the intermediate layer of the music-based model, and training a model for a downstream task of the music-based model using the extracted intermediate features.

2. The information processing method according to claim 1, wherein the music base model comprises a multilevel transformer equipped with a multi-layer encoder.

3. The information processing method according to claim 2, wherein the music base model comprises a first stage model that performs hierarchical representation learning to divide information into different levels based on granularity, and a second stage model that learns a multilevel token stream extracted from the first stage model using autoregressive modeling.

4. The information processing method according to claim 3, wherein the intermediate features are hierarchical features corresponding to different information granularities.

5. The information processing method according to claim 1, wherein the intermediate features include a first intermediate feature corresponding to a first granularity which is the coarsest information granularity, a second intermediate feature corresponding to a second granularity which is the finest information granularity, and a third intermediate feature corresponding to a third granularity which is an information granularity between the first and second granularities.

6. The information processing method according to claim 5, wherein the downstream task includes an understanding task and a generation task.

7. The information processing method according to claim 6, wherein the downstream task includes a first task corresponding to music tagging, and the model for the first task is learned using the first intermediate feature, the second intermediate feature, and the third intermediate feature.

8. The information processing method according to claim 6, wherein the downstream task includes a second task corresponding to music transcription, and the model for the second task is learned using at least the first intermediate feature and the third intermediate feature.

9. The information processing method according to claim 6, wherein the downstream task includes a third task corresponding to music source separation, and the model for the third task is learned using at least the first intermediate features.

10. The information processing method according to claim 6, wherein the downstream task includes a fourth task corresponding to music mixing, and the model for the fourth task is learned using at least the first intermediate features.

11. An information processing device comprising: a first processing unit that inputs music data into a music-based model trained using a dataset and extracts a plurality of intermediate features having different information granularities from the intermediate layer of the music-based model; and a second processing unit that trains a model for a downstream task of the music-based model using the extracted intermediate features.

12. A program that causes a computer to function as a first processing unit that inputs music data into a music-based model trained using a dataset and extracts multiple intermediate features with different information granularities from the intermediate layer of the music-based model, and as a second processing unit that trains a model for a downstream task of the music-based model using the extracted intermediate features.

Images