Generation device and generation method

Abstract

Provided is a generation device comprising a control unit for executing a sound data set generation process for generating a sound data set, which is a set including first sound data and second sound data as elements, on the basis of a large set which is a set including a plurality of small sets, each of which is a set including sound data and a type tag, which is information indicating the type of sound indicated by the sound data, as elements. The first sound data is sound data included in a first small set, which is one of the small sets included in the large set. The second sound data is sound data included in a second small set, which is one of the small sets included in the large set and satisfies the condition that the type indicated by the type tag is different from the type indicated by the type tag included in the first small set. The small sets each further include, as an element, text data of text describing the sound data. The sound data set generated by the sound data set generation process further includes text data, included in the first small set, as an element in addition to the first sound data and the second sound data.

Description

Generating apparatus and generating method

[0001] The present invention relates to a generating apparatus and a generating method.

[0002] There are acoustic separation techniques (see, for example, Non-Patent Document 1).

[0003] X. Liu, H. Liu, Q. Kong, X. Mei, J. Zhao, Q. Huang, MD Plumbley, and W. Wang, “Separate what you describe: Language-queried audio source separation,” Interspeech 2022, pp. 1801-1805, 2022

[0004] In acoustic separation techniques, mathematical models for separation are sometimes obtained through machine learning. In such cases, training data is required, and one type of training data used is mixed sound data. This mixed sound data is obtained by randomly selecting multiple sound data from a large, pre-prepared set of sound data, and then mixing the sounds they represent.

[0005] As a result, the resulting mixed sounds sometimes contained multiple sounds of the same type, such as the sound of a guitar. When such mixed sounds were used for training, the inference accuracy of the resulting trained model (i.e., the accuracy of sound source separation) was sometimes lower than in cases where this was not the case, because it was difficult to separate the sound sources.

[0006] In view of the above circumstances, the present invention aims to provide a technology that improves the accuracy of sound source separation using a mathematical model for acoustic separation.

[0007] One aspect of the present invention is a generation device comprising a control unit that performs a sound data set generation process to generate a sound data set, which is a set containing first sound data and second sound data, based on a large set which is

[0008] One aspect of the present invention is a generation method comprising: a control step in which a computer performs a sound data set generation process to generate a sound data set, which is a set containing first sound data and second sound data, based on a large set which is

[0009] This invention makes it possible to improve the accuracy of sound source separation using a mathematical model for acoustic separation.

[0010] An explanatory diagram for explaining the information processing apparatus according to the embodiment. A diagram showing an example of an algorithm for sound data set generation processing in the embodiment. A diagram showing a first example of experimental results in the embodiment. A diagram showing a second example of experimental results in the embodiment. A diagram showing a third example of experimental results in the embodiment. A diagram showing a fourth example of experimental results in the embodiment. A diagram showing an example of a hardware configuration of the information processing apparatus 1 in the embodiment. A flowchart showing an example of a flow of processing executed by the information processing apparatus 1 in the embodiment.

[0011] (Embodiment) Fig. 1 is an explanatory diagram for explaining the information processing apparatus 1 according to the embodiment. Note that the information processing apparatus 1 is an example of a generation apparatus. The information processing apparatus 1 comprises a control unit 11 including a processor 91 such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Network Processing Unit) connected via a bus, and a memory 92, and executes a program.

[0012] The control unit 11 executes sound data set generation processing. The sound data set generation processing is processing for generating a sound data set based on a large set that is a set including a plurality of small sets. Each small set is a set including, as elements, sound data and a type tag that is information indicating the type of sound represented by the sound data.

[0013] The sound data set includes first sound data s ｔａｒ and second sound data s ｉｎｔ as elements. The first sound data s ｔａｒ is sound data included in a first small set {s ｔａｒ , ID ｔａｒ}, which is one of the small sets included in the large set. Note that ID ｔａｒ is a type tag indicating the type of sound represented by the first sound data s ｔａｒ .

[0014] The second sound data s ｉｎｔ is sound data included in a second small set {s ｉｎｔ , ID ｉｎｔ}. The second small set {s ｉｎｔ , ID ｉｎｔ} is one of the subsets included in the large set, and satisfies the condition that the type indicated by the type tag is different from the type indicated by the type tag included in the first subset. Note that ID ｉｎｔ is the second tone data s ｉｎｔ This is a type tag that indicates the type of sound it represents.

[0015] 1st small set {s ｔａｒ ID ｔａｒ} and the second minor set {s ｉｎｔ ID ｉｎｔ} are different subsets because they have different type tags, and the first sound data s ｔａｒ and second tone data s ｉｎｔ These are two different types of sound data.

[0016] Note that the grand set is a pre-defined set. While the sound data differs for each subset within the grand set, it is not necessary for all subsets to have different type tags. In other words, the grand set may contain subsets that have different sound data but the same type tag.

[0017] In the example in Figure 1, the large set has a small set {s １ ID １} to the subset {s １０ ID １０ It contains 10 subsets up to}. In the example in Figure 1, the result of the sound data set generation process is sound data s １ and sound data s ３ A set of sound data consisting of {s １ s ３} or sound data set {s ５ s ９ Multiple sets of sound data, such as}, have been obtained.

[0018] Sound data set {s} in Figure 1 １ s ３} sound data s １ For example, the first tone data s ｔａｒ This is an example, and in this case the sound data set {s １ s ３} sound data s ３ is the second tone data s ｉｎｔ This is one example. Also, the sound data set {s５ s ９} sound data s ５ For example, the first tone data s ｔａｒ This is an example, and in this case the sound data set {s ５ s ９} sound data s ９ is the second tone data s ｉｎｔ This is one example.

[0019] In the example in Figure 1, the small set {s １ ID １ Type tag ID in} １ and the small set {s ３ ID ３ Type tag ID in} ３ Because they were different, the sound data set generation process resulted in the sound data set {s １ s ３} has been obtained.

[0020] Also, in the example in Figure 1, the subset {s ５ ID ５ Type tag ID in} ５ and the small set {s ９ ID ９ Type tag ID in} ９ Because they were different, the sound data set generation process resulted in the sound data set {s ５ s ９} has been obtained.

[0021] ID １ ID ３ ID ５ and ID ９ These are all examples of type tags. Also, sound data s １ is the first tone data s ｔａｒ If so, the small set {s １ ID １} is the first subset, ID １ is the first tone data s １ A type tag ID indicating the type of sound. ｔａｒ And in this case, the small set {s ３ ID ３} is the second minor set, s ３ is the second tone data s ｉｎｔ ID ３ is the second tone data sｉｎｔ which is the type tag ID indicating the type of sound ｉｎｔ .

[0022] When sound data s ５ is first sound data s ｔａｒ , the subset {s ５ , ID ５} is a first subset, wherein ID ５ is the type tag ID indicating the type of sound represented by the first sound data s ５ which is the type tag ID indicating the type of sound ｔａｒ . In this case, the subset {s ９ , ID ９} is a second subset, wherein s ９ is second sound data s ｉｎｔ , and ID ９ is the type tag ID indicating the type of sound represented by the second sound data s ９ which is the type tag ID indicating the type of sound ｉｎｔ .

[0023] <More Specific Example of Sound Data Set Generation Processing> A more specific example of sound data set generation processing will be described. In an example of sound data set generation processing, a control unit 11 first acquires one subset randomly from the universal set. Next, the control unit 11 acquires another one subset randomly from the universal set.

[0024] Subsequently, the control unit 11 determines whether the respective type tags of the two acquired subsets are different from each other. If the result of this determination shows that the respective type tags of the two acquired subsets are different from each other, the control unit 11 generates, as a sound data set, a set including all sound data of the two acquired subsets as elements. The result of this generation (that is, the generated sound data set) is recorded in a predetermined storage destination.

[0025] On the other hand, if the result of the determination shows that the respective type tags of the two acquired subsets are not different from each other, the control unit 11 does not generate a set including all sound data of the two acquired subsets as elements. Therefore, no sound data set including all sound data of the two acquired subsets as elements is generated.

[0026] This series of processes is repeatedly executed until a predetermined termination condition such as a predetermined number of executions is satisfied. This is an example of sound data set generation processing. Note that this is merely an example. In this example, a small set is randomly acquired from the large set, but the acquisition is not necessarily required to be random. The small set may be acquired according to a predetermined rule other than random. As will be described in detail later, the small set may be acquired based on, for example, a type tag. Further, when text data of a text describing sound data is included in the small set, the small set may be acquired based on the text data.

[0027] <Example where text data is included as an element> As described above, the small set may further include, as an element, text data of a text describing sound data. In this case, the sound data set generated by the sound data set generation processing is first sound data s ｔａｒ and second sound data s ｉｎｔ , and further includes, as an element, text data t included in the first small set ｔａｒ as an element. That is, the sound data set generated when the small set also includes text data as an element is a sound data set {s ｔａｒ , t ｔａｒ , s ｉｎｔ}.

[0028] Note that when the small set also includes text data as an element, the small set may be acquired based on the text data as described above. That is, the first small set {s ｔａｒ , ID ｔａｒ , t ｔａｒ} and the second small set {s ｉｎｔ , ID ｉｎｔ , t ｉｎｔ} may be acquired from the large set by sampling based on text data. An example of sampling based on text data will be described later.

[0029] FIG. 2 is a diagram illustrating an example of an algorithm for sound data set generation processing according to the embodiment. Note that the example of FIG. 2 is an example of an algorithm for sound data set generation processing in a case where text data of a text describing sound data is also included as an element in the small set.

[0030] One of the purposes of forming a set with this algorithm is, for example, S ｔａｒ and S ｉｎｔ The sound of mixing and is separated into S ｔａｒ This is to use it as training data for solving sound source separation, which extracts only the relevant part. Therefore, the set obtained by this algorithm is, for example, S ｔａｒ and S ｉｎｔ The sound of mixing and is separated into S ｔａｒ It is used as training data to solve sound source separation, which extracts only the relevant parts of the sound.

[0031] However, as can be seen from the algorithm in Figure 2, it is possible to generate a set of sound data even if no text data exists. Therefore, the algorithm in Figure 2, with the presence of text data removed, is an example of an algorithm for generating a set of sound data when the subset does not contain text data.

[0032] In Figure 2, D is an example of a large set. ｉ ,t ｉ ) is sound data s ｉ and text data t ｉ This represents a subset whose elements are and . Note that in the example in Figure 2, for the sake of clarity, it is not explicitly stated that the subset includes type tags. However, the subset does include type tags.

[0033] In Figure 2, B represents the mini-batch size. In Figure 2, the phrase "Target and interference samples for a mini-batch" means that an empty list of size B will be created, since the mini-batch size is B.

[0034] Figure 2 shows that the following process is performed for each variable j, from 1 to B. First, the subset {s ｔａｒ ,t ｔａｒ This indicates that} is obtained by sampling. In Figure 2, "sampling" indicates that sampling is performed. Note that t ｔａｒ is sound data s ｔａｒ This is text data of the text that explains it.

[0035] Any type of sampling may be used in the above sampling. This sampling may be, for example, sampling based on type tags, sampling based on text data, or random sampling.

[0036] Sampling based on type tags is the process of obtaining a small set of data based on type tags, while sampling based on text data is the process of obtaining a small set of data based on text data.

[0037] In the example in Figure 2, next we have the subset {s ｉｎｔ ,t ｉｎｔ This indicates that {s} is obtained by sampling. Sampling here is obtained by sampling a small set {s}. ｔａｒ ,t ｔａｒ Sampling is performed using the same method as when obtaining}.

[0038] In the example in Figure 2, the next step is to obtain the subset {s}. ｔａｒ ,t ｔａｒ} and the subset {s ｉｎｔ ,t ｉｎｔ It is determined whether the identification tags for} are different from each other. tar and s int "are overlapped" indicates this. If the identification tags are different from each other, one of the minibatches contains the sound data set {s ｔａｒ ,t ｔａｒ s ｉｎｔ} is added.

[0039] The above process is performed for each j from j=1 to j=B.

[0040] <Sampling Based on Type Tags> Here, we will explain examples of sampling based on type tags. Specifically, we will explain Tag-based weighted sampling (TWBS) and Tag-based group sampling (TGBS) as examples of sampling based on type tags. In the case of sampling based on type tags, the subset does not need to contain text data as long as it contains type tags.

[0041] <<TWBS>> As mentioned above, different subsets have different sound data, but this is not necessarily the case for type tags. As a result, there may be differences in the frequency of occurrence between type tags. Therefore, TWBS is a technology that attempts to eliminate as much as possible the differences in the frequency of occurrence between type tags.

[0042] TWBS is a weight w expressed by the following equation (1). ｉ This sampling is performed with the weights already calculated, specifically the weights w ｉ This is a sampling method using . Note that i is an identifier that identifies each subset included in the large set. ｉ c is a set of tags for identifier i. ｉ This is the weight defined for [the variable].

[0043]

[0044] The N that appears in equation (1) ｋ In this subset, the type tag is the ID. ｋ It represents the total number of things that are [that].

[0045] By the way, each sound data does not necessarily have to be assigned only one type tag; it may be assigned multiple types. Therefore, a subset does not necessarily have to contain only one type tag; it may contain multiple types. Also, the number of type tags contained in a subset does not need to be the same for all subsets included in the larger set.

[0046] When a subset contains multiple type tags, a subset with the same type tag is one in which at least one tag is common among the tags it contains. Therefore, the second subset {s} described above ｉｎｔ ID ｉｎｔ The condition that appears in the definition of}, "one of the subcategories included in the larger set, where the type indicated by the type tag is different from the type indicated by the type tag included in the first subset," means that if there are multiple type tags in the second subset or the first subset, all tags in the second subset are different from the tags included in the first subset.

[0047] In fact, k is an identifier that identifies each type tag contained in the subset of identifier i. Therefore, c ｉThis represents the set of type tag identifiers contained within the subset of identifier i.

[0048] As mentioned above, TWBS is weighted w ｉ This is a sampling method that uses weights w ｉ This sampling method satisfies the condition that smaller sets of values ​​are more likely to be selected as samples.

[0049] Therefore, TWBS is a sampling method in which smaller sets of type tags that are rarer are more likely to be acquired as the first or second set. Thus, TWBS helps to equalize the frequency of occurrence of type tags.

[0050] <<TGBS>> TBGS is a sampling method in which subsets are divided into groups according to type tags. Hereafter, the group formed by assigning a subset to each type tag will be called the first subset group. Also below, the number of first subset groups formed by assigning a subset to each type tag will be denoted as X. In other words, X is the number of type tags. X is greater than or equal to the batch size B.

[0051] Note: audio data ｉ Multiple type tags may be assigned to the same type. When multiple type tags are assigned, the sound data is assigned to the first subgroup for each type tag. Conversely, sound data s that exist in the same group ｉ This is a collection of items that have at least one matching type tag.

[0052] Let's explain TGBS in more detail. TGBS is a sampling method in which B groups of first subgroups are randomly selected from X first subgroups, and one subgroup is taken from each of the selected B groups. In this case as well, the frequency of occurrence between type tags is equalized.

[0053] <Sampling Based on Text Data> Next, we will explain examples of sampling based on text data. Specifically, we will explain Embedding-based weighted sampling (EWBS) and Embedding-based group sampling (EGBS) as examples of sampling based on text data. In the case of sampling based on text data, unlike sampling based on type tags, the subset must contain text data.

[0054] <<EWBS>> EWBS is a weight a expressed by the following equation (2) ｉ This sampling is performed with the assumption that the values ​​have been calculated in advance. i is an identifier that identifies each subset included in the large set. ｉ This is a weight defined for a subset of identifiers i.

[0055]

[0056] The d that appears in equation (2) ｉ，ｌ This is the cosine distance of the sentence embeddings between the text data of the subset of identifier i and the text data of the subset of identifier l. A sentence embedding is a vector that represents the embedding of sentences, and the cosine distance between similar sentences approaches 0, while the cosine distance between different sentences approaches 2.

[0057] The definition of τ appearing in number (2) is a constant greater than 0 that adjusts the weights. ｉ This is a subset determined to be similar to the subset of identifier i. The subset determined to be similar to the subset of identifier i refers to a set of sentence embeddings selected from all sentence embeddings, in any order and of any number, starting with those belonging to the subset with the closest cosine distance to the subset of identifier i, or selected by any method such as having a cosine distance within any range.

[0058] Note that the set with the smallest cosine distance to the set of identifier i is the set with the smallest representative cosine distance. The representative cosine distance is the cosine distance calculated by any method, such as the smallest or average, among the cosine distances obtained between all the text data belonging to the set of identifier j and all the text data belonging to the set of identifier j. This determination is made before the sound data set generation process is executed. N appearing in equation (2) ｎ This is the number of subsets that were determined to be similar to the subset of identifier i. The more similar subsets there are, the higher the weight a. ｉ The smaller and rarer the item, the greater its weight a ｉ It's big.

[0059] As mentioned above, EWBS uses weight a ｉ This is a sampling method that uses weight a ｉ This sampling method satisfies the condition that smaller sets of values ​​are more likely to be selected as samples.

[0060] <<EGBS>> EGBS is a sampling method that is performed after a second set of subgroups has been obtained. Let's explain how the second set of subgroups is obtained. First, embeddings of the text data contained in the subset are obtained. Next, clustering such as K-means clustering is performed on the obtained embeddings to divide the embeddings into clusters. The result of grouping using this result is the second set of subgroups.

[0061] More specifically, a second subgroup is a collection of subgroups that are grouped by the cluster to which the embedding belongs. Therefore, subgroups belonging to different second subgroups have different clusters to which the text data embeddings belong, while subgroups belonging to the same second subgroup have the same cluster to which the text data embeddings belong.

[0062] In light of the above, the definition of a second-tier subgroup is as follows: A second-tier subgroup is a collection of subgroups that satisfy the conditions that the clusters to which the text data belong are the same, and that these clusters are obtained by a predetermined clustering of the text data embeddings.

[0063] Hereafter, the number of second subgroups will be denoted as Y. Y is greater than or equal to the batch size B.

[0064] Let's explain EGBS in more detail. EGBS is a sampling method in which B groups of second subsets are randomly selected from Y groups of second subsets, and one subset is taken from each of the selected B groups.

[0065] <Experimental Results> Examples of experimental results will be explained using Figures 3 to 6. Specifically, Figures 3 to 6 are examples of experimental results using data and evaluation scales used in DCASE 2024 Challenge Task 9 (DCASE24 T9), a challenge to compete on the performance of LASS (Language-queried audio source separation). LASS is one of the mathematical models for performing acoustic separation.

[0066] Figure 3 shows a first example of experimental results in the embodiment. In Figure 3, the vertical axis represents SDR (Signal-to-Distortion Ratio), and the horizontal axis represents the trained LASS. Therefore, Figure 3 shows the SDR for each trained LASS on the horizontal axis. The unit of the vertical axis is dB. SDR is an index used to evaluate the quality of separated sound sources and indicates the ratio of distortion between the separated sound source and the original sound source.

[0067] The differences in the names of the trained LASS models on the horizontal axis represent differences in how the mixed sound data used to train the LASS models was obtained. This is also true for Figures 4 and 5.

[0068] The mixed sound data used to obtain the "Baseline" on the horizontal axis consists of mixed sounds obtained using pairs of audio data and text data that were randomly selected from a pre-prepared set of pairs that did not have type tags.

[0069] The mixed sound data used to obtain "CLS" on the horizontal axis is the mixed sound data obtained when sampling is random in the algorithm shown in Figure 2.

[0070] The mixed sound data used to obtain "CLS + TWBS" on the horizontal axis is the mixed sound data obtained when sampling is TWBS in the algorithm shown in Figure 2.

[0071] The mixed sound data used to obtain "CLS + TGBS" on the horizontal axis is the mixed sound data obtained when sampling is TGBS in the algorithm shown in Figure 2.

[0072] The mixed sound data used to obtain "CLS + EWBS" on the horizontal axis is the mixed sound data obtained when sampling is EWBS in the algorithm shown in Figure 2.

[0073] The mixed sound data used to obtain "CLS + EGBS" on the horizontal axis is the mixed sound data obtained when sampling is EWBS in the algorithm shown in Figure 2.

[0074] Therefore, among the mathematical models on the horizontal axis, all except "Baseline" represent the results of learning using mixed sound data obtained using the sound data set obtained through the sound data set generation process.

[0075] The results in Figure 3 show that the sound data set generation process improves SDR, one of the indicators of the performance of sound source separation using a mathematical model for sound source separation.

[0076] <<Regarding the mixed sound data>> The mixed sound data obtained using the sound data set is sound data that represents a mixed sound obtained by mixing the sound represented by the first sound data and the sound represented by the second sound data included in the sound data set. Any well-known technique for generating mixed sound data by mixing the sounds represented by two sound data can be used as the technique for obtaining the mixed sound data from the sound data set.

[0077] Furthermore, the process of obtaining mixed sound data from the sound data set may be performed by the control unit 11 itself, or by another device. The sound data set obtained by the sound data set generation process may be recorded in a predetermined storage location. Therefore, the acquisition of the mixed sound data does not necessarily have to be performed by the control unit 11, as long as there is a device that can read the sound data set from that storage location and can obtain mixed sound data from the sound data set.

[0078] Figure 4 shows a second example of experimental results in the embodiment. In Figure 4, the vertical axis represents SDRI (SDR improvement). The horizontal axis is the same as the horizontal axis in Figure 3. Therefore, Figure 4 shows the SDRI for each trained LASS on the horizontal axis. The unit of the vertical axis is dB. SDRI is an index that shows the difference between the SDR after processing and the SDR before processing, and is an index that measures the degree of improvement in separation performance. Before processing refers to the SDR of the sound source before sound source separation by the mathematical model of sound source separation.

[0079] The results in Figure 4 show that the sound data set generation process improves SDRI, one of the indicators of the performance of sound source separation using a mathematical model for sound source separation.

[0080] Figure 5 shows a third example of experimental results in the embodiment. In Figure 5, the vertical axis represents SISDR (Scale-Invariant SDR). The horizontal axis is the same as the horizontal axis in Figure 3. Therefore, Figure 5 shows the SISDR for each trained LASS on the horizontal axis. The unit of the vertical axis is dB. Note that SISDR is a more robust SDR index that measures distortion independently of the signal scale (amplitude).

[0081] The results in Figure 5 show that the sound data set generation process improves SISDR, one of the indicators of the performance of sound source separation using a mathematical model for sound source separation.

[0082] Figure 6 shows a fourth example of experimental results in the embodiment. Figure 6 is an example of the results of performance evaluations of LASS-Net, ClapSEP, AudioSep, the "Baseline" technology in Figure 3, the technologies that ranked 5th, 4th, 3rd, 2nd, and 1st in DCASE 2024 Challenge Task 9, and "CLS+EWBS" in Figure 3.

[0083] Furthermore, the technology that came in 5th place in DCASE 2024 Challenge Task 9 is shown as “5” in Figure 6. th It is labeled as "Chung_KT_1". The technology that came in 4th place in DCASE 2024 Challenge Task 9 is labeled as "4" in Figure 6. th It is written as "Romaniuk_SRPOL_2".

[0084] The technology that came in third place in DCASE 2024 Challenge Task 9 is shown in Figure 6 as “3 rd It is labeled as "HanYin_NWPU-JLESS_4". The technology that came in second place in DCASE 2024 Challenge Task 9 is shown as “2 nd It is labeled as "Guan_HEU_2". The technology that came in first place in DCASE 2024 Challenge Task 9 is shown as “1 st It is written as "Kim_GIST-AunionAI_4".

[0085] The "Baseline" technology in Figure 3 is also labeled "Baseline" in Figure 6. The "CLS + EWBS" technology in Figure 3 is also labeled "CLS + EWBS" in Figure 6.

[0086] Of the technologies shown in Figure 6, only "CLS + EWBS" uses mixed sound data generated from a sound data set obtained through a sound data set generation process for training. The other technologies do not.

[0087] The results in Figure 6 show that for SDR, SDRI, and SISDR, "CLS + EWBS" yielded higher results than the others.

[0088] <<Effects of Sound Data Set Generation Process>> As shown in Figures 3 to 6, the sound data set generation process improves the performance of sound source separation using a mathematical model for sound source separation. This is because the sound data sets obtained through the sound data set generation process are sets of different types of sound data.

[0089] Furthermore, because the frequency of sound occurrence differs depending on the type of sound, there is a bias in the number of sounds included in each type tag or text data within that data. When TWBS, TGBS, EWBS, or EGBS are also used, the degradation of sound source separation caused by this bias is improved. This is because when TWBS or TGBS are also used, the frequency of occurrence between type tags is made uniform. Also, when EWBS or EGBS are also used, the sound data is combined in a way that shows a large difference in the text data and used for training.

[0090] Because the sound data set is structured in this way, when creating mixed sound data, the operator or computer simply needs to mix the sounds represented by each sound data in the given sound data set. This alone generates mixed sound data where sounds of the same type are not mixed. Furthermore, when TWBS, TGBS, EWBS, or EGBS are also used, this mixed sound has a balanced number of sounds of each type. As a result, using this mixed sound data to train a mathematical model for source separation yields a trained mathematical model with higher separation performance.

[0091] In this way, the process of generating a sound data set improves the accuracy of sound source separation performed by the mathematical model used for acoustic separation.

[0092] <Example of Hardware Configuration of Information Processing Device 1> Figure 7 shows an example of the hardware configuration of the information processing device 1 in the embodiment. The information processing device 1 includes a control unit 11 and executes a program. The information processing device 1 functions as a device comprising a control unit 11, an interface unit 12, and a storage unit 13 by executing a program.

[0093] More specifically, the processor 91 reads the program stored in the storage unit 13 and stores the read program in the memory 92. By executing the program stored in the memory 92, the information processing device 1 functions as a device comprising a control unit 11, an interface unit 12, and a storage unit 13.

[0094] The control unit 11 controls the operation of each functional unit of the information processing device 1. For example, the control unit 11 performs sound data set generation processing. For example, the control unit 11 acquires information stored in the storage unit 13. Specifically, the process of acquiring information stored in the storage unit 13 is read.

[0095] Furthermore, the control unit 11 may perform a process to generate mixed sound data from the sound data set obtained in the sound data set generation process. The control unit 11 may also perform a process to infer type tags based on the sound data. Such a process may, for example, be a process to execute a trained mathematical model that infers type tags based on sound data (hereinafter referred to as the "trained tag inference model"). In this case, the trained tag inference model may be a well-known mathematical model that infers type tags based on sound data.

[0096] The type tag may be obtained manually for each audio data set in advance, or it may be obtained for each audio data set by performing a process that infers the type tag based on the audio data, such as running a trained tag inference model.

[0097] Furthermore, the control unit 11 may perform training on a mathematical model for acoustic separation.

[0098] The interface unit 12 includes a communication interface for connecting the information processing device 1 to an external device. The interface unit 12 communicates with the external device via wired or wireless connection.

[0099] When the control unit 11 performs sound data set generation processing, the external device is, for example, the device that transmits the large set. In such a case, the interface unit 12 acquires the large set by communicating with the device that transmits the large set. When the control unit 11 performs processing to infer type tags based on sound data, the external device may be the device that transmits the sound data. In such a case, the interface unit 12 acquires the sound data by communicating with the device that transmits the sound data.

[0100] The external device may be, for example, a device that receives the audio data set as an output destination. Such a device acquires the audio data set via the interface unit 12. After acquiring the audio data set, such a device generates audio data for a mixed sound.

[0101] When the control unit 11 performs learning of a mathematical model for acoustic separation, the external device may be, for example, a device that transmits information used for learning, including sound data of mixed sounds.

[0102] The interface unit 12 may include input devices such as a mouse, keyboard, touch panel, and microphone. The interface unit 12 may also be configured as an interface connecting these input devices to the information processing device 1. In this way, the input devices of the interface unit 12 receive various types of information or signals to the information processing device 1 via wired or wireless connections. Note that the information or signals do not necessarily have to be input to the communication interface of the interface unit 12, but may also be input to the input devices of the interface unit 12.

[0103] The interface unit 12 outputs various types of information, for example. The interface unit 12 includes, for example, a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display, as well as a speaker. The interface unit 12 may be configured as an interface for connecting these display devices or speakers to the information processing device 1. Therefore, the interface unit 12 may output information indicated by information or signals input to the input device of the interface unit 12 in the form of an image or sound.

[0104] The storage unit 13 is configured using a computer-readable recording medium such as a magnetic hard disk drive or a semiconductor memory device. The storage unit 13 stores various information related to the information processing device 1. The storage unit 13 stores various information generated by the operation of the control unit 11, for example. The storage unit 13 may reside, for example, on the cloud.

[0105] <Example of the processing flow executed by the information processing device 1> Figure 8 is a flowchart showing an example of the processing flow executed by the information processing device 1 in the embodiment. The control unit 11 acquires a large set (step S101). Next, the control unit 11 executes sound data set generation processing (step S102). The sound data set obtained by the control unit 11 through the execution of sound data set generation processing may be recorded in a predetermined storage location such as the storage unit 13.

[0106] Furthermore, the sound data set may be used to generate mixed sound data, and the generated mixed sound data may be output to a predetermined output destination. The predetermined output destination may be a predetermined storage location such as the memory unit 13, or it may be a mathematical model that performs sound source separation for learning.

[0107] The information processing device 1 configured in this way performs sound data set generation processing. Therefore, as described in the above section on "Effects of Sound Data Set Generation Processing," it is possible to improve the accuracy of sound source separation using a mathematical model that performs acoustic separation.

[0108] (Variant)

[0109] The algorithm illustrated in Figure 2 was designed to prevent the creation of pairs of sound data representing the same type of sound by using type tags. However, when a subset contains text data, it is not always necessary to obtain a set of sound data using type tags. For example, a set of sound data could be generated from sound data of subsets whose text data is not identical.

[0110] Furthermore, the mathematical model used for learning, which performs sound separation and uses a mixed sound data obtained from a sound data set generation process, may be, for example, LASS (Language-queried audio source separation).

[0111] The information processing device 1 may be implemented using multiple information processing devices connected to each other via a network. In this case, each process executed by the control unit 11 may be performed in a distributed manner by the multiple information processing devices.

[0112] Furthermore, all or part of each function of the information processing device 1 may be implemented using hardware such as ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, magnetic disks, magneto-optical disks, optical disks (CD-ROM, DVD-ROM, etc.), portable media such as semiconductor memory (volatile memory, non-volatile memory, etc.) (ROM, RAM, etc.), and storage devices such as hard disks built into computer systems. The program may also be transmitted via a telecommunications line.

[0113] Although embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention.

[0114] 1... Information processing device, 11... Control unit, 12... Interface unit, 13... Storage unit, 91... Processor, 92... Memory

Claims

1. A sound data set generation device comprising: a control unit that performs a sound data set generation process that generates a sound data set, which is a set containing first sound data and second sound data, based on a large set which is 2. The generation apparatus according to claim 1, wherein the control unit, in the sound data set generation process, obtains or selects the first subset and the second subset from the large set by sampling based on the text data, and generates a sound data set whose elements are each sound data contained in the first subset and the second subset and the text data contained in the first subset.

3. The generation apparatus according to claim 1, wherein the control unit, in the sound data set generation process, obtains or selects the first subset and the second subset from the main set by sampling based on the type tag, and generates a sound data set whose elements are each sound data contained in the first subset and the second subset and the text data contained in the first subset.

4. A control step in which a computer performs a sound data set generation process to generate a sound data set, which is a set containing first sound data and second sound data, based on a large set which is

Images