Information processing device, information processing method, and information processing program

Abstract

An information processing device (100) has a first extraction unit (124) and a calculation unit (125). The first extraction unit (124) extracts, from a section of audio data in which a target speaker is speaking, a statistic of time-series feature amounts. The calculation unit (125) calculates, on the basis of the statistic extracted by the first extraction unit (124), a weight that emphasizes, in the time-series feature amounts, a dimension indicating a feature of the target speaker.

Description

Information Processing Device, Information Processing Method, and Information Processing Program

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program.

[0002] Spoken dialogue understanding is a technology for recognizing and analyzing conversational speech, and plays an important role in communication scenarios involving multiple people. For example, spoken dialogue understanding is expected to be applied to accurate recording of the content of meetings and business negotiations with multiple participants, support for consensus building, and improvement of discussion quality.

[0003] Conventionally, a model has been disclosed that uses a neural network which receives multi-speech speech and the utterance segments of a target speaker to be extracted (hereinafter referred to as the target speaker) and speakers not to be extracted (hereinafter referred to as non-target speakers) as inputs, to extract the feature quantity of a specific speaker from the multi-speaker speech (for example, see Non-Patent Document 1).

[0004] S. Horiguchi, T. Moriya, A. Ando, T. Ashihara, H. Sato, N. Tawara, M. Delcroix, "Guided Speaker Embedding," in Proc. ICASSP 2025.

[0005] However, in the conventional technology, there are cases where the speaker feature quantity of the target speaker cannot be extracted appropriately. For example, in the conventional technology, depending on the ratio of the lengths of the utterance segments of the target speaker and the non-target speakers, weighting may be performed such that the information of the target speaker that should originally be emphasized is suppressed, or the information of the target speaker that should originally be suppressed is emphasized, making it impossible to appropriately extract the speaker feature quantity of the target speaker.

[0006] In order to solve the above-described problems and achieve the objective, an information processing device is characterized by comprising: a first extraction unit that extracts a statistic of feature quantities for each time series from a segment in which a target speaker in speech data is speaking; and a calculation unit that calculates, based on the statistic extracted by the first extraction unit, a weight for emphasizing the dimension indicating the feature of the target speaker with respect to the feature quantities for each time series.

[0007] According to one aspect of the present invention, the speaker features of a target speaker can be appropriately extracted.

[0008] Figure 1 is a diagram showing an example of an information processing system according to Embodiment 1. Figure 2 is a diagram illustrating an overview of speaker dialization processing based on two-stage processing related to Embodiment 1. Figure 3 is a diagram illustrating an overview of the second-stage processing related to Embodiment 1. Figure 4 is a diagram illustrating the problems of the invention related to Reference 1, which is related to Embodiment 1. Figure 5 is a diagram illustrating the problems of the invention related to Reference 2, which is related to Embodiment 1. Figure 6 is a diagram illustrating an overview of the invention related to Non-Patent Document 1. Figure 7 is a diagram illustrating the processing within the encoder of the invention related to Non-Patent Document 1. Figure 8 is a diagram showing an example of the configuration of an information processing device according to Embodiment 1. Figure 9 is a diagram illustrating an example of audio data stored in the information processing device according to Embodiment 1. Figure 10 is a diagram illustrating an example of the processing of the first extraction unit and calculation unit according to Embodiment 1. Figure 11 is a diagram showing an example of the processing of the display unit of the information processing device according to Embodiment 1. Figure 12 is a diagram showing the overall processing flow of the information processing device according to Embodiment 1. Figure 13 is a flowchart illustrating an example of the learning process of the information processing device according to Embodiment 1. Figure 14 is a flowchart illustrating an example of the inference process of the information processing device according to Embodiment 1. Figure 15 is a diagram showing an example of the configuration of an information processing device according to Embodiment 2. Figure 16 is a diagram illustrating an example of the processing of the first extraction unit, the second extraction unit, and the calculation unit according to Embodiment 2. Figure 17 is a diagram illustrating an example of the hardware configuration.

[0009] Embodiments of the information processing apparatus, information processing method, and information processing program according to the present application will be described in detail below with reference to the drawings. However, these embodiments do not limit the information processing apparatus, information processing method, and information processing program according to the present application.

[0010] <Embodiment 1> First, the information processing device 100 according to Embodiment 1 will be described.

[0011] <Overall Configuration> First, the overall configuration of the information processing system 10 according to this embodiment will be described. Figure 1 is a diagram showing an example of the information processing system according to Embodiment 1. As shown in Figure 1, the information processing system 10 has an information processing device 100 and an audio data recording terminal 200 that are connected in a communicative manner to send and receive information. Note that the system configuration shown in Figure 1 is just an example, and the connection status and number of each device are not particularly limited.

[0012] The information processing device 100 is an information processing device that extracts speaker characteristics of the target speaker from the acquired audio data, and is implemented by a computer or the like. For example, the information processing device 100 acquires audio data processed in segment units transmitted from the audio data recording terminal 200, and extracts speaker characteristics of the target speaker within each segment.

[0013] The voice data recording terminal 200 is an information processing terminal that records voice data to be processed by the information processing device 100, and can be implemented using a computer, smartphone, voice recorder, etc. For example, the voice data recording terminal 200 divides long voice data with multiple speakers, such as from a meeting, lecture, or business negotiation, into short segments of a few seconds to tens of seconds, and determines the utterance section for each speaker for each divided segment. The voice data recording terminal 200 then transmits the voice data divided into segments and the information on the utterance sections for each speaker to the information processing device 100.

[0014] Here, the series of processes described above, from dividing the audio data into segments to determining the speech intervals for each speaker, are not limited to processes performed by the audio data recording terminal 200. For example, an external device other than the audio data recording terminal 200 may perform the series of processes described above, or the information processing device 100 may perform the series of processes described above. In this case, the audio data recording terminal 200 may, for example, perform the process of transmitting long audio data with multiple speakers, such as recorded meetings, lectures, or business negotiations, to an external device.

[0015] <Challenges of the Conventional Technology> Next, we will describe the conventional technology related to the method for extracting speaker features from speech data associated with this embodiment. As mentioned above, speech dialogue understanding is a technology for recognizing and analyzing conversational speech, and plays an important role in communication settings involving multiple people. For example, speech dialogue understanding is expected to be applied to accurately recording the contents of meetings and business negotiations involving multiple participants, supporting consensus building, and improving the quality of discussions.

[0016] Here, in order to realize speech dialogue understanding, it is necessary not only to transcribe spoken audio using speech recognition, but also to determine who spoke each utterance. For example, a simple implementation would be to first find the utterance segments for each speaker from the audio (hereinafter also referred to as speaker diarization), and then perform speech recognition on each utterance segment.

[0017] One proposed method for speaker diarization is an end-to-end speaker diarization technique that constructs a neural network that independently infers whether each speaker is speaking at each time step from an input sequence of acoustic features. However, this method has several drawbacks: it is difficult to construct a neural network-based model with no limit on the number of speakers; long audio recordings with many speakers used to train such models have a small amount of data and are difficult to annotate; and sequence models commonly used in audio processing, such as Transformers and Conformers, require computational and memory consumption proportional to the square of the sequence length, making it difficult to process long audio recordings at once. For these reasons, it is difficult to handle long audio recordings lasting several hours or to perform inference with no upper limit on the number of speakers.

[0018] To solve these problems, when handling long audio files, inference based on two-stage processing is sometimes performed. Figure 2 is a diagram illustrating the outline of speaker diarization processing based on two-stage processing related to Embodiment 1. In Figure 2, examples of each process are shown for the first stage, which performs "1. Processing on a segment-by-segment basis" for long input audio files, and the second stage, which performs "2. Determination of speaker identity across segments".

[0019] As shown in Figure 2, in the first stage of processing, the audio is divided into short segments of a few seconds to several tens of seconds, and speaker dialization is performed on each segment using the aforementioned model. Then, in the second stage, speaker features are extracted for each speaker speaking within each segment, and the results for each segment obtained by clustering the speaker features extracted from all segments are integrated to obtain the final inference result for the long audio.

[0020] Speaker features refer to vectors extracted from speech that represent the speaker's identity, and based on their similarity, it is possible to determine whether utterances were made by the same speaker.

[0021] Here, the second stage of processing shown in Figure 2 will be explained in two further stages. Figure 3 is a diagram illustrating the overview of the second stage of processing related to Embodiment 1. In Figure 3, examples of each stage of processing are shown for the audio data processed in segments: the first stage, which involves "2-1. Extracting speaker features for each speaker within each segment," and the second stage, which involves "2-2. Clustering speaker features to determine whether the speakers are the same."

[0022] As shown in Figure 3, in the first stage of processing, speaker features are extracted for each speaker speaking within each segment of the audio data that has been processed segment by segment, and multiple speaker features are obtained. Then, in the second stage, the results for each segment obtained by clustering based on the similarity of the speaker features extracted from all segments are integrated to obtain the final inference result of speaker identity for the long audio. In other words, speaker identity across segments is determined by determining that utterances corresponding to speaker features determined to belong to the same cluster are by the same speaker, and utterances corresponding to speaker features determined to belong to a different cluster are by different speakers. Note that in the processing shown in Figure 3, the number of clusters is determined dynamically, so there is no upper limit to the number of speakers that can be inferred.

[0023] In the process shown in Figure 3, as a method for extracting speaker features based on multi-speaker audio and each speaker's utterance section, Reference 1 discloses a method for extracting speaker features using the sections in which each speaker is speaking alone. Figure 4 is a diagram illustrating the problems of the invention according to Reference 1, which relates to Embodiment 1. In the invention shown in Figure 4, the sections in which each speaker is speaking alone are cut, and speaker features are extracted using the cut sections, thereby preventing the inclusion of information from speakers who are not the target of extraction.

[0024] (Reference 1): H. Bredin, “pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe,” in Proc. Interspeech, 2023.

[0025] However, regarding the invention shown in Figure 4, while it is generally better to use the longest possible length of speech for extracting speaker features, there is a problem in that the single-utterance segments are not always sufficiently long, and in some cases, there may be no segments in which a speaker is uttering alone.

[0026] Furthermore, Reference 2 below discloses a method for extracting speaker features using audio from which sound source separation has been performed. Figure 5 is a diagram illustrating the problems of the invention related to Reference 2, which is associated with Embodiment 1. In the invention shown in Figure 5, the speech data of each speaker is separated from a mixed sound uttered by multiple speakers by sound source separation. As a result, the invention in Figure 5 suppresses the utterance of the interfering speaker.

[0027] (Reference 2): N. Tawara, A. Ando, ​​S. Horiguchi, M. Delcroix, “Multi-channel Speaker Counting for EEND-VC-based Speaker Diarization on Multi-domain Conversation,” in Proc. ICASSP 2025.

[0028] However, the invention shown in Figure 5 has several problems, including the fact that the system becomes complex in order to perform sound source separation, that even if sound source separation is performed, speaker characteristics may be impaired due to distortion caused by the separation, and that the separation results do not necessarily correspond one-to-one with the speaker dialization results obtained for each segment.

[0029] In this context, Non-Patent Document 1, mentioned above, discloses a model for extracting features of a specific speaker from multi-speaker audio using a neural network that takes multi-speaker audio and the speech segments of the target speaker and non-target speaker as input. Figure 6 is a diagram illustrating the outline of the invention related to Non-Patent Document 1.

[0030] Figure 6 shows an overview of the overall processing of the invention described in Non-Patent Document 1. As shown in Figure 6, the invention described in Non-Patent Document 1 distinguishes information between the speech segments of the target speaker and the speech segments of non-target speakers from the information of the input speech and the speech segments corresponding to the input speech, and extracts speaker features of the target speaker.

[0031] Here, we will explain the processing within the encoder related to the invention of this embodiment 1, with respect to the invention described in Non-Patent Document 1. Figure 7 is a diagram illustrating the processing within the encoder of the invention described in Non-Patent Document 1.

[0032] As shown in Figure 7, in the encoder of the speaker feature extractor that takes one speaker's voice as input according to Non-Patent Document 1, weights are assigned to each dimension of the time series features based on statistics extracted from the entire sequence. As statistics, for example, the following number (2), which is a vector obtained by averaging the time series features shown in the following number (1) in the time direction, is used as a time series embedding.

[0033]

[0034]

[0035] Then, within the encoder, a multilayer neural network f is used to calculate the following number (3), which is a D-dimensional weight vector, from the calculated statistics. This number is then used in place of the original time-series features, multiplied by the following number (4), to obtain a sequence E', thereby weighting the important dimensions of D.

[0036]

[0037]

[0038] Note that the symbol shown between "w" and "e in Equation (4) １ " represents the Hadamard product. In addition, the important dimensions herein refer to dimensions useful for emphasizing speaker information contained in input speech, wherein based on statistics calculated from the entire sequence, the weights of important dimensions are increased (close to 1), and the weights of unimportant dimensions are decreased (close to 0).

[0039] Here, the "important dimensions" in the speaker feature extractor for multi-speaker speech have two perspectives. One is a dimension important for emphasizing the features of the target speaker, and the other is a dimension important for suppressing the features of non-target speakers. However, it is difficult to determine from simple statistics such as an average from which perspective each dimension is important.

[0040] For example, for speech in which only the target speaker utters, the importance calculated from the statistics is considered to almost represent what should be emphasized. On the other hand, for speech in which a non-target speaker utters for a longer time than the target speaker, the non-target speaker becomes dominant in the statistics, and the importance calculated therefrom is considered to almost represent what should be suppressed. However, an erroneous determination of emphasis / suppression may suppress the features of the target speaker that should be emphasized, or emphasize the features of the non-target speaker that should be suppressed, thus adversely affecting the output speaker feature quantity.

[0041] That is, the invention according to Non-Patent Document 1 has a problem that, depending on the ratio of the lengths of the target speaker section and the non-target speaker section, there is a risk of suppressing the information of the target speaker that should be emphasized or emphasizing the information of the target speaker that should be suppressed, and thus the speaker feature quantity of the target speaker may not be able to be appropriately extracted in some cases.

[0042] <Outline of Processing of Information Processing Apparatus according to Embodiment 1> In view of the aforementioned problems, the information processing apparatus 100 according to the first embodiment extracts the statistic of the feature amount for each time series from the section in which the target speaker in the voice data is speaking, and based on the extracted statistic, calculates a weight for emphasizing the dimension indicating the feature of the target speaker with respect to the feature amount for each time series.

[0043] For example, regarding the processing in the encoder shown in FIG. 6, the information processing apparatus 100 uses a binary variable indicating whether or not the target speaker in the voice data is speaking, to extract the statistic only from the section where the target speaker is speaking. Then, the information processing apparatus 100 calculates a weight for emphasizing the dimension indicating the feature of the target speaker from the extracted statistic using a multilayer neural network.

[0044] As a result, the information processing apparatus 100 makes all the calculated weights based on the viewpoint of dimensions that should be emphasized, and can appropriately extract the speaker feature amount of the target speaker regardless of the ratio of the speech sections of the target speaker / non-target speaker in the input.

[0045] <Configuration of Information Processing Apparatus according to Embodiment 1> Next, the configuration of the information processing apparatus 100 shown in FIG. 1 will be described with reference to FIG. 8. FIG. 8 is a diagram illustrating a configuration example of the information processing apparatus according to the first embodiment. As shown in FIG. 8, the information processing apparatus 100 includes, for example, a communication unit 110, a control unit 120, and a storage unit 140.

[0046] The communication unit 110 is implemented by, for example, a NIC (Network Interface Card) or the like. For example, the communication unit 110 is connected to the voice data recording terminal 200, and transmits and receives data such as voice data and information related to the speech section of the speaker.

[0047] The storage unit 140 is implemented by, for example, a storage device such as a RAM (Random Access Memory) or a hard disk. The storage unit 140 stores data and programs required for various types of processing by the control unit 120. For example, the storage unit 140 includes a speaker feature amount extraction model 141 and a voice data storage unit 142.

[0048] The speaker feature extraction model 141 stores data related to the parameters of the machine learning model used in the speaker feature extraction process described later. For example, the speaker feature extraction model 141 stores data for functions used by each processing unit of the control unit 120 when executing processing, such as weights calculated by the calculation unit 125 described later.

[0049] Here, the speaker feature extraction model 141 is a machine learning model that sequentially executes the processing steps shown in Figure 6 above. It takes speech data containing utterances from multiple speakers, time information of the speaker's utterance interval, and identification information of the target speaker as input, and outputs speaker features of the target speaker.

[0050] The voice data storage unit 142 stores, for example, information about the voice data to be processed transmitted from the voice data recording terminal 200 described above. Figure 9 is a diagram illustrating an example of voice data stored in the information processing device according to Embodiment 1. As shown in Figure 9, the voice data storage unit 142 stores, for example, "data number," "voice data," "segment data," and "utterance section data," each associated with the data to be processed.

[0051] The "Data Number" stores the identification number assigned to each audio data to be processed. The "Audio Data" stores the waveform data of the audio to be processed. The "Segment Data" stores data in which the audio data has been divided into segments. The "Speech Interval Data" stores data indicating the speaker's speech interval in each segment of data.

[0052] Returning to the explanation of Figure 8, the control unit 120 is implemented by a CPU (Central Processing Unit) or MPU (Micro Processing Unit), etc., which executes various programs stored in the device's internal memory using RAM as the working area. Alternatively, the control unit 120 can be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). As shown in Figure 8, the control unit 120 includes, for example, a mixed sound generation unit 121, a speech data acquisition unit 122, a speaker information acquisition unit 123, a first extraction unit 124, a calculation unit 125, a learning unit 126, a feature estimation unit 127, a speaker determination unit 128, an assignment unit 129, and a display unit 130.

[0053] The mixed sound generation unit 121 generates a mixed sound by overlapping parts of the audio data using multiple audio data, each containing recordings of the utterances of multiple speakers. For example, when training the speaker feature extraction model 141, the mixed sound generation unit 121 generates a mixed sound in which the utterances of each speaker partially overlap by adding together the utterance data of multiple speakers selected from the sample data used for training, with the utterance start times shifted by an arbitrary signal ratio.

[0054] Here, the sample data used for processing by the mixed sound generation unit 121 is, for example, waveform data of a recording of a speaker's utterance, to which speaker identification information (hereinafter also referred to as speaker ID) is attached. This allows the mixed sound generation unit 121 to associate the generated mixed sound with the utterance intervals of each speaker and the speaker ID. The mixed sound generation unit 121 can also perform data augmentation on the generated mixed sound, for example, by superimposing noise or convolving an impulse response to add reverberation.

[0055] The voice data acquisition unit 122 acquires voice data containing the voices of multiple speakers. For example, the voice data acquisition unit 122 acquires voice data in segment units, which are obtained by dividing voice data containing multiple speakers transmitted from the voice data recording terminal 200. In addition to acquiring segment units of voice data, the voice data acquisition unit 122 can also acquire long-form voice data before it is divided into segments, and the acquired voice data is stored in the voice data storage unit 142.

[0056] Furthermore, the audio data acquisition unit 122 acquires the mixed sound generated by the mixed sound generation unit 121. For example, when training the speaker feature extraction model 141, the audio data acquisition unit 122 acquires audio data of a mixed sound in which the utterances of each speaker generated by the mixed sound generation unit 121 partially overlap.

[0057] Furthermore, even when training the speaker feature extraction model 141, if a mixed sound containing the utterances of multiple speakers, such as in an actual meeting or lecture, is obtained along with the labels of the speaker's utterance intervals, the audio data acquisition unit 122 can acquire the mixed sound containing the utterances of multiple speakers, such as in an actual meeting or lecture, instead of the mixed sound generated by the mixed sound generation unit 121.

[0058] The speaker information acquisition unit 123 acquires information indicating the sections in which each of the multiple speakers in the audio data is speaking, and information identifying the target speaker. For example, the speaker information acquisition unit 123 acquires the time information of the speaking sections of each speaker included in the audio data, which is linked to the audio data acquired by the audio data acquisition unit 122, and the speaker ID of the target speaker to be processed by the speaker feature extraction model 141, and stores them in the audio data storage unit 142.

[0059] Here, the speaker information acquisition unit 123, for example, associates the time information of each speaker's utterance interval included in the audio data with the audio data through speaker dialization by an external device such as the audio data recording terminal 200, or through processing by the mixed sound generation unit 121.

[0060] Furthermore, the speaker information acquisition unit 123 acquires, for example, one speaker ID selected from among the speaker IDs of each speaker associated with the audio data as the speaker ID of the target speaker. Note that, if the speaker features of the target speaker have been extracted, the speaker information acquisition unit 123 automatically selects one speaker ID from among the remaining speaker IDs from which speaker features have not yet been extracted as the speaker ID of the target speaker. This allows the information processing device 100 to sequentially extract speaker features for all speakers included in the audio data.

[0061] The first extraction unit 124 extracts statistical values ​​of time-series features from the sections in the audio data in which the target speaker is speaking. For example, the first extraction unit 124 uses only the speech sections of each speaker that correspond to the speaker ID of the target speaker, from the speech sections of each speaker acquired by the speaker information acquisition unit 123, for the audio data acquired by the audio data acquisition unit 122, and extracts statistical values ​​from the speech sections of the target speaker.

[0062] The calculation unit 125 calculates weights that emphasize the dimensions representing the characteristics of the target speaker for each time series feature based on the statistics extracted by the first extraction unit 124. For example, the calculation unit 125 uses a multilayer neural network f to calculate a weight vector that emphasizes the dimensions representing the characteristics of the target speaker from the statistics extracted from the speech intervals of the target speaker by the first extraction unit 124.

[0063] Here, the processing of the first extraction unit 124 and the calculation unit 125 will be explained using Figure 10. Figure 10 is a diagram illustrating an example of the processing of the first extraction unit and the calculation unit according to Embodiment 1. Figure 10 shows an example of the processing related to the encoder portion of the speaker feature extraction model 141, and shows the processing of calculating weighted time-series features by adding weights that emphasize the dimension representing the characteristics of the target speaker to the time-series features of the speech data.

[0064] For example, the first extraction unit 124 extracts the time-series features shown in number (1) above from the audio data acquired by the audio data acquisition unit 122. The first extraction unit 124 also uses the time information of each speaker's utterance interval acquired by the speaker information acquisition unit 123 and the speaker ID of the target speaker to generate a binary variable p, shown in number (5) below, which indicates whether or not the target speaker is speaking. ｔ Calculate.

[0065]

[0066] Then, the first extraction unit 124 extracts the above-mentioned binary variable p ｔ Using this method, we extract the statistical quantity z from the speech intervals of the target speaker shown in the following number (6).

[0067]

[0068] Then, the calculation unit 125 calculates a weight vector w that emphasizes the dimension representing the characteristics of the target speaker using the formula shown in number (3) above. Then, the calculation unit 125 calculates the weighted time series feature quantity E' using the calculated weight w and the formula shown in number (4) above.

[0069] Furthermore, the first extraction unit 124 can also extract the statistic z only from the target speaker's solitary utterances if, for example, it is guaranteed that there are sections in the audio data in which the target speaker speaks alone.

[0070] Furthermore, the calculation unit 125 can also apply any transformation shown in the following number (7) to the time series features in parallel with the weight calculation, for example, using the method described in Reference 3 below. In this case, the calculation unit 125 calculates the weighted time series features E' using the formula shown in the following number (8).

[0071] (Reference 3): H. Wang, S. Zheng, Y. Chen, L. Cheng, and Q. Chen, “CAM++: A fast and efficient network for speaker verification using context-aware masking,” in Proc. Interspeech 2023.

[0072]

[0073]

[0074] Furthermore, the calculation unit 125 can perform the same processing even when the time series features are given by a second-order tensor as shown in the number (9) below, or as shown in the number (10) below, by, for example, the method described in Reference 4 below.

[0075] (Reference 4): J. Thienpondt, B. Desplanques, and K. Demuynck, “Integrating frequency translational invariance in TDNNs and frequency positional information in 2D ResNets to enhance speaker verification,” in Proc. Interspeech 2021.

[0076]

[0077]

[0078] Furthermore, the series of processes from the first extraction unit 124 to the calculation unit 125 described above can utilize statistics calculated from the input only during the training of the speaker feature extraction model 141, for example, as described in Reference 5 below. Batch normalization normalizes values ​​using the mean and variance within a batch during the training of a neural network. In the series of processes described above, by calculating these statistics using only the utterance intervals of the target speaker, the features derived from the target speaker that are to be extracted are always normalized to be distributed around the origin, thereby improving the stability of the learning process. Similar operations can also be applied to other normalization methods such as layer normalization.

[0079] (Reference 5): S. Ioffe, C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proc. ICML, 2015.

[0080] The learning unit 126 receives audio data acquired by the audio data acquisition unit 122, information indicating the sections in which each of the multiple speakers is speaking, acquired by the speaker information acquisition unit 123, and information identifying the target speaker, and then trains a machine learning model to output the feature quantities of the target speaker.

[0081] For example, the learning unit 126 updates the parameters used by the calculation unit 125 when calculating weights in order to accurately extract speaker features of the target speaker, based on input information such as the audio data to be processed, the time information of the utterance intervals of each speaker in the audio data, and the speaker ID of the target speaker.

[0082] For example, the learning unit 126 takes the audio data of the mixed speech with K speakers generated by the mixed speech generation unit 121 and outputs speaker features for each of the K speakers sequentially as the target speaker to the speaker feature extraction model 141. Then, the learning unit 126 optimizes the extracted speaker features for the K speakers using, for example, a speaker identification loss based on cross-entropy.

[0083] Here, the segment-based speech data used to train the speaker feature extraction model 141 has a speaker ID associated with each speaker's utterance interval, but the same speaker ID is used for each segment for the same speaker. In other words, the learning unit 126 can classify the speaker features extracted in each segment according to the speaker ID. As a result, the learning unit 126 updates the parameters of the speaker feature extraction model 141 so that the differences in the extracted speaker features for the same speaker ID are reduced.

[0084] For example, the learning unit 126 updates the parameters of the multilayer neural network f used in the processing of the calculation unit 125 so that it is optimized.

[0085] The feature estimation unit 127 extracts feature statistics for each time series from the intervals in the audio data where the target speaker is speaking. Based on the extracted statistics, it inputs the audio data, information indicating the intervals in which each of the multiple speakers is speaking, and information identifying the target speaker into a pre-trained machine learning model that has learned parameters used to calculate weights that emphasize the dimensions representing the characteristics of the target speaker for each time series feature, and obtains the feature quantities of the target speaker output by the machine learning model.

[0086] For example, the feature estimation unit 127 inputs the audio data acquired by the audio data acquisition unit 122, the time information of each speaker's utterance interval acquired by the speaker information acquisition unit 123, and the identification information of the target speaker to a trained speaker feature extraction model 141 that performs the processing of the first extraction unit 124 and the calculation unit 125 within the encoder, and obtains the speaker features of the output target speaker.

[0087] Here, the feature estimation unit 127 can appropriately estimate the speaker features of the target speaker by using the values ​​of the multilayer neural network f used when the calculation unit 125, which has been optimized by the learning unit 126 described above, calculates the weights.

[0088] Furthermore, when estimating speaker features of a target speaker in segment-level speech data using the trained speaker feature extraction model 141, information indicating each speaker's utterance interval is not shared across segments. In other words, it is shown that different speakers exist in each segment, such as the first speaker and the second speaker, so the first speaker in one segment often represents a different speaker than the first speaker in other segments.

[0089] The speaker determination unit 128 classifies the feature quantities of the target speaker estimated by the feature quantity estimation unit 127 for each of the multiple speakers included in the audio data, and determines the identity of the speaker in the multiple audio data. For example, the speaker determination unit 128 classifies the speaker feature quantities of the target speaker extracted for each speaker in the segment-based audio data by the feature quantity estimation unit 127, and determines the identity of the speaker included in each of the multiple segment-based audio data (see 2-2 in Figure 3).

[0090] The assignment unit 129 assigns each of the multiple speakers determined by the speaker determination unit 128 to the corresponding section of the audio data. For example, the assignment unit 129 associates each speaker whose identity has been determined by the speaker determination unit 128 with the information indicating the utterance section acquired by the speaker information acquisition unit 123, and assigns the speaker corresponding to each utterance section of the audio data (see Figure 3). The audio data to which each speaker is assigned may be audio data divided into segments, or it may be long audio data before being divided into segments.

[0091] When playing audio data, the display unit 130 displays speaker information assigned by the assignment unit 129 according to the playback time of the audio data. For example, the display unit 130 uses speaker information assigned to the long audio data before it is divided into segments by the assignment unit 129 to display on the screen the speaker information of the section that is speaking in the audio data playback time.

[0092] The location where the display unit 130 displays speaker information is not particularly limited; for example, it may be displayed on the screen of the information processing device 100, or on the screen of the user's smartphone or other device that plays video data related to the audio data.

[0093] Here, we will describe an example of a screen displayed by the display unit 130. Figure 11 is a diagram showing an example of the processing of the display unit of the information processing apparatus according to Embodiment 1. Figure 11 shows two specific examples, (1) and (2), of the process by which the display unit 130 displays speaker information on the audio data playback screen.

[0094] As shown in Figure 11(1), the display unit 130 displays objects distinguishing by color or pattern for each speaker's utterance section from speaker A to speaker F on the screen where the audio data is played back, thereby indicating which speaker is speaking in the section corresponding to the playback time.

[0095] As shown in Figure 11(2), the display unit 130 displays, on the screen playing the audio data, information about the speaker and the content of their speech for the section corresponding to the playback time, by displaying the objects shown in Figure 11(1) as well as text data that is a transcription of the content of each speaker's speech.

[0096] <Information Processing Flow> Next, we will explain the flow of a series of processes performed by each functional unit of the information processing device 100. Figure 12 is a diagram showing the overall processing flow of the information processing device according to Embodiment 1. In Figure 12, the processing of the information processing device 100 is shown in two distinct stages: learning and inference.

[0097] First, let's explain the series of processes that occur during learning in the information processing device 100. First, the mixed sound generation unit 121 generates a mixed sound in which the utterances of each speaker partially overlap, using speech data associated with speaker IDs. Then, the speech data acquisition unit 122 acquires the generated mixed sound. Finally, the speaker information acquisition unit 123 acquires the time information of the utterance intervals of each speaker associated with the mixed sound.

[0098] The learning unit 126 then updates the parameters of the speaker feature extraction model 141 before training so that the speaker features of each speaker included in the mixed sound are optimized, thereby generating a trained speaker feature extraction model 141. During the encoding process of the speaker feature extraction model 141 during training, the processing of the first extraction unit 124 and the calculation unit 125 is executed.

[0099] Next, a series of processes during inference by the information processing device 100 will be described. First, the voice data acquisition unit 122 acquires voice data by dividing the long voice data to be processed into segments. Then, the speaker information acquisition unit 123 acquires the time information of the utterance interval of each speaker associated with each voice data segment, and the identification information of the target speaker.

[0100] The feature estimation unit 127 then extracts speaker features of the target speaker using the trained speaker feature extraction model 141 generated during the training process. During the inference process, the first extraction unit 124 and the calculation unit 125 are executed in the encoder processing of the trained speaker feature extraction model 141.

[0101] <Flowchart> Next, the processing flow of the information processing device 100 will be described with reference to Figures 13 and 14. Figure 13 is a flowchart showing an example of the processing during learning of the information processing device according to Embodiment 1. Figure 14 is a flowchart showing an example of the processing during inference of the information processing device according to Embodiment 1. Note that each step in the flowcharts shown in Figures 13 and 14 can be executed in a different order, and some processes may be omitted.

[0102] First, the processing during learning will be explained with reference to Figure 13. When the information processing device 100 first acquires audio data with a speaker ID (S101; Yes), it generates a mixed sound by overlapping a part of the audio data (S102). On the other hand, if audio data with a speaker ID has not been acquired (S101; No), the information processing device 100 waits until audio data with a speaker ID is acquired.

[0103] After the processing in S102, the information processing device 100 acquires information on the speech segments of each speaker for the mixed sound (S103). Then, the information processing device 100 learns a speaker feature extraction model so that the speaker features extracted for each speaker are optimized (S104), and the process ends.

[0104] Next, the processing details during inference will be explained with reference to Figure 14. First, if the information processing device 100 acquires audio data (S201; Yes), it acquires information on the utterance segments of each speaker in relation to the audio data, and information to identify the target speaker (S202). On the other hand, if audio data has not been acquired (S201; No), the information processing device 100 waits until audio data is acquired.

[0105] After processing in S202, the information processing device 100 estimates the speaker features of the target speaker using the trained speaker feature extraction model (S203), and then terminates the process.

[0106] <Effects of Embodiment 1> As described above, the information processing device 100 according to this embodiment has a first extraction unit 124 and a calculation unit 125. The first extraction unit 124 extracts statistical values ​​of feature quantities for each time series from the section in the speech data in which the target speaker is speaking. Based on the statistical values ​​extracted by the first extraction unit 124, the calculation unit 125 calculates weights that emphasize the dimensions that represent the characteristics of the target speaker for each time series of feature quantities.

[0107] As a result, the information processing device 100 can appropriately extract speaker features of the target speaker regardless of the ratio of target speaker / non-target speaker utterances in the input, as all calculated weights are based on the dimension that should be emphasized.

[0108] Furthermore, the information processing device 100 includes a voice data acquisition unit 122, a speaker information acquisition unit 123, and a learning unit 126. The voice data acquisition unit 122 acquires voice data containing the voices of multiple speakers. The speaker information acquisition unit 123 acquires information indicating the section in the voice data in which each of the multiple speakers is speaking, and information identifying the target speaker.

[0109] The learning unit 126 receives audio data acquired by the audio data acquisition unit 122, information indicating the speaking intervals of multiple speakers acquired by the speaker information acquisition unit 123, and information identifying the target speaker, and trains a machine learning model to output the feature quantities of the target speaker. In particular, the learning unit 126 updates the parameters used by the calculation unit 125 when calculating weights.

[0110] As a result, the information processing device 100 can update the parameters for weight calculation in the encoder part of the speaker feature extraction model, which outputs speaker features of a target speaker based on the input of speech data from multiple speakers and information on each speaker's utterance interval, to the optimal values, thereby generating a speaker feature extraction model that extracts speaker features of a target speaker with high accuracy.

[0111] Furthermore, the information processing device 100 further includes a mixed sound generation unit 121. The mixed sound generation unit 121 uses multiple audio data, each recording of the utterances of multiple speakers, to generate a mixed sound by overlapping parts of the audio data. In this case, the audio data acquisition unit 122 acquires the mixed sound generated by the mixed sound generation unit 121.

[0112] As a result, when the information processing device 100 is training a speaker feature extraction model, it can easily generate a mixed sound containing the voices of multiple speakers from sample data associated with speaker IDs and use it as training data.

[0113] Furthermore, the information processing device 100 includes a voice data acquisition unit 122, a speaker information acquisition unit 123, and a feature quantity estimation unit 127. The voice data acquisition unit 122 acquires voice data containing the voices of multiple speakers. The speaker information acquisition unit 123 acquires information in the voice data indicating the section in which each of the multiple speakers is speaking, and information identifying the target speaker.

[0114] The feature estimation unit 127 extracts feature statistics for each time series from the intervals in the audio data where the target speaker is speaking, and based on the extracted statistics, it inputs the audio data, information indicating the intervals in which each of the multiple speakers is speaking, and information identifying the target speaker into a machine learning model that has already been trained with parameters used to calculate weights that emphasize the dimensions representing the characteristics of the target speaker for each time series feature, and obtains the feature quantities of the target speaker output by the machine learning model.

[0115] As a result, the information processing device 100 can accurately extract speaker features of the target speaker from the audio data using a pre-trained speaker feature extraction model that has been trained to optimize the parameters used in calculating weights based on statistics extracted only from the speech intervals of the target speaker in the encoder processing.

[0116] The information processing device 100 further includes a speaker determination unit 128, an assignment unit 129, and a display unit 130. The speaker determination unit 128 classifies the feature quantities of the target speaker estimated by the feature quantity estimation unit 127 for each of the multiple speakers included in the audio data, and determines the identity of the speakers in the multiple audio data. The assignment unit 129 assigns each of the multiple speakers determined by the speaker determination unit 128 to the corresponding section of the audio data. When the audio data is played back, the display unit 130 displays the speaker information assigned by the assignment unit 129 according to the playback time of the audio data.

[0117] As a result, the information processing device 100 assigns speaker information to long audio data spanning segments based on the determination result of speaker identity for the audio data in segments, and displays each speaker's utterance section in correspondence with the playback time at the same time as the audio data is played, making it easy to understand at what point in the audio data each speaker is speaking.

[0118] <Embodiment 2> Next, an information processing device 100A according to Embodiment 2 will be described. In Embodiment 1 described above, a process was described in which a weight is calculated to emphasize the dimension that represents the characteristics of the target speaker based on statistical quantities extracted using only the speech interval of the target speaker. However, the processes performed by the information processing device are not limited to this process. For example, the information processing device 100A according to Embodiment 2 uses not only statistical quantities extracted using only the speech interval of the target speaker, but also statistical quantities extracted using only the speech interval of non-target speakers to calculate a weight that emphasizes the dimension that represents the characteristics of the target speaker.

[0119] The following describes the processing content of the information processing device 100A according to Embodiment 2. Note that the information processing device 100A according to Embodiment 2 differs from the information processing device 100 according to Embodiment 1 in that it has a second extraction unit 131, so the processing mainly related to the second extraction unit 131 will be described.

[0120] <Configuration of the Information Processing Device of Embodiment 2> The configuration of the information processing device 100A according to Embodiment 2 will now be described. Figure 15 is a diagram showing an example of the configuration of the information processing device according to Embodiment 2. As shown in Figure 15, the information processing device 100A differs from the information processing device 100 shown in Figure 8 in that it has a second extraction unit 131.

[0121] The second extraction unit 131 extracts statistical features for each time series from the sections of speech in the audio data spoken by speakers who are not the target speakers. For example, the second extraction unit 131 uses only the speech sections of each speaker that correspond to the speaker ID of the non-target speaker, from the speech sections of each speaker acquired by the speaker information acquisition unit 123, for the audio data acquired by the audio data acquisition unit 122, and extracts statistical values ​​from the speech sections of the non-target speakers.

[0122] In this case, the calculation unit 125 calculates weights using a multilayer neural network from the statistics extracted by the first extraction unit 124 and the statistics extracted by the second extraction unit 131. For example, the calculation unit 125 uses a multilayer neural network f to calculate a weight vector that emphasizes the dimension representing the characteristics of the target speaker, from the statistics extracted from the target speaker's utterance section extracted by the first extraction unit 124 and the statistics extracted from the non-target speaker's utterance section extracted by the second extraction unit 131.

[0123] Here, using Figure 16, we will explain the series of processes of the first extraction unit 124, the second extraction unit 131, and the calculation unit 125. Figure 16 is a diagram illustrating an example of the processes of the first extraction unit, the second extraction unit, and the calculation unit according to Embodiment 2. In Figure 16, similar to Figure 10 described above, an example of the process related to the encoder portion of the speaker feature extraction model 141 is shown, and it shows the process of calculating weighted time-series features by adding weights to the time-series features of the speech data to emphasize the dimension that represents the characteristics of the target speaker.

[0124] For example, the first extraction unit 124 performs a series of processes shown in Figure 10 on the audio data acquired by the audio data acquisition unit 122, and extracts a statistical quantity Z from the speech interval of the target speaker shown in the number (6) above.

[0125] In response, the second extraction unit 131 uses, for example, the time information of each speaker's utterance interval obtained by the speaker information acquisition unit 123 and the speaker ID of the non-target speaker to create a binary variable q that indicates whether or not the non-target speaker shown in the following number (11) is speaking. ｔ Calculate.

[0126]

[0127] Then, the second extraction unit 131 extracts the above-mentioned binary variable q ｔ Using this method, we extract the statistical quantity v from the utterance segments of non-directive speakers shown in the following number (12).

[0128]

[0129] Note that ε in the above number (12) is a small constant used to prevent division by zero when audio data is input that is spoken only by the target speaker, as shown in the following number (13).

[0130]

[0131] Then, the calculation unit 125 calculates a weight vector w that emphasizes the dimension representing the characteristics of the target speaker using the formula shown in number (14) below. As shown in number (14) below, in the process according to Embodiment 2, the calculation of the weight vector w is performed using both the statistic z from the first extraction unit 124 and the statistic v from the second extraction unit 131. Then, the calculation unit 125 calculates the weighted time series feature E' using the calculated weight w and the formula shown in number (4) above.

[0132]

[0133] Furthermore, it is preferable that the utterance interval of the non-target speaker used in the processing of the second extraction unit 131 is an interval that does not overlap with the utterance interval of the target speaker. For example, the second extraction unit 131 has a binary variable q defined to be "1" when "at time t, the non-target speaker is speaking and the target speaker is not speaking". ｔ By using this method, the statistic v can be extracted only from intervals in which the non-target speaker is speaking and the target speaker is not speaking. This makes it easier for the second extraction unit 131 to identify information to be suppressed when the target speaker is also speaking during the interval in which the non-target speaker is speaking.

[0134] <Effects of Embodiment 2> As described above, the information processing device 100A according to Embodiment 2 has a first extraction unit 124, a calculation unit 125, and a second extraction unit 131. The first extraction unit 124 extracts statistical amounts of time-series features from the section in the audio data in which the target speaker is speaking. The second extraction unit 131 extracts statistical amounts of time-series features from the section in the audio data in which a speaker other than the target speaker is speaking. The calculation unit 125 calculates weights using a multilayer neural network from the statistical amounts extracted by the first extraction unit 124 and the statistical amounts extracted by the second extraction unit 131.

[0135] As a result, the information processing device 100A can not only assign greater weight to dimensions that should be emphasized based on statistics calculated from the target speaker's utterance interval, but also reduce the weight of dimensions that should be suppressed based on statistics calculated from the non-target speaker's utterance interval, thereby improving the accuracy of speaker feature extraction from the target speaker.

[0136] <System Configuration, etc.> Of the processes described in each of the above embodiments, some of the processes described as being performed automatically can also be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0137] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. Moreover, each processing function performed by each device can be implemented, in whole or in any part, by a CPU and the program that is analyzed and executed by that CPU, or by hardware using wired logic.

[0138] For example, some or all of the storage unit 140 shown in Figures 8 and 15 may be stored in a storage server or the like, rather than being held by the information processing device 100 or the information processing device 100A. In this case, the information processing device 100 or the information processing device 100A acquires various types of information by accessing the storage server.

[0139] <Hardware Configuration> Figure 17 shows an example of a hardware configuration. The information processing device 100 and information processing device 100A according to each embodiment described above are implemented by a computer 1000 having a configuration such as that shown in Figure 17.

[0140] Figure 17 shows an example of the configuration of a computer that implements an information processing device according to each embodiment. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0141] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0142] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define each process of the information processing device 100 or the information processing device 100A are implemented as program modules 1093 in which code executable by the computer 1000 is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, a program module 1093 for performing processes similar to the functional configuration of the information processing device 100 or the information processing device 100A is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0143] Furthermore, the configuration data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes them.

[0144] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN, WAN, etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070.

[0145] The functions of the elements disclosed herein may be implemented using circuits or processing circuitry that include general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), conventional circuits, and / or combinations thereof, which are programmed using one or more programs stored in one or more memories, or otherwise configured to perform the disclosed functions. A processor is considered processing circuitry or circuitry because it includes transistors and other circuits. A processor may be a programmed processor that executes programs stored in memory.

[0146] In this disclosure, a circuit, unit, or means is hardware that performs the enumerated functions, or hardware programmed to perform the enumerated functions. Hardware may be any hardware disclosed herein that is programmed or configured to perform the enumerated functions.

[0147] There is a memory for storing a computer program that includes computer instructions. These computer instructions provide logic and routines that enable hardware (e.g., processing circuitry or circuitry) to perform the methods disclosed herein. This computer program can be implemented in commonly known forms, such as computer-readable storage media, computer program products, memory devices, recording media such as CD-ROMs and DVDs, and / or memory in FPGAs and ASICs.

[0148] Furthermore, the following additional information is disclosed regarding the above embodiments.

[0149] (Note 1) An information processing device comprising: a memory; and at least one processor connected to the memory, wherein the processor extracts statistical quantities of time-series features from the intervals in the speech data where the target speaker is speaking; and calculates weights to emphasize the dimensions representing the characteristics of the target speaker for each time-series feature based on the extracted statistical quantities.

[0150] (Appendix 2) A non-temporary storage medium storing a program executable by a computer to perform information processing, wherein the information processing involves extracting statistical quantities of time-series features from the intervals in speech data spoken by a target speaker, and calculating weights to emphasize the dimensions representing the characteristics of the target speaker for each time-series feature based on the extracted statistical quantities.

[0151] (Appendix 3) An information processing device comprising: memory; and at least one processor connected to the memory, wherein the processor acquires audio data containing the voices of multiple speakers; acquires information in the audio data indicating the interval in which each of the multiple speakers is speaking, and information identifying the target speaker; extracts time-series feature statistics from the interval in the audio data in which the target speaker is speaking, and inputs the audio data, the information indicating the interval in which each of the multiple speakers is speaking, and the information identifying the target speaker to a machine learning model that has been trained with parameters used to calculate weights that emphasize the dimension representing the characteristics of the target speaker for the time-series feature based on the extracted statistics, and acquires the feature quantities of the target speaker output by the machine learning model.

[0152] (Appendix 4) A non-temporary storage medium storing a program executable by a computer to perform information processing, wherein the information processing includes: acquiring audio data containing the voices of multiple speakers; acquiring information in the audio data indicating the intervals in which each of the multiple speakers is speaking, and information identifying the target speaker; extracting time-series feature statistics from the intervals in the audio data in which the target speaker is speaking; and inputting the audio data, the information indicating the intervals in which each of the multiple speakers is speaking, and the information identifying the target speaker into a machine learning model that has been trained with parameters used to calculate weights that emphasize the dimension representing the characteristics of the target speaker for the time-series feature based on the extracted statistics, and obtaining the feature quantities of the target speaker output by the machine learning model.

[0153] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually described as being incorporated by reference.

[0154] 10 Information Processing System 100, 100A Information Processing Device 110 Communication Unit 120 Control Unit 121 Mixed Sound Generation Unit 122 Voice Data Acquisition Unit 123 Speaker Information Acquisition Unit 124 First Extraction Unit 125 Calculation Unit 126 Learning Unit 127 Feature Estimation Unit 128 Speaker Determination Unit 129 Assignment Unit 130 Display Unit 131 Second Extraction Unit 140 Storage Unit 141 Speaker Feature Extraction Model 142 Voice Data Storage Unit 200 Voice Data Recording Terminal

Claims

1. An information processing device comprising: a first extraction unit that extracts statistical quantities of time-series features from the section in audio data in which a target speaker is speaking; and a calculation unit that calculates weights to emphasize the dimensions representing the characteristics of the target speaker for each time-series feature based on the statistical quantities extracted by the first extraction unit.

2. The information processing apparatus according to claim 1, further comprising a second extraction unit that extracts statistical quantities of the time-series features from the section in the audio data in which a speaker other than the intended speaker is speaking, wherein the calculation unit calculates the weights using a multilayer neural network from the statistical quantities extracted by the first extraction unit and the statistical quantities extracted by the second extraction unit.

3. An information processing device according to claim 1, further comprising: an audio data acquisition unit that acquires audio data containing the voices of multiple speakers; a speaker information acquisition unit that acquires information indicating the interval in which each of the multiple speakers is speaking in the audio data and information identifying a target speaker; and a learning unit that, upon receiving the audio data acquired by the audio data acquisition unit, the information indicating the interval in which each of the multiple speakers is speaking acquired by the speaker information acquisition unit, and the information identifying the target speaker, learns a machine learning model to output the feature quantities of the target speaker, wherein the learning unit updates the parameters used when the calculation unit calculates the weights.

4. The information processing apparatus according to claim 3, further comprising a mixed sound generation unit that generates a mixed sound by overlapping a portion of the audio data using multiple audio data obtained by recording the utterances of multiple speakers, wherein the audio data acquisition unit acquires the mixed sound generated by the mixed sound generation unit.

5. An information processing device comprising: an audio data acquisition unit that acquires audio data containing the voices of multiple speakers; a speaker information acquisition unit that acquires information indicating the intervals in which each of the multiple speakers is speaking and information identifying the target speaker in the audio data; and a feature estimation unit that inputs the audio data, the information indicating the intervals in which each of the multiple speakers is speaking, and the information identifying the target speaker into a machine learning model that has been trained with parameters used to extract statistical amounts of time-series features from the intervals in which the target speaker is speaking in the audio data, and to calculate weights that emphasize the dimensions representing the characteristics of the target speaker for each time-series feature based on the extracted statistical amounts, and acquires the features of the target speaker output by the machine learning model.

6. The information processing apparatus according to claim 5, further comprising: a speaker determination unit that classifies the feature quantities of the target speaker estimated by the feature quantity estimation unit for each of the plurality of speakers included in the audio data and determines the identity of the speakers in the plurality of audio data; an assignment unit that assigns each of the plurality of speakers determined by the speaker determination unit to a corresponding section of the audio data; and a display unit that, when playing back the audio data, displays the speaker information assigned by the assignment unit according to the playback time of the audio data.

7. An information processing method in which a computer performs the following steps: extracts statistical values ​​of time-series features from the section in audio data in which a target speaker is speaking; and calculates weights to emphasize the dimensions that represent the characteristics of the target speaker for each time-series feature based on the extracted statistical values.

8. An information processing program that causes a computer to perform the following steps: extract statistical values ​​of time-series features from the sections in audio data in which the target speaker is speaking; and, based on the extracted statistical values, calculate weights to emphasize the dimensions that represent the characteristics of the target speaker for each time-series feature.

Images