Method for estimating geometry-invariant sound source direction on basis of deep neural network, and system and computer program for executing same

Abstract

The present invention provides a method for estimating a geometry-invariant sound source direction on the basis of a deep neural network, and a system and a computer program for executing same. According to an embodiment, the disclosed method for estimating a geometry-invariant sound source direction on the basis of a deep neural network is a method for estimating a geometry-invariant sound source direction on the basis of a deep neural network, which is performed by a system comprising at least one processor, the method comprising: an audio signal acquisition process of acquiring a multi-channel audio signal for one or more speakers from a microphone array including one or more microphones; a frequency transformation process of transforming, for each channel, the acquired multi-channel audio signal into frequency domain information through a short-time Fourier transform (STFT); a feature extraction process of extracting independent features for each channel from the transformed frequency domain information; a microphone position encoding process of transforming the multi-channel audio signal into an embedding vector, and encoding microphone coordinate information of each channel to add a position encoding embedding for the microphone coordinate information to the embedding vector; a spatio-temporal dual-path process of combining the independent features of each channel through channel-wise multi-head self-attention and a frame-wise gated recurrent unit by using the microphone coordinate information as an input value; and a spatial spectrum mapping process of estimating a spatial spectrum for each sound source direction by performing alignment between space and time corresponding to a sound source direction of an audio signal for each channel.

Description

Method for estimating geometrically invariant sound source direction based on deep neural networks, and a system and computer program for executing the same

[0001] The disclosed invention relates to a method for estimating a geometrically invariant sound source direction based on a deep neural network that enables the estimation of the sound source direction regardless of the shape of the microphone array through microphone coordinate encoding, and to a computing device and a computer program for executing the same.

[0002] Speech recognition is a technology that receives human speech through a microphone, converts it into text, or executes corresponding commands.

[0003] Recently, with the advancement of voice recognition technology, various voice recognition services such as Apple's Siri, Google Now, Microsoft's Cortana, and Amazon's Alexa have been launched. To accurately recognize user commands provided by these voice recognition services, it is required to track the location of target sound sources, such as the speaker's voice, in real time, and to separate each sound source from signals received by one or more microphones in noisy environments to identify their locations.

[0004] Meanwhile, as lectures and presentations using equipment such as video conferencing programs and beamforming microphones in indoor spaces have become commonplace, the adoption of directional array microphones that adapt to changes in the speaker's position in voice and speaker recognition is increasing, and machine-human communication through Automatic Speech Recognition (ASR) systems is taking place in various ways.

[0005] ASR systems statistically model input speech signals and utilize this model to estimate and generate text corresponding to the input signals. Typical statistical models involve constructing acoustic modeling (AM) and language modeling (LM) using deep neural networks (DNNs) or recurrent neural networks (RNNs). Furthermore, end-to-end recognition methods have recently been proposed, such as Connectionist Temporal Classification (CTC), which reflects the continuous nature of actual speech rather than using two-stage models.

[0006] Deep learning-based models are actively used in speech recognition systems due to their advantage of being able to model the non-linear and complex relationship between analog speech signals and digital text signals. Speech parameters in speech recognition systems are primarily composed of feature vectors in Mel units that reflect auditory characteristics, such as Mel-filterbank cepstrum coefficients (MFCC) or log melfilterbank coefficients, and phoneme information represented by the corresponding frame is mapped through the distribution of power for each Mel frequency. However, since the articulation pattern changes due to linguistic influences such as surrounding phonemes, syllables, and words even when pronouncing the same phoneme, and the speaker's voice exhibits various physical characteristics, it is known that it is highly effective to comprehensively process various variables such as situation, environment, speaker, and utterance content through a neural network structure.

[0007] The technology for estimating speech direction in speech recognition systems estimates the direction from which speech is coming when speech is present in each frame. Existing deep learning-based models are designed, trained, and used only for specific microphone arrays. The method of estimating speech direction using such deep learning-based models involves the inconvenience of having to retrain the deep learning-based model whenever the shape or number of microphone arrays changes, which incurs unnecessary time and cost and reduces generalization performance.

[0008] To solve these problems, the disclosed embodiment provides a method for estimating geometrically invariant sound source directions based on a deep neural network and a system for executing the same, which can learn by gradually increasing the difficulty of various shapes and numbers of microphone arrays by applying a microphone position encoding process that encodes the spherical coordinates of a microphone using sine and cosine functions, thereby enabling effective direction estimation for various microphone arrays.

[0009] A method according to one disclosed embodiment is a method for estimating a geometrically invariant sound source direction based on a deep neural network, performed by a system comprising at least one processor, comprising: an audio signal acquisition process for acquiring a multichannel audio signal for one or more speakers from a microphone array comprising one or more microphones; a frequency transformation process for converting the acquired multichannel audio signal into frequency domain information by channel through a Short-Time Fourier Transform (STFT); a feature extraction process for extracting independent features by channel from the converted frequency domain information; a microphone position encoding process for converting the multichannel audio signal into an embedding vector, encoding microphone coordinate information of each channel, and adding a position encoding embedding for the microphone coordinate information to the embedding vector; and a spatial-time dual-path process for combining independent features of each channel through a channel-wise multi-head self-attention and a frame-wise gated recurrent unit, using the output value from the feature extraction process and the position encoding embedding as input values. It includes a spatial spectrum mapping process that estimates the spatial spectrum for each sound source direction by performing spatial and temporal alignment corresponding to the sound source direction of the audio signal for each channel.

[0010] Alternatively, the multichannel audio signal comprises a left channel audio signal, a center channel audio signal, and a right channel audio signal, and the microphone array acquires the center channel audio signal.

[0011] Alternatively, the microphone position encoding process uses sine and cosine functions to generate a periodic pattern, thereby causing the position encoding embeddings to have different periods.

[0012] Alternatively, the above-mentioned space-time dual-path process is composed of a block including a frame-wise gated recurrent unit obtained by channel-wise softmax aggregation of the results of channel-wise multi-head self-attention, and is constructed by stacking at least one block up to N (N>0, N is a natural number) blocks sequentially.

[0013] Alternatively, the spatial spectrum estimation process above estimates the spatial spectrum for the direction of each sound source using a deep neural network-based model, and the deep neural network-based model is trained by applying a complexity gradual training (CGT) method that gradually increases the difficulty regarding the number of channels and shape of the microphone array.

[0014] Alternatively, a method for progressively learning the complexity includes a multi-stage geometry learning (MSGL) process that learns by increasing the difficulty regarding the shape of the microphone array or the number of microphones, and a deeply-supervised curriculum learning (DSCL) process that changes the target spatial spectrum according to the depth of the deep neural network-based model and the progress of learning.

[0015] Alternatively, the deep neural network-based model applies a permute-invariant operation to the microphone channel axis to perform operations independent of the number and order of microphones.

[0016] The disclosed system is a system for estimating geometrically invariant sound source directions based on a deep neural network, comprising: a microphone array for acquiring multichannel audio signals for one or more speakers; a frequency converter for converting the acquired multichannel audio signals into frequency domain information through a short-time Fourier transform (STFT) for each channel; a feature extraction unit for extracting independent features for each channel from the converted frequency domain information; a microphone position encoding unit for converting the multichannel audio signals into embedding vectors, encoding microphone coordinate information for each channel, and adding a position encoding embedding for the microphone coordinate information to the embedding vectors; a spatial-time dual-path unit for combining independent features of each channel through a channel-wise multi-head self-attention and a frame-wise gated recurrent unit using the output value from the feature extraction unit and the position encoding embedding as input values; and a spatial spectrum mapping unit for estimating a spatial spectrum for each sound source direction by performing alignment between space and time corresponding to the sound source direction of the audio signal for each channel.

[0017] Alternatively, the feature extraction unit is composed of a Channel Independent Feature Extractor (CIFE).

[0018] Alternatively, the space-time dual path section is composed of blocks including channel-wise multi-head self-attention, channel-wise softmax aggregation, and frame-wise gated recurrent unit, and is constructed by stacking at least one of the blocks sequentially up to N (N>0, N is a natural number).

[0019] A computer program according to a disclosed embodiment is a computer program stored on a computer-readable storage medium, wherein, when executed on one or more processors, the computer program performs operations for estimating a geometrically invariant sound source direction based on a deep neural network, the operations comprising: an audio signal acquisition operation for acquiring a multi-channel audio signal for one or more speakers from a microphone array including one or more microphones; a frequency conversion operation for converting the acquired multi-channel audio signal into frequency domain information through a Short-Time Fourier Transform (STFT) for each channel; a feature extraction operation for extracting independent features for each channel from the converted frequency domain information; and a microphone position encoding operation for converting the multi-channel audio signal into an embedding vector, encoding microphone coordinate information of each channel, and adding a position encoding embedding for the microphone coordinate information to the embedding vector. It includes a spatial-temporal dual-path operation that combines independent features of each channel through a channel-wise multi-head self-attention and a frame-wise gated recurrent unit using the above-mentioned microphone coordinate information as input values; and a spatial spectrum mapping operation that estimates the spatial spectrum for the direction of each sound source by performing spatial and temporal alignment corresponding to the sound source direction of the audio signal for each channel.

[0020] A method for estimating geometrically invariant sound source directions based on a deep neural network according to the disclosed embodiment and a system for executing the same apply a permute-invariant operation to the microphone channel axis so as to perform operations independent of the number and order of microphones, thereby enabling a deep neural network-based model to learn robustly for various microphone arrays and thereby effectively perform direction estimation for various microphone arrays.

[0021] FIG. 1 is a control block diagram of a system that executes a method for estimating geometrically invariant sound source directions based on a disclosed deep neural network.

[0022] FIG. 2 is a diagram specifically illustrating a system that implements a method for estimating geometrically invariant sound source directions based on a disclosed deep neural network.

[0023] FIG. 3 is a diagram illustrating the configuration of a space-time dual path section according to an embodiment of the present invention.

[0024] Figure 4 is an example diagram showing setting values ​​for a multi-geometry learning process in a method of progressively learning complexity for a disclosed deep neural network-based model.

[0025] FIG. 5 is a diagram showing a deep neural network-based model learned through a method of progressively learning complexity according to one embodiment of the present disclosure.

[0026] Figure 6 is a flowchart illustrating a method for estimating geometrically invariant sound source directions based on a disclosed deep neural network.

[0027] Figure 7 is an example diagram showing the results of comparing the sound source direction estimation performance between a method for estimating geometrically invariant sound source directions based on the disclosed deep neural network and a control group.

[0028] Figure 8 is an example diagram showing a method for estimating geometrically invariant sound source directions based on the disclosed deep neural network and the spatial spectra output from each model of the control group.

[0029] Figure 9 is an example diagram showing the performance of a model according to the number of microphones in a control group and a method for estimating geometrically invariant sound source directions based on the disclosed deep neural network.

[0030] Figure 10 is an example diagram showing the results of measuring the learning speed of a control group and a method for estimating geometrically invariant sound source directions based on the disclosed deep neural network.

[0031] Throughout the specification, the same reference numerals refer to the same components. This specification does not describe all elements of the embodiments, and general content in the art to which the invention pertains or content that overlaps between embodiments is omitted.

[0032] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are directly connected but also cases where they are indirectly connected, and indirect connections include connections made via a wireless communication network.

[0033] Furthermore, when it is stated that a part "includes" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0034] Singular expressions include plural expressions unless there is an obvious exception in the context.

[0035] In addition, terms such as "~part," "~unit," "~block," "~part," and "~module" may refer to a unit that processes at least one function or operation. For example, the above terms may refer to at least one piece of hardware such as an FPGA (field-programmable gate array) or an ASIC (application specific integrated circuit), at least one piece of software stored in memory, or at least one process processed by a processor.

[0036] The symbols attached to each step are used to identify each step and do not indicate the order of the steps relative to one another; the steps may be performed differently from the specified order unless a specific order is clearly indicated in the context.

[0037] Hereinafter, with reference to the attached drawings, embodiments relating to a method for estimating a geometrically invariant sound source direction based on a deep neural network according to the disclosed embodiment and a system for executing the same will be described in detail.

[0038] FIG. 1 is a control block diagram of a system that executes a method for estimating geometrically invariant sound source directions based on a disclosed deep neural network.

[0039] A system (1) for executing the method can be implemented as a computer or portable terminal (hereinafter user terminal, 3) capable of connecting to a communication network such as the internet. Here, the computer includes, for example, a laptop, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser, and the portable terminal can be implemented as any type of handheld-based wireless communication device, such as a smartphone, etc., as a wireless communication device that ensures portability and mobility.

[0040] Referring to FIG. 1, a user terminal (3) may include a communication interface (11) for receiving various data, such as a neural network model or training data required for speech recognition, from an external server (2); a memory (12) for storing the received training data and neural network model; an input unit (13) for receiving text to be converted into speech or other various user input commands; a processor (10) for controlling the overall system; and an output unit (14) for outputting the synthesized speech performed by the processor (10) as sound. However, since FIG. 1 is merely an example, the user terminal (3) may include other configurations for implementing a computing environment. Additionally, only some of the disclosed configurations may be included in the user terminal (3).

[0041] Specifically, the communication interface (11) may include one or more components that enable communication with an external communication network, and may include, for example, at least one of a short-range communication module, a wired communication module, and a wireless communication module.

[0042] The short-range communication module may include various short-range communication modules that transmit and receive signals using a wireless communication network at short range, such as a Bluetooth module, an infrared communication module, an RFID (Radio Frequency Identification) communication module, a WLAN (Wireless Local Access Network) communication module, an NFC communication module, and a Zigbee communication module.

[0043] Wired communication modules may include various wired communication modules such as Local Area Network (LAN) modules, Wide Area Network (WAN) modules, or Value Added Network (VAN) modules, as well as various cable communication modules such as USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).

[0044] In addition to Wi-Fi modules and WiBro (Wireless broadband) modules, the wireless communication module may include wireless communication modules that support various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), and LTE (Long Term Evolution).

[0045] The memory (12) may store a neural network model required to implement a method for estimating a geometrically invariant sound source direction based on the disclosed deep neural network, an algorithm required for the operation of the processor (10), or a program for implementing the algorithm. To this end, the memory (12) may be implemented as at least one of a non-volatile memory device such as a cache, ROM (Read Only Memory), PROM (Programmable ROM), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and flash memory, a volatile memory device such as RAM (Random Access Memory), or a storage medium such as a hard disk drive (HDD) and a CD-ROM, but is not limited thereto.

[0046] The input unit (13) can receive various input commands. Specifically, the input unit (13) may include a microphone that receives voice that speaks into a video containing text as subtitles, in addition to receiving input text. Furthermore, the input unit (13) may include hardware devices such as various buttons, switches, pedals, keyboards, mice, trackballs, various levers, handles, or sticks for receiving execution commands required for the operation of the processor (10). Additionally, the input unit (13) may include a GUI (Graphical User Interface), i.e., a software device such as a touch pad, for user input commands. The touch pad may be implemented as a touch screen panel (TSP) and form a layered structure with the display of the output unit (14).

[0047] The output unit (14) may include a speaker that outputs the recognized voice as sound, as well as a display for outputting the learning results or inference results of a neural network model. The display may be provided as a Cathode Ray Tube (CRT), Digital Light Processing (DLP) panel, Plasma Display Panel, Liquid Crystal Display (LCD) panel, Electro Luminescence (EL) panel, Electrophoretic Display (EPD) panel, Electrochromic Display (ECD) panel, Light Emitting Diode (LED) panel, or Organic Light Emitting Diode (OLED) panel, but is not limited thereto.

[0048] The processor (10) can fine-tune or train a neural network model stored in memory (12) and perform speech recognition functions, including sound source direction estimation, through the trained neural network model of input text.

[0049] Specifically, the processor (10) can add microphone coordinate information to the embedding vector through a microphone position encoding process so that it can estimate the geometrically invariant sound source direction using a deep neural network-based model, thereby enabling the estimation of the sound source direction regardless of the shape of the microphone array and the number of microphones. The processor (10) can perform a method of gradually learning complexity by increasing the difficulty regarding the number and shape of the microphones.

[0050] A specific method for the processor (10) to estimate the geometrically invariant sound source direction based on a deep neural network will be described later through other drawings below.

[0051] Meanwhile, the processor (10) may refer to a data processing device embedded in hardware having a physically structured circuit to perform a function expressed by code or instructions included in a program. Examples of such data processing devices embedded in hardware may include, but are not limited to, processing devices such as a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a Graphics Processing Unit (GPU), and a neural Processing Unit (NPU). The processor (10) may be provided in multiple units.

[0052] The hardware configurations provided in the user terminal (10) can exchange data and signals via Network Termination (NT) of a digital network such as ISDN (Integrated Services Digital Network).

[0053] FIG. 2 is a diagram specifically illustrating a system for executing a method for estimating a geometrically invariant sound source direction based on a disclosed deep neural network, and FIG. 3 is a diagram illustrating the configuration of a space-time dual path section according to an embodiment of the present invention.

[0054] Referring to FIG. 2, the system (1) includes, but is not limited to, a microphone array (110), a frequency converter (120), a feature extraction unit (130), a microphone position encoding unit (140), a spatial-temporal dual path unit (150), and a spatial spectrum mapping unit (160).

[0055] A microphone array (110) containing one or more microphones acquires a multi-channel audio signal for one or more speakers. In this case, each microphone is designed to occupy a specific coordinate in space, and each microphone may form one channel or form one or more channels per microphone. Accordingly, the microphone array (110) acquires a multi-channel audio signal including the audio signal of each microphone.

[0056] The multichannel audio signal consists of a left channel audio signal, a center channel audio signal, and a right channel audio signal, and the microphone array (110) receives the center (c) channel audio signal among the multichannel audio signals.

[0057] The frequency conversion unit (120) converts the acquired multi-channel audio signal into frequency domain information by channel through a short-time Fourier transform (STFT).

[0058] The feature extraction unit (130) extracts independent features for each channel of frequency domain information. The feature extraction unit (130) operates as a Channel Independent Feature Extractor (CIFE).

[0059] The microphone position encoding unit (140) converts the multi-channel audio signal into an embedding vector and encodes the microphone coordinate information of each channel to add a position encoding embedding for the microphone coordinate information to the embedding vector. When the microphone array is represented in a spherical coordinate system, the position of the microphone can be represented by an azimuth, an elevation angle, and a distance (or radius). Here, the radius in the spherical coordinate system represents the distance from the center of the sphere to the point.

[0060]

[0061] In mathematical equation 1, P c is a position encoding matrix, and the spherical coordinate information (r) of each microphone c ,θ c ,Φ c θ in ) c , Φ c , r c represents the azimuth, elevation, and distance, respectively. Also, in Equation 1, M represents the feature size of the deep learning model, v represents a vector divided equally into M / 4 over the range [0, 1], α is the size of the position encoding matrix, and β is a scaling factor that controls the frequency (repetition period) of the position encoding matrix.

[0062] The formulas substituted into the sine and cosine functions used in the microphone position encoding unit (140) are selected to generate periodic patterns, and can generate unique values ​​for different positions while generating periodic values. These unique values ​​can reflect the relative positions of the microphone in the embedding space.

[0063] The microphone position encoding unit (140) generates a periodic pattern using a sine function and a cosine function, so that the position encoding embeddings have different periods, which helps each dimension to encode different position information. In addition, the microphone position encoding unit (140) ensures diversity between dimensions by having the sine function handle even dimensions and the cosine function handle odd dimensions, so that each dimension handles different information, and enables the model to effectively learn the relative position of the microphone.

[0064] Spherical coordinate information of each microphone (r c ,θ c ,Φ c The distance from ) can be provided through Amplitude Modulation (AM) by multiplying it by a sine / cosine function, and the azimuth and elevation angles can be provided by applying Phase Modulation (PM), which modulates the phase of the sine / cosine function, or Frequency Modulation (FM), which modulates the frequency.

[0065] The spatial-temporal dual path section (150) is composed of a channel-wise multi-head self-attention (CW-MHSA) (151) that is applied to all channels to share information between channels, and a frame-wise gated recurrent unit (FW-GRU) (152) that is applied to each frame to share time information. The spatial-temporal dual path section (150) uses the output of the feature extraction section (130) and the position encoding embedding as input values, and can effectively combine the features of each channel through the channel-wise multi-head self-attention (151) and the frame-wise gated recurrent unit (152).

[0066] The space-time dual path section (150) can be composed of at least one to N blocks, but to increase the accuracy of the model, N=4 blocks can be stacked sequentially.

[0067] Channel-specific multi-head self-attention (151) calculates Q, K, V, which are query, key, and value vectors, calculates an attention score by taking the inner product of the query and the key, calculates an attention weight which is the weighted average of the attention scores by applying a softmax function, and then calculates a weighted sum of the values. In FIG. 3, P represents the position encoding for microphone coordinate information, R represents the entire set of real numbers, Y represents the DNN input and output tensors, and the superscripts (C, M, T) indicate the input and output of a specific module. C represents the total number of channels, M represents the feature size, and T represents the total number of frames of the audio input.

[0068] Layer Normalization (or Layer Norm) is a training technique designed to train models faster by calculating the mean and standard deviation for each data sample within a single batch and performing normalization. This can also be expressed as the transpose of batch normalization, which performs normalization on a feature-by-feature basis within a single batch.

[0069] Self-attention is not performed using embedding vectors that have the dimension of the initial input model, but rather first obtains Q, K, and V vectors from each vector, and the Q, K, and V vectors have a smaller dimension than the embedding vectors that have the dimension of the initial input model. Therefore, channel-wise multi-head self-attention (151) performs a dimension permutation in which the input embedding vector is received as a c-dimensional channel, attention is performed in g-dimensional, and then output again in c-dimensional.

[0070] The frame-by-frame gated circulation unit (152) performs channel-by-channel softmax aggregation, frame-by-frame GRU (Gated Recurrent Unit), ELU (Exponential Linear Unit), linear layer, and layer normalization.

[0071] The spatial spectrum mapping unit (160) is composed of N RCBs and a linear layer and estimates the spatial spectrum for each direction. First, the spatial spectrum mapping unit (160) compresses channel information using Channel-wise Softmax Aggregation (CWSA). The output of the CWSA passes through N RCBs in succession and simultaneously passes through the linear layer connected to each RCB to finally output the spatial spectrum. To effectively learn about various microphone arrays, the present invention may apply a complexity gradual training method (CGT). The complexity gradual training method consists of a multi-stage geometry learning (MSGL) process in which the microphone array learns by gradually increasing the difficulty from one form or type to multiple forms or numbers, and a deeply-supervised curriculum learning (DSCL) process in which the target spatial spectrum changes according to the depth of the model and the progress of learning.

[0072] FIG. 4 is an example diagram showing setting values ​​for a multi-geometry learning process in a method of progressively learning complexity for a disclosed deep neural network-based model, and FIG. 5 is a diagram showing a deep neural network-based model learned through a method of progressively learning complexity according to one embodiment of the present disclosure.

[0073] As shown in Fig. 4, in the multi-geometry learning process, various array types, initial learning rates, weight decays, and applied epochs can be applied for learning.

[0074] In the multi-geometry learning process, the model learns only one type of 4-channel array in the first stage, learns various forms of 4-channel arrays in the second stage, and learns various array forms ranging from 4 to 12 channels in the third stage. In this way, the model can learn robustly for various arrays. Through this, direction estimation can be performed effectively for various arrays.

[0075] The deep supervised curriculum learning process can be structured as a combination of the curriculum learning method and the deep supervision method. Deep supervision refers to a deep learning model approach that provides additional supervised information (e.g., intermediate outputs) to intermediate layers, rather than the conventional method of training which utilizes only the outputs at the end to calculate the target and loss. Models using the deep supervision method may have an auxiliary loss function in the intermediate layers. This auxiliary loss function can predict the error at each layer based on the outputs of the intermediate layers. The predicted error can then be combined with the final loss function to facilitate learning. The curriculum learning method gradually narrows the width of the correct direction spectrum used for training, initially increasing the correlation with the periphery, and eventually inducing the output to have a direction spectrum that accurately points only to the correct direction.

[0076] A deep neural network-based model for estimating the direction of a sound source always applies a permute-invariant operation to the microphone channel axis to perform operations independent of the number and order of microphones. Here, permutation invariant refers to a model that generates the same output regardless of the order of the input vector elements.

[0077] Referring to FIG. 5, after the multi-channel audio signal undergoes STFT, instantaneous relative transfer function (iRTF) features are extracted. The extracted features are passed through a shared module (410) composed of a convolutional neural network (CNN), batch normalization (BN), and an exponential linear unit (ELU). The features passed through the shared module (510) are transmitted to n skip connection modules (SCMs), which are mapping modules (520), and then transmitted to each output layer after learning. Here, the SCMs may have a structure of CNN+BN+ELU, and the output layer may be composed of CNN+Sigmoid operation steps. In the deep supervised learning module (530), the Cross-entropy loss (CE Loss) can be measured using labeled information (or supervised information, data values ​​corresponding to specific σ values) that are different from the values ​​output from each output layer. CE Loss can represent the difference between the model's predicted probability distribution and the actual label probability distribution. For example, in the first SCM, the loss can be measured by providing label information with the largest σ value of 16 as map information. In the next step, the second SCM, the loss can be measured by providing label information with a σ value of 6 as map information. In the following step, the third SCM, the loss can be measured by providing label information with a σ value of 2 as map information. This method may induce clearer outputs as the layers deepen by assigning progressively smaller values ​​to the label information around the target. Separately from assigning label information as intermediate information, curriculum learning can be performed by determining the initial and final σ values ​​within a single module to ensure that the σ value gradually decreases.For example, a curriculum learning method can be performed in the first SCM, where the labeling information is initially assigned as 16 but is ultimately assigned as 2, thereby training to decrease consistently. Additionally, in the algorithm module of the first layer, a σ value is assigned to maximize the initial value, and learning is performed through curriculum learning via scheduling until a predetermined final σ value is reached. In the module of the next layer, an initial σ value smaller than that of the first module is assigned, so that as the layers increase, the initial σ value gradually decreases, while the final σ value is set to be the same to train the artificial intelligence algorithm. Here, the scheduling may include a method that maintains a constant rate of decrease in σ, setting the σ value to decrease from the initial σ value to the final σ value. Regarding the deep supervision method, the result output from the first SCM is output to the output layer, but it is also output to the next layer, the second SCM. In the second SCM, learning is performed by reflecting the result from the first SCM. Therefore, the learning result of the second SCM derives a result that additionally reflects the learning result from the first SCM. Even if the number of layers increases thereafter, results can be derived by similarly reflecting the learning results up to the previous layer.

[0078] FIG. 6 is a flowchart illustrating a method for estimating geometrically invariant sound source directions based on a disclosed deep neural network. Each step described below is a process performed by each functional part constituting the system, but for a concise and clear explanation of the invention, the subject of each step is collectively referred to as the system.

[0079] The system (1) obtains a multi-channel audio signal for one or more speakers from a microphone array (110) including one or more microphones (S1).

[0080] The system (1) converts the acquired multi-channel audio signal into frequency domain information by channel through a short-time Fourier transform (STFT) (S2), and extracts independent features for each channel from the converted frequency domain information (S3).

[0081] The system (1) converts multi-channel audio signals into embedding vectors and performs a microphone position encoding process by encoding microphone coordinate information of each channel and adding a position encoding embedding for the microphone coordinate information to the embedding vectors (S4).

[0082] The system (1) uses microphone coordinate information as input values ​​and combines independent features of each channel through channel-specific multi-head self-attention and frame-specific gated circulation units (S5).

[0083] The system (1) performs spatial and temporal alignment corresponding to the sound source direction of the audio signal for each channel to estimate the spatial spectrum for each sound source direction (S6).

[0084] Figure 7 is an example diagram showing the results of comparing the sound source direction estimation performance between a method for estimating geometrically invariant sound source directions based on the disclosed deep neural network and a control group.

[0085] Mean absolute error (MAE) represents the absolute difference between the correct answer and the estimate, and a lower value indicates better performance. Accuracy (ACC) represents the percentage of frames judged as correct when the MAE is within 10 degrees for each frame.

[0086] As shown in Figure 7, it can be seen that the method for estimating geometrically invariant sound source directions based on the disclosed deep neural network demonstrates better performance compared to the control group that does not apply the microphone position encoding process by applying the microphone position encoding process.

[0087] Figure 8 is an example diagram showing a method for estimating geometrically invariant sound source directions based on the disclosed deep neural network and the spatial spectra output from each model of the control group.

[0088] As shown in Fig. 8, the spatial spectrum output from each model represents the energy for each direction, allowing us to see how accurately the direction is estimated compared to the correct answer. It can be seen that the method for estimating geometrically invariant sound source directions based on the disclosed deep neural network shows a more accurate and clearer spatial spectrum compared to a control group that does not apply the microphone position encoding process by applying the microphone position encoding process.

[0089] Figure 9 is an example diagram showing the performance of a model according to the number of microphones in a control group and a method for estimating geometrically invariant sound source directions based on the disclosed deep neural network.

[0090] Figure 9 shows the performance (MAE, ACC) of the model for different signal-to-ratio (SNR), reverberation time (RT60), and number of microphones. A lower SNR indicates a more noisy environment, a higher RT60 indicates a more reverberant environment, and a smaller number of microphones indicates a more difficult environment, as less information can be acquired.

[0091] It can be seen that the method for estimating geometrically invariant sound source orientation based on the disclosed deep neural network demonstrates more stable performance in challenging environments compared to a control group that does not apply the microphone position encoding process by applying the microphone position encoding process.

[0092] Figure 10 is an example diagram showing the results of measuring the learning speed of a control group and a method for estimating geometrically invariant sound source directions based on the disclosed deep neural network.

[0093] Figure 10 shows the number of parameters (# params.), floating point operations per second (FLOPS), and inference time on the CPU and GPU for each model. It can be seen that the model to which the method for estimating geometrically invariant sound source directions based on the disclosed deep neural network is applied has an intermediate number of parameters and low FLOPS compared to other models, and has fast inference time on the CPU and GPU. In particular, as the number of channels increases, other models show an increase in FLOPS proportional to the square of the number of channels, whereas the present invention shows an increase in FLOPS proportional to the number of channels, indicating efficiency.

[0094] Although FIG. 6 describes each process as being executed sequentially, this is merely an illustrative explanation of the technical concept of one embodiment of the present invention. In other words, a person skilled in the art to which one embodiment of the present invention belongs can apply various modifications and variations by changing the order described in each figure or executing one or more of the processes in parallel, as long as they do not deviate from the essential characteristics of one embodiment of the present invention. Therefore, FIG. 6 is not limited to a chronological order.

[0095] Meanwhile, the processes illustrated in FIG. 6 can be implemented as computer-readable code on a computer-readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. That is, a computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.). Additionally, computer-readable recording media can be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner.

Claims

1. A method for estimating a geometrically invariant sound source direction based on a deep neural network, performed by a system comprising at least one processor, wherein An audio signal acquisition process for acquiring a multi-channel audio signal for one or more speakers from a microphone array including one or more microphones; A frequency conversion process that converts an acquired multi-channel audio signal into frequency domain information through a Short-Time Fourier Transform (STFT) for each channel; A feature extraction process for extracting channel-independent features from transformed frequency domain information; A microphone position encoding process that converts the above multi-channel audio signal into an embedding vector, encodes microphone coordinate information of each channel, and adds a position encoding embedding for the microphone coordinate information to the embedding vector; A spatial-temporal dual-path process that combines independent features of each channel through a channel-wise multi-head self-attention and a frame-wise gated recurrent unit, using the output value from the above feature extraction process and the above position encoding embedding as input values; and A method comprising a spatial spectrum mapping process that estimates the spatial spectrum for each sound source direction by performing spatial and temporal alignment corresponding to the sound source direction of the audio signal for each channel.

2. In Paragraph 1, The above multichannel audio signal is, It includes a left channel audio signal, a center channel audio signal, and a right channel audio signal, A method in which the above microphone array acquires the center channel audio signal.

3. In Paragraph 1, The above microphone position encoding process is, A method of generating a periodic pattern using a sine function and a cosine function so that the position encoding embeddings have different periods.

4. In Paragraph 1, The above space-time dual-path process is, A method comprising a block including a frame-wise gated recurrent unit obtained by channel-wise softmax aggregation of the results of channel-wise multi-head self-attention, and stacking at least one block up to N (N>0, N is a natural number) blocks sequentially.

5. In Paragraph 1, The above spatial spectrum estimation process is, Using a deep neural network-based model to estimate the spatial spectrum for the direction of each sound source, and The above deep neural network-based model is a method trained by applying a complexity gradual training (CGT) method that gradually increases the difficulty regarding the number of channels and shape of the microphone array.

6. In Paragraph 5, The method for gradually learning the above complexity is, A method comprising a multi-stage geometry learning (MSGL) process for learning by increasing the difficulty of the shape of the microphone array or the number of microphones, and a deeply-supervised curriculum learning (DSCL) process for changing the target spatial spectrum according to the depth of the deep neural network-based model and the progress of learning.

7. In Paragraph 5, The above deep neural network-based model is, A method of applying a permute-invariant operation to the microphone channel axis to perform an operation independent of the number and order of microphones.

8. A system for estimating geometrically invariant sound source directions based on a deep neural network, A microphone array for acquiring multi-channel audio signals for one or more speakers; A frequency converter that converts acquired multi-channel audio signals into frequency domain information through a short-time Fourier transform (STFT) for each channel; A feature extraction unit that extracts independent features for each channel from transformed frequency domain information; A microphone position encoding unit that converts the above multi-channel audio signal into an embedding vector, encodes microphone coordinate information of each channel, and adds a position encoding embedding for the microphone coordinate information to the embedding vector; A spatial-temporal dual-path unit that combines independent features of each channel through a channel-wise multi-head self-attention and a frame-wise gated recurrent unit, using the output value from the feature extraction unit and the position encoding embedding as input values; and A system comprising a spatial spectrum mapping unit that estimates the spatial spectrum for each sound source direction by performing spatial and temporal alignment corresponding to the sound source direction of the audio signal for each channel.

9. In Paragraph 8, The above feature extraction unit is, A system composed of a Channel Independent Feature Extractor (CIFE).

10. In Paragraph 8, The above space-time dual path section is, A system comprising blocks including channel-wise multi-head self-attention, channel-wise softmax aggregation, and frame-wise gated recurrent unit, wherein the blocks are stacked sequentially from at least one to N (N>0, N is a natural number).

11. A computer program stored on a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs operations for estimating a geometrically invariant sound source direction based on a deep neural network, and The above operations are, An audio signal acquisition operation for acquiring a multi-channel audio signal for one or more speakers from a microphone array including one or more microphones; Frequency conversion operation that converts acquired multi-channel audio signals into frequency domain information through Short Time Fourier Transform (STFT) for each channel; Feature extraction operation for extracting channel-independent features from transformed frequency domain information; A microphone position encoding operation that converts the above multi-channel audio signal into an embedding vector, encodes microphone coordinate information of each channel, and adds a position encoding embedding for the microphone coordinate information to the embedding vector; A spatial-time dual-path operation that combines independent features of each channel through channel-wise multi-head self-attention and a frame-wise gated recurrent unit using the above microphone coordinate information as input values; and A computer program comprising a spatial spectrum mapping operation that estimates the spatial spectrum for each sound source direction by performing spatial and temporal alignment corresponding to the sound source direction of the audio signal for each channel.

Images