Methods, non-transitory computer-readable media, and apparatuses with neural network that reduces spatial aliasing

Abstract

In some examples, a method of reducing spatial aliasing in a beamformed signal includes generating an anti-aliasing spectral mask by feeding a spectrogram of the beamformed signal to a neural network trained, using a plurality of aliased beamformed signals and a corresponding plurality of aliased-reduced beamformed signals, to generate anti-aliasing spectral masks. The method also includes generating an aliased reduced spectrogram of the beamformed signal by applying the anti- aliasing spectral mask to the spectrogram of the beamformed signal.

Description

METHODS, NON-TRANSITORY COMPUTER-READABLE MEDIA, AND APPARATUSES WITH NEURAL NETWORK THAT REDUCES SPATIAL ALIASINGCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of priority from Spanish Application No.P202530098, filed on February 10, 2025, and US Provisional Application Serial No. 63 / 969,942, filed on 28 January 2026, each of which is incorporated by reference herein in its entirety.FIELD OF THE DISCLOSURE

[0002] Various example embodiments relate to audio-signal processing and, more specifically but not exclusively, to reducing the effects of spatial aliasing on audio signals captured using microphone arrays.BACKGROUND

[0003] Herein, “spatial aliasing” refers to a phenomenon that causes a sound wave’s spatial information (such as its direction of arrival) to become ambiguous when sampled by a microphone array with a relatively large spacing between the constituent microphones, which causes the system to misinterpret the true location of the sound source. For example, high-frequency spatial information from a certain direction can be misinterpreted as representing spatial information at the same frequency from another direction, due to the insufficient spacing of the array. As a result, two different directions that should otherwise be unique may disadvantageously become indistinguishable from one another. In other words, when spatial aliasing occurs, the direction of arrival of the acoustic signal, as judged by the system, becomes ambiguous.BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS

[0004] In the context of recording sound with microphone arrays, several microphone signals may be combined with respective frequency dependent gains to obtain signals with desired directional characteristics. When the microphones of the array are non-coincident, spatial aliasing occurs at frequencies whose corresponding wavelengths are smaller than twice the spacing between the microphones. In some examples, spatial aliasing manifests itself as an undesired frequency-and / or direction-dependent boost or attenuation above the aliasing frequency fA.

[0005] Various embodiments disclosed herein provide a method directed at mitigating the effects of spatial aliasing using a deep neural network (DNN) trained in a supervised way. In some embodiments, the DNN receives an aliased spectrogram and produces a spectral mask that helps to reduce or eliminate the effects of aliasing when applied to the aliased spectrogram. In some other embodiments, the DNN receives an aliased spectrogram and generates a plausible corresponding substantially alias-free spectrogram.

[0006] In some aspects, the techniques described herein relate to a method of reducing spatial aliasing in a beamformed signal, the method including: generating an anti-aliasing spectral mask by feeding a spectrogram of the beamformed signal to a neural network trained, using a plurality of aliased beamformed signals and a corresponding plurality of aliased-reduced beamformed signals, to generate anti-aliasing spectral masks; and generating an aliased-reduced spectrogram of the beamformed signal by applying the anti-aliasing spectral mask to the spectrogram of the beamformed signal.

[0007] In some aspects, the techniques described herein relate to an apparatus for reducing spatial aliasing in a beamformed signal, the apparatus including: at least one processor; and at least one memory including program code, wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: generate an antialiasing spectral mask by feeding a spectrogram of the beamformed signal to a neural network trained, using a plurality of aliased beamformed signals and a corresponding plurality of aliased-reduced beamformed signals, to generate anti-aliasing spectral masks; and generate an aliased-reduced spectrogram of the beamformed signal by applying the anti-aliasing spectral mask to the spectrogram of the beamformed signal.

[0008] In some aspects, the techniques described herein relate to a method of reducing spatial aliasing in a beamformed signal, the method including: generating the beamformed signal by applying beamforming to a set of microphone array signals; and generating an aliased-reduced spectrogram of the beamformed signal by feeding a spectrogram of the beamformed signal to a neural network trained, using a plurality of aliased beamformed signals and a corresponding plurality of aliased-reduced beamformed signals, to generate anti-aliasing spectral masks.

[0009] In some aspects, the techniques described herein relate to an apparatus for reducing spatial aliasing in a beamformed signal, the apparatus including: at least one processor; and at leastone memory including program code, wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: generate the beamformed signal by applying beamforming to a set of microphone array signals; and generate an aliased-reduced spectrogram of the beamformed signal by feeding a spectrogram of the beamformed signal to a neural network trained, using a plurality of aliased beamformed signals and a corresponding plurality of aliased-reduced beamformed signals, to generate anti-aliasing spectral masks.

[0010] As used in this specification, the term "spectrogram" refers to a complex spectrogram, for example obtained by applying the Short-Time Fourier Transform (STFT) to a time-domain signal. While the disclosed embodiments primarily describe operations using complex spectrograms, the methods and systems described herein are also applicable to real-valued spectrograms, which can be obtained using the absolute values (or magnitudes) of the pixels of the complex spectrogram. When utilizing real spectrograms, phase information may be reconstructed through implementation of a phase retrieval algorithm. Representative examples of the phase retrieval algorithm include, but are not limited to, a Griffin-Lim algorithm and a neural vocoder.

[0011] In at least some examples, the complex spectrogram contains sufficient information to uniquely determine and reconstruct the original time-domain signal, thereby establishing substantial mathematical equivalence between reconstruction of the complex spectrogram and reconstruction of the corresponding time-domain signal. Consequently, embodiments in which a DNN receives an aliased spectrogram and generates a corresponding substantially alias-free spectrogram may be alternatively implemented as embodiments in which the DNN receives an aliased time-domain signal and generates a corresponding substantially alias-free time-domain signal. Such alternative time- and frequency-domain implementations should therefore be acknowledged as being substantial functional equivalents of one another.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Other aspects, features, and benefits of various disclosed embodiments will become more fully apparent, by way of example, from the following detailed description and the accompanying drawings, in which:

[0013] FIG. l is a diagram illustrating a schematic three-dimensional (3D) perspective view of a mobile phone with three microphones according to some examples.

[0014] FIG. 2 is a diagram illustrating an example of the effects of spatial aliasing on sound field decomposition using conventional beamforming according to some examples.

[0015] FIG. 3 is a diagram illustrating an example of an aliased beamformer signal according to some examples.

[0016] FIG. 4 is a diagram illustrating an example of a corresponding ideal alias-free signal according to some examples.

[0017] FIG. 5 is a diagram illustrating an example of an estimated filtering mask that removes aliasing according to some examples.

[0018] FIG. 6 is a diagram illustrating an example of a reconstructed spectrogram that is a result of applying the filtering mask of FIG. 5 to the aliased beamformer signal of FIG. 3 according to some examples.

[0019] FIG. 7 is a block diagram illustrating an example audio-signal processing pipeline according to some examples.

[0020] FIG. 8 is a block diagram illustrating an example generation method of an anti-aliasing DNN according to some examples.

[0021] FIG. 9 is a diagram illustrating an example of an output of a beamformer configured to obtain the same directional response but affected by spatial aliasing according to some examples.

[0022] FIG. 10 is a diagram illustrating an example of an ideal, alias-free output of a directional microphone recording a mixture of sources from different directions according to some examples.

[0023] FIG. 11 is a diagram illustrating an example of the anti-aliasing mask produced by the DNN according to some examples.

[0024] FIG. 12 is a diagram illustrating an example of a filtered output, showing reduced spatial aliasing according to some examples.

[0025] FIG. 13 is a block diagram illustrating an example of an audio-signal processing pipeline according to some additional examples.

[0026] FIG. 14 is a block diagram illustrating an immersive voice and audio services (IV AS) coder / decoder ("codec") framework for encoding and decoding IVAS bitstreams according to some examples.

[0027] FIG. 15A illustrates a schematic block diagram of an example device architecture (e.g., an apparatus) that may be used to implement various aspects of the present disclosure.

[0028] FIG. 15B illustrates a schematic block diagram of an example CPU implemented in the device architecture of FIG. 15A that may be used to implement various aspects of the present disclosure.DETAILED DESCRIPTION

[0029] Spaced microphone arrays are commonly found in devices such as smartphones, laptops, headsets, action cameras and surround microphone setups. The microphones M= m, mNare typically combined with frequency-dependent complex gains G to obtain one or more signals V = G -M, where V= v1,..., vvare directional signals corresponding to virtual microphones with a desired polar pattern, looking direction, or null direction. The virtual microphones are typically obtained using beamforming or spherical harmonics encoding, and some common applications include audio zoom, noise suppression, and Ambisonics encoding.

[0030] It is often the case that the microphone array is non-coincident, in which case the underlying principle on which many DSP techniques are based is that the incoming sources produce phase differences between the microphones, and for a single source the phase difference is an injective function (i.e., a one-to-one mapping) of the direction of arrival (DOA). However, this condition holds true only when the wavelength is larger than twice the spacing between the microphones. When the wavelength is smaller than the 2x spacing, it is no longer possible to associate a phase difference to a single DOA. Hence, the beamforming and spherical harmonics encoding result in spatial aliasing or a degradation of the desired directional output signal where sources coming from other directions are amplified instead of being attenuated, and sources coming from the direction of interest may be attenuated, all in a frequency- and direction-dependent manner.

[0031] FIG. l is a diagram illustrating a schematic three-dimensional (3D) perspective view of a mobile phone 100 with three microphones 102-106 according to some examples. In mobile devices, it is common to have microphones separated by a few centimeters. For example, in FIG. 1, the spacing between the top microphone 102, the back microphone 104, and the bottom microphone 106 may be in the approximate range between 3 cm and 15 cm, which corresponds to aliasing frequencies between approximately 2 kilohertz (kHz) and 10 kHz.

[0032] Beamforming enables the extraction of signals from desired directions using an array of microphones. While often used as a preprocessing step, such as denoising and separation for automatic speech recognition, beamforming plays a central role in spatial audio, where beamforming enables audio encoding, e.g., obtaining directional stereo signals from a spaced pair of omnidirectional microphones or converting a multi -mi crophone recording into Ambisonics.

[0033] Similar to how the discrete sampling interval in time domain determines the frequency limit above which time domain aliasing occurs, spatial sampling is also subject to an analogous limitation, known as spatial aliasing. Spatial aliasing affects spaced microphone arrays, where processing based on phase differences suffers from directional ambiguity above the spatial aliasing frequency. Spatial aliasing drastically degrades spatial and spectral accuracy of beamformers.

[0034] In some examples of signal processing, efforts to mitigate spatial aliasing in beamforming focus on wide-band and multi-stage solutions, rotating arrays, and modeling of spatial aliasing for spherical arrays. Despite the significant interest in deep learning approaches to signal processing, the topic of solving spatial aliasing has previously attracted little attention in the deep learning audio community.

[0035] Various embodiments disclosed herein address the above-indicated problems in the state of the art by addressing spatial aliasing reduction in conventional beamforming using a deep learning approach. Some embodiments use a U-Net-based filter configured to reduce the effects of spatial aliasing in the signals obtained as a result of sound field encoding and decoding. Some examples allow for direct evaluation of the alias-reducing capabilities of the system and have high potential for straightforward integration into popular spatial audio processing applications, e.g., obtaining first-order Ambisonics (FOA) signals from a tetrahedral microphone array or obtaining directional stereo signals from compact omnidirectional arrays. In some examples, alias-reducingincludes de-aliasing. Additionally, in some examples, alias-reduced includes de-aliased and / or alias-free.

[0036] Various examples implement two distinct approaches. In the first approach, a DNN is trained to produce a suitable spectral mask given an aliased input spectrogram and, optionally, the conditioning signals of the microphone array. The spectral mask acts as a filter where the linear gain g(f, t) is a complex number whose magnitude is not constrained to be less than one, thereby enabling a boost of the input signal where / when needed. In the second approach, a DNN is trained to generate a plausible alias-free spectrogram based on the aliased beamformer signal, and the alias-free microphone array signals, which are used as conditioning inputs.

[0037] Various embodiments implement the following noteworthy features / aspects:i. A DNN may be used to remove spatial aliasing from a beamformer.ii. Spatial aliasing may be removed from a beamformed signal by applying a filter (spectral mask) that can boost or attenuate the aliasing components to reduce or eliminate them. iii. A second alias-free band (the band above fA) may be generated based on a first alias-free band (the band below fA) with the conditioning of the microphone array signals. iv. A training scheme that allows a DNN to learn the alias-reducing spectral mask of (ii) or to generate the alias-free band of (iii).

[0038] Spatial processing of audio signals captured with spaced microphones is in general based on phase differences between the sensors, leveraging the property that in a certain frequency range they map univocally to the DOA of an acoustic wavefront. The upper frequency limit of the aforementioned range depends on the spacing between the microphones, such that fA= c / (2d), where fAis the spatial aliasing frequency, c is the speed of sound and d is the microphone spacing. Above fA, the phase differences do not unambiguously map to the DOA because sound originating from different directions can cause identical shift through phase warping.

[0039] FIG. 2 is a diagram illustrating an example 200 of the effects of spatial aliasing on sound field decomposition using conventional beamforming according to some examples. Therein, two omnidirectional microphones 202 and 204 are used to form a cardioidal response at a frequency above fA. For each curve, the distance from the center of the microphone array represents the magnitude response for the corresponding direction.

[0040] In the example 200, the spatial response is deformed due to spatial aliasing and illustrated as the spatially aliased directivity 206. As a result, with respect to the spatially aliased directivity 206 and an alias-free directivity 208, the signal of source i is over attenuated.Additionally, with respect to the spatially aliased directivity 206 and the alias-free directivity 208, the signal of source s2is overly amplified.

[0041] Since these alterations caused by spatial aliasing are frequency-dependent, source signals will be subject to direction-dependent filtering. In contrast, in the alias-free case, they would be scaled by frequency- independent directional factors. This implies that, in case of sound field decomposition, where the goal is to obtain a specified frequency -independent spatial response, a frequency- and direction-dependent combination of both attenuation and amplification is needed to reduce spatial aliasing with a filtering approach.

[0042] FIG. 3 is a diagram illustrating an example 300 of an aliased beamformer signal according to some examples. FIG. 4 is a diagram illustrating an example 400 of a corresponding ideal alias-free signal according to some examples. FIG. 5 is a diagram illustrating an example 500 of an estimated filtering mask that removes aliasing according to some examples. FIG. 6 is a diagram illustrating an example 600 of a reconstructed spectrogram that is a result of applying the filtering mask of FIG. 5 to the aliased beamformer signal of FIG. 3 according to some examples.

[0043] For illustration purposes and without any implied limitations, it is assumed that the input signals, output signals, and the filters are spectrograms, and the time and frequency variables are omitted when the time and frequency variables are not explicitly needed to convey the pertinent concepts more-clearly.

[0044] In some examples, the research community is primarily focusing on using Al to build beamformers blindly, often with the goal of spatial source separation, to replace DSP-based techniques. In contrast, an approach implemented in various embodiments disclosed herein focuses on improving existing DSP-based beamformers by filtering or regenerating the band affected by spatial aliasing. In some cases, this approach is motivated by the relevance of existing DSP-based beamformers and psycho-acoustically optimized Ambisonics encoders, which:• Feature adequate and predictable objective and subjective performance suitable both for UGC mobile capture and for professional mic-array immersive capture, in particular below fA-• Aim at producing a spatial decomposition of the sound field instead of a source decomposition of the sound field.

[0045] FIG. 7 is a block diagram illustrating an example audio-signal processing pipeline 700 according to some examples. The example 700 of FIG. 7 implements the above-mentioned spectral mask approach. The example 700 includes input signals 702, a DNN 704, beamformers 706, aliased directional microphone signals 708, anti-aliasing masks 710, filtering 712, and alias-reduced directional microphone signals 714.

[0046] In the example 700, the input signals 702 are a set of virtual microphones v1,...,vvobtained as V = G -M, where M is a set of microphone array signals m,..., mNand G is a matrix of complex gains.

[0047] In various examples, the virtual microphone signals can be obtained from microphone array signals using one of the following techniques:• Delay and sum beamforming with a peak or a null in a predefined direction (e.g., to obtain a fixed polar pattern).• Minimum variance distortion-less response (MVDR) beamforming with a peak or null in a predefined direction (e.g., to maximize the pickup in a region of interest and suppress sources outside the region of interest, as in audio zoom applications).• Spherical harmonics encoding (e.g., to obtain first-order Ambisonics components W, X, Y, Z from a circular microphone array).

[0048] The pipeline 700 of FIG. 7 includes the DNN 704 that is designed and trained to receive the input signals 702 and the aliased directional microphone signals 708 (e.g., the complex input spectrogram of FIG. 3) and determine the anti-aliasing masks 710 (e.g., the estimated filtering mask of FIG. 5). The anti-aliasing masks 710, when applied to the aliased directional microphone signals 708 with the filtering 712, produce the alias-reduced directional microphone signals 714 (i.e., modified spectrograms) wherein the effects of spatial aliasing are reduced (e.g., the reconstructed spectrogram of FIG. 6).

[0049] In various examples, reducing spatial aliasing may involve a suitable combination of one or more of: attenuating unwanted source components from undesired directions, amplifying components that have been attenuated, and compensating for phase differences introduced by thebeamformer. In some examples, the amplifying operation is enabled by a design feature that causes the magnitude of the mask not to be constrained to the range smaller than one, and the compensating for phase differences is enabled by a design feature that causes the mask to be a complex mask.

[0050] In some examples, it may be preferred for the input signals and output masks to retain the full frequency resolution of the short-time Fourier transform employed. That is, the corresponding method is configured to operate on frequency bins rather than on bands obtained by grouping multiple bins. This feature also helps to modify the phase of the signals more precisely. In some examples, the employed DNN can be implemented using various convolutional neural networks (CNNs), such as a CNN-based encoder-decoder structure or a U-Net.

[0051] In some examples, to ensure that the anti-aliasing masks 710 output by the DNN 704 allows for a magnitude greater than one, an unbounded (or a loosely bounded) activation function may be used in the output layer. Examples of such function include a scaled and shifted sigmoid, a scaled hyperbolic tangent, an identity function, or other suitable function. For practical purposes, the anti-aliasing mask 710 may be bound between a minimum and a maximum amount of attenuation and boost by clipping its values as a post processing.

[0052] FIG. 8 is a block diagram illustrating an example generation method 800 of an antialiasing DNN according to some examples. The example generation method 800 has the following features:a) a circuit for generating N training microphone array signals 808.b) a beamformer for generating V aliased beamformer outputs obtained by applying a suitable algorithm (delay and sum, MVDR, spherical harmonics encoding, etc.) to the various training signals.c) a circuit for generating an alias-free version of the V beamformer outputs. Since these signals are not affected by aliasing, they can serve as the ground truth for computing the loss function, which is then used to drive the learning process for the DNN 704.

[0053] With respect to the circuit for generating N training microphone array signals, inputs source signals 802A-802N (e.g., Source 1 through Source S) are individually input into one of the microphone array encoders 804A-804N, respectively. Each of the microphone array encoders 804A-804N output N microphone array signals that are fed into a summing operation 806. The summing operation 806 sums together the N microphone array signals from all of the microphonearray encoders 804A-804N to generate the N training microphone array signals 808 that are fed into beamformers 810 (Nx V) and aDNN 812.

[0054] With respect to the beamformer for generating V aliased beamformer outputs, beamformers 810 (N x V) generate V aliased directional microphone signals 814 based on the N training microphone array signals 808. The V aliased directional microphone signals 814 are fed into the DNN 812.

[0055] With respect to the circuit for generating the alias-free version of the V beamformer outputs, the inputs source signals 802A-802N (e.g., Source 1 through Source S) are also individually input into one of the directional encoders 816A-816N, respectively. Each of the directional encoders 816A-816N output V microphone directional signals that are fed into a summing operation 818. The summing operation 818 sums together the V microphone directional signals from all of the directional encoders 816A-816N to generate summed V microphone directional signals that are fed into a decoder 820 that transforms the summed V microphone directional signals into L alias-free decoded signals 822. The L alias-free decoded signals 822 are used in a loss function operation 824 to generate a loss that is fed into the DNN 812.

[0056] The DNN 812 generates V anti-aliasing masks 826 based on the N training microphone array signals 808, the loss, and V aliased directional microphone signals 814. The V anti-aliasing masks 826 are applied in a filtering operation 828 to the V aliased directional microphone signals 814 to generate V alias-reduced directional microphone signals 830.

[0057] The V alias-reduced directional microphone signals 830 are fed into a second decoder 832 that transforms the V alias-reduced directional microphone signals 830 into L alias-reduced decoded microphone signals 834. The L alias-reduced decoded microphone signals 834 are fed back into the loss function operation 824. The loss function operation 824 may be performed repeatedly until the DNN 812 meets applicable stoppage criteria (e.g., a fixed number of epochs).

[0058] In some examples, although the V anti-aliasing masks 826 are described as based on the N training microphone array signals 808, the loss, and V aliased directional microphone signals 814, the V anti-aliasing masks 826 may initially be based on the N training microphone array signals 808 and V aliased directional microphone signals 814 without the loss because the loss is only availableonce an initial set of the L alias-reduced decoded microphone signals 834 is available to fed back into the loss function operation 824.

[0059] In other examples, although the V anti-aliasing masks 826 are described as based on the N training microphone array signals 808, the loss, and V aliased directional microphone signals 814, the V anti-aliasing masks 826 may initially be based on the N training microphone array signals 808 and V aliased directional microphone signals 814 in addition to a default loss because a computed loss is only available once an initial set of the L alias-reduced decoded microphone signals 834 is available to fed back into the loss function operation 824.

[0060] In some examples, the training signals mentioned in a) may be obtained by recording real sound events with a microphone array, or by simulating such events via convolution of a source signal y with position-dependent impulse responses:= y * / (ri, fy), where riand fyare the positions of the i-th microphone and the sound source respectively. For some use cases, simulations may be preferred, e.g., because the simulations may be used to generate a selectable number of examples from any desired source type, source position, and array geometry.

[0061] In some examples, the alias-free ground-truth signals mentioned in c) may be obtained using the following operations:• Choosing a DO A corresponding to fy.• For each virtual microphone, choosing a polar pattern that corresponds to the pattern of the corresponding beamformer. In various examples, the pattern can be frequency dependent (as is the case with at least some beamformers) or frequency independent.• Encoding the source into the virtual microphone signals by applying the polar patterns and the DOA information (e.g., multiplying the source signal by the value of the polar pattern at the desired DOA).• For a scene with more than one source, repeating the above operations for every source and sum the resulting signals.

[0062] In some examples, the real and imaginary parts representing complex spectrograms are stacked into the channel dimension of the input convolutional layer of the DNN 812.

[0063] In some examples, the example generation method 800 may be performed without the decoder 820 and the decoder 832. In these examples, the loss function operation 824 computes theloss from the summed V microphone directional signals from the summing operation 818 and the V alias-reduced directional microphone signals 830.

[0064] In some examples, the decoder is configured to generate a frequency-dependent linear combination of the beamformed signals. When the virtual microphones are Ambisonics components, the decoder may be an Ambisonics decoder to a pre-defined set of loudspeakers.When the virtual microphones are directional microphones pointing in several directions, the decoder may be a downmixer configured to output the corresponding 5.1 surround-sound channel set.

[0065] For at least some use cases, the example generation method 800 of FIG. 8 is beneficial, e.g., because the example generation method 800 efficiently obtains modified virtual microphones that result in alias-free decoded signals, rather than trying to obtain alias-free virtual microphones themselves. Also note that a method that results in alias-free virtual microphones would not need to account for a decoder because the decoding of alias-free signals is also alias-free. The former method is employed in the example generation method, e.g., because the direction- and frequencydependent nature of spatial aliasing makes it unlikely to be removed fully. Hence, in some cases, it is more important to favor better-decoded signals at the expense of potentially slightly degraded virtual microphone signals. An example corresponding scenario is a use case, in which the decoder does not preserve or reconstruct the entire sound field, e.g., when decoding Ambisonics to stereo. In such a use case, adding the decoder at training time may lead to better alias-reducing in stereo, e.g., because the DNN 812 may be better suited to model the stereo aliasing than to model the aliasing in the entire set of Ambisonics components.

[0066] In various examples, the decoder (e.g., the decoder 820 and / or the decoder 832) is not a trainable component. However, different types of decoders may be used on different training set items in the example generation method 800 of FIG. 8 to improve the robustness and generalization during the subsequent inference.

[0067] In some examples, the anti-aliasing masks 826 generated by the DNN 812 include a set of V x V masks Tt jif, t), where the indices i and j span the dimension of virtual microphone signals. The filtering operation 828 (e.g., an alias-reducing operation) may be represented by matrix multiplication of the set of beamformed signals 7(, t) with the set of masks T(f, t). In thismanner, the masks T(f, t) operate to perform alias-reducing on a given channel based on the signals from all available channels.

[0068] In some examples, a phase-aware loss function may be used in the example generation method 800 of FIG. 8, and the DNN 812 is trained until it meets applicable stoppage criteria (e.g., a fixed number of epochs).

[0069] The inference process is implemented using the pipeline of FIG. 7 includes forward propagation through the trained DNN 812 of the given input spectrogram.

[0070] It is worth noting that, in some examples, the DNN 812 learns the transformation that leads from the input beamformed signal to the chosen reference. In such examples, the DNN 812 may also improve aspects of beamforming that are not related to spatial aliasing. For example, some beamformers have a frequency-dependent polar pattern (e.g., a directivity that decreases with frequency below fA). When the chosen reference pattern has a frequency -independent directivity, the DNN 812 will also learn to scale the input spectrogram below fAin a frequency- and directiondependent way so that the DNN 812 may recover the frequency-independent directivity pattern corresponding to the target.

[0071] FIGS. 9-12 are diagrams illustrating the additional utility (benefit) of the DNN 704 of FIG. 7 according to some examples. FIG. 9 is a diagram illustrating an example 900 of an output of a beamformer configured to obtain the same directional response but affected by spatial aliasing according to some examples. FIG. 10 is a diagram illustrating an example 1000 of an ideal, alias-free output of a directional microphone recording a mixture of sources from different directions according to some examples. FIG. 11 is a diagram illustrating an example 1100 of the anti-aliasing mask 710 produced by the DNN 704 according to some examples. FIG. 12 is a diagram illustrating an example 1200 of a filtered output, showing reduced spatial aliasing according to some examples.

[0072] In the examples 900 and 1000, the target and aliased spectrograms are different below fAdue to having a different directivity. The anti-aliasing mask 710 acts on the full spectrogram to correct both the aliasing and directivity anomalies, because the DNN 704 has implicitly learned how to perform this type of correction during training (FIG. 8) from the chosen target signals.

[0073] In some examples, the matrix Ttj is diagonal. In such examples, each beamformed virtual microphone signal is alias-reduced independently from the others. In some examples, the decoder D can be represented by the identity matrix. Such examples correspond to removing the encoder and optimising the alias-reducing process to obtain the optimal beamformed signals rather than the optimal decoded beamformed signals. In some examples, the microphone array signals are not fed into the DNN 704.

[0074] FIG. 13 is a block diagram illustrating an example 1300 of an audio-signal processing pipeline according to some additional examples. The example 1300 of FIG. 13 implements the above-mentioned generative approach. Under the generative approach, instead of removing aliasing by modifying an aliased spectrogram, the audio-signal processing pipeline 1300 operates to remove aliasing by generating a plausible alias-free spectrogram, e.g., based on the following conditioning information:• The beamformed spectrogram (which is alias-free below fAand aliased above fA• The spectrograms of the microphone array signals.

[0075] The example 1300 includes a DNN 1308 that is designed and trained to receive the input signals 1302 and the aliased directional microphone signals 1306 (e.g., the complex input spectrogram of FIG. 9), the aliased directional microphone signals 1306 from the beamformer 1304. The DNN 1308 produces the alias-reduced directional microphone signals 1310 (i.e., modified spectrograms) wherein the effects of spatial aliasing are reduced (e.g., the reconstructed spectrogram of FIG. 12).

[0076] In some examples, the training of the audio-signal processing pipeline of FIG. 13 is supervised. The corresponding training scheme may be analogous to that shown in FIG. 8. The training input, reference, and aliased signals may be obtained as described above in reference to FIG. 8. Training variants that are with, or without, the decoder may also be implemented.

[0077] In some examples, the DNN 1308 used in the pipeline 1300 of FIG. 13 includes an encoder configured to map the inputs into the corresponding latent space, a generative diffusionbased neural -network model configured to operate in the latent space, and a decoder configured to map the latent-space output of the model back into the signal space.

[0078] In some examples, the encoder operates to encode an input signal (which can be a mel-spectrogram instead of an FFT spectrogram) into a latent domain representation. Multiple beamformed virtual microphones (aliased) as well as the microphone array spectrograms are stacked along the channel dimension and encoded into the latent domain representation.

[0079] In some examples, the generative model, such as a diffusion model based on the Transformer U-Net, acts on the latent representation, and generates a new latent representation. The generated latent representation may have a number of channels that is equal to the number of the beamforming virtual microphones, so that each generated channel in the latent space is the alias-reduced (e.g., alias-free) representation of the corresponding input virtual microphone.

[0080] In some examples, the decoder operates to decode the generated latent representation into an output mel-spectrogram or an output spectrogram. A vocoder (for example, a neural vocoder) is used to turn the output spectrogram into an audio waveform.

[0081] In various examples, supervised training of the pipeline 1300 of FIG. 1300 with alias-free targets ensures that the generative model learns to generate alias-reduced latent representations (e.g., alias-free latent representations), which are then converted into alias-reduced spectrograms (e.g., alias-free spectrograms).

[0082] In some examples, the the pipeline 1300 of FIG. 13 is trained and configured to generate only the portion of the spectrogram that corresponds to the frequencies above fA. The portion of the spectrogram that corresponds to the frequencies lower than fAis passed through and is concatenated with the generated spectrogram to construct a spectrogram representing the full range of frequencies of interest.

[0083] FIG. 14 is a block diagram illustrating an immersive voice and audio services (IV AS) coder / decoder (“codec”) framework 1400 for encoding and decoding IVAS bitstreams, according to some examples. IVAS is expected to support a range of audio service capabilities, including but not limited to mono to stereo upmixing and fully immersive audio encoding, decoding, and rendering. IVAS is also intended to be supported by a wide range of devices, endpoints, and network nodes, including but not limited to: mobile and smart phones, electronic tablets, personal computers, conference phones, conference rooms, virtual reality (VR) and augmented reality (AR) devices, home theatre devices, and other suitable devices.

[0084] The example IVAS codec 1400 includes an IVAS encoder 1401 and an IVAS decoder 1404. The IVAS encoder 1401 may be considered a first device that is located upstream from a second device (e.g., a mobile device) such as the IVAS decoder 1404. Thus, the first device may also be referred to as an upstream device or an upstream encoder device, while the second device may be referred to as a downstream device or a downstream decoder device.

[0085] The IVAS encoder 1401 includes a spatial encoder 1402 and a core audio encoder 1403. The input of the spatial encoder 1402 corresponds to a first path 1410. The spatial encoder 1402 receives input audio (e.g., input audio content) via the first path 1410. The spatial encoder 1402 processes and encodes the received input audio. In some implementations, the spatial encoder 1402 includes the DNN 704 of FIG. 7 or the DNN 1308 of FIG. 13 that may reduce spatial aliasing of the input audio.

[0086] Additionally, in some examples, the spatial encoder 1402 implements SPAR and DirAC for analy zing / downmixing N dmx spatial audio channels, as described in further detail below. The outputs of the spatial encoder 1402 correspond to a second path 1411 and a third path 1412. The spatial encoder 1402 is coupled to the core audio encoder 1403 via the second path 1411. The spatial encoder 1402 is coupled to the IVAS decoder 1404 via the third path 1412. The output of the spatial encoder 1402 includes a spatial metadata (MD) bitstream (BS) and N dmx channels of spatial downmix. The N dmx channels of spatial downmix are provided by the spatial encoder 1402 to the core audio encoder 1403 via the second path 1411. The spatial MD BS is provided by the spatial encoder 1402 to the IVAS decoder 1404 via the third path 1412. The spatial MD is quantized and entropy coded. In some implementations, quantization can include fine, moderate, coarse, and extra coarse quantization strategies and entropy coding can include Huffman or Arithmetic coding. The framework permits not more than three levels of quantization at a given operating mode; however, with decreasing bitrates, the three levels become increasingly coarser overall, to meet bitrate requirements.

[0087] The input of the core audio encoder 1403 corresponds to the second path 1411. The output of the core audio encoder 1403 corresponds to a fourth path 1413. The core audio encoder 1403 (e.g., based on mono Enhanced Voice Services (EVS) encoding unit) encodes N dmx channels (TV dmx = 1-16 channels) of the spatial downmix into an audio bitstream, which is combined (via thefourth path 1413) with the spatial MD bitstream into an IVAS encoded bitstream transmitted to IVAS decoder 1404 via the third path 1412.

[0088] The IVAS decoder 1404 includes a core audio decoder 1405 (e.g., an EVS decoder) and a spatial decoder / renderer 1406 (e.g., SPAR / DirAC). The input of the core audio decoder 1405 corresponds to a fifth path 1414. The core audio decoder 1405 receives the audio bitstream via the fifth path 1414. The core audio decoder 1405 is configured to decode the audio bitstream extracted from the IVAS bitstream to recover the N dmx audio channels. The output of the core audio decoder 1405 corresponds to a sixth path 1415. The core audio decoder 1405 is coupled to the spatial decoder / renderer 1406 via the sixth path 1415. The core audio decoder 1405 is configured to provide the N dmx audio channels (e.g., the decoded spatial downmix) to the spatial decoder / renderer 1406 via the sixth path 1415.

[0089] The inputs of the spatial decoder / renderer 1406 correspond to the third path 1412 and the sixth path 1415. The spatial decoder / renderer 1406 receives the spatial MD bitstream from the IVAS encoder 1401 via the third path 1412 and receives the decoded spatial downmix from the core audio decoder 1405 via the sixth path 1415. The spatial decoder / renderer 1406 decodes the spatial MD bitstream extracted from the IVAS bitstream to recover the spatial MD and synthesizes (e.g., renders) output audio channels using the spatial MD and a spatial upmix for playback on various audio systems with different speaker configurations and capabilities. In some implementations, when spatial aliasing is not reduced by the spatial encoder 1402, the spatial decoder / renderer 1406 includes the DNN 704 of FIG. 7 or the DNN 1308 of FIG. 13 that may reduce spatial aliasing of the decoded spatial downmix.

[0090] The output of the spatial decoder / renderer 1406 corresponds to a seventh path 1416. The spatial decoder / renderer 1406 provides the output audio (e.g., decoded audio) via the seventh path 1416.

[0091] FIG. 15A illustrates a schematic block diagram of an example device architecture 1500 (e.g., an apparatus 1500) that may be used to implement various aspects of the present disclosure. Architecture 1500 includes but is not limited to servers and client devices, systems, and methods as described in reference to FIGS. 1-14. As shown, the architecture 1500 includes central processing unit (CPU) 1501 which is capable of performing various processes in accordance with a program stored in, for example, read only memory (ROM) 1502 or a program loaded from, for example,storage unit 1508 to random access memory (RAM) 1503. The CPU 1501 may be, for example, an electronic processor 1501. In RAM 1503, the data required when CPU 1501 performs the various processes is also stored, as required. CPU 1501, ROM 1502, and RAM 1503 are connected to one another via bus 1504. Input / output interface 1505 is also connected to bus 1504.

[0092] The following components are connected to I / O interface 1505: input unit 1506, that may include a keyboard, a mouse, or the like; output unit 1507 that may include a display such as a liquid crystal display (LCD) and one or more speakers; storage unit 1508 including a hard disk, or another suitable storage device; and communication unit 1509 including a network interface card such as a network card (e.g., wired or wireless).

[0093] In some implementations, input unit 1506 includes one or more microphones in different positions (depending on the host device) enabling capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).

[0094] In some implementations, output unit 1507 include systems with various number of speakers. Output unit 1507 (depending on the capabilities of the hose device) can render audio signals in various formats (e.g., mono, stereo, immersive, binaural, and other suitable formats).

[0095] In some embodiments, communication unit 1509 is configured to communicate with other devices (e.g., via a network). Drive 1510 is also connected to I / O interface 1505, as required. Removable medium 1511, such as a magnetic disk, an optical disk, a magneto-optical disk, a flash drive or another suitable removable medium is mounted on drive 1510, so that a computer program read therefrom is installed into storage unit 1508, as required. A person skilled in the art would understand that although apparatus 1500 is described as including the above-described components, in real applications, it is possible to add, remove, and / or replace some of these components and all these modifications or alteration all fall within the scope of the present disclosure.

[0096] In accordance with example embodiments of the present disclosure, the processes described above may be implemented as computer software programs or on a computer-readable storage medium. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program including program code for performing methods. In such embodiments, thecomputer program may be downloaded and mounted from the network via the communication unit 1509, and / or installed from the removable medium 1511, as shown in FIG. 15 A.

[0097] FIG. 15B illustrates a schematic block diagram of an example CPU 1501 implemented in the device architecture 1500 of FIG. 15A that may be used to implement various aspects of the present disclosure. The CPU 1501 includes an electronic processor 1520 and a memory 1521. The electronic processor 1520 is electrically and / or communicatively connected to the memory 1521 for bidirectional communication. The memory 1521 stores an anti-aliasing DNN generation software 1522 and an immersive audio decoding software 1523. In some examples, memory 1521 may be located internal to the electronic processor 1520, such as for an internal cache memory or some other internally located ROM, RAM, or flash memory. In other examples, memory 1521 may be located external to the electronic processor 1520, such as in a ROM 1502, a RAM 1503, flash memory or a removable medium 1511, or another non-transitory computer readable medium that is contemplated for device architecture 1500. In some instances, the electronic processor 1520 may implement the anti-aliasing DNN generation software 1522 stored in the memory 1521 to perform, among other things, the example anti-aliasing DNN generation method 800 of FIG. 8. In some instances, the electronic processor 1520 may implement the immersive audio decoding software 1523 stored in the memory 1521 to perform, among other things, the example audio-signal processing pipeline 700 of FIG. 7 and / or the example audio-signal processing pipeline 1300 of FIG. 13.

[0098] Generally, various example embodiments of the present disclosure may be implemented in hardware or special purpose circuits (e.g., control circuitry), software, logic or any combination thereof. For example, the units and modules discussed above can be executed by control circuitry (e.g., CPU 1501 in combination with other components of FIG. 15A), thus, the control circuitry may be performing the actions described in this disclosure. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device (e.g., control circuitry). While various aspects of the example embodiments of the present disclosure are illustrated and described as block diagrams, flowcharts, or using some other pictorial representation, it will be appreciated that the blocks, apparatus, systems, techniques or methods described herein may be implemented in, as nonlimiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0099] Additionally, various blocks shown in the flowcharts may be viewed as method steps, and / or as operations that result from operation of computer program code, and / or as a plurality of coupled logic circuit elements constructed to carry out the associated function(s). For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program containing program codes configured to carry out the methods as described above.

[0100] In the context of the disclosure, a machine-readable medium may be any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may be non-transitory and may include but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a randomaccess memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0101] Computer program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These computer program codes may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus that has control circuitry, such that the program codes, when executed by the processor of the computer or other programmable data processing apparatus, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may execute entirely on a computer, partly on the computer, as a stand-alone software package, partly on the computer and partly on a remote computer or entirely on the remote computer or server or distributed over one or more remote computers and / or servers.

[0102] With regard to the processes, systems, methods, heuristics, etc. described herein, it should be understood that, although the steps of such processes, etc. have been described as occurring according to a certain ordered sequence, such processes could be practiced with thedescribed steps performed in an order other than the order described herein. It further should be understood that certain steps could be performed simultaneously, that other steps could be added, or that certain steps described herein could be omitted. In other words, the descriptions of processes herein are provided for the purpose of illustrating certain embodiments and should in no way be construed so as to limit the claims.

[0103] Accordingly, it is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided would be apparent upon reading the above description. The scope should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technologies discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In sum, it should be understood that the application is capable of modification and variation.

[0104] All terms used in the claims are intended to be given their broadest reasonable constructions and their ordinary meanings as understood by those knowledgeable in the technologies described herein unless an explicit indication to the contrary is made herein. In particular, use of the singular articles such as “a,” “the,” “said,” etc. should be read to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.

[0105] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in fewer than all features of a single disclosed embodiment.

[0106] The following are enumerated examples of methods, apparatuses, and non-transitory computer-readable media as described herein.

[0107] Example 1: a method of reducing spatial aliasing in a beamformed signal, the method comprising: generating an anti-aliasing spectral mask by feeding a spectrogram of the beamformed signal to a neural network trained, using a plurality of aliased beamformed signals and a corresponding plurality of aliased-reduced beamformed signals, to generate anti-aliasing spectral masks; and generating a an aliased-reduced spectrogram of the beamformed signal by applying the anti-aliasing spectral mask to the spectrogram of the beamformed signal.

[0108] Example 2: the method of Example 1, further comprising: generating the beamformed signal by applying beamforming to a set of microphone array signals; and transforming the beamformed signal into a frequency domain to generate the spectrogram of the beamformed signal.

[0109] Example 3: the method of Example 2, wherein the beamforming is implemented using a beamforming method selected from the group consisting of: delay and sum beamforming with a peak or a null in a predefined direction; minimum variance distortionless response (MVDR) beamforming with a peak or null in the predefined direction; spherical harmonics encoding; and applying a vector of complex gains to the set of microphone array signals.

[0110] Example 4: the method of any one claim of Examples 1-3, further comprising generating an alias-reduced beamformed signal by transforming the aliased-reduced spectrogram into a time domain.

[0111] Example 5: the method of any one claim of Examples 1-4, wherein the spectral mask represents a filter wherein a linear gain is a complex number whose magnitude is not constrained to be smaller than one.

[0112] Example 6: the method of any one claim of Examples 1-5, wherein the neural network has a U-Net architecture.

[0113] Example 7: the method of any one of Examples 1-6, wherein the anti-aliasing spectral mask is a combination of anti-aliasing spectral masks generated by the neural network in response to a plurality of spectrograms of a plurality of beamformed signals produced by applying a plurality of different beamforming operations to a set of microphone array signals, where each of indices and spans a full count of the plurality of beamformed signals.

[0114] Example 8: the method of Example 7, wherein the set of anti-aliasing spectral masks includes a spectral mask represented by a diagonal matrix.

[0115] Example 9: the method of Example 7 or 8, wherein the neural network is trained using a decoder configured to render the full count of the plurality of beamformed signals to a different number of rendered signals.

[0116] Example 10: a non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising any one of the methods of Examples 1-9.

[0117] Example 11: an apparatus for reducing spatial aliasing in a beamformed signal, the apparatus comprising: at least one processor; and at least one memory including program code, wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: generate an anti-aliasing spectral mask by feeding a spectrogram of the beamformed signal to a neural network trained, using a plurality of aliased beamformed signals and a corresponding plurality of aliased-reduced beamformed signals, to generate anti-aliasing spectral masks; and generate an aliased-reduced spectrogram of the beamformed signal by applying the anti-aliasing spectral mask to the spectrogram of the beamformed signal.

[0118] Example 12: a method of reducing spatial aliasing in a beamformed signal, the method comprising: generating the beamformed signal by applying beamforming to a set of microphone array signals; and generating an aliased-reduced spectrogram of the beamformed signal by feeding a spectrogram of the beamformed signal to a neural network trained, using a plurality of aliased beamformed signals and a corresponding plurality of aliased-reduced beamformed signals, to generate anti-aliasing spectral masks.

[0119] Example 13: the method of Example 12, further comprising transforming the beamformed signal into a frequency domain to generate the spectrogram of the beamformed signal.

[0120] Example 14: the method of Example 13, wherein the beamforming is implemented using a beamforming method selected from the group consisting of: delay and sum beamforming with a peak or a null in a predefined direction; minimum variance distortionless response (MVDR)beamforming with a peak or null in the predefined direction; spherical harmonics encoding; and applying a vector of complex gains to the set of microphone array signals.

[0121] Example 15: the method of any one claim of Examples 12-14, further comprising generating an alias-reduced beamformed signal by transforming the aliased-reduced spectrogram into a time domain.

[0122] Example 16: the method of any one claim of Examples 12-15, wherein the neural network comprises a diffusion model configured to perform alias-reducing using a latent domain representation of the spectrogram of the beamformed signal.

[0123] Example 17: the method of Example 16, wherein the diffusion model is implemented using a transformer U-Net architecture.

[0124] Example 18: the method of any one claim of Examples 12-17, wherein the aliased-reduced spectrogram is a combination of aliased-reduced spectrograms generated by the neural network in response to a plurality of spectrograms of a plurality of beamformed signals produced by applying a plurality of different beamforming operations to a set of microphone array signals.

[0125] Example 19: the method of Example 18, wherein the neural network is trained using a decoder configured to render the full count of the plurality of beamformed signals to a different number of rendered signals.

[0126] Example 20: the method of any one claim of Examples 12-19, wherein the aliased-reduced spectrogram excludes a spectral range below an aliasing frequency of the microphone array.

[0127] Example 21: a non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising any one of the methods of Examples 12-20.

[0128] Example 22: an apparatus for reducing spatial aliasing in a beamformed signal, the apparatus comprising: at least one processor; and at least one memory including program code, wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: generate the beamformed signal by applying beamforming to a set of microphone array signals; and generate an aliased-reduced spectrogram of thebeamformed signal by feeding a spectrogram of the beamformed signal to a neural network trained, using a plurality of aliased beamformed signals and a corresponding plurality of aliased-reduced beamformed signals, to generate anti-aliasing spectral masks.

[0129] While this disclosure includes references to illustrative embodiments, this specification is not intended to be construed in a limiting sense. Various modifications of the described embodiments, as well as other embodiments within the scope of the disclosure, which are apparent to persons skilled in the art to which the disclosure pertains are deemed to lie within the principle and scope of the disclosure, e.g., as expressed in the following claims.

[0130] Some embodiments may be implemented as circuit-based processes, including possible implementation on a single integrated circuit.

[0131] Some embodiments can be embodied in the form of methods and apparatuses for practicing those methods. Some embodiments can also be embodied in the form of program code recorded in tangible media, such as magnetic recording media, optical recording media, solid state memory, floppy diskettes, CD-ROMs, hard drives, or any other non-transitory machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the patented invention(s). Some embodiments can also be embodied in the form of program code, for example, stored in a non-transitory machine-readable storage medium including being loaded into and / or executed by a machine, wherein, when the program code is loaded into and executed by a machine, such as a computer or a processor, the machine becomes an apparatus for practicing the patented invention(s). When implemented on a general -purpose processor, the program code segments combine with the processor to provide a unique device that operates analogously to specific logic circuits.

[0132] Unless explicitly stated otherwise, each numerical value and range should be interpreted as being approximate as if the word “about” or “approximately” preceded the value or range.

[0133] The use of figure numbers and / or figure reference labels in the claims is intended to identify one or more possible embodiments of the claimed subject matter in order to facilitate the interpretation of the claims. Such use is not to be construed as necessarily limiting the scope of those claims to the embodiments shown in the corresponding figures.

[0134] Although the elements in the following method claims, if any, are recited in a particular sequence with corresponding labeling, unless the claim recitations otherwise imply a particular sequence for implementing some or all of those elements, those elements are not necessarily intended to be limited to being implemented in that particular sequence.

[0135] Reference herein to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the disclosure. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments necessarily mutually exclusive of other embodiments. The same applies to the term “implementation.”

[0136] Unless otherwise specified herein, the use of the ordinal adjectives “first,” “second,” “third,” etc., to refer to an object of a plurality of like objects merely indicates that different instances of such like objects are being referred to, and is not intended to imply that the like objects so referred-to have to be in a corresponding order or sequence, either temporally, spatially, in ranking, or in any other manner.

[0137] Unless otherwise specified herein, in addition to its plain meaning, the conjunction “if’ may also or alternatively be construed to mean “when” or “upon” or “in response to determining” or “in response to detecting,” which construal may depend on the corresponding specific context. For example, the phrase “if it is determined” or “if [a stated condition] is detected” may be construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event] ”

[0138] Also, for purposes of this description, the terms “couple,” “coupling,” “coupled,” “connect,” “connecting,” or “connected” refer to any manner known in the art or later developed in which energy is allowed to be transferred between two or more elements, and the interposition of one or more additional elements is contemplated, although not required. Conversely, the terms “directly coupled,” “directly connected,” etc., imply the absence of such additional elements.

[0139] As used herein in reference to an element and a standard, the term compatible means that the element communicates with other elements in a manner wholly or partially specified by the standard and would be recognized by other elements as sufficiently capable of communicating withthe other elements in the manner specified by the standard. The compatible element does not need to operate internally in a manner specified by the standard.

[0140] The functions of the various elements shown in the figures, including any functional blocks labeled as “processors” and / or “controllers,” may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processor” or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read only memory (ROM) for storing software, random access memory (RAM), and nonvolatile storage. Other hardware, conventional and / or custom, may also be included. Similarly, any switches shown in the figures are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.

[0141] As used in this application, the terms “circuit,” “circuitry” may refer to one or more or all of the following: (a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry); (b) combinations of hardware circuits and software, such as (as applicable): (i) a combination of analog and / or digital hardware circuit(s) with software / firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions); and (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.” This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processorintegrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0142] It should be appreciated by those of ordinary skill in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0143] “BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS” in this specification is intended to introduce some example embodiments, with additional embodiments being described in “DETAILED DESCRIPTION” and / or in reference to one or more drawings. “BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS” is not intended to identify essential elements or features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.SUPPLEMENTARY MATERIALS APPENDIXExample De-Aliasing FilteringThe decoding step within the filtering model, given by equation (1), is inspired by

[0017] and stems from the fact that some decoding strategies might lead to a loss of spatial information in the decoded signals compared to the set of virtual microphones; for example, decoding Ambisonic to stereo typically discards the vertical component. In these cases, it might be In this work, we consider de-aliasing of pre-beamformed sigadvantageous to put emphasis on de-aliasing of the parts of the nals, which we refer to as the virtual microphone signals. The sound field that dominate the decoded signals, while reducing following equation presents the adopted filtering model: the attention paid to the less significant components. In other words, the filter is generally optimized for an alias-free decod¬ d = D Fp v / t, (1) ing instead of alias-free virtual microphones, although the latter where vyte CVx lis a virtual microphone vector of Short can also be achieved with our approach.Time Fourier Transform (STFT) coefficients, f is a frequency Note that the de-aliasing filter acts on the virtual microindex, t is a frame index and V is the number of virtual microphones, rather than directly on the decoded signals; by using a phones. F G CV x Vis the predicted de-aliasing matrix filter, multichannel matrix formulation where the virtual microphones D e KD xVis a decoder matrix and dp 6 CD x 1is a vector of can interact, the system is given the added capacity of leveragSTFT coefficients for the D decoded signals. ing a plurality of virtual microphones to de-alias each individual Linear spatial audio decoding, represented by the decodermatrix D in equation (1), is a processing step that converts virtual microphone, effectively performing an implicit intermevirtual microphone signals, that describe the general decomdiate beamforming step. However, if intermediate beamforming posed sound field, into a specific reproduction setup. An examis to be prevented, the filter can be reduced to a diagonal maple would be mid-side

[0016] or Ambisonic decoding [3]. When trix. In the remaining parts, we refer to the former filter setting the virtual microphone signals correspond directly to the deas full and to the latter as diag. The general formulation of sired decoded signals, e.g. for two left- and right-facing carequation (1) provides a flexible way to encapsulate a variety of dioid beamformers to be rendered in stereo, the decoding is not possible scenarios in one model; as an example, if the goal is necessary and D reduces to an identity matrix Ix V. In this to produce alias-free non-decoded Ambisonics, the training can paper, we consider only frequency-independent decoding, albe augmented with different decoders and decoding positions, though equation (1) could easily generalize by making the depotentially increasing the system robustness and indirectly encoding matrix frequency-dependent. abling de-aliasing on the full sphere.U-Net-Based Filter PredictionFor supervised learning, we define the target alias-free beamWe propose to predict the de-aliasing filter matrix F based former signals using the following equation:on the virtual microphone signals v / t, leveraging supervisedtraining of a deep neural network, similarly to

[0013] , Since the t / t = E Sft, (2) virtual microphone signals are characterized by different spatial responses or looking directions, as in the case of different where t / t6 CD x lis a vector of alias-free decoded STFT Ambisonic components, spatial aliasing will affect each virtual coefficients, E e RD x Sis an alias- free encoder matrix, and microphone in a different way. This useful diversity and redunSft 6 CSx lis an STFT vector for S source signals. dancy makes virtual microphone signals a suitable regressor for In this work, we define a frequency-independent alias-free the proposed filter-predicting model. encoder matrix E as the first-order directional response[E = p + (l - p) iIj.,, (3)where p is a shape coefficient and ij, jsare directional unit vecThe PHASEN

[0020] loss between the de-aliased d and the tors in Cartesian coordinates. The directional vector id points alias-free t / tdecoded signals is optimized by minimizing its towards the d-th decoding direction, realized by the d-th row ofmean value across the decoding directions. The model is trained the decoder matrix D, while jspoints towards the s-th soundsource, p is a coefficient between 0 and 1 that defines the shape for 100 epochs using the Adam optimizer with p = 0.9, 2 = of the spatial response, such that 0 results in a figure-of-eight 0.999, a batch size of 24 and a learning rate of 0.001, which is pattern, 0.5 produces a cardioid and 1 corresponds to an omnihalved if the batch loss does not decrease for two epochs. directional characteristics. The application of the alias-free enThe training is based on simulated microphone array sigcoder matrix E to the source signal vector Sftcan be illustrated nals obtained by applying delays to source signals, based on using Figure 1, where the intersection of the dashed source anechoic far field propagation with respect to the center of the array. We simulate a 4-channel omnidirectional microphone arDoA lines with the target spatial directivity determines the valray, with the sensors arranged on a cross-like plan, where one ues for the weighted summation. Note that E does not model pair is aligned with the x-axis and the second pair is aligned the direction-dependent phase shifts potentially introduced by with the y-axis. In the experimental setups with a fixed aliasing a beamformer. Therefore, the complex-valued filter F / twill frequency, the microphone spacing d for each pair is equal to also be tasked with relative phase restoration, approximating a 3cm, while in the case of varying aliasing d is chosen randomly spatially uniform phase response of the de-aliased signals. between 1 and 10cm, independently for each pair. Microphone mixture signals are obtained by choosing between one and four Following state-of-the-art neural processing of audio

[0018] , the sources, simulating the capsule signals with the sources ranU-Net

[0019] , an encoder-decoder architecture based on 2D condomly placed around the array and summing their contributions volutional filters, is employed as the filter-predicting model. for each sensor. The signals are then transformed by an STFT The only difference between the adopted model and the origwith 2048-point Hann window and 1024 overlapping samples. inal U-Net is the final activation function, which in our case isan identity. This choice was dictated by the required complexAs source signals we use 5s excerpts of speech, music, valued filter properties, as described in Section 3. noise and environmental signals, sampled at 44.1kHz. The Similarly to

[0013] , real and imaginary parts of the virtual speech samples are randomly drawn from

[0021] and additional microphone signals are used as an input to the model, repre300 hours of speech recordings. The music signals are ransented by real- valued features unfolded across the channel didomly drawn from a set of 4000 songs, while the noise and mension. Based on this representation, the model predicts a set environmental recordings are randomly drawn from [22, 23]. of time-frequency masks Mcft€ R, where c is the channel index, which are then reshaped to create the filter matrix F ft. Asdiscussed in Section 3, the filter F ftcan be either diagonal, withoff-diagonal terms zeroed out, or it can be a full matrix filter thatincludes the cross-terms. Depending on the filter configuration,the number of output channels C is either equal to 2 V or 2 V2,respectively, where the factor 2 accounts for the aforementionedreal- valued representation of the complex-valued masks.Experimental EvaluationWe investigate two experimental setups: i) two opposite-facingExperiment ii) addresses the de-aliasing of horizontal FOA, cardioids and ii) 2D First-Order Ambisonics, aka. planar, horizontal or circular FOA [3]. Additionally, both experiments are obtained from the aforementioned array of four microphones. also subject to two variants, one with fixed and one with varying Similarly to experiment i), pressure gradient is calculated along microphone spacing, which we refer to as fix and var. the x- and y-axis pairs to obtain X and Y figure-of-eight signals Experiment i) considers the case of two opposite-facing respectively, while the omnidirectional component W is taken as an average of the four microphone signals. In this expericardioid beams, a common setup for obtaining stereo signals ment, D is an in-phase decoder [3], which produces four carfrom a pair of closely spaced omnidirectional microphones [2], dioids along the front, back, left and right directions, such that Given raw mixture recordings, the virtual microphone signals V = 3 and D = 4.are obtained by calculating pressure gradient for the x-axis pair In both experiments, the raw mixtures are encoded to virtual of the 4-channel microphone array described in Section 4.2, microphone signals using conventional beamforming, based which results in quasi-frequency-independent spatial characteron calculation of the pressure gradient [2]. We have decided istics. In this experimental setup, we use an identity decoder, to use this type of beamformer to ensure a quasi-frequency-thus the spatial responses of the virtual microphone signals are independent spatial response, while generalization is left as a explicitly optimized, as discussed in Section 3. To obtain suit¬ future research possibility. We report the results of both objecable alias-free target signals, we follow the procedure described tive and subjective evaluation, described in the following secin Section 4, with p = 0.5 in equation (3), while V = D = 2. tions. Additionally, we provide audio samples'.The objective evaluation is based on the Complex Scale-Invariant Signal-to-Noise-Ratio (C-Si-SNR)

[0024] between the Table 1: C-Si-SNR values in dB for the experiments with i) carde-aliased dytand the target decoded alias-free tftsignals, dioid pair and ii) 2D Ambisonics. The numbers in parentheses averaged across the D decoding directions. Additionally, we denote improvement over the aliased conditions. heuristically calculate spatial responses for the trained modelby simulating one source at a time for a grid of directions, prei)fix i) var iiffix ii) var dicting and applying the de-aliasing filter, and calculating the 30.3±2.5 15.4±2.5 29.3-2.6 19.7±2.2 magnitude of the de-aliased signal. The procedure is repeated diagwith 100 different source signals for each grid direction, the (19.5) (7.6) (11.3) (7.8) magnitudes are aggregated and the results are normalized. 27.3±2.6 28.9±2.6 19.5±2.2 full 20.2±2.5Table 1 reports the C-Si-SNR values in dB for experiments(16.6) (12.4) (10.2) (7-6) i) and ii) with fixed and varying microphone spacing. Both experiments show an average C-Si-SNR close to 30dB for fix andbetween 15dB and 20dB for var, which is consistent with the expected increasing difficulty of determining a varying spatial aliasing frequency. In case of the 2D FOA experiment ii), where cross-microphone information is already utilized in the decoding step, the differences between diag and full seem negligible. Regarding the cardioid pair experiment i), the full filter shows an advantage over diag for var conditions, while for fix the difference is less pronounced. We hypothesize that this improvement could be explained by the re-introduction of the crosschannel interaction, which is not present in case of i) var diag, neither in filter or in decoder.Figures 2 and 3 present polar plots of spatial responses, incase of the aliased and de-aliased virtual microphones, for exFigure 2: Spatial responses of the left- and right-facing carperiments i) fix and ii) fix, respectively. The patterns are plotted dioids of experiment i) fix, respectively in the top and bottom in four frequency bands, the first of which is below / . Right- row, in four frequency bands.and rear-facing cardioids of experiment ii) are symmetrical totheir counterparts and were omitted for the sake of space. Inboth experimental setups, the filters are effective in restoring the desirable spatial responses. Even in the upper band, where the polar patterns are not accurately restored, the filters reduce the spurious components at the back and sides. To validate the perceptual impact of the proposed approach, we have designed two listening tests using the MUSHRA methodology

[0025] and the webMUSHRA interface

[0026] , The first test studies the impact on the timbre of a single sound source witha random azimuth, decoded to mono using cardioid beamforming. Subjects are presented with the reference alias-free signal,the aliased cardioid, an anchor low-passed 3.5kHz, and the de- Figure 3: Spatial responses of the decoded front- and left-facing aliased versions trained with the diag or full approaches, under cardioids of experiment ii) fix, respectively in the top and botthe varying distance. In the second test we study the ability to tom row, in four frequency bands.recover the correct stereo localization by applying the networksto a pair of left- and right-facing cardioids; here, a single soundsource is positioned at a random azimuth between ±90°, encoded into the aliased and alias-free cardioids, and processed by the same networks; the anchor is a version where the source has been hard-panned to the left or right channel, whichever is more distant from the random source position. Each test comprised nine excerpts of music, speech, or noise, evaluated by 17 expert listeners. Average ratings for all test excerpts were calculated per listener, with results presented in Figure 4. Each point identifies the average rating for all test excerpts for one listener.(a) Timbre test (b) Localisation test Figure 4: MUSHRA scores across different conditions.A repeated measures ANOVA for the timbre test revealed a significant effect of rating stimuli (p < 0.001). Post-hoc tests2showed the aliased version scored 32 MUSHR points below reference on average (p < 0.001). The diag approach performed similarly to the reference (0.6 points below, p = 0.4, not significant), while the full approach scored significantly lower (9 points below, p < 0.001). Notably, a per-excerpt analysis revealed that the full method’s difference was primarily due to a single broadband music item, scoring 53 MUSHRA points below reference (p < 0.001), while other excerpts showed no significant differences to the reference (5 points at most, p > 0.05).An analogous analysis for the localisation test revealed a significant effect of rating stimuli (p < 0.001). Post-hoc tests indicated the aliased version scored significantly lower than the reference (46 MUSHRA points below, p < 0.001). Both the ”diag” and ’’full” approaches performed similarly to the reference, with no significant differences with the reference (3 points at most, p > 0.05).These results demonstrate the effectiveness of the presented methods, in most cases rendering the timbre and localization indistinguishable from the reference. The timbre test results align with the C-Si-SNR improvements, while the localization test confirms the successful restoration of polar patterns. These patterns match the desired inter-channel level differences across most frequency bands, with only the highest bands showing deviations, which are perceptually less significant. This comprehensive evaluation underscores the ability to effectively mitigate both the timbral and spatial spatial aliasing artifacts.

Claims

CLAIMSWhat is claimed is:

1. A method of reducing spatial aliasing in a beamformed signal, the method comprising:generating an anti-aliasing spectral mask by feeding a spectrogram of the beamformed signal to a neural network trained, using a plurality of aliased beamformed signals and a corresponding plurality of aliased-reduced beamformed signals, to generate anti-aliasing spectral masks; and generating a an aliased-reduced spectrogram of the beamformed signal by applying the antialiasing spectral mask to the spectrogram of the beamformed signal.

2. The method of claim 1, further comprising:generating the beamformed signal by applying beamforming to a set of microphone array signals; andtransforming the beamformed signal into a frequency domain to generate the spectrogram of the beamformed signal.

3. The method of claim 2, wherein the beamforming is implemented using a beamforming method selected from the group consisting of:delay and sum beamforming with a peak or a null in a predefined direction;minimum variance distortionless response (MVDR) beamforming with a peak or null in the predefined direction;spherical harmonics encoding; andapplying a vector of complex gains to the set of microphone array signals.

4. The method of any one claim of claims 1-3, further comprising generating an alias-reduced beamformed signal by transforming the aliased-reduced spectrogram into a time domain.

5. The method of any one claim of claims 1-4, wherein the spectral mask represents a filter wherein a linear gain is a complex number whose magnitude is not constrained to be smaller than one.

6. The method of any one claim of claims 1-5, wherein the neural network has a U-Net architecture.

7. The method of any one of claims 1-6, wherein the anti-aliasing spectral mask is a combination of anti-aliasing spectral masks Ttjif, t) generated by the neural network in response to a plurality of spectrograms of a plurality of beamformed signals produced by applying a plurality of different beamforming operations to a set of microphone array signals, where each of indices i and j spans a full count of the plurality of beamformed signals.

8. The method of claim 7, wherein the set of anti-aliasing spectral masks Ttjif, t) includes spectral masks represented by a diagonal matrix.

9. The method of claim 7 or 8, wherein the neural network is trained using a decoder configured to render the full count of the plurality of beamformed signals to a different number of rendered signals.

10. A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising any one of the methods of claims 1-9.

11. An apparatus for reducing spatial aliasing in a beamformed signal, the apparatus comprising:at least one processor; andat least one memory including program code,wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to:generate an anti-aliasing spectral mask by feeding a spectrogram of the beamformed signal to a neural network trained, using a plurality of aliased beamformed signals and a corresponding plurality of aliased-reduced beamformed signals, to generate anti-aliasing spectral masks; and generate an aliased-reduced spectrogram of the beamformed signal by applying the antialiasing spectral mask to the spectrogram of the beamformed signal.

12. A method of reducing spatial aliasing in a beamformed signal, the method comprising:generating the beamformed signal by applying beamforming to a set of microphone array signals; andgenerating an aliased-reduced spectrogram of the beamformed signal by feeding a spectrogram of the beamformed signal to a neural network trained, using a plurality of aliased beamformed signals and a corresponding plurality of aliased-reduced beamformed signals, to generate anti-aliasing spectral masks.

13. The method of claim 12, further comprising transforming the beamformed signal into a frequency domain to generate the spectrogram of the beamformed signal.

14. The method of claim 13, wherein the beamforming is implemented using a beamforming method selected from the group consisting of:delay and sum beamforming with a peak or a null in a predefined direction;minimum variance distortionless response (MVDR) beamforming with a peak or null in the predefined direction;spherical harmonics encoding; andapplying a vector of complex gains to the set of microphone array signals.

15. The method of any one claim of claims 12-14, further comprising generating an alias-reduced beamformed signal by transforming the aliased-reduced spectrogram into a time domain.

16. The method of any one claim of claims 12-15, wherein the neural network comprises a diffusion model configured to perform alias-reducing using a latent domain representation of the spectrogram of the beamformed signal.

17. The method of claim 16, wherein the diffusion model is implemented using a transformer U-Net architecture.

18. The method of any one claim of claims 12-17, wherein the aliased-reduced spectrogram is a combination of aliased-reduced spectrograms generated by the neural network in response to a plurality of spectrograms of a plurality of beamformed signals produced by applying a plurality of different beamforming operations to a set of microphone array signals.

19. The method of claim 18, wherein the neural network is trained using a decoder configured to render the full count of the plurality of beamformed signals to a different number of rendered signals.

20. The method of any one claim of claims 12-19, wherein the aliased-reduced spectrogram excludes a spectral range below an aliasing frequency of the microphone array.

21. A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising any one of the methods of claims 12-20.

22. An apparatus for reducing spatial aliasing in a beamformed signal, the apparatus comprising:at least one processor; andat least one memory including program code,wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to:generate the beamformed signal by applying beamforming to a set of microphone array signals; andgenerate an aliased-reduced spectrogram of the beamformed signal by feeding a spectrogram of the beamformed signal to a neural network trained, using a plurality of aliased beamformed signals and a corresponding plurality of aliased-reduced beamformed signals, to generate antialiasing spectral masks.

Images