Dynamic neural network computing

A dynamically scalable neural network with early exit points addresses battery life and latency issues in hearing aids by optimizing processing power usage, ensuring efficient and effective signal denoising.

WO2026238484A1PCT designated stage Publication Date: 2026-11-19WS AUDIOLOGY AS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/EP2026/062889
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-11
Filing Date
2026-05-05
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Implementing neural networks in hearing aids is challenging due to battery life constraints and latency issues, especially with computationally complex deep neural networks, which consume excessive power and time.

Method used

A dynamically scalable neural network with early exit points allows for variable processing power usage, enabling efficient denoising of signals by prematurely terminating computation when sufficient denoising is achieved, using environmental context and predicted denoising levels to select the appropriate exit point.

Benefits of technology

This approach conserves battery life and reduces latency while maintaining effective signal denoising quality, suitable for hearing aids and other portable devices with limited resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2026062889_19112026_PF_FP_ABST
    Figure EP2026062889_19112026_PF_FP_ABST
Patent Text Reader

Abstract

A method (100) for denoising a signal is provided. The method (100) comprises steps of: receiving (S110) an input signal; inputting (S120) the input signal into a neural network trained to output a denoised signal; and outputting (S130) the output of the neural network as a denoised signal. The neural network is dynamically scalable and comprises: at least two sequential blocks of nodes trained to process signals into denoised signals; and at least one early exit point arranged between said at least two sequential blocks configured to be used to output a denoised signal after using only a subset of said at least two sequential blocks. A hearing aid (10) comprising a processing circuit implementing the method (100) is further provided.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] DYNAMIC NEURAL NETWORK COMPUTING

[0002] Technical Field

[0003] The present inventive concept relates to neural network computing. In particular, it relates to a method for using a dynamic neural network for denoising a signal.

[0004]

[0005] The field of neural networking is fast progressing and there are constantly new innovations for using neural networking to process audio signals.

[0006] Modern hearing aids are using more and more computationally complex signal processing to provide the best possible sound quality and perception fortheir users. These hearing aids may benefit from advances in neural networking, but it has proven a challenge to implement a neural network in a hearing aid.

[0007] Hearing aids are by design very small and run on batteries. Even if it would be possible to implement a neural network on a chip in a hearing aid, it would likely drain the battery so fast that the hearing aid would not be usable the whole day, causing frustrations for the user.

[0008] Additionally, latency is an important factor for hearing aid users to be able to keep up in normal conversation. Hence, long computational times associated with some neural networks such as deep neural networks are especially unsuited for hearing aid implementation.

[0009] It is thereby an object of the present inventive concept to implement a neural network that uses a variable or minimal amount of processing power. This and other objects are achieved by the features set out in the appended independent claims, with embodiments set out in the dependent claims.

[0010] Accordingly, a first aspect of the inventive concept is provided by a method for denoising a signal. The method comprises steps of: receiving an input signal; inputting the input signal into a neural network trained to output a denoised signal; and outputting the output of the neural network as a denoised signal. The neural network is dynamically scalable and comprises: at least two sequential blocks of

[0011]

[0012] nodes trained to process signals into denoised signals; and at least one early exit point arranged between said at least two sequential blocks configured to be used to output a denoised signal after using only a subset of said at least two sequential blocks.

[0013] By having an early exit point, the computation may end prematurely without using the full neural network. This saves computer processing power, battery life, and time compared to using the whole neural network.

[0014] The early exit points are part of the network during training. The neural network is trained to be able to use any of the early exit points or the full network and still provide a sufficient denoising of the input signal.

[0015] By using a later early exit point or the full network, a better denoising may be expected than using an earlier early exit point, as more of the neural network has been used to process the signal.

[0016] However, the network may also be trained to prioritize a specific, most likely to be used early exit point, such that the difference between the prioritized early exit point and the next early exit point is relatively minor.

[0017] The signal may be an audio signal.

[0018] While the early exit point concept is applicable to any sort of signal processing, it is especially suitable for audio signals as denoising is an important process for audio and audio devices are often portable with less available processing power and limited battery life.

[0019] The neural network may be a deep neural network.

[0020] As the early exit point(s) allow for early exit should the whole network not be needed, the neural network may benefit from being a deep neural network with all the processing power that entails should it be needed, while still being able to use less processing power if it is not needed.

[0021] The method may further comprise choosing one of said at least one early exit point or the full neural network to be used when inputting the input signal based on environmental context.

[0022] The environmental context may be that the sound has very little or a lot of relative noise preferring an earlier or later early exit point respectively, that the signal comprises a conversation where low latency is desirable preferring an earlier early exit point, or that a battery is getting low so processing power should be saved preferring an earlier early exit point. The environmental context may be automatically

[0023]

[0024] determined based on previous processing, contents of the input signal, or other data relating to a location or environment of a user. Additionally or alternatively the environmental context may be set by the user, possibly as a result of a prompt or menu on a smart device.

[0025] The method may further comprise dynamically choosing one of said at least one early exit point or the full neural network to be used while the neural network is processing the input signal based on a predicted denoising level reaching a predetermined threshold.

[0026] While a specific early exit point may be preferred or defaulted to, the used early exit point may be determined during processing of the signal to ensure a sufficient minimum quality of the output.

[0027] The network may generate a scalar metric that represents a quality improvement achieved (e.g. in terms of mixture SNR or denoising improvement) that is compared to a predetermined threshold. As soon as the scalar metric reached above the predetermined threshold, the next early exit point is used.

[0028] Additionally or alternatively, the network may model a distribution over a quality metric (e.g. in terms of mixture SNR or denoising improvement) to predict a probability that the output meets a target improvement level. An early exit point that has a high enough confidence in that the output achieves or exceeds the predetermined threshold is used.

[0029] A first and second sequential early exit points of said at least one early exit point may be differentiated by the second early exit point being configured to be used to output a denoised signal after reusing at least one block of said at least two sequential blocks of nodes more than the first early exit point.

[0030] This provides a methodology for implementing a neural network with infinite scalability as a block designed and trained to be iteratively applied for iterative improvement or refinement may be reused as needed. This enables the method to use very little memory, which may be especially important in small devices such as hearing aids.

[0031] A primary and secondary sequential early exit points of said at least one early exit point may be differentiated by the secondary early exit point being configured to be used to output a denoised signal after using one more block of said at least two sequential blocks of nodes more than the primary early exit point.

[0032] In this model an early exit point may be placed between each block of nodes.

[0033]

[0034] This has the benefit of being predictable, and each block of nodes may be trained for a specific purpose or desired effect, such that an early exit points may be chosen based on the minimum set of effects desired.

[0035] An alpha and beta sequential early exit points of said at least one early exit point may be differentiated by the beta early exit point being configured to be used to output a denoised signal after using more nodes of a block of said at least two sequential blocks of nodes more than the alpha early exit point.

[0036] The network may scale by width instead of or as well as scaling in depth, hence using a larger subset of the neural network may comprise using more nodes of the same block.

[0037] This kind of network is relatively easy to train, and the early exit points mitigate a conventional downside of this type of scaling, i.e. its inflexibility and power consumption.

[0038] The neural network may be configured to encode the input signal before processing the signal using said at least two sequential blocks of nodes and to decode the signal before outputting a denoised signal, wherein two sequential early exit points of said at least one early exit point share a decoder.

[0039] Encoding and decoding is common practice in neural networks. By sharing a decoder that is functionally identical, additional memory space may be saved.

[0040] Some or every early exit point may have their own individual decoder or share a decoder with one or more other early exit points and / or the end of the neural network.

[0041] The method may further comprise steps of: separating the input signal into at least two separated input signals and inputting said at least two separated input signals into the neural network; and combining the outputs of the neural network from said at least two separated input signals into a speech enhanced signal and outputting the speech enhanced signal as a denoised signal.

[0042] Separating the input into multiple inputs may improve denoising and may be used for speaker separation and speech enhancement.

[0043] A second aspect of the inventive concept is provided by a hearing aid. The hearing aid comprises: a microphone configured to receive sound; a processing circuit configured to convert the received sound into a hearing loss compensated signal; and a receiver configured to receive the hearing loss compensated signal and output said signal as audio.

[0044]

[0045] The processing circuit is configured to: input the received sound from the microphone as an input signal into a neural network trained to output a denoised signal; and output to the receiver the output of the neural network as a hearing loss compensated signal.

[0046] The neural network is dynamically scalable and comprises: at least two sequential blocks of nodes trained to process signals into denoised signals; and at least one early exit point arranged between said at least two sequential blocks configured to be used to output a denoised signal after using only a subset of said at least two sequential blocks.

[0047] All the benefits and alternatives of the first aspect apply to the second aspect mutatis mutandis.

[0048] The early exit points are especially useful in a hearing aid, as hearing aids are small and battery-powered while still benefiting from sophisticated denoising. A hearing impaired user may be especially sensitive to noisy audio, and the noise may be especially disruptive when amplified.

[0049] Alternatively or additionally, the processing of the received sound using the neural network may be done in a separate microphone connected to the hearing aid. The separate microphone is configured to input the received sound from the microphone as an input signal into a neural network trained to output a denoised signal; and output to the hearing aid the output of the neural network as a hearing loss compensated signal, e.g. as an audio stream.

[0050] The processing circuit may be further configured to process the denoised signal output from the neural network using signal processing before outputting the processed signal to the receiver as a hearing loss compensated signal.

[0051] Such signal processing may e.g. be beamforming, binaural sound, amplification or gain, processing the audio to sound more natural by e.g. adding back some noise, or any other signal processing known to a skilled person to be used in hearing aids.

[0052] The processing circuit may be further configured to input the received sound from the microphone as several input signals into the neural network; combine the several output signals of the neural network into a speech enhanced signal; and output to the receiver the speech enhanced signal as a hearing loss compensated signal.

[0053]

[0054] Separating the input into multiple inputs may improve denoising and may be used for speaker separation and speech enhancement.

[0055] The several input signals may be signals received from multiple microphones of the hearing aid and / or another microphone connected to the hearing aid. The several input signals may be one signal received from one microphone and split into several signals, e.g. into different frequency bands.

[0056] The processing circuit may be further configured to choose one of said at least one early exit point or the full neural network to be used when inputting the received sound from the microphone as an input signal into the neural network based on an environmental context of the hearing aid.

[0057] The hearing aid may have one or more parameters relating to environmental context, such as location, ambient volume or user input. The hearing aid may further have access to additional sensors to indicate remaining battery life, a type of location the user is in, the content of the received audio, and / or history of the user. Some of all this data may influence the environmental context of the hearing aid, and thereby also the early exit point chosen.

[0058] Additionally, the user may set a desired level of denoising from a menu that influences the chosen early exit point. This may influence a predetermined threshold for dynamically choosing an early exit point, and / or this may influence a default early exit point. This choice may be based on an expected battery life of the hearing aid, and this choice may be automatically updated or prompted to be updated if the expected battery life changes or the battery reaches a predetermined charge level.

[0059]

[0060] The invention will be described in further detail with reference to preferred aspects and the accompanying drawing, in which:

[0061] Fig. 1 shows a schematic view of a hearing aid according to an embodiment;

[0062] Fig. 2 shows a flowchart representation of a neural network according to an embodiment;

[0063] Fig. 3 shows a flowchart representation of training a neural network according to an embodiment; and

[0064] Fig. 4 shows a flowchart of a method for denoising a signal according to an embodiment.

[0065]

[0066] Description of embodiments

[0067] In the following, terms such as a / an / the and comprising are intended to be interpreted as non-limiting. Any processor or computing unit may be implemented as an electronic circuit and electronic connections may be wired or wireless unless otherwise explicitly stated. Unless explicitly specified, a wireless connection may be implemented in any number of standards known to a person skilled in the art, such as Wi-Fi, Bluetooth®, Zigbee, Matter, 4G / LTE, 5G and so on.

[0068] Fig. 1 shows an embodiment of a hearing aid 10. The hearing aid 10 comprises a microphone 14, a processing circuit 16, and a receiver 18.

[0069] The microphone 14 is configured to receive sound, e.g. ambient sound and speech. There may be more than one microphone 14 on the hearing aid 10 and a pair of hearing aids 10 may communicate such that both hearing aids 10 receive sound from microphone(s) of both hearing aids 10.

[0070] The processing circuit 16 is configured to convert the received sound, either from the microphone(s) 14 or a separate source, into a hearing loss compensated signal.

[0071] The receiver 18 is configured to receive the hearing loss compensated signal and output said signal as audio for a user. The user may suffer from hearing loss and may therefore benefit from a compensated signal e.g. boosting volume, removing noise, shifting frequencies, and / or separating speech sources. The received 18 may receive this signal from the processing circuit 16 and / or a separate source, such as a standalone microphone connected to the hearing aid 10.

[0072] The processing circuit 16 is configured to input the received sound as an input signal into a neural network trained to output a denoised signal (to be disclosed in more detail in relation to Figs. 2-3 and the appendix); and output (to the receiver) the output of the neural network as a hearing loss compensated signal.

[0073] The processing circuit 16 may further be configured to process the denoised signal output from the neural network using signal processing before outputting the processed signal as a hearing loss compensated signal.

[0074] This may comprise applying source separation, loss compensation, and / or filtering algorithms to the denoised signal.

[0075] The processing circuit 16 may further be configured to input the received sound as several input signals into the neural network; combine the several output

[0076]

[0077] signals of the neural network into a speech enhanced signal; and output the speech enhanced signal as a hearing loss compensated signal.

[0078] This may comprise beamforming or transforming (e.g. Fourier transforming) the received sound before inputting it to the neural network as several input signals. The different input signals may have different sources, frequency ranges and / or directions.

[0079] The hearing aid 10 shown in Fig. 1 is a generic representation of a hearing aid. The inventive concepts herein apply to many different types of hearing aids, such a behind-the-ear (BTE) type, receiver-in-canal (RIC), and completely-in-canal (CIC) type hearing aids.

[0080] Fig. 2 shows an example of a neural network according to the invention.

[0081] The neural network is dynamically scalable and may be a recurring neural network, a deep neural network, a convolutional neural network, or any other suitable neural network.

[0082] The neural network comprises: at least two sequential blocks of nodes (four are shown in the example of Fig. 2) trained to process signals into denoised signals; and at least one early exit point (three are shown in the example of Fig. 2) arranged between said at least two sequential blocks configured to be used to output a denoised signal after using only a subset of said at least two sequential blocks.

[0083] The blocks of nodes comprise conventional neural network nodes that have a weight or gain value and apply a simple modification to the signal. The blocks may not be physically or computationally represented when the network is implemented, and may merely represent a way to structure and conceptualize the network for the purposes of this description.

[0084] This neural network structure is very versatile and works for any type of signal, processing, and implementation. The examples in this document are focused on processing audio for hearing aids such as the hearing aid 10 in Fig. 1, however many other applications are possible, such as processing video for a camera surveillance system or processing health data fora remote health monitoring system.

[0085] The neural network is designed to enable early exit, thereby only utilising a portion of the network’s processing power but also only a portion of its resource expenditure in terms of time and energy. In real-time applications especially, a maximum output of the neural network is not needed depending on the signal to be

[0086]

[0087] processed, and may further not be beneficial due to constraints in terms of both latency and energy consumption.

[0088] Early exit points thereby allow the neural network to scale dynamically depending on how much processing power is needed and / or available.

[0089] The end of the neural network may be similar to an early exit point, but since it by definition is not early it may be considered an exit point.

[0090] The scaling the early exit points provide depends on how the neural network is structured. In Fig. 2, every sequential early exit point is differentiated by adding a block of nodes.

[0091] By running the signal through an additional block of nodes, the network applies more processing and the resulting denoised signal is improved. However, by stopping early the processing may be good enough for the implementation and signal being processed, so time and energy may thereby be saved.

[0092] An alternative scaling that may be used alternatively or in addition to the scaling shown in Fig. 2, is scaling the width of the block used.

[0093] In traditional neural network architecture the width of the network represents how many nodes are utilized simultaneously before feeding the signal through to the next layer, whereas the depth represents the number of layers. In traditional neural network architecture the width of the network is fixed and in order to add computational power the depth is increased. Since width is fixed, it is usually kept relatively thin to allow for implementation on lower grade hardware and to give more granularity and flexibility to scaling in depth.

[0094] However, with early exit points the width is more easily scalable and may be variable as well as or instead of the depth. One early exit point may represent utilizing the full width of the available network and another early exit point may represent using only a portion of the full width. One early exit point may use one portion of the full width and another early exit point may use an entirely different portion of the full width. The two early exit points may overlap in the nodes they use when defining their widths or they may not, their ranges may or may not be mutually exclusive.

[0095] Two sequential early exit points may thereby utilise more nodes of a block than each other, and / or they may use different nodes and / or different blocks.

[0096] This adds a degree of freedom to scalability that enables much more efficient and custom early exit points that achieve results approaching what a full deep neural

[0097]

[0098] network may achieve using only a fraction of computational power. This is achieved by only using the most relevant or efficient path through the neural network depending on the signal received or processing that is requested.

[0099] Techniques such as mixture-of-experts style projections may be used where only a subset of nodes / pathways (experts) are used under given circumstances.

[0100] Tensor-train representations may be used to decompose weight matrices, effectively enabling the network to operate with variable width.

[0101] Another way to scale the neural network that is not shown in Fig. 2 is to use iterative processing. A block of nodes may be designed to continue to provide iterative improvement by reusing it, i.e. running the same signal through the block multiple times. By having an early exit point after different amounts of iterations, a different amount of processing is available while maintaining a low cost in terms of infrastructure and memory space as additional unique parameters are not introduced.

[0102] Heuristics or data-driven criteria may be used to determine a number of repetitions used.

[0103] This may be especially important in implementations with high space and memory constraints, such as in hearing aids such as the one in Fig. 1.

[0104] The neural network of Fig. 2 further comprises an encoder / decoder set. The encoder is located at the start of the neural network and all nodes process the encoded signal. Decoders are located at the end of the neural network as well as at each early exit point.

[0105] Each individual early exit point in Fig. 2 has a decoder, however in other embodiments decoders may be shared between early exit points and / or the end of the network.

[0106] Each early exit point in Fig. 2 comprises some finishing touches and a decoder. These features may be shared or individual for each early exit point. Early exit points may further be simpler than the ones shown in Fig. 2, and e.g. only comprise a decoder or only the finishing touches to produce an output.

[0107] Different early exit points and the end of the network may share some infrastructure to save memory space. It is especially easy and efficient to share infrastructure when using iterative processing.

[0108] In Fig. 2 the neural network receives an input with speech and noise and has been trained to remove the noise to output denoised speech. The different outputs

[0109]

[0110] are then post-processed (e.g. by a processing circuit 16 of a hearing aid 10 in Fig. 1) by adding some loss to make the resulting signal more realistic.

[0111] There may be any number of early exit points available so choosing which to use may be very relevant. The implementation of the neural network may have a default early exit point based on experiments or training results. The early exit point to use may be chosen when inputting the signal and / or while processing the signal. In some embodiments, one early exit point is initially chosen and later changed as a result of processing the signal in the neural network.

[0112] An initial choice of an early exit point may be based on environmental contexts. This may include that the signal is audio from a quiet park, thereby requiring relatively less processing and thereby an earlier early exit point.

[0113] Alternatively, the signal may be audio from a loud cafe, thereby requiring relatively more processing and thereby a later early exit point (or of course the final exit point, i.e. the end of the network).

[0114] The environmental context may be determined from the signal itself, e.g. by analyzing and estimating how much noise it includes or comparing it to an earlier received signal, or from limitations on the network such as available battery power or time.

[0115] The environmental context may be determined from a separate source, such as a hearing aid (such as in Fig. 1) or a connected smart device knowing where it is based on history, location data, internet connection, and / or user input.

[0116] The early exit point may be dynamically chosen while the neural network is processing the input signal. This may be based on predicting a denoising level achieved so far for some or each early exit point, and choosing an early exit point where the signal is predicted to have a denoising level above a predetermined threshold.

[0117] The denoising level may be some quality metric of the denoising achieved so far, such as signal-to-noise ratio. This may be a scalar metric or a distribution model predicting a probability that the output meets a target improvement level.

[0118] For a distribution model, a Bayesian framework such as Bayesian linear regression may be used, or any other similar probabilistic model. A predicted variance (or other suitable distribution parameters) may be translated into a probability that the denoised output achieves or exceeds a predetermined threshold. This may therefore be used as a quality metric to select an early exit point.

[0119]

[0120] The predetermined threshold may depend on the implementation of the neural network, a user preference, a battery level of the device, or manufacturer recommendation.

[0121] Probabilisic formulation allows use of cumulative probability function to directly evaluate the probability of an exit prediction’s performance being above a predetermined threshold, taking modelled uncertainty into account.

[0122] This may also be used with chunks of the signal for local, real-time predictions.

[0123] A general formula of the family of problems that the neural network may solve is:

[0124]

[0125] given a mixture signal Xmix(such as sound) comprising some noise n (from the environment) and a sum of sources x (from e.g. speakers).

[0126] When denoising, all sources x are combined into a single source and noise is removed from that single source. When separating sources, the noise is removed from each source relative to the others.

[0127] If noise is considered as a separate source (i.e. sourced from the environment), source separation is also denoising. Accordingly, the neural network is denoising also when separating sources.

[0128] The neural network with early exit points is trained to produce a denoised signal no matter which early exit point is used. This is different from e.g. using several neural networks in series that are individually trained, this is one complete, singular neural network that produces high quality and predicable output.

[0129] This dynamic framework may be difficult to train and require different methodologies that may take more time and / or more power.

[0130] Fig. 3 shows one possible way to train a neural network like in Fig. 2. Training such a dynamic neural network uses methodologies that account for variable processing depths and conditional execution.

[0131] The training in Fig. 3 uses a series of blocks of nodes as in Fig. 2 with the intent to be able to output a denoised signal after each block, i.e. at early exit points and the end of the network.

[0132]

[0133] To do this real-world partial processing is simulated by progressively trimming an input audio across each network block.

[0134] In Fig. 3 the first block processes a full 10-second clip, the next block processes a trimmed segment of the clip (such as 0.5-9.5 seconds), and the final block processes a further trimmed segment of the clip (such as 1 -9 seconds).

[0135] This training regime teaches the model to generate meaningful outputs even when later blocks are skipped, mimicking the dynamics of early exit and real-time adaptive processing. This helps bridge the gap between full forward pass and dynamic inference scenarios.

[0136] The neural network may be conditioned on a target SNR level to invest model capacity in one good prediction. During training, the first exit point to surpass the target level gets a higher weight in the loss function, while earlier or later exits get reduced weights.

[0137] This allows the model to dynamically adjust its model capacity to better model the prediction that will actually be used — the model conceptually is incentivized to go “all in” on the prediction it thinks will achieve the higher reward. This improves the performance of the first exit to surpass the target at the cost of all other exits.

[0138] Another training regime is soft-probability training. This is inspired by conditional execution and incorporates soft probability assignments to simulate dynamic execution paths.

[0139] Instead of imposing hard decisions (such as a binary exit or continue), the training objective includes a soft probability measure that indicates how likely a particular processing path should be taken.

[0140] This mechanism informs the network of a cost-benefit trade-off of continuing and encourages graceful degradation or refinement based on actual improvements.

[0141] When integrated with Bayesian approaches, this training further refines the early exit decision-making process by providing probabilistic confidence in output quality.

[0142] This approach allows the network to learn conditional execution in a smoother fashion, ensuring that transitions between computational depths / widths are robustly managed.

[0143] Fig. 4 shows a flowchart representing a method 100 for denoising a signal. The method 100 comprises a number of steps, some of which are optional. The optional steps are shown with dashed boxes. The flowchart in Fig. 4 is shown as

[0144]

[0145] linear, however certain steps may overlap, be skipped, or run simultaneously or in parallel depending on the implementation of the method 100.

[0146] The first step is receiving S110 an input signal.

[0147] This may be an audio signal, a video signal or any other noisy signal.

[0148] The next step is choosing S115 an early exit point or the full neural network to be used when inputting the input signal based on environmental context.

[0149] This may be based on the content of the signal, a location, available battery and / or processing power, or any number of other environmental factors.

[0150] This is a simple and effective way to use an appropriate exit when conditions are well known.

[0151] The next step is separating S118 the input signal into at least two separated input signals.

[0152] This may be used to improve the denoising of the neural network for specific frequency ranges, signal types, or processing goals.

[0153] The next step is inputting S120 the input signal(s) into a neural network trained to output a denoised signal.

[0154] The denoised signal may be a signal with less noise, with separated sources, or any other use for denoising. The neural network may be as described in relation to Fig. 2.

[0155] The next step is dynamically choosing S125 an early exit point or the full neural network to be used while the neural network is processing the input signal based on a predicted denoising level reaching a predetermined threshold.

[0156] This allows a more adaptive selection of how much processing power to use based on how the processing is going.

[0157] The next step is combining S128 the outputs of the neural network if the input signal was separated S118 into at least two separated input signals.

[0158] The final step is outputting S130 the output of the neural network as a denoised signal.

[0159] This may further comprise post-processing.

[0160] The preceding description has been a non-exhaustive disclosure of different exemplifying embodiments of the present inventive concept. This is not to be misconstrued as limiting the scope of the protection sought for the inventive concept, which is defined by the appended claims.

[0161]

[0162] This final part of the description further details some of the maths and implementation of the neural network according to the inventive concept is hereby appended.

[0163] In recent years, deep learning-based single-channel speech separation has made dramatic progress on the classic cocktail party problem, in large part driven by increasingly compute- and parameter-efficient neural network architectures.

[0164] Currently, most such architectures are designed with a fixed compute- and parameter budget, and consequently cannot scale to varying compute demands or resources, which limits their use in embedded and heterogeneous devices such as mobile phones and hearables. We design a speech separation architecture for early exit, and propose an uncertainty-aware probabilistic framework to jointly model the clean speech signal and error variance. Using our framework, we derive probabilistic early-exit conditions in terms of desired signal-to-noise ratios. We evaluate our methods across common speech reconstruction tasks, namely speech separation and enhancement, and show that an early exit model can match individual State-of-the-Art models on reconstruction performance and that dynamic early exit models can learn competitive sub-networks.

[0165] Our framework enables prediction of separation quality and allows for finegrained dynamic scaling.

[0166] Introduction

[0167] The cocktail party problem concerns the separation of a (possibly unknown) number of overlapping speakers, possibly corrupted by environmental noise and reverberation, which has many important applications in e.g. telecommunications and hearables. Single-channel speech separation and speech enhancement is the simplified setup of separating a known number of speakers from a single-channel mixture, typically without noise or reverberation.

[0168] Single-channel speech separation has recently become predominantly deep learning-based, beginning with the introduction of the TasNet and the follow-up Conv-TasNet, which showed that competitive speech separation performance could be achieved at significantly lower computational cost.

[0169] In this work we make the following contributions:

[0170]

[0171] • We propose a speech separation architecture for early exit and endow the model uncertainty awareness exploring a probabilistic model formulation accounting both for the clean speech signal, as well as the variance of the error and its associated uncertainty.

[0172] • We demonstrate that our approach provides a simple and principled framework to balance reconstruction quality and early exit-conditions while circumventing the need for conventional scale and permutation invariant training. We further explore that our probabilistic formulation enables to predict the SNR (improvement) and we derive an exit criteria based on desired SNR level.

[0173] • We validate our approach for both speech separation and speech enhancement on widely used datasets, demonstrating significant compute savings when evaluating with targeted SNR improvement goals. We further demonstrate the viability of training a single neural network with early exits to achieve performance competitive with state-of-the-art single-exit models.

[0174] Related Work

[0175] TasNet Linear RNNs: encoder / decoder / masker structure overview of otherTasNet-family models.

[0176] Scale-Invariant Signal-to-Noise Ratio (SI-SNR): SI-SNR is a used evaluation metric and loss function in both speech separation and enhancement defined as, 1 = 7-73, CD

[0177]

[0178] where II II2 is the L2-norm. SI-SNR can be understood as computing the signal-to-noise ratio (SNR) assuming an unknown scale, which is chosen to maximize the SNR.

[0179] SepReformer: The SepReformer introduces the idea of early splitting, where rather than projecting a shared encoded representation into separate estimated sources late in the network as in TasNet, the projection into separate sources happens early in the network, and further processing happens (mostly) independently for each source.

[0180] SepReformer is constructed as a U-net composed of a convolutional block, a time-mixing attention block, and a cross-speaker attention block in a transformer-like stack, and uses Layer-Scale to train comparatively deep and slim networks.

[0181]

[0182] Method

[0183] Bayesian Speech Separation

[0184] We start by considering only a single source and target pair, then consider multiple sources and targets, and finally multiple exits.

[0185] We use a Bayesian objective where the target signal x e RTis modelled by jointly predicting an estimated signalAx e RTand a variance o2in the prediction, where we assume a Gaussian prior on the signal error and an inverse-gamma prior on the variance. Marginalizing out the realized variance, we obtain a multivariate Student t-distribution likelihood:

[0186]

[0187] where !"(•) is the gamma function and we use the shape-scale inverse-gamma parametrization. This objective strikes a simple balance between reducing the signal error through the last term, while also being penalized for underestimating the variance by the second-to-last term.

[0188] Scale Invariance: Similar to scale-invariant SNR (SI-SNR), we can construct a scale-invariant version of our likelihood by introducing a scaling parameter on the estimated signalAx which yields a modified likelihood,

[0189]

[0190] Any conjugate prior can be imposed on y, but its effect would quickly be overwhelmed by the posterior for non-trivial signal length so we opt for a simple maximum-likelihood estimate of y,

[0191]

[0192] which closely resembles the scale-invariance parameter in the conventional SI-SNR, but normalized by the energy in the prediction rather than the target. This scale-invariance formulation has also been used with SI-SNR to train TF-GridNet.

[0193] Predictive Signal-to-Noise Ratios: Using a Bayesian objective that directly models both signal and error magnitude allows us to construct early exit conditions expressed directly as predictive signal-to-noise ratios. The regular signal-to-noise

[0194]

[0195] ratio (SNR), and SNR improvement (i.e. the improvement in SNR by using the estimated signal over the input mixture ~x G RT) can be expressed as

[0196]

[0197] Per our model assumptions, we have

[0198]

[0199] where X2T (A) is a non-central chi-square distribution with non-centrality parameter A and T degrees of freedom. The SNR / SNRi expressions can then be written as ratios of (non-central) chi-square distributions,

[0200]

[0201] Unfortunately, these chi-square distributions are not independent, but for sufficiently large T the chi-square variates converge to their means by the central limit theorem, enabling us to approximate the distribution as such:

[0202]

[0203] We can also assume that x - x (the error) is independent from x and / or x - ~x to arrive at a scaled, non-central F-distribution which also converges to the below gamma distributions in the limit of T ->

[0204] These SNR distributions can be used as early exit conditions by evaluating the CDF of the Gamma distribution, or by comparing quantiles of the distribution to a target SNR level.

[0205] Decibel-scaled SNR: Signal-to-noise ratios are usually reported in decibels. When making use of the CDF of the SNR / SNRi distributions we can simply rescale the target to non-decibel scale, and when considering the expectation of the distribution we use a second-order Taylor expansion,

[0206]

[0207] for gamma-distributed z. The distributions can also be approximated well by a normal distribution for moderate a (which we have) using a Wilson-Hilferty transform.

[0208] Mixture Model Source Separation: When predicting multiple sources from a single mixture, the most common way to match predictions to targets is to use utterance-level Permutation Invariant Training (uPIT), which evaluates the total loss for every possible permutation of sources, and only backpropagates through the permutation with the minimum total loss. Taking advantage of our Bayesian formulation, we can instead cast this problem as a mixture model such that for targets Xs e RTand estimated sourcesAXs G RTthe log-likelihood of each target is,

[0209]

[0210] which can be stably computed in terms of per-source log-likelihoods using the LogSum Exp trick on the forward pass, but at initialization the per-source likelihoods in Equation 13 can exhibit dramatically different scales, which can cause vanishing gradient problems when one estimated source is much more likely than all other sources. Due to this we use a scaled LogSumExp variant with temperature parameter T in early training, where we anneal T from T to 1 with an exponential schedule over the first 0.5% of training,

[0211]

[0212] The temperature parameter T effectively controls the “sharpness” of the LogSumExp, with T > 1 increasing gradient flow to less likely sources, and recovers the regular LogSumExp at T = 1.

[0213] While this mixture likelihood does not guarantee a 1 :1 mapping between sources and targets like UPIT, it does not require the number of predicted sources and targets to match, and scales computationally as O(S2) compared to O(S!) for uPIT which potentially opens up applications with larger numbers of speakers, or variable speakers.

[0214] From a generative standpoint, this says that each target must have at least one likely source.

[0215] Model Architecture

[0216]

[0217] Figure 5a: System overview

[0218] Figure 5b: Logical flow of model

[0219] We design a novel TasNet-family model suitable for early-exit.

[0220] We base the initial design of the BayesSep architecture on building blocks from the SepReformer model due to its strong performance in speech separation. Like the SepReformer, the network consists of (1) a shallow audio encoder stack which downsamples the input mixture ~x e RTby a factor F (RT-> RDxT / F, (2) a latent encoder module which processes the encoded mixture, (3) a latent split module which splits the encoded mixture into separate sources (RDxT / F-> RSXDXT / F^ (4)adecoder stack which is applied to each latent source, and (5) a shallow decoder which upsamples the signal by a factor F, following high-level changes:

[0221] • To allow for reconstruction at multiple exit points with high fidelity, we do not downsample the signal anywhere except in the very first and last layers.

[0222] • We replace time-mixing attention layers with a simple linear RNN based on real-gated linear recurrent units (RG-LRU)

[0223] Figure 6: Detailed architecture.

[0224] Figure 7: Detailed architecture of the blocks used.

[0225] (a) Temporal mixing block

[0226] (b) Local convolutional block

[0227] Encoder-Separator-Decoder: Following recent trends, we opt for an encoderseparator-decoder design where a shallow encoder / decoder pair down- / upsamples the audio signals and a separator module directly maps produces the encoder output to the decoder, rather than masking the encoder output as in TasNet.

[0228] Separator: We construct our separator module as a deep transformer-like stack with pre-norm, skip connections, and LayerScale. Specifically, for every layer f with input x in the stack we compute its output as:

[0229]

[0230] where norm( ) is an RMSNorm layer with per-channel scales initialized to 1, and a G RDis another vector of per-channel scaling factors, which we initialize to 10-5. This allows very deep networks to be trained stably as the separator is approximately a skip connection at initialization and only later learns to integrate non-linearity as a grows.

[0231] Figure 8: Speaker attention block (if needed)

[0232] We use early split as in SepReformer, where the first NEnc layers in the stack process the encoded speech mixture after which a speaker split module SpeakerSplit: RT xD-> RT xSxDprojects the refined speech mixture into S separate groups of channels, where the speaker dimension is considered a batch dimension in later processing.

[0233] Snake activations: We use the Snake activation function, which has previously been used to model audio data.

[0234]

[0235] where a is a learnable per-channel parameter.

[0236] Linear RNN block: As our primary long-range time-mixing block, we use a linear recurrent neural network (RNN) based on real-gated linear recurrent units (RG-LRU). The core recurrence in the RNN is,

[0237] fco = 0, (17)

[0238]

[0239] for sequences of scalars at, bt, and ht, where bt is the input, at e [0, 1] are decay factors, and ht is the output.

[0240]

[0241] Feed-forward block: Similar to SepReformer: a pointwise block which applies gated non-linear activations to each embedding vector independently, a (long) convolution (CLA), and a cross-speaker attention layer.

[0242] Discussion

[0243] Setting a to an integer allows the CDF to be computed in terms of a exponential functions. It seems likely to me that forcing all levels to PIT together is beneficial because it means the speaker attention layers cannot swap the sources, which

[0244]

[0245] would cause conflicting information when the speaker attention output is combined with its skip connection, which would contain information from the other speaker. Forcing the layers not to do this would split features even more, which is argued to be really good in sepreformer.

[0246]

Claims

CLAIMS1. A method (100) for denoising a signal, the method (100) comprising steps of: receiving (S110) an input signal;inputting (S120) the input signal into a neural network trained to output a denoised signal; andoutputting (S130) the output of the neural network as a denoised signal; wherein the neural network is dynamically scalable and comprises:at least two sequential blocks of nodes trained to process signals into denoised signals; andat least one early exit point arranged between said at least two sequential blocks configured to be used to output a denoised signal after using only a subset of said at least two sequential blocks.

2. The method according to claim 1 , wherein the signal is an audio signal.

3. The method according to claim 1 or 2, wherein the neural network is a deep neural network.

4. The method according to any one of the preceding claims, further comprising choosing (S115) one of said at least one early exit point or the full neural network to be used when inputting the input signal based on environmental context.

5. The method according to any one of the preceding claims, further comprising dynamically choosing (S125) one of said at least one early exit point or the full neural network to be used while the neural network is processing the input signal based on a predicted denoising level reaching a predetermined threshold.

6. The method according to any one of the preceding claims, wherein a first and second sequential early exit points of said at least one early exit point are differentiated by the second early exit point being configured to be used to output a denoised signal after reusing at least one block of said at least two sequential blocks of nodes more than the first early exit point.

7. The method according to any one of the preceding claims, wherein a primary and secondary sequential early exit points of said at least one early exit point are differentiated by the secondary early exit point being configured to be used to output a denoised signal after using one more block of said at least two sequential blocks of nodes more than the primary early exit point.

8. The method according to any one of the preceding claims, wherein an alpha and beta sequential early exit points of said at least one early exit point are differentiated by the beta early exit point being configured to be used to output a denoised signal after using more nodes of a block of said at least two sequential blocks of nodes more than the alpha early exit point.

9. The method according to any one of the preceding claims, wherein the neural network is configured to encode the input signal before processing the signal using said at least two sequential blocks of nodes and to decode the signal before outputting a denoised signal, wherein two sequential early exit points of said at least one early exit point share a decoder.

10. The method according to any one of the preceding claims, further comprising steps of:separating (S118) the input signal into at least two separated input signals and inputting (S120) said at least two separated input signals into the neural network; andcombining (S128) the outputs of the neural network from said at least two separated input signals into a speech enhanced signal and outputting (S130) the speech enhanced signal as a denoised signal.

11. A hearing aid (10) comprising:a microphone (14) configured to receive sound;a processing circuit (16) configured to convert the received sound into a hearing loss compensated signal; anda receiver (18) configured to receive the hearing loss compensated signal and output said signal as audio;wherein the processing circuit (16) is configured to:input the received sound from the microphone (14) as an input signal into a neural network trained to output a denoised signal; andoutput to the receiver (18) the output of the neural network as a hearing loss compensated signal;wherein the neural network is dynamically scalable and comprises:at least two sequential blocks of nodes trained to process signals into denoised signals; andat least one early exit point arranged between said at least two sequential blocks configured to be used to output a denoised signal after using only a subset of said at least two sequential blocks.

12. The hearing aid according to claim 11, wherein the processing circuit (16) is further configured to process the denoised signal output from the neural network using signal processing before outputting the processed signal to the receiver (18) as a hearing loss compensated signal.

13. The hearing aid according to claim 11 or 12, wherein the processing circuit (16) is further configured to:input the received sound from the microphone (14) as several input signals into the neural network;combine the several output signals of the neural network into a speech enhanced signal; andoutput to the receiver (18) the speech enhanced signal as a hearing loss compensated signal.

14. The hearing aid according to any one of the claims 11-13, wherein the neural network is a deep neural network.

15. The hearing aid according to any one of the claims 11-14, wherein the processing circuit (16) is further configured to choose one of said at least one early exit point or the full neural network to be used when inputting the received sound from the microphone (14) as an input signal into the neural network based on an environmental context of the hearing aid (10).

16. The hearing aid according to any one of the claims 11-15, wherein the processing circuit (16) is further configured to dynamically choose one of said at least one early exit point or the full neural network to be used while the neural network is processing the input signal based on a predicted denoising level reaching a predetermined threshold.

17. The hearing aid according to any one of the claims 11-16, wherein a first and second sequential early exit points of said at least one early exit point are differentiated by the second early exit point being configured to be used to output a denoised signal after reusing at least one block of said at least two sequential blocks of nodes more than the first early exit point.

18. The hearing aid according to any one of the claims 11-17, wherein a primary and secondary sequential early exit points of said at least one early exit point are differentiated by the secondary early exit point being configured to be used to output a denoised signal after using one more block of said at least two sequential blocks of nodes more than the primary early exit point.

19. The hearing aid according to any one of the claims 11-18, wherein an alpha and beta sequential early exit points of said at least one early exit point are differentiated by the beta early exit point being configured to be used to output a denoised signal after using more nodes of a block of said at least two sequential blocks of nodes more than the alpha early exit point.

20. The hearing aid according to any one of the claims 11-19, wherein the neural network is configured to encode the input signal before processing the signal using said at least two sequential blocks of nodes and to decode the signal before outputting a denoised signal, wherein two sequential early exit points of said at least one early exit point share a decoder.