Method for adaptive active noise suppression

Abstract

The invention relates to a method for adaptive active noise suppression for a portable audio system that has a microphone unit comprising at least one first microphone and a loudspeaker, a memory unit and a computing unit, the method comprising the following features: - providing evaluation data comprising audio data, the audio data being recorded using the microphone unit, the audio data comprising at least one first audio signal that is recorded using the first microphone of the microphone unit; - determining filter coefficients on the basis of the provided evaluation data using the computing unit; - generating filtered audio data from the recorded audio data using the filter coefficients and the computing unit; and - reproducing the filtered audio data using the loudspeaker.

Description

[0001] Methods for adaptive active noise suppression

[0002] The present invention relates to a method for adaptive active noise suppression for an audio system and a corresponding audio system.

[0003] Noise reduction methods are widely known in various forms. Generally, a distinction is made between passive and active noise reduction. While passive noise reduction uses sound-absorbing materials to shield external noise and thus reduce the noise perceived by the user, active noise reduction uses a microphone to pick up external noise and analyzes it with a processing unit. This analysis then generates an out-of-phase signal that is output through a loudspeaker. This out-of-phase signal interferes with the noise in such a way that destructive interference occurs, and the noise is either eliminated or at least significantly reduced. Active noise reduction methods are particularly common in headphones (also known as ANC or A / C headphones).The method according to the invention is used in active noise-canceling headphones, headsets, hearing aids, and other audio systems. However, the use of the method according to the invention is explicitly not limited to the aforementioned devices. On the contrary, the method according to the invention can also be used in other audio systems where active noise reduction is desired.

[0004] Active noise suppression methods can employ digital filters described by filter coefficients. The recorded noise is passed through the digital filter, which determines how the frequency components of the noise signal are modified in terms of magnitude and phase. Subsequently, an out-of-phase signal can be generated, which then interferes destructively with the noise, reducing the noise perceived by the user.

[0005] From US 2023 / 0223001 Al, a signal processing device, a signal processing method, a signal processing program, a method for generating a signal processing model, and a sound reproduction device are known.

[0006] Determining suitable filter coefficients for an individual application scenario (different background noise, different fit of the audio device (e.g., the fit of headphones), different directions of incidence of the background noise, etc.) presents a considerable challenge in practice.

[0007] Based on the problem described above, the object of the present invention is to provide a method for adaptive active noise reduction for a portable audio device that allows for optimized provision of the filter coefficients.

[0008] To solve the described problem, the present invention proposes a method for adaptive active noise suppression for a portable audio system, which comprises a microphone unit including at least one first microphone, a loudspeaker, a storage unit and a computing unit, wherein the method has the following features:

[0009] Providing evaluation data comprising audio data, wherein the audio data is recorded using the microphone unit, wherein the audio data includes at least one first audio signal that is recorded with the first microphone of the microphone unit;

[0010] Determining filter coefficients depending on the provided evaluation data using the computing unit;

[0011] Generation of filtered audio data from the recorded audio data using the filter coefficients and the processing unit; playback of the filtered audio data using the loudspeaker; wherein the determination of the filter coefficients is based on the use of a machine learning-based model; the machine learning-based model is stored in the storage unit; the machine learning-based model is trained with training data, wherein the training data includes training audio data and training filter coefficients associated with the training audio data, wherein the provided evaluation data is fed into the machine learning-based model and, depending on the provided evaluation data, filter coefficients are output by the machine learning-based model.

[0012] The method according to the invention offers the advantage that the optimal filter coefficients required for adaptive active noise suppression can be determined automatically and efficiently. The filter coefficients are determined depending on the current application scenario. For example, the spectral properties of the existing noise can be taken into account. Furthermore, as will be explained in detail below, the direction of incidence of the noise and the fit or positioning of the audio device can also be considered, particularly when multiple microphones are used, designed to record several audio signals at different positions.

[0013] The audio system can be, in particular, headphones, a headset, or a hearing aid. However, the present invention is explicitly not limited to use in connection with the aforementioned audio systems, but can be used with any audio system where active noise reduction is desired. The audio systems can be worn either on the ear (for example, in the case of headphones, a headset, or a smartphone) or in the ear (for example, in the case of a hearing aid).

[0014] The microphone unit can include one or more microphones designed to record audio signals. Specifically, the microphone unit can include a first microphone, a second microphone, and a third microphone, as will be explained in detail below. The first microphone can be designed, in particular, to record ambient noise from outside the user's ear canal. In other words, the first microphone can be configured as an external microphone, meaning it is designed to capture sound from outside the ear canal.

[0015] The training audio data also contains audio signals. Depending on the specific embodiment of the method according to the invention, the training audio data may each contain a first training audio signal and optionally also a second and / or a third training audio signal. The first training audio signal can be recorded by a first training microphone, while the optional second and third training audio signals are recorded by a second and a third training microphone, respectively. These training microphones are part of the training microphone unit used during the training process. In particular, it may be provided that the training microphone unit (i.e., the microphone unit used during the preceding training process) and the microphone unit of the audio system according to the invention (i.e., the microphone unit used during the training process) are combined.The microphone unit (which is used in the audio system for active noise reduction) is of identical construction or at least has similar properties. The individual embodiments of the method according to the invention are explained in detail below. The method according to the invention may include a training process for training the machine learning-based model. Different approaches can be pursued during the training process to ensure a particularly efficient training process, as explained in detail below.

[0016] According to one embodiment of the invention, the optimal filter coefficients for various training audio data can be determined experimentally during the training process. For this purpose, the training audio data can include at least one initial training audio signal. In this approach, different filter coefficients are experimentally applied to the initial training audio signal, and it is determined which filter coefficients lead to an optimal result with the given training audio data. The optimal result can be determined, for example, by asking a test subject about the quality of the noise reduction using the respective filter coefficients. The filter coefficients that lead to an optimal result for one (or preferably several) test subjects can then be considered the optimal filter coefficients.

[0017] Alternatively, a training microphone can be positioned in the ear canal of a user or a dummy head, with the first training microphone designed to capture sound near the eardrum. This training microphone, also referred to as the eardrum microphone, can be positioned at a maximum distance of 5 mm, 10 mm, 15 mm, or 20 mm from the eardrum. Subsequently, various filter coefficients can be tested for the respective training audio data. Experimental measurements are taken to determine which filter coefficients result in a lower sound pressure level at the eardrum microphone for each training audio data. This allows for a measurement of how well a set of filter coefficients is suited to reducing background noise near the eardrum. This enables an objective evaluation of the filter coefficients.The optimal filter coefficients can be determined, for example, by varying the filter coefficients while reducing a cost function representing the sound at the eardrum. This cost function can be derived from a time-domain audio signal or represent a time-averaged amplitude of the audio signal. For instance, if the time-averaged amplitude of the audio signal is particularly low for a given set of filter coefficients, this can be considered an indication that the selected set of filter coefficients is especially suitable for efficient noise reduction. Alternatively, the cost function can be determined by transforming the time-dependent audio signal into the frequency domain and then averaging the transformed signal.For example, a short-time Fourier transform can be used to transform the time-dependent audio signal into the frequency domain. The mean value of the transformed signal is then calculated for each frame, and subsequently, the average value across all frames is determined. The filter coefficients are then set to reduce this average value.

[0018] The filter coefficients can be varied during the training process until sufficiently good noise reduction is achieved. This can be accomplished, in particular, by comparing the noise reduction measure used (for example, a time-averaged amplitude of an audio signal) with a predefined threshold and varying the filter coefficients until the noise reduction measure falls below the predefined threshold. In the embodiment described above, it is therefore possible to experimentally determine during the training process which filter coefficients are particularly well-suited for noise reduction with specific training audio data.

[0019] Furthermore, in some embodiments of the method according to the invention, the optimal filter coefficients can be determined computationally or by simulation. In this process, the audio signals expected at the microphones of the audio system, as well as the noise expected at the eardrum, can be determined computationally (instead of measuring them), so that a cost function can be derived which can be reduced or minimized during the process of determining the optimal filter coefficients. In this case, no physical microphones and loudspeakers are required. Instead, transfer functions are used that describe the transmission of sound along a transmission path.Starting with a background noise, typically output via an external loudspeaker during the training process, it's possible to calculate the characteristics of the background noise captured by an external microphone of an audio system. This involves using a first transfer function that describes the sound transmission from the external loudspeaker to the external microphone of the audio system. The background noise captured by the external microphone can then be passed through a digital filter defined by filter coefficients. Subsequently, an out-of-phase audio signal can be generated, designed to reduce the background noise received at the eardrum. This out-of-phase audio signal can then be simulated and output through a loudspeaker.Using a second transfer function that describes the transmission over a second transmission path, the total signal received at the eardrum can then be calculated. This calculated "eardrum signal" can subsequently be fed into a cost function, which is reduced or minimized during the training process to determine optimal filter coefficients. The determination of these transfer functions can be carried out according to methods known from the prior art. For example, to determine a transfer function, a measurement signal can be played through an external loudspeaker, and the audio signal received there can be recorded via a microphone. The measurement signal and the received audio signal can then be transformed into the frequency domain.The transfer function for a transmission path (for example, from an external loudspeaker to an external microphone or from an internal loudspeaker to a microphone positioned at the eardrum) can then be determined by spectral division of the signals.

[0020] In some embodiments of the method according to the invention, the training process may include the following steps:

[0021] Providing training audio data comprising at least an initial audio signal;

[0022] Determining one set of filter coefficients for each of the provided training audio data; wherein the determination of each set of filter coefficients comprises a reduction of a cost function, the cost function preferably describing a sound pressure level or a sound in the ear canal of a user;

[0023] Training the machine learning-based model using the training audio data and the specified filter coefficient sets; and

[0024] Storing the trained model in a training memory unit.

[0025] The training audio data (also generally referred to as training input data within the scope of the present invention) can comprise a first audio signal that is recorded by a first training microphone, wherein the first training microphone can be configured as an external microphone designed to detect external sound (i.e., sound outside the ear canal). Furthermore, the training audio data can comprise a second audio signal that is recorded by a second training microphone, wherein the second training microphone can be configured as an internal microphone designed to detect internal sound (i.e., sound within the ear canal).In addition, the training audio data may include a third audio signal recorded by a third training microphone, the third training microphone being designed as an additional external microphone designed to capture external sound (i.e., sound outside the ear canal).

[0026] The advantage of using only a single initial audio signal is that the processing effort is lower, allowing for a particularly computationally efficient implementation of the method. The advantage of using multiple audio signals during the training process is that additional information can be used, enabling an optimized determination of the filter coefficients. For example, if a first and a second audio signal are used, with the first signal representing a sound outside the ear canal and the second representing a sound inside the ear canal, additional information can be derived. This information can describe, for instance, the fit of the audio device to the ear (the so-called "fitting" of the audio system). This allows the determination of the optimal filter coefficients to also take into account the fitting characteristics of the audio device.

[0027] An additional advantage of using multiple training microphones and multiple audio signals is that it allows for the consideration of additional information about the direction of the noise's arrival. This enables the optimization of the filter coefficients to take the noise's direction into account. For example, this can be achieved by using two external training microphones to capture two audio signals, from which the noise's direction of arrival can be determined (e.g., by analyzing the time-of-arrival differences). In practice, it has been shown that the noise's direction of arrival has a significant influence on the optimization of the filter coefficients.

[0028] In the embodiment of the method according to the invention described above, the training audio data preferably comprises a plurality of training audio signals. For example, the training audio data may comprise several hundred, 1,000, or 10,000 training audio signals in total. For example, the training audio data may comprise 1,000 audio signals recorded with the first training microphone. Alternatively, the training audio data may comprise 2,000 audio signals, with 1,000 audio signals recorded by the first training microphone and 1,000 by the second training microphone. The training audio data and the filter coefficient sets may together also be referred to as the training data set. The audio signals described above may comprise or be derived from time signals or frequency signals.For example, the audio signals can be provided by applying a short-time Fourier transform (STFT). This allows the spectral characteristics of the interference signal to be taken into account.

[0029] As described earlier, the training process for determining the filter coefficients can be carried out using either a measurement-based approach (using physical microphones and loudspeakers) or a simulation-based approach. While the measurement-based approach relies on the perception of test subjects or an audio signal recorded by an internal microphone (positioned near the eardrum) to determine the cost function to be reduced, the simulation-based approach requires neither test subjects nor physical microphones or loudspeakers for the training process. Therefore, the simulation-based approach offers the advantage of generating a large amount of training data (including training audio data and training filter coefficients) in a short time, thus enabling an efficient implementation of the training process overall.

[0030] When reducing the cost function, it may be particularly desirable to minimize this cost function in order to achieve optimal noise reduction. However, in some embodiments of the invention, it may also be desirable not to minimize the cost function, but rather to reduce it to a value close to the minimum.

[0031] The cost function can, in particular, describe the sound pressure level in the ear canal or a sound or sound event. For example, the cost function can describe an audio signal recorded in the ear canal or calculated for the ear canal, where the audio signal can be in the time or frequency domain. The cost function can also be a function derived from the aforementioned audio signal, which can describe a loudness in the ear canal. For example, the cost function can be defined by an average of the power levels over the individual frequency bins. As explained at the beginning, the cost function can preferably describe a sound pressure level in the ear canal. The sound pressure level in the user's ear canal can, in particular, be a sound pressure level in the outer ear canal at a distance d from the eardrum, where d < 5 mm, d < 10 mm, d < 15 mm, or d < 20 mm.

[0032] The reduction of the cost function can be achieved by adjusting the filter coefficients so that the cost function either reaches a minimum value or, alternatively, a value below a predefined limit. For example, the sound pressure level at or near the eardrum can be determined by measurement or simulation, and the filter coefficients adjusted so that the cost function falls below a predefined limit. Alternatively, the cost function can be derived from a frequency analysis. For example, a short-time Fourier transform (STFT) can be used, which transforms a measured or simulated audio signal describing sound received at the eardrum into the time-frequency domain. The cost function can then, for example, describe the average power density of the STFT signal.Alternatively, the power density can be averaged onto a weighted frequency signal. This allows psychoacoustic insights into the perception of an audio signal in different frequency ranges to be utilized. Furthermore, the STFT signal can be processed to simulate psychoacoustic masking effects.

[0033] In some embodiments of the method according to the invention, the training process may include the following steps:

[0034] Determination of filter coefficients by the model based on the audio data;

[0035] Simulation of a sound pressure level in the ear canal based on the audio data and the determined filter coefficients;

[0036] Evaluation of a cost function based on the simulated sound pressure level, which correlates with the perceived loudness; adjustment of the model parameters to reduce the cost function.

[0037] Furthermore, the inventive method may provide that the training process comprises the following steps:

[0038] Providing training audio data comprising at least one initial audio signal; determining one set of filter coefficients for each of the provided training audio data;

[0039] Simulation of a sound pressure level in the ear canal based on the audio data and the determined filter coefficients;

[0040] Evaluation of a cost function based on the simulated sound pressure level, where the cost function represents the perceived loudness; and

[0041] Adjustment of the model parameters taking into account the cost function.

[0042] In particular, the method according to the invention may include a reduction of the cost function that describes the sound pressure level in the outer ear canal of a user.

[0043] In particular, the cost function can describe a sound pressure level near the eardrum (also called the eardrum sound pressure level), where the eardrum sound pressure level describes the sound pressure in the external auditory canal at a distance d < 5 mm, d < 10 mm, d < 15 mm or d < 20 mm.

[0044] In some embodiments of the method according to the invention, the first audio signal may be provided as an audio signal converted into the frequency domain, the conversion preferably being carried out via a short-time Fourier transform. This allows the spectral properties of the interference signal to be taken into account, so that the filter coefficients can be determined as a function of the characteristics of the interference noise.

[0045] Furthermore, the method according to the invention may include a second audio signal in the audio data, which is recorded by a second microphone of the microphone unit. This second microphone may be designed to record an audio signal within the ear canal. Therefore, the second microphone may be designed, in particular, as an internal microphone, i.e., a microphone positioned within the ear canal, or as a microphone that, when the audio system is worn, is directed towards the eardrum. In this case, the training audio data also includes a second audio signal recorded by a second training microphone. The second microphone and the second training microphone may preferably be identical in construction or have similar properties.The advantage of using two microphones designed to record two audio signals (both during the training process and when using the method to determine the filter coefficients) is that the adaptation characteristics of the audio device (the so-called "fitting") and also the direction of incidence of the noise can be taken into account when determining the optimal filter coefficients.

[0046] In some embodiments of the method according to the invention, the audio data may also include a third audio signal, which is recorded by a third microphone of the microphone unit. This third microphone may be designed, in particular, to record an additional audio signal outside the ear canal. It may therefore be configured as an additional (second) external microphone. This makes it possible to consider additional directional information when determining the filter coefficients. The two microphones can be used to determine the direction of incidence of a sound. For this purpose, methods known in the prior art can be used. In particular, the time difference of the signals recorded by the two microphones can be determined.In some embodiments, the signals recorded by the two microphones can be correlated to determine the time difference. It can also be provided that the signals are transferred to the frequency domain to determine the phase difference for different frequencies. If a third audio signal is used, the training audio data also includes a third (training) audio signal, which was previously recorded with a third (training) microphone. The third microphone and the third training microphone can preferably be identical in construction or at least have similar characteristics.

[0047] By using three audio signals recorded by three microphones, the spectral properties of the noise, the matching characteristics (tight fit, loose fit) of the audio system, and the directional characteristics of the noise can all be considered when determining the optimal filter coefficients. In this case, the machine learning-based model "learns" during the training process which filter coefficients are best suited for specific situations to enable efficient noise reduction in the given context.

[0048] In some embodiments of the method according to the invention, it may be provided that the evaluation data have at least one transfer function that describes a transmission path between an external noise and the eardrum.

[0049] For example, the evaluation data can include a first transfer function that describes the transmission of a sound from an external loudspeaker to an external microphone of the audio system. Alternatively, the evaluation data can include a second transfer function that describes the transmission of a sound from an internal loudspeaker of the audio system to an internal microphone of the audio system located near the eardrum (also known as an eardrum microphone). The evaluation data can also include a third transfer function that describes the transmission of a sound from an external loudspeaker to the eardrum microphone. The transfer function provides additional information about the direction of incidence of the noise as well as about the fit characteristics of the audio system (tight fit, loose fit, etc.).Various methods already known from the prior art can be used to determine the transfer functions. For example, an audio signal recorded via an internal microphone in the outer ear canal can be compared to an audio signal recorded via an external microphone. Alternatively, an audio signal recorded via an internal microphone in the inner ear canal can be compared to an audio signal recorded via an external microphone. Using the transfer function allows additional information to be considered in order to optimize the determination of the filter coefficients.

[0050] In some embodiments of the method according to the invention, the evaluation data may include fitting data (also referred to as "fitting data" or "fitting parameters") that characterize the fit of the audio system to the user's head and / or directional data that characterize the direction of incidence of an unwanted noise. The fitting data may, for example, contain information about whether an audio system (such as headphones or a headset) is securely or loosely fitted. This can also be represented by a parameter that is, for example, normalized to a value between 0 and 1. The directional data may describe an angle of incidence from which an unwanted noise is received.Alternatively, the directional data can describe two angles of incidence, where a first angle defines a horizontal angle of incidence of the noise (also referred to as the azimuth angle) and a second angle describes a vertical angle of incidence of the noise (zenith angle). In the embodiment described above, the machine learning-based model is previously trained with matching data and / or directional data so that the model can take this data into account when determining suitable filter coefficients. In this way, an optimization of the filter coefficients is achieved depending on the aforementioned matching data and / or the directional data. Furthermore, the method according to the invention can additionally include training a machine learning-based model, wherein the training comprises the following steps:

[0051] Reduction of a cost function that characterizes a sound pressure level at a user's eardrum.

[0052] In particular, it may be intended that the aforementioned cost function is minimized.

[0053] Furthermore, the inventive method may include a machine learning-based model comprising an artificial or neural network, in particular a deep neural network, a convolutional neural network or a recursive neural network with memory, a support vector machine or an adaptive algorithm.

[0054] Furthermore, to solve the problem described above, an audio system for adaptive active noise reduction is proposed, comprising a microphone unit with at least one first microphone, a loudspeaker, a storage unit, and a processing unit, wherein the microphone unit is designed to record evaluation data comprising audio data, wherein the audio data includes at least one first audio signal, and the first microphone of the microphone unit is designed to record the first audio signal; the processing unit is designed to determine filter coefficients depending on the provided evaluation data; the processing unit is designed to generate filtered audio data from the recorded audio data using the filter coefficients; the loudspeaker is designed to output the filtered audio data; wherein

[0055] - the computing unit is designed to determine the filter coefficients using a machine learning-based model;

[0056] - the machine learning-based model is stored in the memory unit;

[0057] - the machine learning-based model is trained with training data, where the training data includes training audio data and training filter coefficients associated with the training audio data,

[0058] - the computing unit is designed to input the provided evaluation data into the machine learning-based model and, depending on the provided evaluation data, to determine filter coefficients through the machine learning-based model.

[0059] The audio system according to the invention can in particular be configured as headphones, a headset, or a hearing aid. The first microphone can in particular be configured as an external microphone.

[0060] In the audio system according to the invention, the microphone unit may include a second microphone designed to record a second audio signal of the evaluation data. This second microphone may, in particular, be an internal microphone (i.e., a microphone facing the ear canal) designed to capture sound within the ear canal. The inclusion of the second microphone allows for the consideration of additional information, particularly regarding the adaptation characteristics of the audio system. This enables a more precise determination of the optimal filter coefficients.

[0061] Furthermore, the audio system according to the invention may include a third microphone unit designed to record a third audio signal of the evaluation data, preferably designed to detect sound outside a user's ear canal. This provides an additional external microphone capable of evaluating additional information about the background noise. In particular, the use of the second microphone allows for the recording and evaluation of directional information about the background noise, enabling a particularly precise determination of the optimal filter coefficients for the given scenario.

[0062] More the

[0063] Further preferred aspects of the present invention will be explained in more detail below.

[0064] As explained earlier, previous methods for active noise cancellation (ANC) in headphones are often based on static filters to generate a compensation signal. Designing these filters is complex, as they must be optimized for various fit conditions and noise scenarios.

[0065] While static filters can deliver generally good ANC performance, their effectiveness can decrease in specific situations or changing environments. Current technologies offer various approaches to adjusting filter coefficients and thus optimizing ANC performance. However, each of these approaches has its own specific limitations, as explained below.

[0066] The so-called Fx-LMS algorithm is a commonly used algorithm for fully adaptive FIR filters in ANC systems. Due to their computational complexity, these are rarely implemented in low-latency audio codecs, especially since filtering down to low frequencies requires long filters. However, the Fx-LMS algorithm has some weaknesses: Ensuring robust ANC performance at the eardrum is difficult, as individual differences in the ear canal must be taken into account. Furthermore, the convergence behavior of the Fx-LMS algorithm is difficult to configure and control, and can lead to unwanted temporal artifacts or even instability. The Fx-LMS algorithm also carries a risk of over-adaptation.The high number of degrees of freedom, determined by the FIR filter length, can cause the algorithm to adapt too precisely to specific situations, leading to unpleasant artifacts even with slight system changes. Conversely, excessively short FIR filters can result in insufficient filter performance at low frequencies. Furthermore, in a multi-rate system, sample rate conversion of FIR filter coefficients may be necessary, introducing additional complications and increased computational costs. Instead of FIR filter coefficients, a single weighting factor can also be adapted. However, this offers insufficient flexibility and degrees of freedom to respond to diverse noise scenarios.

[0067] Another state-of-the-art method is the calibration of ANC filters, for example, when inserting headphones by playing a test signal and subsequently determining or selecting optimal filter coefficients. A limited number of pre-optimized filters may be available, from which an algorithm or the user selects those that best suit their headphone fit and environment. However, this selection often only occurs once when inserting or putting on the headphones and otherwise requires user interaction.

[0068] Many state-of-the-art adaptive methods are designed to minimize the power output of an ANC headphone's internal microphone, rather than considering the sound pressure level at the eardrum. While the pressure chamber effect suggests that the sound pressure level at the eardrum and the internal microphone are comparable up to a certain cutoff frequency, this assumption no longer holds true for higher frequencies. Research literature presents methods using virtual sensors (ENGI, "virtual sensing"), where virtual sensors estimate the output signal of physical sensors that cannot be physically implemented at runtime, for example, due to space constraints. State-space models are used to describe the sensor behavior. However, the model accuracy can be less than ideal. Furthermore, approaches using virtual sensors require additional components, such as the previously mentioned Fx-LMS algorithm.

[0069] It is well known that active noise cancellation (ANC) systems in headphones are designed to reduce unwanted ambient noise and create a more comfortable listening experience. However, current solutions have significant weaknesses. They are either not flexible enough to adapt to constantly changing circumstances, such as the fit of the headphones, the individual anatomy of the ear, or the spectral power density and direction of ambient noise, or they are too complex and too sensitive to even the slightest changes. This can lead to temporary artifacts in the audio signal or reduced effectiveness. Furthermore, existing ANC systems are susceptible to interference at the internal microphone from body vibrations, which further impairs noise reduction. Particularly with binaural ANC systems, aggressive adaptation can distort the user's spatial hearing.Another problem is the lack of consideration for human perception, as many systems do not adequately account for the sound pressure level at the eardrum. These factors lead to suboptimal performance and an unnatural listening experience.

[0070] Within the scope of the present invention, a fully adaptive ANC system for headphones based on machine learning methods is presented, which, among other things, achieves robust performance even under challenging conditions. The invention solves the aforementioned problems through an adaptive ANC system based on the principles of machine learning.

[0071] Ideally, measurement data on various parameters such as headphone fit, individual ear anatomy, spectral power density of ambient noise, and the direction of incidence of the noise can be collected and combined into a comprehensive dataset. Based on this data, a model can be trained to calculate filter coefficients for the ANC system. This training is performed using machine learning methods and can preferably reduce a specially developed cost function that explicitly considers human perception at the eardrum. By incorporating subjective hearing perception, a more comfortable audio experience is achieved.

[0072] The invention comprises various embodiments to optimize the adaptability, robustness, and performance of the system. These can include different filter structures, input features for the model, and model structures. Model architectures are presented that enable robust and artifact-free adaptation behavior.

[0073] Some preferred embodiments of the invention are explained in more detail below with reference to the figures. These show the

[0074] Fig. 1 schematically shows an in-ear headphone in the ear canal with essential electronic components;

[0075] Fig. 2 shows the process steps of an embodiment of the method according to the invention;

[0076] Fig. 3 shows a block diagram of a device according to the invention with an adaptation based on a reference microphone; Fig. 4 shows an example of the signal flow for a cost function that imitates the human auditory perception of background noise;

[0077] Fig. 5 shows exemplary process steps for generating training data and training a machine learning model;

[0078] Fig. 6 shows a block diagram of a device according to the invention with an adaptation based on a reference and measuring microphone; and

[0079] Fig. 7 shows an example of a machine learning model for estimating coefficients.

[0080] To better understand the principles of the present invention, embodiments of the invention are explained in more detail below with reference to the figures. It is understood that the invention is not limited to these embodiments and that the described features can also be combined or modified without departing from the scope of protection of the invention as defined in the claims. In particular, it is taken for granted that the features described in connection with the method according to the invention can also be implemented in connection with the system according to the invention.

[0081] Figure 1 shows an example of a portable audio system 10 in the form of an in-ear headphone (also referred to as an in-ear headphone or earphone), in which the method according to the invention can be used. As already explained at the outset, the method according to the invention can also be used in other types of headphones, hearing aids, hearing protectors, headsets, glasses, or smartphones with audio functionality. The in-ear headphone audio system 10 sits in the ear canal 14 of a user, held in place by an ear tip 12, and acoustically seals it completely or partially. Depending on the fit of the ear tip 12 for an individual user, the ear canal 14 can be more or less open by a ventilation opening 16. The portable audio system 10 can also have targeted ventilation in the housing or in the ear tip 12.

[0082] The audio system 10 is equipped with at least one first microphone 18, which, within the scope of the present invention, is also referred to as a reference sensor or reference microphone. In the illustrated embodiment, the first microphone 18 is designed as an external microphone that detects ambient sound and generates a reference signal based on this. Furthermore, the audio system 10 has at least one second microphone 20, which, within the scope of the present application, is also referred to as a measuring sensor or measuring microphone and is preferably designed as an inward-facing microphone that, during operation, faces the eardrum 22 of a user. The second microphone 20 is designed to detect the sound in the ear canal 14 of the user and generate a measurement signal based on this.The audio system 10 is equipped with at least one processing unit 24, which processes the signals recorded by the first microphone 18 (reference sensor) and the second microphone 20 (measuring sensor) according to the invention. The processing unit 24 can, in particular, be one or more digital signal processors. Based on the sensor signals processed by the processing unit 24, an audio signal is then reproduced via the internal loudspeaker 26 of the headphones (also referred to as the internal loudspeaker). The processed ambient sound reproduced via the loudspeaker 26 can then, for example, reduce the ambient sound at the eardrum 22 by means of destructive interference.

[0083] Figure 1 also includes a third microphone 28, designed to measure the sound pressure level at or near the eardrum 28 of a user. Within the scope of the present invention, the third microphone 28 is also referred to as the eardrum microphone, since it is arranged in close proximity to the eardrum 22. The third microphone 28 is particularly necessary for recording measurement data for the design of the audio system 10, but not for real-time operation. Depending on the anatomy of the ear and the fit of the audio system 10, the transfer functions, which describe the transmission of sound from the loudspeaker 26 or from an external source to the third microphone 28, can vary.

[0084] Figure 2 shows a schematic flowchart of the method 100 according to the invention. In a first method step 110, evaluation data, including audio data, is provided. The audio data is recorded using a microphone unit, wherein the audio data includes at least one first audio signal, which is recorded with a first microphone of the microphone unit. In a second method step 120, filter coefficients are determined as a function of the provided evaluation data using a processing unit. In a third method step 130, filtered audio data is generated from the recorded audio data using the filter coefficients and the processing unit. In a fourth method step 140, the filtered audio data is played back using the loudspeaker. The filtered audio data ensures that the noise caused by destructive interference is reduced.In the method 100 according to the invention, the filter coefficients are determined using a machine learning-based model. The model is trained with training data, wherein the training data comprises training audio data and training filter coefficients assigned to the training audio data.

[0085] Figure 3 shows a first block diagram of a processing operation according to the invention. The reference signal x(n) of a first reference sensor, here in the form of an external microphone of the headphones, is recorded and filtered by a forward filter W(z,ri) to generate a compensation signal. The compensation signal is then played back via a loudspeaker of the headphones. The loudspeaker signal is transmitted via the electro-acoustic transfer function G(z) between the loudspeaker and a measuring sensor, here in the form of an internal microphone of the headphones facing the ear canal, and G(z) between the loudspeaker and a tympanic microphone, where it interferes with the passive sound signal d(n) and d(n) at the measuring and tympanic microphones. The measuring and tympanic microphones then record the signals e(n) = d(n) + y(n) and e(n) = d(n) + y(n) accordingly.

[0086] In the embodiment shown in Fig. 3, a machine learning-based model is also provided, which analyzes the reference signal and, based on this analysis, adjusts the coefficients w(n) of the forward filter to minimize a measure of the sound pressure level at the user's eardrum. The sound pressure level at the eardrum is described by the eardrum signal d(n) for the passive system, i.e., for y(n) = 0, and e(n) for the active system of the eardrum microphone. The eardrum microphone serves solely for data acquisition for training the model. It is not part of the real-time system.

[0087] Figure 4 illustrates a possible design for a measure of sound pressure level or perceived loudness at a user's eardrum. The eardrum microphone signal d(n) describes the sound pressure level at the eardrum with a passive system, while e(ri) describes the sound pressure level with an active system. First, the signals e(ri) and d(n) are transformed into a time-frequency representation using a short-time Fourier transform (STFT), allowing analysis of the intensity of different frequency components and their temporal evolution. Another form of time-frequency analysis, for example, using a filter bank, is also possible. Subsequently, the output spectrogram of the STFT can be aggregated into frequency bands using a Mel filter bank, which serves to mimic human frequency perception. The Mel spectrogram of e(ri) is referred to here as M sf,k) denotes the frequency index f and the frame index k of the STFT. Alternatively, a more complex model for human hearing can be used instead of the Mel filter bank, which models psychoacoustic masking effects in the frequency and time domains as well as an absolute hearing threshold.

[0088] The Mel spectrogram is then reduced to a cost value l per frame by calculating the mean of the powers of the values ​​of M. s f,k) is calculated as follows:

[0089] For human loudness perception, which is crucial for the perceived performance of ANC systems, the magnitude of a frequency spectrum is particularly important. Therefore, in the embodiment shown here, only the magnitude of the Mel spectrogram is considered. The exponent p determines the weighting in the frequency domain. It is typically chosen in the range pe (0, 2). An exponent of p = 2 results in a typical root mean square (RMS) value. However, in this case, large values ​​of M have a significant impact. S The influence of certain frequencies on the mean is so strong that weaker frequency components receive insufficient attention. Therefore, choosing p < 2 is advantageous. In particular, choosing p = 0.3 is advantageous because the dynamic behavior of the function of x 0 3, at least for a certain range of values, comparable to that of the logx function, and the logarithmic function has proven useful for describing the human perception of loudness. Of course, the logarithmic function can also be used to calculate the mean. However, the power of the absolute value is numerically somewhat better suited and less sensitive to very small values.

[0090] The cost value l is calculated for both the signal e(ri) and d(n).

[0091] In a final step, the normalized cost function can then be used. The cost function can be calculated by normalizing the cost values ​​for the active system using the corresponding values ​​for the passive system and then converting them to decibels, taking the exponent p into account. The normalized cost function thus approximates the perceived loudness difference between the passive and active systems. Normalization is optional, but has the advantage that the cost function is less sensitive to the absolute level of the ambient noise. Normalization can also be performed taking into account the active eardrum signal, for example, for a static ANC system.

[0092] Fig. 5 describes by way of example a method 200 for training the model according to an embodiment of the present invention.

[0093] In a first process step 210, a training dataset is generated. As already described in the introduction, the present invention offers various approaches for generating the training data. On the one hand, this data can be provided experimentally by manually determining which filter coefficients are optimally suited for a specific audio signal. For this purpose, a test subject can be asked at which filter coefficients optimal noise reduction is perceived. For realistic results, a series of measurements with different test subjects using different ear fittings is recommended. Alternatively, the training data can also be provided automatically (without a test subject) by reducing or minimizing a cost function, where the cost function describes a detected sound or a sound pressure level in the ear canal of a user.For this purpose, a dummy head modeled on a human head can be used, with one or more microphones arranged in the ear canal of the dummy head. In particular, a tympanic microphone can be arranged near the eardrum (for example, at a distance of less than 5 mm, 10 mm, 15 mm, or 20 mm from the eardrum) to capture the sound in the ear canal. During the training process, it can then be experimentally determined which filter coefficients lead to a reduction or minimization of the background noise. In this way, the optimal filter coefficients for each audio signal can be experimentally determined during the training process. According to a particularly preferred embodiment of the method according to the invention, which will be explained in detail below, the training data are determined by a simulation approach.

[0094] During the measurement, signals such as exponential sweeps can be played back via external speakers and the integrated headphone speaker. Simultaneously, the signals from the reference, measurement, and eardrum microphones can be recorded. From this data, direction-dependent transfer functions between the respective external speakers and all microphones, as well as the secondary path G(z) from the headphone speaker to the measurement microphone and the eardrum path G(z) from the headphone speaker to the eardrum microphone, can be determined.

[0095] Based on external transfer functions, virtual, passive microphone signals can be simulated for various interference signals by convolving an interference signal with the corresponding transfer functions. Interference signals for different directional measurements can also be combined. Taking the secondary and tympanic pathways into account, an active measurement and tympanic signal can then be simulated for a specific compensation signal.

[0096] A data point in the training dataset then consists of the virtual passive microphone signals as well as the secondary and tympanic pathways. To ensure an accurate representation of the acoustic behavior, the data for a given data point should preferably be based on a single measurement, ideally with identical headphone fitting. The training dataset should be as comprehensive as possible and cover a wide range of situations so that the model is able to react to previously unseen situations. It is recommended to generate a test dataset, ideally based on exclusive measurement data and noise signals, to validate the model.

[0097] Instead of acoustic measurements, computer simulations of the acoustic behavior are also possible.

[0098] In a second process step 220, coefficients for the forward filter are determined for each magazine using an optimization procedure. These coefficients minimize or at least reduce a cost function. A magazine corresponds, for example, to the frame index k of an STFT or a sampling point n. This cost function could, for instance, be the measure of the sound pressure level at the eardrum shown in Fig. 3. However, other types of cost functions can also be used. In particular, the cost function can be extended by a regularization term, which, for example, penalizes large filter coefficients and ensures that the coefficients are within a reasonable range of values. Through regularization, coefficients that have little or no effect on the cost function can converge to 0 or to other predefined values. This behavior is advantageous for the estimation process in the next step.The optimal coefficients can then be stored together with the passive microphone signals and the transfer functions G(z) and G(z) of the headphone loudspeaker. Additionally, the values ​​of the cost function, as well as a local approximation of the gradient of the cost function around the optimum as a function of the filter coefficients, can be stored, so that the cost function can be estimated when there are deviations from the optimal coefficients.

[0099] In a third process step, 230, the model's parameters are trained using machine learning methods. While observing simulated microphone signals with the system active, the model estimates the optimal coefficients for each journal in the training dataset. The model parameters are chosen to minimize the distance between the estimated coefficients and the previously determined optimal coefficients. This distance measure can also be calculated using the stored values ​​of the cost function and the gradient information. Thus, the distance measure can approximate the cost function, taking the estimated coefficients into account.

[0100] To robustly parameterize the model, it is advantageous not to simulate the active microphone signals solely based on the optimal coefficients. The coefficients can also be set to 0, 1, or random values ​​for specific time points. Furthermore, the coefficients can be smoothed over time. This allows the model to learn to estimate the optimal coefficients even from a suboptimal state.

[0101] The model parameters are then output in a fourth process step 240.

[0102] The training of the model parameters can also be performed in other ways, for example, by explicitly simulating the active microphone signals for each journal of the training data and evaluating the cost function. However, since models that use a measurement microphone signal require recursion during such training (because the measurement microphone signal depends on the coefficients and the coefficients, in turn, depend on the measurement microphone signal), this training approach can take more computation time. In the previously discussed training method, where optimal coefficients are calculated first, this recursion can be decoupled, thus speeding up the training.

[0103] Figure 6 shows another block diagram of an embodiment of the method according to the invention. In addition to the example from Figure 3, a measuring microphone is used, which is fed to both a feedback controller K(z) and the model. Such a controller can also be used in Figure 3. Furthermore, the measuring microphone signal can also be fed to the model without using a controller. The controller can be designed to provide perceived attenuation of external sound in addition to the forward filter. The controller can also be designed to compensate for the occlusion effect. When using a controller, its influence must be taken into account when simulating the active microphone signals, in particular the measuring and tympanic microphone signals. The measuring microphone signal can, in particular, provide information about the fit of the headphones and the anatomy of the user's ear.It is also possible to input additional signals, such as the speaker signal. Alternatively, other audio signals (such as music or telephone calls), which are also played back via the audio system's internal speaker, can be fed into the model.

[0104] Figure 7 shows an exemplary embodiment of a model for estimating coefficients based on a reference and measurement microphone signal. A corresponding model can be used, for example, based on Figure 6. Subcomponents of this model can also be omitted, exchanged, or modified. In real-time operation, the model can be executed for each frame of an STFT analysis of the reference and measurement microphone signal. Based on the spectra Xk and Ef,k for the current frame index k, magnitude and phase characteristics are then calculated.

[0105] The normalized absolute value curve can be used, for example, as a magnitude characteristic. where X and E represent the frequency response of the current STFT frame in the form of a column vector, and the square brackets represent the composition of the corresponding vectors. The division here is to be understood element-wise, i.e., per frequency. Regularizing the denominator to avoid division by zero is advantageous. Such normalization has the benefit that the range of values ​​of the quotient is well-defined between 0 and 1, and that such a division, unlike, for example, a logarithm, is numerically superior.

[0106] For example, the real and imaginary parts of the normalized cross-correlation between X and E can be used as a phase characteristic.

[0107] In this case too, the value range is likely defined between -1 and +1. Normalization can also be achieved using the absolute value |XE*|.

[0108] The magnitude spectra and cross-correlation values ​​can be time-smoothed before being incorporated into the features to reduce noise in the features. The magnitude spectra or cross-correlation values ​​can also be grouped into frequency bands beforehand to reduce the number of features. Such a grouping is also possible after the features have been calculated.

[0109] Another input to the model is the weighting factors from the last journal w( / c - l). These are combined with the magnitude and phase features to form a feature vector. Before combination, each individual feature and weighting factor can be processed by a neural network to obtain modified features.

[0110] The feature vector can then be transformed into an internal state n(Ji) by a deep neural network. This state can contain encoded information about the current fit of a headphone or about the nature of the ambient sound field.

[0111] The internal state can then be converted into a first update vector by two further deep neural networks. and a gate ö(k) is transferred. The first update vector is determined by transformed into a second activation vector. Here, iv describes an estimate of the mean and a wAn estimate of the standard deviation of the optimal weighting factors. By appropriately choosing the initial activation function of the deep neural network to generate ju( / c), for example as the hyperbolic tangent, the range of values ​​of the second activation vector and the coefficients can be restricted, which is beneficial for the stability and robustness of the system.

[0112] As a final step, the coefficients are updated by the following rule: Here, ye(0, 1) describes a decay factor that ensures coefficients currently considered unimportant by the model tend towards 0. This behavior is equivalent to a regularization of the coefficients. The gate ö(k) controls how much new information, in the form of the second activation vector n'(k), flows into the current coefficients. If an element of ö(k) is equal to 0, the corresponding coefficient is not modified, so that, for example, the last state is maintained for y = 1.

[0113] The present invention further comprises the following aspects.

[0114] Aspect 1

[0115] Method for adaptive active noise cancellation for an on-ear or in-ear wearable audio system, wherein

[0116] - a reference signal is generated from a captured ambient sound;

[0117] - a measurement signal is generated from the sound captured in the ear canal of a user of the wearable audio system;

[0118] - a frequency domain transformation of the reference and measurement signals is performed to obtain the reference and measurement spectrum;

[0119] - filtering of the reference signal using certain coefficients to generate a compensation signal, the coefficients being calculated or determined based on the reference and measurement spectrum in such a way as to minimize a measure of the sound pressure level at the user's eardrum;

[0120] - the compensation signal is output.

[0121] Aspect 2

[0122] Method according to aspect 1, wherein the coefficients are calculated by an algorithm whose parameters are optimized offline based on measurement data that include synchronous recordings of the reference sensor, the measurement sensor and an additional microphone on the eardrum.

[0123] Method according to aspect 2, where a short-time Fourier transform is used as the frequency domain transformation.

[0124] Procedure according to aspect 3, wherein

[0125] - the measure of the power of the sound pressure level at the user's eardrum is calculated by transforming the recording of the additional microphone at the eardrum through a short-time Fourier transform to obtain the eardrum spectrum;

[0126] - the eardrum spectrum is divided into discrete frequency bands, the magnitude of which is raised to a power between 0 and 2, and then the mean value of the raised magnitudes is calculated.

[0127] Procedure according to aspect 2, whereby the coefficients are smoothed over time after their update.

[0128] REFERENCE MARK LIST

[0129] Audio system

[0130] Ear insert

[0131] ear canal

[0132] Ventilation opening, first microphone, second microphone

[0133] eardrum

[0134] computing unit

[0135] loudspeaker third microphone inventive method first process step of the inventive method second process step of the inventive method third process step of the inventive method fourth process step of the inventive method training method first step of the training method second step of the training method third step of the training method fourth step of the training method

Claims

REQUIREMENTS 1. Method (100) for adaptive active noise suppression for a portable audio system (10) comprising a microphone unit including at least a first microphone (18) and a loudspeaker (26), a storage unit and a computing unit (24), wherein the method (100) has the following features: Providing (110) evaluation data comprising audio data, wherein the audio data are recorded using the microphone unit, wherein the audio data include at least one first audio signal which is recorded with the first microphone (18) of the microphone unit; Determining (120) filter coefficients depending on the provided evaluation data using the computing unit (24); Generation (130) of filtered audio data from the recorded audio data using the filter coefficients and the computing unit (24); Playback (140) of the filtered audio data using the loudspeaker (26); wherein the determination of the filter coefficients is based on the use of a machine learning-based model; the machine learning-based model is stored in the memory unit; the machine learning-based model is trained with training data, wherein the training data includes training audio data and training filter coefficients associated with the training audio data, wherein the provided evaluation data are input into the machine learning-based model and, depending on the provided evaluation data, filter coefficients are output by the machine learning-based model.

2. Method (100) according to claim 1, characterized in that the method (100) comprises a training process for training the machine learning-based model.

3. Method (200) according to claim 2, wherein the training process comprises the following steps: Providing (210) training audio data comprising at least one initial audio signal; Determine (220) one set of filter coefficients for each of the provided training audio data; wherein the determination of each set of filter coefficients comprises a reduction of a cost function, wherein the cost function preferably describes a sound pressure level in the ear canal (14) of a user; Training (230) the machine learning-based model using the training audio data and the determined filter coefficient sets; and Storing (240) the trained model in a training memory unit.

4. Method (200) according to claim 3, characterized in that the reduction of the cost function comprises a reduction of a cost function that describes the sound pressure level in the external auditory canal (14) of a user.

5. Method (200) according to claim 2, wherein the training process comprises the following steps: Determination of filter coefficients by the model based on the audio data; Simulation of a sound pressure level in the ear canal based on the audio data and the determined filter coefficients; - 40 - Evaluation of a cost function based on the simulated sound pressure level, which correlates with the perceived loudness; Adjusting the model parameters to reduce the cost function.

6. Method (200) according to claim 5, wherein the training process comprises the following steps: Providing training audio data comprising at least an initial audio signal; Determine one set of filter coefficients for each of the provided training audio data; Simulation of a sound pressure level in the ear canal based on the audio data and the determined filter coefficients; Evaluation of a cost function based on the simulated sound pressure level, where the cost function represents the perceived loudness; and Adjustment of the model parameters taking into account the cost function.

7. Method (200) according to one of the preceding claims, characterized in that the first audio signal is provided as an audio signal converted into the frequency domain, wherein the conversion into the frequency domain is preferably carried out via a short-time Fourier transform.

8. Method (200) according to one of the preceding claims, characterized in that the audio data includes a second audio signal which is recorded with a second microphone (20) of the microphone unit.

9. Method (200) according to claim 8, characterized in that the audio data includes a third audio signal which is recorded with a third microphone (28) of the microphone unit.

10. Method (200) according to one of the preceding claims, characterized in that the evaluation data includes at least one transfer function that describes the transmission path between an external noise and the inner ear.

11. Method (200) according to one of the preceding claims, characterized in that the evaluation data includes adaptation data which characterize the adaptation of the audio system to the user's head and / or direction data which characterize the direction of incidence of a disturbance noise.

12. Method (200) according to one of the preceding claims, characterized in that the machine learning-based model comprises an artificial neural network, a support vector machine or a linear regression.

13. Method (200) according to one of the preceding claims, characterized in that the training audio data comprises several hundred, preferably several 1,000, particularly preferably several 10,000 training audio signals.

14. Audio system (10) for adaptive active noise reduction, comprising a microphone unit with at least one first microphone (18), a loudspeaker (26), a storage unit and a computing unit (24), wherein the microphone unit is designed to record evaluation data comprising audio data, wherein the audio data includes at least one first microphone (18). exhibit an audio signal, and the first microphone (18) of the microphone unit is designed to record the first audio signal; the processing unit (24) is designed to determine filter coefficients depending on the provided evaluation data; the processing unit is designed to generate filtered audio data from the recorded audio data using the filter coefficients; the loudspeaker (26) is designed to output the filtered audio data; wherein the processing unit (24) is designed to determine the filter coefficients using a machine learning-based model; the machine learning-based model is stored in the storage unit;the machine learning-based model is trained with training data, wherein the training data includes training audio data and training filter coefficients associated with the training audio data, the computing unit (24) is designed to input the provided evaluation data into the machine learning-based model and, depending on the provided evaluation data, to determine filter coefficients through the machine learning-based model.

15. Audio system (10) according to claim 14, characterized in that the microphone unit has a second microphone (20) designed to record a second audio signal of the evaluation data.

16. Audio system (10) according to claim 15, characterized in that the second microphone (20) is designed to detect sound within an ear canal (14) of a user.

17. Audio system (10) according to one of claims 14 to 16, characterized in that the microphone unit has a third microphone (28) designed to record a third audio signal of the evaluation data.

18. Audio system (10) according to claim 17, characterized in that the third microphone (28) is preferably designed to detect sound outside of an ear canal (14) of a user.

19. Audio system (10) according to one of claims 14 to 18, characterized in that it is a headphone, headset, smartphone or hearing aid.

Images