A system for hearing assistance to a user and associated computer-implemented method

Abstract

A system (20) for hearing assistance to a user, said system comprising: - a microphone array (23, 24) providing audio signal, - at least one hearing device (28), - an eye-tracking device (21) configured to provide user's gaze direction and stabilization data, - a vergence device configured to provide user's vergence data, - a processing unit (26) comprising: an attention estimation module (260) configured to generate attention data of the user representative of a point of attention of the user based on the user's gaze direction and stabilization data and vergence data, an audio beamforming module (262) configured to generate a beamformed audio signal intended to enhance sounds for the user through said at least one hearing device (28) based on the audio signals of the microphone array (23, 24) and the attention data of the user representative of a point of attention of the user.

Description

A system for hearing assistance to a user and associated computer-implemented methodDESCRIPTIONTechnical field

[0001] The invention relates generally to hearing aids, and particularly to systems and methods for improving directional hearing.Backaround information and prior art

[0002] Speech understanding in noisy environments is a significant problem for the hearing- impaired person. Hearing impairment is usually accompanied by a reduced time resolution of the sensorial system in addition to a gain loss. These characteristics further reduce the ability of the hearing-impaired to filter the target sound source from the background noise and particularly to understand speech in noisy environments. It can also be challenging for non-hearing-impaired person.

[0003] Some newer hearing aids offer a directional hearing mode to improve speech intelligibility in noisy environments. This mode makes use of an array of microphones and applies beamforming technology to combine multiple microphone audio inputs into a single, directional audio output channel. Beamforming is a signal processing technique in which multiple channels of audio can be processed to generate a beamformed audio signal in which audio from different directions may be amplified, attenuated or cleaned. Therefore, the audio output channel has spatial characteristics that increase the contribution of audio waves arriving from the target sound source direction which has the user’s attention relative to those of the audio waves from other directions.

[0004] To determine the user’s attention, some existing solutions rely on an eye-tracking system that determines the gaze direction of a user. For example, US2016 / 0080874A1 discloses a hearing assistance system incorporating an eye tracker and microphones which audio outputs are conditioned to obtain an enhanced audio. In crowded environments with multiple stimuli, maintaining control over user attention through gaze alone can be challenging, potentially limiting its effectiveness in improving audio localization.

[0005] Other existing solutions rely on audio systems using head tracking to guide audio beamforming toward a specific target sound source.

[0006] No evidence-based support was provided to suggest that any of the identified alternatives would be more successful in determining the user’s attention.

[0007] Moreover, these systems often face challenges in accurately identifying a user’s point of attention, particularly in dynamic or intricate visual environments.

[0008] One object of the present invention is to enhance the accuracy of the capture of attention of the user, especially in dynamic of intricate visual environments.

[0009] Another object of the present invention is to determine the best combination of sensors to use in different contexts of beamforming.

[0010] Another object of the invention is to improve the accuracy of the selection of the target sound source the user would like to listen to and tailor the beamforming audio experience accordingly, especially in dynamic of intricate visual environments, the user being hearing impaired or not.Summary

[0011] These goals are obtained according to the present disclosure by proposing a system for hearing assistance as set forth in claim 1 , a head mountable device as set forth in claim 10, an eyewear as set forth in clam 11 and a method as set forth in claim 12.

[0012] Further characteristics of the systems, devices and method are the objects of the dependent claims.Description of the drawings

[0013] For a more complete understanding of the description provided herein and the advantages thereof, reference is now made to the brief descriptions below, taken in connection with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0014] FIG. 1 represents is a schematic illustration showing a system for hearing assistance to a user shown as an eyewear;

[0015] FIG. 2 is a schematic illustration of a processing unit implemented within the system of FIG. 1 with its inputs and outputs;

[0016] FIG. 3 is a schematic illustration of another part of the processing unit implemented within the system of FIG.1 ;

[0017] FIG. 4 is a scheme illustrating the main steps of a computer-implemented method for hearing assistance to a user, according to the disclosure.Detailed of embodiments

[0018] In the description which follows the drawing figures are not necessarily to scale and certain features may be shown in generalized or schematic form in the interest of clarity and conciseness or for informational purposes. In addition, although making and using various embodiments are discussed in detail below, it should be appreciated that as described herein are provided many inventive concepts that may be embodied in a wide variety of contexts. Embodiments discussed herein are merely representative and do not limit the scope of the disclosure. It will also be obvious to one skilled in the art that all the technical features that are defined relative to a method can be transposed, individually or in combination, to a device and conversely, all the technical features relative to a device can be transposed, individually or in combination, to a method.

[0019] FIG.1 is a schematic illustration of a hearing assistance system 20 that is at least in part integrated into an eyewear. The eyewear may include at least a frame 22 with a front piece 30 and temples 32 connected to respective edges of the front piece 30.

[0020] In other embodiments, the hearing assistance system 20 can be at least in part integrated in other head mountable devices. Head-mountable devices encompass any type of head mountable device designed to be worn in front of / in at least one eye of a user and / or head mountable hearing device, in any suitable form factor. It includes headmounted display device (e.g. near eye display device, augmented reality devices, virtual reality devices and / or mixed reality devices), helmet, headset, eyewear smart or not (e.g. eyeglasses, googles, masks, visors, clip-on, contact lens, intraocular implant). It includes also earwear (e.g. headphones, earbuds), hearing aid, speakers, bone conduction transducers.

[0021] The hearing assistance system 20 comprises an array of microphones 23, 24 mounted at predetermined distance of each other on the frame 22.

[0022] The microphones 23,24 are configured to detect and capture at least one audio signal from sound sources in the environment of the user. Each microphone 23, 24 in the array of microphones can be of the same type or different types. In particular, all microphones23, 24 in the microphone array may be omnidirectional or directional. A microphone array having one dimension (for example, either the array of microphones 23 or the array of microphones 24) is capable of beamforming in one dimension, while a microphone array in two-dimensions (namely where for example there is both the array of microphones 23and the array of microphones 24) may be capable of beamforming in one or two dimensions.

[0023] In the illustrated example, microphones 23 are mounted on the front piece 30 of the frame 22, while microphones 24 are mounted on temples 32. Although the extensive array of microphones 24 that is shown in FIG. 1 is useful in some applications of the present disclosure, the principles of hearing assistance that are described herein may alternatively be applied, mutatis mutandis, using smaller of higher numbers of microphones. For example, these principles may be applied using an array of microphones 23 on the front piece 30, alternatively or additionally to the array of microphones 24 that are on the temples. In some embodiments, preferably, a single microphone 23 on the front piece 30 and a single microphone 24 on each temple 32 may be used to triangulate in a simple manner the audio signal sources.

[0024] The hearing assistance system 20 comprises one or more speakers 28, which are in the illustrated embodiment mounted on the frame 20 and more precisely on the end of each temple 32, typically in proximity to the user's ears. Although only a single speaker 28 is shown on each temple 32 in FIG. 1, the system 20 may alternatively comprise two or more speakers that can be grouped in arrays mounted on or attached to one or both temples 32. In other embodiments, only one speaker 28 is mounted on or attached to the frame 22.

[0025] Preferably, the speakers 28 are monodirectional or omnidirectional.

[0026] In other embodiments, the system 20 comprises alternative hearing devices 28. Such hearing devices can be headphones, earbuds, headsets or any audio transducers like bone conduction transducers.

[0027] The hearing assistance system 20 also comprises one or more physiological sensor devices.

[0028] The system comprises at least an eye-tracking device including at least an eye tracking sensor 21 ,21’ and an eye tracking module operatively coupled to the sensor to provide gaze direction data and eye movements data of the user’s eyes.

[0029] The eye tracking module comprises at least a processor and a memory.

[0030] The eye tracking sensor can track the eye using any eye tracking technique. In particular, it may comprise imaging systems like cameras including depth camera, LIDAR, time-of-flight sensor, infrared sensor, wavefront sensor and / or a combination thereof.

[0031] In one example, as shown in FIG. 1 , the eye-tracking sensors 21 ,21 ’ are cameras. The cameras 21 ,21 ’ may be embedded or attached to the frame 22 especially in the front piece 30. One single camera 21 ,21 ’ may be sufficient.

[0032] The eye tracking module is configured to determine gaze direction data and eye movements of the user’s eyes based on the eye images (including pupil, corneal and / or light reflection images) and / or videos captured.

[0033] Preferably, the eye tracking module further processes the eye movements and in particular the saccadic movements of the user’s eyes and determine the fixations of the eyes. The module is configured to generate features relative to the stability of the gaze of the user’s eyes. Preferably, it generates features like focus intensity and duration of the gaze on target sound source(s) in the environment the user is looking at.

[0034] In embodiments, the eye tracking module further process the eye movements and in particular the pupil movements i.e. dilation / constriction variation of the eyes to generate features relative to a level of cognitive and emotional engagement of the user on target sound source(s) the user is interested in.

[0035] The eye tracking module uses any algorithm and in particular machine learning based algorithm to process data output from the eye tracking sensors and determine gaze direction vectors, gaze stability vectors as well as user cognitive engagement vectors associated with target sound source(s) the user is interested in.

[0036] In addition, the system comprises at least one vergence sensor and a vergence module operatively coupled to the sensor to provide vergence data of the user’ eyes, including vergence movements.

[0037] Vergence movements align the fovea of each eye of the user with attentional target sound sources located at different distances from the user. Unlike other types of eye movements in which the two eyes move in the same direction, vergence movements are disconjugate i.e. they involve either a convergence or divergence of the lines of sight of each eye to see an object that is nearer or farther away from the user.

[0038] The vergence sensor may be any eyetracking systems already described above. It may be the same eyetracking sensor or an additional one.

[0039] The vergence module comprises at least a processor and a memory.

[0040] The vergence module is configured to determine vergence movements, in particular the direction of each eye as well as the vergence angle of the user’s eyes i.e. the angle of convergence of the lines of sight of the eyes as they focus at variable distances from the user. The vergence module is configured to estimate the 3D distance or depth of attentional target sound source(s) the user is interested in based on these vergence angle data.

[0041] The vergence module uses any algorithm and in particular machine learning based algorithm to process vergence data to output depth of focus of user’s eyes vectors.

[0042] The set of physiological sensor devices may also include at least one additional device configured to provide cortical activity data of the user. This device comprises at least a cortical activity sensor and cortical activity module operatively coupled to the sensor.

[0043] The cortical activity module comprises at least one processor and a memory.

[0044] Cortical activity signals captured by the sensor can be characterized by their frequency, amplitude and phase and can give rise electrical impulses indicative of brain activity pattern. The signal properties can be analyzed using time frequency analysis.

[0045] The cortical activity sensor(s) may comprise electroencephalography (EEG) sensor and / or magnetoencephalography MEG sensor to measure cortical activity signals.

[0046] In one example shown in FIG.1 the cortical activity sensor comprises at least a pair of electrodes 29,29’ configured to detect at least one EEG response signal to a scene / target sound source(s) viewed and listened by the user.

[0047] The electrodes 29,29’ are arranged at a predetermined distance to each other and at least partially in contact with the user’s head when the user wears the eyewear.

[0048] Preferably, the electrodes 29,29’ are temporal electrodes placed in temporal regions of the user ‘s head to detect in a simple and accurate way the EEG response signals.

[0049] The electrodes 29,29’ can be embedded in the frame 20 or attached to said frame 20.

[0050] The cortical activity module processes the cortical activity signals to extract features in different frequency bands (e.g. alpha, beta and gamma oscillations) including features related to amplitude variations, spectral power densities, power ratio, and event-related potentials, in particular Visual Evoked Potential (VEP) and Auditory Evoked Potential (AEP).

[0051] In an embodiment, the cortical activity signals are filtered to remove artifacts and generate useful signals to be processed to extract features.

[0052] Based on the features extracted, the cortical activity module is configured to infer cortical activity patterns associated to several states representative of user’s level of engagement with attentional target sound source(s) in his environment.

[0053] For example, the states may be whether the user is focusing intently on a single target sound source (concentration, sustained attention) or switching focus between various sound sources (selective attention or distraction). It may also encompass any emotional state like whether the user is relaxed, stressed, excited, sad or not.

[0054] The cortical activity module uses any algorithm and in particular machine learning based algorithm to process cortical activity signal data to output user’s level of engagement with target sound source(s) in his environment.

[0055] The set of physiological sensor devices may also include an additional head device configured to provide head data of the user. The device comprises a head tracking sensor and a head tracking module operatively coupled to the sensor.

[0056] The head data can be head movement (e.g. head fixed, tilted, up / down / tilt, left movement, right movement) or head orientation (e.g. pitch, yaw, and roll) and / or position.

[0057] The head tracking module comprises a processor and a memory.

[0058] The head tracking sensor may comprise an inertial measurement sensor, an acceleration sensor, a gyroscope, a depth camera or a combination thereof.

[0059] In another embodiment, the head data and in particular head orientation data may be inferred based on the geometry of the microphone array to infer the head orientation data.

[0060] The head tracking module process outputs signals from the head tracking sensor to extract features related to the user's physical orientation and movement in the environment, providing context to the direction and range of potential attentional targets sound sources the user is looking at.

[0061] The module uses any algorithm and in particular machine learning based algorithm to process head data to output user's physical orientation and / or movement vectors representative of the attentional targets sound sources the user is looking at.

[0062] In one embodiment, head orientation data may also be used to optimize depth of focus estimation.

[0063] In embodiments, the set of physiological sensor devices may also include a device to provide context information relative to the environment of the user. For example, the context information may include the objects and the person's location in the environment, the type of sound source the user is interested in (e.g. person speaking, conversation, device, etc..). Preferably, the context information device may include at least a camera.

[0064] A context information module of the device uses any algorithm and in particular machine learning based algorithm to process context information detected and provide context information vector.

[0065] In embodiments, the set of physiological sensor devices may also include heart rate sensor, pulse rate sensor, temperature sensor, blood pressure sensor, ambient light sensor to collect additional vectors representative of the user’ level of engagement on attentional targets sound sources in his environment.

[0066] The sensors can be embedded or attached in the frame or be part of a wearable device on any device in the user’s environment.

[0067] In embodiments, all the sensors described can provide signals in real time. In other embodiments, the sensors provide signals based on an event triggered by the user or triggered by a predetermined threshold value.

[0068] In addition, the system 20 comprises a processing unit 26.

[0069] The processing unit 26 comprises at least one memory to store data and computer executable instructions and at least one processor.

[0070] As illustrated in the embodiment of FIG.1 the processing unit is embedded in or attached to at least one of the temples of the frame 22. In other embodiments, the processing unit is embedded in the front piece 30 of the frame 22.

[0071] The processing unit 26 is configured to identify the user’s attention state based on data gathered from at least the eye tracking device and the vergence device.

[0072] In addition, the processing unit 26 is configured to enhance the audio signals from the array of microphones by applying a beamforming function based on the user’s attention state estimated. For example, the processing unit emphasizes the sounds that originate from the attentional target sound source the user is looking at. while suppressing background noise originating outside this target sound source.

[0073] The processing unit 26 is described in greater detail hereinbelow, with the aid of FIG.2.

[0074] As shown in FIG.2, the processing unit 26 comprises an attention estimation module 260.

[0075] The attention estimation module 260 is configured to generate attention data of the user representative of a point of attention of the user. Attention data are at least the 3D localization of the target sound source(s) the user is interested in and the associated level of attentional engagement of the user.

[0076] In embodiments, the attention estimation module 260 integrates and processes at least as inputs gaze direction and head data and depth of focus data determined from vergence data. Based on these information, the attention estimation device 260 is configured to apply a corresponding weight to the gaze direction data, head data and to the depth of focus data, in order to generate the attention data of the user representative of a point of attention of the user.

[0077] Preferably, the head data are head orientation data.

[0078] Moreover, the attention estimation module 260 is configured to process the vergence data to decide which weight to apply respectively to the gaze direction vector and to the head vector. The vergence data taken into account in the attention estimation module helps to identify the best combination of sensors based on the region of interest the user is looking at. For example, in specific situations, the head vector may have a higher weight than the gaze vector or the contrary.

[0079] Preferably, context information data are used to smooth gaze direction and head vectors to adjust these vectors to point at the closest object and / or person of interest within the environment of the user.

[0080] In other embodiments, user level of engagement data defined from cortical activity data are added to module 260 with a corresponding weight to generate the attention data of the user representative of a point of attention of the user. Adding cortical activity data allows to optimize the identification of the level of engagement of the user on target sound source(s).

[0081] Preferably, gaze stability data are also considered to generate attention data., The module 260 is configured to receive said gaze stability data and to apply a corresponding weight to the gaze stability data.

[0082] The module 260 may consider further physiological data or context information data if needed, as a function of the type of sensors implemented.

[0083] The processing unit determines attention data of the user representative of a point of attention of the user using any model-based algorithm, and in particular machine learningbased algorithm.

[0084] In one embodiment, the attention estimation module 260 comprises a neural network.Gaze direction vectors, user orientation and / or movement in space vector defined from head data and the depth of focus vectors are joined as inputs in a neural network, fused in the network whose output is the attention vector representative of the point of attention of the user.

[0085] In another embodiment, an additional input corresponding to the user level of engagement vector defined from cortical activity data is added to the set of input vectors in the network.

[0086] All these data are combined to infer with accuracy the attention of the user in his environment.

[0087] In other embodiments, an additional input corresponding to the gaze stability vector is added to the set of input vectors in the network.

[0088] The module 260 may consider further data if needed, in particular context information vector, pupil data, user’s level of fatigue thanks to eyetracking monitoring.

[0089] As also shown in FIG. 2, the processing unit 26 comprises an audio beamforming module 262. The audio beamforming module 262 is configured to receive audio signals from the microphone array 23, 24 as well as attention data generated by the attention estimation module 260. Based on the attention data and the audio signals, the audio beamforming module 262 generates at least beamformed audio signal intended to enhance sounds for the user through said at least one speaker 28. For that, beamformedsignal outputting from the beamforming module 262 is directed towards a drive signal module 264, typically comprising a numerical - analogic converter and filters, which drive signal (output) is sent to said at least one speaker 28.

[0090] The audio beamforming module 262 is described in greater detail hereinbelow.

[0091] In one embodiment, the audio beamforming module 262 is configured to select a set of beamforming processing settings to perform with respect to the audio signals captured by the microphones. In particular, the audio beamforming module 262 comprises a beamforming model that receives as inputs at least the attention vector and features associated to the audio signals captured from the microphones. The model identifies as output a selection of beamforming processing settings to apply to the signals to spatially filter the audio signal inputs.

[0092] In several embodiments, the settings may include dynamic noise cancellation algorithms to isolate attentional target sound sources and / or real time 3d spatial audio algorithms to localize and enhance multiple target sound sources simultaneously.

[0093] For example, the settings may include parameters relative to beamforming directionality and / or intensity to apply to the audio signals associated with the target sound source’s), to the amplification of sounds from the point of attention of the user and / or the suppression of background noise and unrelated audio sources. In another example, it may relate to a general enhancement of auditive acuity of people suffering hearing loss and / or the decision to broaden the beamforming to help the user to listen to the whole environment. All the settings enhance the listening experience in alignment with the user's attention determined.

[0094] The audio beamforming module 262 uses any algorithm and in particular machine learning based algorithm to process data to output the beamforming processing settings.

[0095] In one embodiment, it uses a deep neural network trained to map attention vectors to optimal beamforming settings. For example, this network determine the directionality and intensity of the beamforming focus. The audio signals captured by the microphones are then processed using the beamforming processing settings identified.

[0096] Preferably, the audio signal processing is made in real time as the user’s attention shifts or auditory conditions of the environment change.

[0097] Within the beamforming module 262, any beamforming function known in art may be considered. For example, a time delay algorithm may be used to combine the audio signals from the microphones 23, 24, with time shifts between the signals that are equal to the propagation times of the acoustic waves between the microphone locations with respect to the desired beam direction and to the location of the speaker 28. Alternatively, a Minimum Variance Distortionless Response (MVDR) beamforming algorithm may beapplied in order to achieve better spatial resolution. Other applicable beamforming techniques are based on Linear Constraint Minimum Variance (LCMV) and General Sidelobe Canceller (GSC) algorithms.

[0098] FIG.3 shows a part of the processing unit 26 comprising a beamforming activation device 50.

[0099] The hearing assistance system 20 may comprise an ON / OFF activation device 50 for the audio beamforming module 262. The user has therefore the choice to activate or deactivate the beamforming processing of audio signals.

[0100] Alternatively, or additionally, the processing unit 26 may comprise an activation device 51 for the audio beamforming module 262 configured to:- receive at least gaze direction data of the user and / or- receive at least vergence angle data of the user and / or- receive at least the head orientation data- activate the audio beamforming module when gaze direction data are above a first preset threshold value THR1 and / or the vergence angle data are above a second preset threshold value THR2, and / or head orientation data are above a fourth present threshold value THR4.

[0101] Where cortical activity data are provided, the activation device 51 is also configured to receive cortical activity data of the user and to activate the audio beamforming module 262 where, additionally, said cortical activity data are above a third preset threshold value THR3.

[0102] The system 20, the processing unit 26, the different devices and modules may use diverse techniques and models to carry out the determination and estimation described above. In particular, they can use any algorithm to determine the outputs based on the inputs provided. The algorithms may include fuzzy logic, classification models, clustering algorithms, data regressions algorithms, image segmentation algorithms, Bayesian belief networks, data fusion engines machine learning algorithms like Support Vector Machines SVMs ( linear, cubic, ..), neural network like Medium Neural Network (MNN), wide Neural Network (WNN), recurrent neural networks (RNNs) like long short term memory (LSTM), deep learning algorithms, convolutional neural network (CNN) and / or Canonical Correlation Analysis (CCA) algorithm and so forth.

[0103] In alternative embodiments, all the modules described above may be implemented in the processing unit 26.

[0104] In other embodiments, the processing unit 26 can be at least part of any external computing device or can be a virtual machine located on a cloud or edge server. All the devices or modules of the processing unit 26 described previously do not need to be allimplemented within the processing unit 26. Indeed, to optimize the memory and power capabilities, at least some of the devices / modules may be implemented in an external computing device communicating with the devices present in the frame through a network.

[0105] Computing devices may include mobile devices (e.g., smartphones, tablet computers.) and / or wearable devices (e.g., smartwatches), laptops, server computers and / or head-mounted display devices.

[0106] The network can be a physical or wireless connection, such as Bluetooth®, Wi-Fi®, a GSM network or any global positioning system (GPS), private data network, near field communications (NFC), LAN or WAN network.

[0107] Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, systems on a chip (SoC), baseband processors, field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionality described throughout this disclosure.

[0108] Memory refers to refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. Memory may include a random-access memory (RAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, combinations of the aforementioned types of computer- readable media, or any other medium that can be used to store computer executable code in the form of instructions or data structures that can be accessed by the different modules and processing units.Computer implemented method for hearing assistance to a user

[0109] FIG.4. is a schematic diagram of the main steps of a computer implemented method 100 for hearing assistance to a user according to the disclosure.

[0110] In particular, the method 100 can be implemented by means of a computer program or software loaded into the memory of the processing unit 26. Such a computer program thus comprises instructions that induce the processing unit 26 to implement the method 100 when the processing unit runs the program.

[0111] The method 100 comprises the steps of:- receiving 110 at least an audio signal captured with a microphone array,- providing 120 gaze direction of the user,- providing 130 vergence data of the user,- providing head data of the user,- determining 150 attention data of the user representative of a point of attention of the user within his environment,- generating 160 a beamformed audio signal from features associated with the audio signal and from the attention data,- broadcasting 170 the beamformed audio signal to the user through a hearing device.

[0112] Preferably, the head data are head orientation data.

[0113] In step 110, the processing unit 26 determines also features associated with the audio signals. For example, the features can be the intensity, and the power of the audio signals received from each microphone and / or the direction of arrival of the audio signals.

[0114] In some embodiments, in step 150, the method determines the attention data by using a model. More precisely, the method comprises steps of- inputting at least a gaze direction vector, head vector and a depth of focus vector in a model and- outputting at least an attention vector representative of the attention of the user. Preferably, the model is a convolutional neural network CNN.Preferably, the head data are head orientation data.

[0115] In some embodiments, the computer implemented method may comprise before step 150:- a step 140 where cortical activity data are provided,- a step 141 where a user level of engagement data is determined from cortical activity data.

[0116] In step 140, cortical activity signals are collected, and the cortical activity module determines features associated with the cortical activity signals. For example, the features can be the signal amplitude and power. A step of applying a frequency domain filtering to the cortical activity signal may be implemented to select useful signals about user’s attention.

[0117] In these embodiments, in step 150, the method determines the attention data by using a model. More precisely, the method comprises steps of:- inputting at least a gaze direction vector, a head vector, a depth of focus vector and a user level of engagement vector in a model and- outputting at least an attention vector representative of the attention of the user. Preferably, the model is a convolutional neural network CNN.Preferably, the head vector is a head orientation vector.

[0118] In step 160, the method comprises the steps of selecting a set of beamforming processing settings to perform with respect to the audio signals captured by the microphones by using a model. More precisely, the method comprises steps of :Inputting at least the attention vector and features associated to the audio signals captured from the microphones- outputting a set of beamforming processing settings to apply to the signals to spatially filter the audio signal inputs.Preferably, the model is a deep learning network.

[0119] In other embodiments, the method comprises a step of activating the audio beamforming experience when gaze direction data and / or vergence data and or cortical activity data determined respectively in steps 120, 130 and 140 are above a respective first preset threshold value THR1 , a second preset threshold value THR2 and / or a preset threshold value THR3.

[0120] In some embodiments, the method comprises a step of providing feedback about the beamforming processing settings selected by determining a user’s level of satisfaction of the audio signals broadcasted.

[0121] In embodiments, the level of satisfaction may be computed from the cortical activity data and / or eye tracking data like pupil dilation data to define the level of satisfaction of the user. Any method to determine the level of satisfaction can be used. In other embodiments, a feedback answer from the user by voice command and / or user movement is provided.

[0122] In embodiments, If the level of satisfaction is below a threshold THR5, the method comprises a step to determine again the attention data in step 150 and / or to select new beamforming processing settings in step 160.

[0123] Although representative methods, devices and modules have been described in detail herein, those skilled in the art will recognize that various substitutions and modifications may be made without departing from the scope of what is described and defined by the appended claims.

Claims

CLAIMS1. A system (20) for hearing assistance to a user, said system comprising:- a microphone array (23, 24) providing audio signals,- at least one hearing device (28),- an eye-tracking device (21) configured to provide user’s gaze direction,- a vergence device (21 ) configured to provide user’s vergence data,- a head tracking device configured to provide head data,- a processing unit (26) comprising:- an attention estimation module (260) configured to generate attention data of the user representative of a point of attention of the user based on the user’s gaze direction, head data and vergence data,- an audio beamforming module (262) configured to generate a beamformed audio signal intended to enhance sounds for the user through said at least one hearing device (28) based on the audio signals and the attention data of the user representative of a point of attention of the user.

2. System according to claim 1 further comprising a cortical activity device (29,29’) configured to provide cortical activity data of the user, the attention estimation module (260) being configured to generate attention data based on the cortical activity data provided.

3. System according to claim 2, wherein the cortical activity device (29,29’) is configured to determine user level of engagement data within his environment.

4. System according to one of claims 1 to 3 further comprising a device configured to provide gaze stability data of the user, the attention estimation module (260) being configured to generate attention data based on the gaze stability data provided.

5. System according to one of the preceding claims wherein the head data comprise head orientation data.

6. System according to one of the preceding claims wherein the audio beamforming module (262) comprises a beamforming model configured to receive as inputs at least the attention data and audio signals of the microphone array (23, 24) and to determine as output a selection of beamforming processing settings to apply to the audio signals.

7. System according to claim 1 wherein the attention estimation module (260) comprises a model configured to receive as inputs the user’s gaze direction, the head data and the vergence data and to determine as output attention data of the user representative of a point of attention of the user.

8. System according to one of the preceding claims, comprising an activation device (50,) for the audio beamforming module (262), said activation device (50) being configured to:- receive gaze direction data of the user,- receive at least vergence data of the user,- activate the audio beamforming module (262) when gaze direction data are above a first preset threshold (THR1) and / or vergence data are above a second preset threshold (THR2).

9. System according to claim 8 wherein the activation device (50) is also configured to receive cortical activity data of the user and to activate the audio beamforming device where, additionally, said cortical activity data are above a third preset threshold (THR3).

10. Head mountable device for hearing assistance to a user comprising:- a microphone array (23, 24) providing audio signals,- at least one hearing device (28),- an eye-tracking device (21) configured to provide user’s gaze direction data,- a vergence device configured to provide user’s vergence data,- a head tracking device configured to provide user’s head data,- a processing unit (26) comprising:- an attention estimation module (260) configured to generate attention data of the user representative of a point of attention of the user based on the user’s gaze direction data, head data and vergence data,- an audio beamforming module (262) configured to generate a beamformed audio signal intended to enhance sounds for the user through said at least one hearing device (28) based on the audio signals and the attention data of the user representative of a point of attention of the user.

11. Eyewear for hearing assistance to a user comprising:- a microphone array (23, 24) providing audio signals,- at least one hearing device (28),17- an eye-tracking device (21) configured to provide user’s gaze direction and stabilization data,- a vergence device configured to provide user’s vergence data,- a head tracking device configured to provide user’s head data- a processing unit (26) comprising:- an attention estimation module (260) configured to generate attention data of the user representative of a point of attention of the user based on the user’s gaze direction data, head data and vergence data,- an audio beamforming module (262) configured to generate a beamformed audio signal intended to enhance sounds for the user through said at least one hearing device (28) based on the audio signals and the attention data of the user representative of a point of attention of the user.

12. A computer implemented method for hearing assistance to a user, said method comprising the steps of:- receiving (110) at least an audio signal captured with a microphone array,- providing (120) gaze direction data of the user,- providing (130) vergence data of the user,- providing head data of the user,- determining (150) attention data of the user representative of a point of attention of the user within his environment based on the user’s gaze direction data, head data and vergence data,- generating (160) a beamformed audio signal from the audio signal and the attention data,- delivering (170) the beamformed audio signal to the user through a hearing device.

13. A computer implemented method according to claim 12, comprising the following steps:- providing (140) cortical activity data of the user,- determining (150) attention data of the user representative of a point of attention of the user within his / her environment based on the cortical activity data.

14. A computer implemented method according to one of the claims 12 to 13, wherein steps 150, 160 and / or 170 are implemented with machine learning based algorithms.

15. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by at least one processor, cause the processor to perform operations comprising:- receiving (110) at least an audio signal captured with a microphone array,- providing (120) gaze direction data of the user,- providing (130) vergence data of the user,- providing head data of the user,- determining (150) attention data of the user representative of a point of attention of the user within his environment based on the user’s gaze direction data, head data and vergence data,- generating (160) a beamformed audio signal from the audio signal and the attention data,- delivering (170) the beamformed audio signal to the user through a hearing device.

Images