Input adjustment

Abstract

Presented herein are techniques for delivering stimulation signals to a user of a hearing device system to evoke perception of modified speech. Sound signals are received that include speech. The received speech comprises auditory comprehension attributes. The sounds signals are converted to stimulation signals for delivery to a user of a hearing device system. During the converting, one or more of the auditory comprehension attributes associated with the received speech are modified to generate modified speech that has the same meaning as the received speech. The stimulation signals are delivered to the user to evoke perception of the modified speech.

Description

Atty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1INPUT ADJUSTMENT BACKGROUNDField of the Invention[oooi] The present invention relates generally to adjusting inputs signals, such as sound signals, in a medical device.Related Art

[0002] Medical devices are devices that are intended to be used for medical purposes. They can vary in both their intended use and indications for use. Examples range from simple, low-risk medical supplies to complex, potentially high-risk devices that are implanted and / or sustain life, such as deep brain stimulators and brain-computer interfaces. Other categories of medical device include diagnostic equipment.

[0003] Hearing devices act on an actual or potential auditory perception of an individual, including to improve perception of sound signals, to reduce perception of sound signals, etc. In particular, a hearing device can deliver sound signals to a user in any form, including in the form of acoustical stimulation, mechanical stimulation, electrical stimulation, etc., and / or can operate to suppress all or some sound signals. As such, a hearing device can be a device for use by a hearing-impaired person (e.g., hearing aids, middle ear auditory prostheses, bone conduction devices, direct acoustic stimulators, electro-acoustic hearing prostheses, auditory brainstem stimulators, bimodal hearing prostheses, bilateral hearing prostheses, dedicated tinnitus therapy devices, tinnitus therapy devices, etc.) or a device for use by a person with normal hearing (e.g., a consumer device that provides audio streaming, a consumer headphone, an earphone, etc.), a hearing protection device (e.g., a noise cancellation headset, a loudness reduction apparatus, etc.), etc.SUMMARY

[0004] In one aspect, a method is provided. The method comprises: receiving, at a device associated with a user, one or more audio signals that include speech; processing the one or more audio signals to adjust at least one of a wording or inflection of the speech to produce modified speech that has a same meaning as the speech; and delivering stimulation signals to the user to evoke perception of the modified speech.

[0005] In another aspect, another method is provided. The method comprises: receiving sound signals that include speech, wherein the speech comprises auditory comprehension attributes;Atty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1converting the sound signals to stimulation signals for delivery to a user; during the converting, modifying one or more of the auditory comprehension attributes associated with the speech to generate modified speech; and delivering the stimulation signals to the user to evoke perception of the modified speech.

[0006] In another aspect, a system is provided. The system comprises: a memory, and at least one processor operable coupled to the memory, wherein the at least one processor is configured to: receive sound signals that include received speech, wherein the received speech comprises auditory comprehension attributes; during the converting, modify one or more of the auditory comprehension attributes associated with the received speech to generate modified speech having, relative to the received speech, at least one of a decreased associated cognitive load or an increased engagement level; using the modified speech to generate stimulation signals for delivery to a user; and deliver the stimulation signals to the user.

[0007] In another aspect, one or more non-transitory computer readable storage media including instructions are provided that, when executed by a processor, cause the processor to: receive sound signals that include received speech, wherein the received speech comprises auditory comprehension attributes; convert the sound signals to stimulation signals for delivery to a user; during the converting, modify one or more of the auditory comprehension attributes associated with the received speech to generate modified speech, wherein the modified speech has a same meaning as the received speech; and deliver the stimulation signals to the user to evoke perception of the modified speech.

[0008] In another aspect, a method is provided. The method comprises: receiving, at a device associated with a user, one or more audio signals that include speech; modifying, using a machine learning model, the speech based on an auditory cognitive capacity or preference associated with the user to produce modified speech; and delivering stimulation signals to the user to evoke perception of the modified speech.

[0009] In another aspect, a method is provided. The method comprises: receiving, at a sensory device, spatial signals that include target content, wherein the target content comprises comprehension attributes; converting the spatial signals to stimulation signals for delivery to a user of the sensory device; during the converting, modifying one or more of the comprehension attributes associated with the target content to generate modified target content, wherein the modified target content has a same meaning as the modified target content; and delivering the stimulation signals to the user to evoke perception of the modified target content.Atty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Embodiments of the present invention are described herein in conjunction with the accompanying drawings, in which:

[0011] FIG. 1A is a schematic diagram illustrating a cochlear implant system with which aspects of the techniques presented herein can be implemented;

[0012] FIG. IB is a side view of a user wearing a sound processing unit of the cochlear implant system of FIG. 1A;

[0013] FIG. 1C is a schematic view of components of the cochlear implant system of FIG. 1 A;

[0014] FIG. ID is a block diagram of the cochlear implant system of FIG. 1 A;

[0015] FIG. IE is a schematic diagram illustrating a computing device with which aspects of the techniques presented herein can be implemented;

[0016] FIG. 2 is a block diagram illustrating a portion of a hearing device system with which aspects of the techniques presented herein can be implemented;

[0017] FIG. 3 is a block diagram of a machine learning model with which aspects of the techniques presented herein can be implemented;

[0018] FIG. 4A illustrates an exemplary user interface of an audio transformation application, according to techniques presented herein;

[0019] FIG. 4B illustrates a user interface that shows an audio simplification log, according to techniques presented herein;

[0020] FIG. 5 is a flow chart that illustrates a method of delivering stimulation signals to a user to evoke perception of speech that is modified to adjust a wording or inflection of received speech, according to techniques presented herein;

[0021] FIG. 6 is a flow chart of a method of delivering stimulation signals to a user to evoke perception of speech with modified auditory comprehension attributes, according to techniques presented herein;

[0022] FIG. 7 is a schematic diagram illustrating a vestibular stimulator system with which aspects of the techniques presented herein can be implemented;

[0023] FIG. 8 is a schematic diagram illustrating a retinal prosthesis system with which aspects of the techniques presented herein can be implemented;Atty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1

[0024] FIG. 9 is a schematic diagram illustrating a tinnitus therapy device with which aspects of the techniques presented herein can be implemented; and

[0025] FIG. 10 is a perspective view of an upper airway stimulation device with which aspects of the techniques presented herein can be implemented.DETAILED DESCRIPTION

[0026] Presented herein are techniques for signal processing in user devices, such a hearing devices. According to embodiments described herein, incoming audio signals that include speech are received (e.g., from a person talking, a television streamer, a music player, etc.). The audio signals (e.g., speech) is analyzed using a model, such as a machine learning model, and adjustments / modifications are made to the speech without changing the meaning of the speech. The modified / transformed speech is delivered to the device user.

[0027] In certain embodiments, the speech adjustments / modifications are configured (made in a manner) to reduce the auditory cognitive load associated with the user successfully comprehending the speech or message in auditory form through a hearing device. In other embodiments, the adjustments are configured to, for example, increase humor or engagement, adjust a tone or context associated with the speech, perform a translation, or adjust the speech in another manner. The user of the device can control how the speech is adjusted using, for example, an application on a user device (e.g., a smartphone). For example, the user can choose to simplify the speech, increase humor associated with the speech, increase engagement, etc., by making a selection in the application. The modifications to the speech are configured to increase the user’s comprehension of the speech.

[0028] According to techniques described herein, the speech adjustment can be made based on factors associated with the device user. For example, the adjustment can be made based on factors such as an age of the user, an amount of time since the user received the device, a cognitive ability associated with the user, an auditory cognitive capacity or preference associated with the user, and / or additional factors.

[0029] There are a number of different types of devices in / with which the techniques presented herein can be implemented. Merely for ease of description, the techniques presented herein are primarily described with reference to a specific device, namely a hearing device in the form of a cochlear implant. However, it is to be appreciated that the techniques presented herein can also be partially or fully implemented by any of a number of different types of devices or systems, including consumer electronic devices (e.g., consumer hearing devices, consumerAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1computing devices such as mobile phones and tablets, audio equipment such as home theatre and car audio systems, etc.), computing systems (e.g., servers in data centers, Internet-of-Things (loT) devices), various types of software systems, such as databases, machine learning and artificial intelligence systems, other medical devices, such as diagnostic equipment or life sustaining equipment, etc. For example, the techniques presented herein could be used in or with sensory protheses, including hearing aids and cochlear implants, and various medical devices, such as pacemakers, drug delivery systems, implantable defibrillators, functional electrical stimulation devices, sleep disorder devices (e.g., sleep apnea devices), seizure devices (e.g., devices for monitoring and / or treating epileptic events), balance or movement disorder devices (e.g., vestibular stimulation devices), tinnitus management devices, visual implants (e.g., bionic eyes), and other neuromodulation devices (e.g., brain-computer interfaces).

[0030] FIGs. 1 A-1D illustrate an example cochlear implant system 102 with which aspects of the techniques presented herein can be implemented. The cochlear implant system 102 comprises an external component 104 that is configured to be directly or indirectly attached to the body of the user, and an internal / implantable component 112 that is configured to be implanted in or worn on the head of the user. In the examples of FIGs. 1 A-1D, the implantable component 112 is sometimes referred to as a “cochlear implant.” FIG. 1A illustrates the implantable component 112 implanted in the head 154 of a user, while FIG. IB is a schematic drawing of the external component 104 worn on the head 154 of the user. FIG. 1C is another schematic view of the cochlear implant system 102, while FIG. ID illustrates further details of the cochlear implant system 102. For ease of description, FIGs. 1A-1D will generally be described together.

[0031] In the examples of FIGs. 1A-1D, the external component 104 comprises a sound processing unit 106, an external coil 108, and generally, a magnet system (e.g., one or more magnets) 150 fixed relative to the external coil 108. The implantable component 112 includes an implantable coil 114, an implant body 134, and an elongate stimulating assembly 116 configured to be implanted in the user’s cochlea. In one example, the sound processing unit 106 is an off-the-ear (OTE) sound processing unit, sometimes referred to herein as an OTE component, which is configured to send data and power to the implantable component 112. In general, an OTE sound processing unit is a component having a generally cylindrically shaped housing 111 and which is configured to be magnetically coupled to the user’s head 154 (e.g., includes an integrated external magnet 150 configured to be magnetically coupled to anAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1internal / implantable magnet system 152 in the implantable component 112). The OTE sound processing unit 106 also includes an integrated external (headpiece) coil 108 (the external coil 108) that is configured to be inductively coupled to the implantable coil 114.

[0032] It is to be appreciated that the OTE sound processing unit 106 is merely illustrative of the external devices that could operate with implantable component 112. For example, in alternative examples, the external component 104 can comprise a behind-the-ear (BTE) sound processing unit configured to be attached to, and worn adjacent to, the user’s ear. A BTE sound processing unit comprises a housing that is shaped to be worn on the outer ear of the user. In certain examples, the BTE is connected to a separate external coil assembly via a cable, where the external coil assembly is configured to be magnetically and inductively coupled to the implantable coil 114, while in other embodiments the BTE includes a coil disposed in or on the housing worn on the outer ear of the user. It is also to be appreciated that alternative external components could be located in the user’s ear canal, worn on the body, etc. For example, the external component could be a micro-BTE unit, an in-the-ear (ITE) unit, etc. In addition, as described elsewhere herein, the techniques could be implemented without an external component (e.g., in a totally-implantable arrangement, a mostly-implantable arrangement, etc.).

[0033] Although the cochlear implant system 102 includes the sound processing unit 106 and the implantable component 112, as described below, the implantable component 112 can operate independently from the sound processing unit 106, for at least a period, to stimulate the user. For example, the implantable component 112 can operate in a first general mode, sometimes referred to as an “external hearing mode,” in which the sound processing unit 106 captures sound signals which are then used as the basis for delivering stimulation signals to the user. The implantable component 112 can also operate in a second general mode, sometimes referred as an “invisible hearing” mode, in which the sound processing unit 106 is unable to provide sound signals to the implantable component 112 (e.g., the sound processing unit 106 is not present, the sound processing unit 106 is powered-off, the sound processing unit 106 is malfunctioning, etc.). As such, in the invisible hearing mode, the implantable component 112 captures sound signals itself via implantable sound sensors and then uses those sound signals as the basis for delivering stimulation signals to the user. Further details regarding operation of the implantable component 112 in the external hearing mode are provided below, followed by details regarding operation of the implantable component 112 in the invisible hearing mode. It is to be appreciated that reference to the external hearing mode and the invisible hearing modeAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1is merely illustrative and that the implantable component 112 could also operate in alternative modes.

[0034] In FIGs. 1 A and 1C, the cochlear implant system 102 is shown with an external device 110, configured to implement aspects of the techniques presented. The external device 110, which is shown in greater detail in FIG. IE, is a computing device, such as a personal computer (e.g., laptop, desktop, tablet), a mobile phone (e.g., smartphone), a remote control unit etc. The external device 110 and the cochlear implant system 102 (e.g., sound processing unit 106 or the implantable component 112) wirelessly communicate via a bi-directional communication link 126. The bi-directional communication link 126 can comprise, for example, a short-range communication, such as Bluetooth link, Bluetooth Low Energy (BLE) link, a proprietary link, etc.

[0035] Returning to the example of FIGs. 1 A-1D, the sound processing unit 106 of the external component 104 also comprises one or more input devices configured to capture and / or receive input signals (e.g., sound or data signals) at the sound processing unit 106. The one or more input devices include, for example, one or more sound input devices 118 (e.g., one or more external microphones, audio input ports, telecoils, etc.), one or more auxiliary input devices 128 (e.g., audio ports, such as a Direct Audio Input (DAI), data ports, such as a Universal Serial Bus (USB) port, cable port, etc.), and a short-range wireless transmitter / receiver (wireless transceiver) 120 (e.g., for communication with the external device 110), each located in, on or near the sound processing unit 106. However, it is to be appreciated that one or more input devices can include additional types of input devices and / or less input devices (e.g., the short-range wireless transceiver 120 and / or one or more auxiliary input devices 128 could be omitted).

[0036] The sound processing unit 106 also comprises the external coil 108, a charging coil 130, a closely-coupled radio frequency transmitter / receiver (RF transceiver) 122, at least one rechargeable battery 132, and an external sound processing module 124. The external sound processing module 124 can be configured to perform a number of operations that are represented in FIG. ID by a speech transformer 131, an output signal generator 133, and an audio transcriber (transcription module) 135. Each of the speech transformer 131, the output signal generator 133, and the audio transcriber 135 can be formed by one or more processors (e.g., one or more Digital Signal Processors (DSPs), one or more uC cores, etc.), firmware, software, etc. arranged to perform operations described herein. That is, the speech transformer 131, the output signal generator 133, and the audio transcriber 135 can each be implementedAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1as firmware elements, partially or fully implemented with digital logic gates in one or more application-specific integrated circuits (ASICs), partially or fully in software, etc. Although FIG. ID illustrates the speech transformer 131, an output signal generator 133, and the audio transcriber 135 as being implemented / performed at the external sound processing module 124, it is to be appreciated that these elements (e.g., functional operations) could also or alternatively be implemented / performed as part of the implantable sound processing module 158, as part of the external device 110, etc.

[0037] Returning to the example of FIGs. 1A-1D, the implantable component 112 comprises an implant body (main module) 134, a lead region 136, and the stimulating assembly 116, all configured to be implanted under the skin (tissue) 115 of the user. The implant body 134 generally comprises a hermetically-sealed housing 138 that includes, in certain examples, at least one power source 125 (e.g., one or more batteries, one or more capacitors, etc.), in which the RF interface circuitry 140 and a stimulator unit 142 are disposed. The implant body 134 also includes the internal / implantable coil 114 that is generally external to the housing 138, but which is connected to the RF interface circuitry 140 via a hermetic feedthrough (not shown in FIG. ID).

[0038] As described further below, at least the implantable coil 114 is disposed in a biocompatible encapsulation layer (encapsulation) 101. The encapsulation layer 101 includes a magnet pocket (not shown in FIG. 1C), which can be any suitable feature in the encapsulation layer 101 configured to receive and retain an implantable magnet system 152. The implantable magnet system 152 comprises one or more magnets / magnetic components, potentially disposed in a hermetic housing. The magnets / magnetic materials disposed of an implantable magnet system can take any of a number of different forms / arrangements and can include, for example, fixed magnets / magnetic materials, uniaxially or biaxially rotatable magnets (e.g., diametric magnets), flexible omnidirectional rotating magnetic arrays (FORMAs) (e.g., miniaturized magnetic balls made sealed insides interconnected cavities where the magnetic balls can rotate freely in the cavities), etc.

[0039] As noted, the stimulating assembly 116 is configured to be at least partially implanted in the user’s cochlea. The stimulating assembly 116 includes a plurality of longitudinally spaced intra-cochlear electrical stimulating contacts (electrodes) 144 that collectively form a contact array (electrode array) 146 for delivery of electrical stimulation (current) to the user’s cochlea. The stimulating assembly 116 extends through an opening in the user’s cochlea (e.g., cochleostomy, the round window, etc.) and has a proximal end connected to stimulator unitAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1142 via lead region 136 and a hermetic feedthrough (not shown in FIG. ID). Lead region 136 includes a plurality of conductors (wires) that electrically couple the electrodes 144 to the stimulator unit 142. The implantable component 112 also includes an electrode outside of the cochlea, sometimes referred to as the extra-cochlear electrode (ECE) 139.

[0040] As noted, the cochlear implant system 102 includes the external coil 108 and the implantable coil 114. The external magnet 150 is positioned adjacent to the external coil 108, and the intemal / implantable magnet system 152 is positioned adjacent to the implantable coil 114. The external magnet 150 and the internal / implantable magnet system 152 adjacent to the external coil 108 and the internal / implantable coil 114, respectively, facilitate the operational alignment of the external coil 108 with the implantable coil 114. This operational alignment of the coils enables the external component 104 to transmit data and power to the implantable component 112 via a closely-coupled wireless link 148 formed between the external coil 108 with the implantable coil 114. In certain examples, the closely-coupled wireless link 148 is an RF link. However, various other types of energy transfer, such as infrared (IR), electromagnetic, capacitive, and inductive transfer, can be used to transfer the power and / or data from an external component to an implantable component and, as such, FIG. ID illustrates only one example arrangement.

[0041] As noted above, the sound processing unit 106 includes the external sound processing module 124. The external sound processing module 124 is configured to process the received input audio signals (received at one or more of the input devices, such as sound input devices 118 and / or auxiliary input devices 128) and convert the received input audio signals into output control signals for use in stimulating a first ear of a user or user (i.e., the external sound processing module 124 is configured to perform sound processing on input signals received at the sound processing unit 106). Stated differently, the one or more processors (e.g., processing element(s) implementing firmware, software, etc.) in the external sound processing module 124 are configured to execute sound processing logic in memory to convert the received input audio signals into output control signals (stimulation signals) that represent electrical stimulation for delivery to the user.

[0042] As noted, FIG. ID illustrates an embodiment in which the external sound processing module 124 in the sound processing unit 106 generates the output control signals. In an alternative embodiment, the sound processing unit 106 can send less processed information (e.g., audio data) to the implantable component 112, and the sound processing operations (e.g.,Atty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1conversion of input sounds to output control signals 156) can be performed by a processor within the implantable component 112.

[0043] In FIG. ID, according to an example embodiment, output control signals (stimulation signals) are provided to the RF transceiver 122, which transcutaneously transfers the output control signals (e.g., in an encoded manner) to the implantable component 112 via the external coil 108 and the implantable coil 114. That is, the output control signals (stimulation signals) are received at the RF interface circuitry 140 via the implantable coil 114 and provided to the stimulator unit 142. The stimulator unit 142 is configured to utilize the output control signals to generate electrical stimulation signals (e.g., current signals) for delivery to the user’s cochlea via one or more of the stimulating contacts 144. In this way, the cochlear implant system 102 electrically stimulates the user’s auditory nerve cells, bypassing absent or defective hair cells that normally transduce acoustic vibrations into neural activity, in a manner that causes the user to perceive one or more components of the input audio signals (the received sound signals).

[0044] As detailed above, in the external hearing mode, the implantable component 112 receives processed sound signals from the sound processing unit 106. However, in the invisible hearing mode, the implantable component 112 is configured to capture and process sound signals for use in electrically stimulating the user’s auditory nerve cells. In particular, as shown in FIG. ID, an example embodiment of the implantable component 112 can include a plurality of implantable sound sensors 165(1), 165(2) that collectively form a sensor array 160, and an implantable sound processing module 158. Similar to the external sound processing module 124, the implantable sound processing module 158 can comprise, for example, one or more processors and a memory device (memory) that includes sound processing logic. The memory device can comprise any one or more of Non-Volatile Memory (NVM), Ferroelectric Random Access Memory (FRAM), read only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. The one or more processors are, for example, microprocessors or microcontrollers that execute instructions for the sound processing logic stored in memory device.

[0045] In the invisible hearing mode, the implantable sound sensors 165(1), 165(2) of the sensor array 160 are configured to detect / capture input sound signals 166 (e.g., acoustic sound signals, vibrations, etc.), which are provided to the implantable sound processing module 158. The implantable sound processing module 158 is configured to convert received input sound signals 166 (received at one or more of the implantable sound sensors 165(1), 165(2)) intoAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1output control signals 156 for use in stimulating the first ear of a user or user (i.e., the implantable sound processing module 158 is configured to perform sound processing operations). Stated differently, the one or more processors (e.g., processing element(s) implementing firmware, software, etc.) in the implantable sound processing module 158 are configured to execute sound processing logic in memory to convert the received input sound signals 166 into output control signals 156 that are provided to the stimulator unit 142. The stimulator unit 142 is configured to utilize the output control signals 156 to generate electrical stimulation signals (e.g., current signals) for delivery to the user’s cochlea, thereby bypassing the absent or defective hair cells that normally transduce acoustic vibrations into neural activity.

[0046] It is to be appreciated that the above description of the so-called external hearing mode and the so-called invisible hearing mode are merely illustrative and that the cochlear implant system 102 could operate differently in different embodiments. For example, in one alternative implementation of the external hearing mode, the implantable component 112 could use signals captured by the sound input devices 118 and the implantable sound sensors 165(1), 165(2) of sensor array 160 in generating stimulation signals for delivery to the user.

[0047] FIG. IE is a block diagram illustrating one example arrangement for an external computing device 110 configured to perform one or more operations in accordance with certain embodiments presented herein. As shown in FIG. IE, in its most basic configuration, the external computing device 110 includes at least one processing unit 183 and a memory 184. The processing unit 183 includes one or more hardware or software processors (e.g., Central Processing Units) that can obtain and execute instructions. The processing unit 183 can communicate with and control the performance of other components of the external computing device 110. The memory 184 is one or more software or hardware-based computer-readable storage media operable to store information accessible by the processing unit 183. The memory 184 can store, among other things, instructions executable by the processing unit 183 to implement applications or cause performance of operations described herein, as well as other data. The memory 184 can be volatile memory (e.g., RAM), non-volatile memory (e.g., ROM), or combinations thereof. The memory 184 can include transitory memory or non-transitory memory. The memory 184 can also include one or more removable or non-removable storage devices. In examples, the memory 184 can include RAM, ROM) EEPROM (Electronically-Erasable Programmable Read-Only Memory), flash memory, optical disc storage, magnetic storage, solid state storage, or any other memory media usable to store information for laterAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1access. By way of example, and not limitation, the memory 184 can include wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared, other wireless media, or combinations thereof. In certain embodiments, the memory 184 comprises logic 195 that, when executed, enables the processing unit 183 to perform aspects of the techniques presented.

[0048] In the illustrated example of FIG. IE, the external computing device 110 further includes a network adapter 186, one or more input devices 187, and one or more output devices 188. The external computing device 110 can include other components, such as a system bus, component interfaces, a graphics system, a power source (e.g., a battery), among other components. The network adapter 186 is a component of the external computing device 110 that provides network access (e.g., access to at least one network 189). The network adapter 186 can provide wired or wireless network access and can support one or more of a variety of communication technologies and protocols, such as Ethernet, cellular, Bluetooth, near-field communication, and RF, among others. The network adapter 186 can include one or more antennas and associated components configured for wireless communication according to one or more wireless communication technologies and protocols. The one or more input devices 187 are devices over which the external computing device 110 receives input from a user. The one or more input devices 187 can include physically-actuatable user-interface elements (e.g., buttons, switches, or dials), a keypad, keyboard, mouse, touchscreen, and voice input devices, among other input devices that can accept user input. The one or more output devices 188 are devices by which the external computing device 110 is able to provide output to a user. The output devices 188 can include a display 190 (e.g., a liquid crystal display (LCD)) and one or more speakers 191, among other output devices for presentation of visual or audible information to the user, a clinician, an audiologist, or other user.

[0049] It is to be appreciated that the arrangement for the external computing device 110 shown in FIG. IE is merely illustrative and that aspects of the techniques presented herein can be implemented at a number of different types of systems / devices including any combination of hardware, software, and / or firmware configured to perform the functions described herein. For example, the external computing device 110 can be a personal computer (e.g., a desktop or laptop computer), a hand-held device (e.g., a tablet computer), a mobile device (e.g., a smartphone), a surgical system, and / or any other electronic device having the capabilities to perform the associated operations described elsewhere herein.Atty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1

[0050] As noted, embodiments described herein provide for processing audio signals received at a hearing device to adjust or optimize the speech in a manner indicated by a user of the hearing device. The hearing device adjusts the speech without changing a meaning of the speech and provides a stimulation signal to the cochlea of the user that simulates the modified / transformed speech.

[0051] Referring to FIG. 2, shown is a block diagram of a sound processor 200 of a medical device, such as a hearing device or cochlear implant. The sound processor 200 includes an audio transcriber 202, a speech transformation block (speech transformer) 204, and an output signal generator block (output signal generator) 206. In the example illustrated in FIG. 2, sound processor 200 processes received speech to provide signals to a user of a hearing device, but where the output signals are modified in some manner, without changing the meaning of the speech. For example, in certain examples, the sound processor 200 can simplify the received speech, adjust a tone associated with the received speech (e.g., to provide humor, change a formality of the speech, emphasis aspects of the speech, etc.), translate the speech, etc. Although the modules of FIG. 2 are illustrated as being presented in a sound processor of a hearing device, the algorithm to transform received speech to adjust / modify the speech for a user can be present at any location in a hearing device system between an audio source and a cochlea of a user (e.g., a TV streamer, custom user device or TV software, paired smartphone, in a totally implantable cochlear implant (TICI), etc.).

[0052] In the illustrated embodiment of FIG. 2, at 210, sound processor 200 receives an incoming audio signal (e.g., from a speaker or other source), where the incoming audio signal 210 includes speech. The sound processor 200 includes an audio transcriber 202 that transcribes the received speech. For example, as a user of a hearing device is listening to an audio source (e.g., a person, television, etc.), a microphone of the hearing device receives the audio signals and audio transcriber 202 transcribes the speech of the audio signals. In one embodiment, audio transcriber 202 transcribes the speech after the audio signals have been pre-processed (e.g., to reduce noise and perform other pre-processing).

[0053] In one embodiment, prior to transmitting the speech to the speech transformer 204, the incoming audio signal is split into optimal “segments” or “chunks” to allow the audio signal to be fed into an algorithm for transformation or modification. In one embodiment, a segment is defined as a unit of time, such as one second. In another embodiment, the segment is defined as a word count, such as four to ten words. In another embodiment, a constant real-time stream of speech is transmitted to the speech transformer 204. In this embodiment, the algorithm ofAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1the speech transformer 204 is constantly “listening” and constantly adjusting the segmentation internally prior to transforming or modifying the speech.

[0054] Speech transformer 204 analyzes the received transcribed speech and transforms the speech. For example, speech transformer 204 uses a machine trained model or computer algorithm capable of language manipulation to analyze the incoming speech and predict an adjustment to the content, tone, or inflection of the speech to be delivered in synthesized auditory form to the user of the hearing device. In other words, the speech transformer 204 transforms the speech (e.g., by modifying the wording and / or tone of the speech) without changing a meaning of the speech for output of synthesized audio representative of the modified speech to the user. In one example, the computer algorithm is a large language model (LLM). In one embodiment, the model is custom pre-trained on a small sample of text. Generally speaking, larger models take longer to respond, so using a smaller model reduces the latency of providing stimulation signals associated with the modified speech to the user of the hearing device.

[0055] In some embodiments, the sound processor 200 does not include audio transcriber 202 and speech transformer 204 receives the audio signals directly (e.g., from a microphone, a preprocessing module of the hearing device, an output of another device, etc.). In this embodiment, speech transformer 204 interprets the speech in the audio without the speech being transcribed.

[0056] Speech transformer 204 modifies, transforms, or adjusts auditory comprehension attributes associated with the speech in a number of different ways and based on different factors to create target content. As described further with respect to FIG. 3, speech transformer 204 receives an audio snippet of the speech or a transcription of the speech, an instruction, and optionally other inputs (e.g., user information, spatial (e.g., auditory) environment information, user feedback, etc.), transforms the speech based on the instruction (and other inputs), and outputs text or speech corresponding to the modified speech / target content. The instruction can be, for example, “Here is a sentence, please simplify it <the sentence>” (or please add humor, please add context from earlier snippets, please remove gendered language, please apply these 25 modifications as per our user’s preference, etc.). The speech transformer 204 outputs the text of the modified speech or signals representative of the modified speech (e.g., within a fraction of milliseconds).Atty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1

[0057] In one embodiment, the adjustments to the speech are determined / selected based on an input from the user. For example, the user selects auditory comprehension attributes associated with the speech to modify the speech for the user by, for example, decreasing the cognitive load associated with perception, by the user, of the speech or increasing an engagement of / for the user with the speech or speaker. In one embodiment, the user uses an application on a user device (e.g., a smartphone) to indicate how to adjust the speech. For example, the application provides a list of options for adjusting or processing the speech and the user selects an option. In this embodiment, the speech transformer 204 adjusts the speech based on the input received from the user.

[0058] In another embodiment, the adjustments to the speech are based on other factors, such as environmental factors (e.g., spatial, such as auditory, environmental factors). In this embodiment, the hearing device may determine, based on environmental factors (e.g., sound or noise in the vicinity of the hearing device), how to modify the speech for the user. For example, if the user is in a stressful environment, the speech transformer 204 adjusts the speech to reduce the cognitive load on the user. As another example, if it is determined that the user is at a theater watching a movie, play, or concert, the speech transformer 204 determines not to adjust the speech so that the user can experience the speech unaltered.

[0059] In another embodiment, the adjustments to the speech are determined based on factors associated with the user, such as age, cognitive ability, auditory comprehension faculties, or other factors. In this embodiment, the adjustments can be configured (made in a manner) for reducing the auditory cognitive load associated with the user successfully comprehending the message in auditory form through the hearing device. In one example, in which the incoming speech includes content having relatively high complexity compared to the user’s cognitive and / or auditory comprehension faculties, the adjustment includes paraphrasing and / or word replacement in the synthesized auditory output. For example, speech transformer 204 may receive a transcript of original received speech “We have considered your application at this time, and I would like to extend an offer for this role” and modify the speech to output “You got the job” for the user.

[0060] In another example, the adjustments can be determined based on an amount of time since the user has been fitted for the hearing device. In this example, the speech adjustments are configured for reducing the auditory cognitive load for the user when the user is first fitted for the hearing device. When the user first receives the hearing device, the user is adjusting to the hearing device and can be allocating brain resources to focusing on “hearing” and not onAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1comprehending words or complex sentences. During this period, the speech can be processed to produce simplified speech with the same meaning as the original speech to reduce the cognitive load on the user during the period of adjustment. Over time, the user adjusts to the hearing device and the processing of the speech changes. For example, the simplification of the speech can be decreased the longer the user has the hearing device (e.g., until the speech is no longer simplified for the user).

[0061] In another embodiment, the adjustments to the speech are determined based on inputs from the user. For example, if the user has difficulties in perceiving the difference between the spoken words ‘accept’ and ‘except,’ then the output of the speech transformer 204 rephrases incoming speech to avoid those words or otherwise reduce the chances of confusion. In this example, a user indicates to the system (e.g., using an application on a user device) which words are troublesome or data could be acquired from hearing tests. As another example, the hearing device receives user feedback after processing speech and processes subsequent speech based on the user feedback.

[0062] In other embodiments, the speech transformer 204 modifies the original speech in other ways to adjust auditory comprehension attributes associated with the speech. In one example, the speech transformer 204 modifies the tone of the message to suit different contexts (e.g., changing a formal tone to a more casual format or changing a casual format to a more formal tone). In one example, the received speech “I would like to request your presence at this meeting” is adjusted to output “Can you come to the meeting?”

[0063] In another example, longer texts are condensed into shorter, more digestible summaries while retaining key information. In this example, the received speech “The quarterly financial report indicates a significant increase in revenue, primarily driven by the successful launch of our new product line” is adjusted to output “Our new product line boosted quarterly revenue.”

[0064] In another example, explanations or definitions are added for complex terms or jargon. In this example, the received speech “The API will return a JSON object” is adjusted to output “The API (Application Programming Interface) will return a JSON (JavaScript Object Notation) object, which is a format for structuring data.”

[0065] In another example, emotional content is identified and adjusted to be more neutral or positive. In this example, the received speech “I’m really upset about the delay” is adjusted to output “I’m concerned about the delay.” In another example, language is modified to ensureAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1the language is inclusive and free from bias. In this example, the received speech “The best man for the job” is adjusted to output “The best person for the job.”

[0066] In another example, messages are tailored to the user’s preferences or past interactions. In this example, the generic speech “Thank you for your purchase” is adjusted to output “Thank you for your purchase, Alex! We hope you enjoy your new headphones.” In another example, language is modified to be culturally appropriate and sensitive. In this example, the original transcribed speech “Let’s have a Christmas party” is adjusted to output “Let’s have a holiday party.”

[0067] In another example, the level of formality of the speech is changed or adjusted to match the context or audience. In this example, the formal speech “We regret to inform you that your application was unsuccessful” is modified to the more informal speech “Sorry, but you didn’t get the job this time.” In another example, language is modified to ensure that the language is gender neutral. In this example, the speech “Each employee must submit his report by Friday” is modified to the neutral speech “Each employee must submit their report by Friday.”

[0068] In another example, text is converted to match a specific voice or style, such as by making text sound like it was written by a particular author or in a specific genre. In this example, the original speech “The weather is nice today” is adjusted to the Shakespearean-style speech “The weather doth shine fair this day.” In another example, key points or important information is highlighted in a message. In this example, the original speech “The meeting is scheduled for 3 PM tomorrow” is modified to emphasize “Don’t forget, the meeting is at 3 PM tomorrow ."

[0069] In another example, natural pauses and pacing are added to text to make it sound more like natural speech when converted to audio. In this example, the speech “Let’s meet at the cafe at 3 PM” is modified with pacing to output “Let’s meet at the cafe... at 3 PM.” In another example, emotional cues are added to text to convey feelings more effectively. In this example, the original text “I’m happy to see you” is modified to infuse emotion by outputting “I’m so happy to see you!”

[0070] In another example, sentences are rephrased to fit different contexts or audiences. In this example, the original speech “Please submit your report by Friday” is modified for a friend to output “Hey, can you get your report to me by Friday?” In another example, prompts or questions are added to make the text more engaging. In this example, the original speech “WeAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1need to discuss the project” is modified to be more interactive by outputting “Can we discuss the project? What time works for you?”

[0071] In another example, storytelling elements are added to make the text more engaging and vivid. In this example, the original speech “The team worked hard to meet the deadline” is modified to output the enhanced “The team pulled together, working late into the night, to meet the deadline.” In another example, a tone and / or pitch of the text is adjusted to convey different emotions or emphasis when read aloud. In this example, the original speech “I can’t believe it!” is adjusted to output the modulated “I can ’t believe it!”

[0072] In another example, additional context or background information is provided to make a message clearer. In this example, the speech “The results were unexpected” is modified with context to output “Given the previous trends, the results were unexpected.” In another example, light humor is added to make the received speech more engaging and relatable. In this example, the original speech “The meeting was long” is modified with humor to output “The meeting was so long, I almost grew a beard!”

[0073] In another example, the pace of speech is adjusted by slowing or speeding up the speaker’s words. In another example, speech is translated from one langue, dialect, or broken language into preferred language. In this example, the original “Ich mag Montags nichts” is translated into “I don’t like Mondays.” As another example, the original “We go to shopping please now?” is translated into “Can we please go shopping now?” In another example, the speech is modified to remove swear words or profanity from language.

[0074] In one embodiment, the user provides feedback to the system, and the feedback is used when transforming subsequently received speech. In one example, the user provides the feedback using, for example, an application on a user device. In another example, the hearing device receives a verbal or non-verbal indication from the user as feedback (e.g., head nodding or shaking, gaze direction, etc.). In this example, the hearing device receives negative feedback if the user verbally expresses dissatisfaction (e.g., says “I don’t understand,” “Can you repeat that?,” etc.), if the user appears to strain to understand, or if the user provides another type of verbal or non-verbal feedback.

[0075] Referring back to FIG. 2, speech transformer 204 generates a modified text transcription and provides the modified text transcription to the output signal generator 206 for use in generating an output signal to the user. Output signal generator 206 converts the modified text transcription into stimulation signals for delivery to the user to for perception, byAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1the user, of the modified speech. In some embodiments, speech transformer 204 is capable of synthesizing speech from text. For example, LLMs can be trained to sound like a person. With enough time listening to a person, the output of speech transformer 204 can reach closer and closer to mimicking the natural voice of a person. In addition, the LLM is capable of outputting speech with voice variation to implement some of the above-described speech manipulations (e.g., adding emphasis, speaking slower, etc.). In this embodiment, speech transformer 204 outputs the modified text transcription. In some embodiments, if text to output signal generator 206 converts the received text back to speech signals or if speech transformer 204 outputs the modified text transcription, the modified text transcription is output to the user in the speaker’s voice. In this way, the user experiences the speaker’s voice with the modified speech.

[0076] When the audio snippet of the modified speech has been output from the sound processor or other portion of a hearing device system (e.g., smartphone, TV streamer, TICI, mini microphone, etc.), at 212, the output modified speech may be converted into stimulation signals to output to the user to simulate the modified speech. The user of the hearing device hears the words of the modified speech and understands their meaning, leading to enhanced listening comprehension. In this way, the techniques described herein improve not just hearing but also improve communication and comprehension.

[0077] Reference is now made to FIG. 3, which is a block diagram of a machine learning model 300 in accordance with certain aspects presented herein. In one example, machine learning model 300 is a LLM or another type of computer algorithm that has been trained to manipulate language. In the example illustrated in FIG. 3, machine learning model is a pre-trained machine learning model used to transform speech received in an audio signal. Machine learning model 300 is operationally located between an audio source and the inner ear of a user of a hearing device, such as in the sound processor of the hearing device, a TV streamer, custom laptop or TV source, paired mobile phone, a TICI, or in another device that is part of a hearing device system.

[0078] As illustrated in FIG. 3, the machine learning model 300 receives speech 302 from an audio source. In one example, the speech 302 is transcribed speech. In another example, machine learning model 300 is capable of interpreting audio and the speech 302 is not transcribed. In one embodiment, the speech 302 is a segment of speech (e.g., defined by time or word count). In another embodiment, the speech 302 is a real-time stream of audio and the machine learning model 300 adjusts the segmentation internally.Atty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1

[0079] In certain examples, machine learning model 300 receives instructions and additional data 304. The instructions indicate howto transform the speech 302 to produce modified speech 306. In one embodiment, the instructions are based on a selection from a user of the hearing device (e.g., using an application on a smartphone). Some examples of instructions from the user include simplification (in which the language of the speech 302 is simplified to reduce the cognitive load on the user), summarization (in which the speech 302 is summarized), emotion adjustment (in which an emotion of speech 302 is adjusted to be more neutral or positive), personalization (in which speech 302 is personalized for the user), formality adjustment (in which the formality of speech 302 is adjusted to match the context or audience), pacing and pausing (in which natural pauses are added to speech 302 to make the speech sound more natural), emotion infusion (in which emotional cues are added to speech 302), humor addition (in which humor is added to speech 302), or additional types of instructions. In one example, the instructions include an indication to automatically transform the speech 302 based on factors associated with the user (e.g., based on biometric data associated with the user) or environmental factors associated with the user (e.g., based on an environment that the user is currently in).

[0080] In one embodiment, the additional data 304 includes information associated with the user of the hearing device. For example, the information includes an age of the user, cognitive and / or comprehension faculties associated with the user, an amount of time that the user has had the hearing device, biometric data associated with the user, or other data. The information associated with the user is used to control the transformation of the speech for the user. In some embodiments, biometric data or other measurements are used to determine how to adjust the received speech 302. In one example, if data indicates that the user is overwhelmed or under stress, the speech 302 is simplified to reduce a cognitive load on the user. In another example, if the user has recently been fitted for the hearing device and is adjusting to having the hearing device, the speech 302 is simplified. In this example, as the user continues to get used to the hearing device, the simplification of the speech 302 is reduced until the user is comfortable with the hearing device. In this way, a cognitive load on the user is reduced to help the user adjust to the hearing device.

[0081] In another embodiment, the additional data 304 includes environmental data. For example, audio of the environment surrounding the user is analyzed to determine how to transform or adjust the speech 302. In one example, if it is determined that the user is in a theater watching a play or movie or listening to a concert, the speech 302 will not beAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1transformed to ensure that the user experiences the show in the original form. In another example, if the audio indicates that the user is in a busy or overwhelming environment, speech 302 may be simplified to reduce the cognitive load on the user. In this example, the biometric data associated with the user can be used in combination with the environmental data to determine whether the user is stressed or overwhelmed.

[0082] In another embodiment, the additional data 304 includes feedback from the user. In one example, the user provides feedback using an application on a user device to indicate whether the user is satisfied with the speech adjustment. In another example, the user provides spoken feedback, either explicitly or implicitly. In this example, the user provides verbal indications to indicate whether the user is satisfied with the speech adjustment. In some example, the user can say “I like this adjustment,” “I like how this speech is simplified,” “I enjoy the added humor,” “I don’t like this transformation,” or other explicit feedback. In other example, the user can implicitly provide feedback by saying “I don’t understand,” “I understand what you’re saying,” or other statements. In some embodiments, other data associated with the user can indicate the user’s feedback. For example, data that indicates that the user is straining to understand (e.g., body movement or other stress indicators) are construed as negative feedback. The user feedback is used by machine learning model 300 when providing speech adjustment for subsequently received speech.

[0083] As illustrated in FIG. 3, the speech 302 is transformed by machine learning model 300 based on the instructions and additional data 304 to produce transformed or modified speech 306. The modified speech 306 is output to the processing chain of the hearing device for additional processing and for stimulation of electrodes in the cochlea of the user to simulate the modified speech 306 for the user. For example, the electrodes implanted in the cochlea of the user are stimulated so that the user electrically hears the modified speech 306. In some embodiments, the modified speech 306 matches a voice of the speaker of the speech 302.

[0084] The modified speech 306 is additionally fed back into machine learning model 300 and uses in conjunction with user feedback for training the machine learning model 300 to control / determine the speech transformation for the user.

[0085] In this way, adjustments to received speech are optimized for a user of a hearing device. In some embodiments, the speech adjustments are configured (made in a manner) to reduce a cognitive load or increase comprehension for the user. By adjusting or optimizing the speech, comprehension and / or engagement is improved based on user input, user selections, and / orAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1user feedback. In addition, speech modifications help a user adapt to a new hearing device by reducing a cognitive load on the user while the user is adjusting to the new hearing device.

[0086] Reference is now made to FIGs. 4A and 4B. FIGs. 4A and 4B illustrate user interfaces of applications associated with the techniques described herein. In particular, referring first to FIG. 4A, shown is a user interface 400 of an audio transformation application. User interface 400 includes a menu 402 that lists options for how to transform received speech. The user interface 400 is presented to a user of a hearing device on a user device, such as a smartphone. In this example, the user makes a selection from menu 402 to indicate how to transform received speech. Menu 402 includes the selections “Simplification,” “Summarization,” “Emotion Adjustment,” “Personalization,” “Formality Adjustment,” “Pacing and Pausing,” “Emotion Infusion,” “Humor Addition,” “Automatic,” and “Off.” The options illustrated in menu 402 are exemplary and additional and / or different options may be presented to a user. The options illustrated in menu 402 have been described above with respect to FIGs. 2 and 3. In this example, when the “Automatic” option is selected, the speech is adjusted based on user and / or environmental factors. In addition, the “Off’ options allow the user to turn off the speech adjustment. In the example illustrated in FIG. 4A, the user has selected the “Simplification” option to simplify speech to reduce a cognitive load on the user.

[0087] FIG. 4B illustrates a user interface 410 that shows an audio simplification log. Audio simplification log presents a log of original speech and modified speech to the user. The audio simplification log illustrates an example in which the user has selected the “Simplification” option (e.g., from menu 402). The audio simplification log includes the time 412 that speech was received, the original speech 414, and the “after” or modified speech 416. In the example illustrated in FIG. 4B, at 12:03 pm, a speaker said, “We have considered your application and at this time I would like to extend an offer for this role.” The user has selected an option to simplify the speech and, after simplification, “You got the job” was output to the user.

[0088] In some embodiments, an audio log is presented for each different selection made by a user on menu 402. For example, if the user chose the “Humor Addition” selection on menu 402, an additional audio log is presented in which humor has been added to the speech. In other embodiments, a single audio transformation log is presented that includes a log of all audio transformations and a time at which each transformation took place.

[0089] By presenting the audio logs to a user, the user has an option to review the original speech that was received. In this way, the user, for example, can see how the audio wasAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1transformed, provide feedback based on the transformation, and review additional or different data that may have been in the original audio.

[0090] Reference is now made to FIG. 5, which is a flow chart that illustrates a method 500 of delivering stimulation signals to a user to evoke perception of speech that is modified to adjust a wording or inflection of received speech.

[0091] At 502, one or more audio signals that include speech are received at a hearing device system associated with a user. The audio signals are received, for example, from a microphone or an external device (e.g., smartphone, audio device, TV streamer, etc.) of the hearing device system.

[0092] At 504, the one or more audio signals are processed to adjust at least one of a wording or inflection of the speech to produce modified speech that has a same meaning as the speech. For example, the one or more audio signals are processed based on instructions to adjust the wording or inflection of the speech without changing a meaning of the speech. The instructions indicate audio comprehension attributes to modify based on, for example, user selections, user data, environmental data, and / or user feedback.

[0093] At 506, stimulation signals are delivered to the user to evoke perception of the modified speech. For example, stimulation signals are delivered to electrodes in the cochlea of the user to evoke perception of the modified speech. The modified speech can be perceived in a voice of the speaker of the speech. The modified speech increases the comprehension or engagement of the speech for the user.

[0094] Reference is now made to FIG. 6, which is a flow chart of a method 600 of delivering stimulation signals to a user to evoke perception of speech with modified auditory comprehension attributes.

[0095] At 602, sound signals are received that include speech. The speech comprises auditory comprehension attributes. At 604, the sound signals are converted to stimulation signals for delivery to a user of a hearing device system. At 606, during converting, one or more of the auditory comprehension attributes associated with the received speech are modified to generate modified speech. The modified speech has the same meaning as the received speech. For example, a wording, tone, inflection, pace / rate, pitch, or another auditory comprehension attribute is modified to generate modified speech. The modified speech provides a different auditory comprehension to the user without modifying a meaning of the received speech. ForAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1example, the modified speech can be simplified to reduce an auditory comprehension load on the user without changing a meaning of the words.

[0096] At 608, stimulation signals are delivered to the user to evoke perception of the modified speech, he modified speech can be perceived in a voice of the speaker of the speech. The modified speech increases the comprehension or engagement of the speech for the user without changing a meaning of the speaker’s words.

[0097] As previously described, the technology disclosed herein can be applied in any of a variety of circumstances and with a variety of different devices. Example devices that can benefit from technology disclosed herein are described in more detail in FIGs. 7-10 for a vestibular stimulator system, a retinal prosthesis system, a tinnitus therapy device, and an upper airway stimulation device, respectively. However, the techniques of the present disclosure can be applied to other devices, such as neurostimulators, cardiac pacemakers, cardiac defibrillators, seizure therapy stimulators, as well as other medical devices that deliver stimulation to tissue. Further, technology described herein can also be applied to consumer devices. These different systems and devices can benefit from the technology described herein.

[0098] FIG. 7 illustrates an example vestibular stimulator system 702, with which embodiments presented herein can be implemented. As shown, the vestibular stimulator system 702 comprises an implantable component (vestibular stimulator) 712 and an external device / component 704 (e.g., external processing device, battery charger, remote control, etc.). The external device 704 comprises a transceiver unit 760. As such, the external device 704 is configured to transfer data (and potentially power) to the vestibular stimulator 712.

[0099] The vestibular stimulator 712 comprises an implant body (main module) 734, a lead region 736, and a stimulating assembly 716, all configured to be implanted under the skin / tissue (tissue) 715 of the user. The implant body 734 generally comprises a hermetically-sealed housing 738 in which RF interface circuitry, one or more rechargeable batteries, one or more processors, and a stimulator unit are disposed. The implant body 734 also includes an internal / implantable coil 714 that is generally external to the housing 738, but which is connected to the transceiver via a hermetic feedthrough (not shown).[ooioo] The stimulating assembly 716 comprises a plurality of electrodes 744(l)-(3) disposed in a carrier member (e.g., a flexible silicone body). In this specific example, the stimulating assembly 716 comprises three (3) stimulation electrodes, referred to as stimulation electrodes 744(1), 744(2), and 744(3). The stimulation electrodes 744(1), 744(2), and 744(3) function asAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1an electrical interface for delivery of electrical stimulation signals to the user’s vestibular system.[ooioi] The stimulating assembly 716 is configured such that a surgeon can implant the stimulating assembly adjacent the user’s otolith organs via, for example, the user’s oval window. It is to be appreciated that this specific embodiment with three stimulation electrodes 744 is merely illustrative and that the techniques presented herein can be used with stimulating assemblies having different numbers of stimulation electrodes, stimulating assemblies having different lengths, etc.

[0102] Moreover, vestibular stimulator 712 can be configured to implement the techniques presented herein. In one example, input signals (e.g., spatial input signals, motion input signals, environmental input signals, etc.) may be modified in the manner described herein and provided to the user. For example, the input signals could be modified / optimized for the specific attributes or preferences of the user.

[0103] FIG. 8 illustrates a retinal prosthesis system 801 that comprises an external device 810 configured to communicate with an implantable retinal prosthesis 800 via signals 851. The retinal prosthesis 800 comprises an implanted processing module 825, and a retinal prosthesis sensor-stimulator 890 is positioned proximate the retina of a user. The external device 810 and the processing module 825 can communicate via coils 808, 814.

[0104] In an example, sensory inputs (e.g., photons entering the eye) are absorbed by a microelectronic array of the sensor-stimulator 890 that is hybridized to a glass piece 892 including, for example, an embedded array of microwires. The glass can have a curved surface that conforms to the inner radius of the retina. The sensor-stimulator 890 can include a microelectronic imaging device that can be made of thin silicon containing integrated circuitry that convert the incident photons to an electronic charge.

[0105] The processing module 825 includes an image processor 823 that is in signal communication with the sensor-stimulator 890 via, for example, a lead 888 that extends through surgical incision 889 formed in the eye wall. In other examples, processing module 825 is in wireless communication with the sensor-stimulator 890. The image processor 823 processes the input into the sensor-stimulator 890 and provides control signals back to the sensor-stimulator 890 so the device can provide an output to the optic nerve. That said, in an alternate example, the processing is executed by a component proximate to, or integrated with, the sensor-stimulator 890. The electric charge resulting from the conversion of the incidentAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1photons is converted to a proportional amount of electronic current which is input to a nearby retinal cell layer. The cells fire and a signal is sent to the optic nerve, thus inducing a sight perception.

[0106] The processing module 825 can be implanted in the user and function by communicating with the external device 810, such as a BTE unit, a pair of eyeglasses, etc. The external device 810 can include an external light / image capture device (e.g., located in / on a behind-the-ear device or a pair of glasses, etc.), while, as noted above, in some examples, the sensor-stimulator 890 captures light / images, in which sensor-stimulator 890 is implanted in the user.

[0107] Moreover, retinal prosthesis system 801 can be configured to implement the techniques presented herein. In one example, input signals (e.g., spatial input signals, motion input signals, environmental input signals, etc.) may be modified in the manner described herein and provided to the user. For example, the input signals could be modified / optimized for the specific attributes or preferences of the user.

[0108] FIG. 9 illustrates a tinnitus therapy device 900 (e.g., a tinnitus implant, a tinnitus management stimulator) including a sound input unit 902 (e.g., a microphone) configured to receive acoustic inputs. In some embodiments, the sound input unit 902 is implanted adjacent to an outer ear 903 to position a diaphragm 916 of the sound input unit 902 such that the diaphragm 916 is configured to be displaced (vibrate) in response to the acoustic inputs. The tinnitus therapy device 900 further includes an implant body 904 in which circuitry, such as a processor and / or a memory, is disposed. The implant body 904 is also coupled to a coil 908 to enable transfer of power / data between the tinnitus therapy device 900 and an external device. The implant body 904 is electrically coupled to the sound input unit 902 to receive the acoustic input. The tinnitus therapy device 900 is then configured to convert the acoustic input to tinnitus therapy control signals (e.g., based on a classification of the acoustic input).

[0109] The tinnitus therapy control signals are provided to an actuator 906 electrically coupled to the implant body 904 for delivery to the user. By way of example, a coupling member 940 couples the actuator 906 to an ossicular chain 936 (i.e., the malleus, the incus, and the stapes bones) positioned in a middle ear cavity between a tympanic membrane 913 and a cochlea 938 of the user, and the actuator 906 is configured to deliver the tinnitus therapy control signals. The actuator 906 is attached to a temporal bone 915 of the user via a fixation system 942 and is configured to impart motion to (e.g., vibrate) the ossicular chain 936, which is typicallyAtty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1configured to amplify sound waves received via an ear canal 911. In operation, the actuator 906 is configured to impart motion based on the tinnitus therapy control signals, and such vibration creates waves of fluid motion of perilymph within the cochlea 938 to activate hair cells within the cochlea 938. Activation of the hair cells causes nerve impulses to be generated and transferred through spiral ganglion cells, an auditory nerve, and a brain, where the vibration is perceived as sounds to provide relief of tinnitus symptoms experienced by the user.

[0110] Moreover, tinnitus therapy device 900 can be configured to implement the techniques presented herein. In one example, input signals (e.g., spatial input signals, motion input signals, environmental input signals, etc.) may be modified in the manner described herein and provided to the user. For example, the input signals could be modified / optimized for the specific attributes or preferences of the user.[oom] FIG. 10 illustrates an upper airway stimulation device 1000 (e.g., an upper airway implant, a sleep apnea management stimulator, a sleep disorder system) that includes an implant body 1002, a sensor 1004, and a stimulator 1006. The upper airway stimulation device 1000 is implantable in a user 1008 to position the sensor 1004 adjacent to lungs 1010 of the user 1008. Thus, the sensor 1004 is able to receive input that indicates breathing performed by the user 1008. The implant body 1002 includes a housing in which circuitry, such as a processor and / or a memory, is disposed. The sensor 1004 transmits electrical signals in response to receipt of the input, and the upper airway stimulation device 1000 is configured to convert the electrical signals to stimulation signals, which are provided to the stimulator 1006. The stimulator 1006 is positioned adjacent to a hypoglossal nerve 1012 of the user 1008 and is configured to deliver the stimulation signals to the hypoglossal nerve 1012, which fires nerve cells of a tongue of the user 1008, thereby causing the tongue to contract and move (e.g., in an anterior direction) and increase a size of an opening of an airway of the user 1008. Consequently, the upper airway stimulation device 1000 generates stimulation signals based on the input to help the user 1008 breathe more easily (e.g., while the user 1008 is asleep to mitigate sleep apnea).

[0112] Moreover, upper airway stimulation device 1000 can be configured to implement the techniques presented herein. In one example, input signals (e.g., spatial input signals, motion input signals, environmental input signals, etc.) may be modified in the manner described herein and provided to the user. For example, the input signals could be modified / optimized for the specific attributes or preferences of the user.Atty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1

[0113] The above embodiments have been primarily described with reference to use of the techniques presented herein during implantation of a specific implantable component, namely a cochlear implant stimulating assembly. However, as described above with reference to FIGs.7, 8, 9, and 10, the techniques presented herein can be used to modify received inputs to improve operation or use of a variety of implantable components, including sleep disorder devices (e.g., sleep apnea devices), balance or movement disorder devices (e.g., vestibular stimulation devices), tinnitus management devices, visual implants (e.g., bionic eyes), and other neuromodulation devices (e.g., brain-computer interfaces).

[0114] As should be appreciated, while particular uses of the technology have been illustrated and discussed above, the disclosed technology can be used with a variety of devices in accordance with many examples of the technology. The above discussion is not meant to suggest that the disclosed technology is only suitable for implementation within systems akin to that illustrated in the figures. In general, additional configurations can be used to practice the processes and systems herein and / or some aspects described can be excluded without departing from the processes and systems disclosed herein.

[0115] This disclosure described some aspects of the present technology with reference to the accompanying drawings, in which only some of the possible aspects were shown. Other aspects can, however, be embodied in many different forms and should not be construed as limited to the aspects set forth herein. Rather, these aspects were provided so that this disclosure was thorough and complete and fully conveyed the scope of the possible aspects to those skilled in the art.

[0116] As should be appreciated, the various aspects (e.g., portions, components, etc.) described with respect to the figures herein are not intended to limit the systems and processes to the particular aspects described. Accordingly, additional configurations can be used to practice the methods and systems herein and / or some aspects described can be excluded without departing from the methods and systems disclosed herein.

[0117] According to certain aspects, systems and non-transitory computer readable storage media are provided. The systems are configured with hardware configured to execute operations analogous to the methods of the present disclosure. The one or more non-transitory computer readable storage media comprise instructions that, when executed by one or more processors, cause the one or more processors to execute operations analogous to the methods of the present disclosure.Atty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1

[0118] Similarly, where steps of a process are disclosed, those steps are described for purposes of illustrating the present methods and systems and are not intended to limit the disclosure to a particular sequence of steps. For example, the steps can be performed in differing order, two or more steps can be performed concurrently, additional steps can be performed, and disclosed steps can be excluded without departing from the present disclosure. Further, the disclosed processes can be repeated.

[0119] Although specific aspects were described herein, the scope of the technology is not limited to those specific aspects. One skilled in the art will recognize other aspects or improvements that are within the scope of the present technology. Therefore, the specific structure, acts, or media are disclosed only as illustrative aspects. The scope of the technology is defined by the following claims and any equivalents therein.

[0120] It is also to be appreciated that the embodiments presented herein are not mutually exclusive and that the various embodiments can be combined with another in any of a number of different manners.

Claims

1. Atty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC1CLAIMSWhat is claimed is:

1. A method comprising:receiving, at a device associated with a user, one or more audio signals that include speech;processing the one or more audio signals to adjust at least one of a wording or inflection of the speech to produce modified speech that has a same meaning as the speech; anddelivering stimulation signals to the user to evoke perception of the modified speech.

2. The method of claim 1, further comprising:receiving an input from the user that indicates a manner of adjusting the at least one of the wording or inflection of the speech, and wherein processing the one or more audio signals includes:adjusting the wording or the inflection of the speech based on the input from the user.

3. The method of claim 1, wherein processing the one or more audio signals to adjust at least one of the wording or inflection of the speech comprises:simplifying the speech.

4. The method of claim 1, wherein processing the one or more audio signals to adjust at least one of the wording or inflection of the speech comprises:modifying a humor associated with the speech.

5. The method of claim 1, wherein processing the one or more audio signals to adjust at least one of the wording or inflection of the speech comprises:adjusting the wording or inflection to increase user engagement.

6. The method of claim 1, wherein processing the one or more audio signals to adjust at least one of the wording or inflection of the speech comprises:adjusting the wording or inflection to decrease a cognitive load associated with perception of the speech.Atty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC17. The method of claim 1, 2, 3, 4, 5, or 6, wherein processing the one or more audio signals to adjust at least one of the wording or inflection of the speech comprises:processing the one or more audio signals with a pre-trained machine learning model.

8. The method of claim 7, wherein the pre-trained machine learning model is a pretrained large language model (LLM).

9. The method of claim 1, 2, 3, 4, 5, or 6, further comprising:receiving user information associated with the user, and wherein processing the one or more audio signals includes processing the one or more audio signals based on the user information.

10. The method of claim 1, 2, 3, 4, 5, or 6, further comprising:receiving environmental information associated with an auditory environment of the user, and wherein processing the one or more audio signals includes processing the one or more audio signals based on the environmental information.

11. The method of claim 1, 2, 3, 4, 5, or 6, further comprising:receiving feedback from the user in response to processing the one or more audio signals; andadjusting processing of subsequently received audio signals based on the feedback.

12. The use of the method of claim 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or 11 in a hearing aid, an implantable auditory prosthesis, a sleep disorder device, a seizure device, a balance or movement disorder device, a tinnitus management device, or a visual device.

13. The method of claim 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or 11, wherein the device is hearing aid, an implantable auditory prosthesis, a sleep disorder device, a seizure device, a balance or movement disorder device, a tinnitus management device, or a visual device.Atty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC114. A method comprising:receiving sound signals that include speech, wherein the speech comprises auditory comprehension attributes;converting the sound signals to stimulation signals for delivery to a user;during the converting, modifying one or more of the auditory comprehension attributes associated with the speech to generate modified speech; anddelivering the stimulation signals to the user to evoke perception of the modified speech.

15. The method of claim 14, wherein modifying the one or more of the auditory comprehension attributes associated with the speech to generate modified speech comprises:modifying the one or more of the auditory comprehension attributes to adjust a wording of the speech without changing a meaning of the speech.

16. The method of claim 14, wherein modifying the one or more of the auditory comprehension attributes associated with the speech to generate modified speech comprises:modifying the one or more of the auditory comprehension attributes to adjust an inflection associated with the speech.

17. The method of claim 14, wherein modifying the one or more of the auditory comprehension attributes associated with the speech to generate modified speech comprises:modifying the one or more of the auditory comprehension attributes to adjust a context of the speech.

18. The method of claim 14, wherein modifying the one or more of the auditory comprehension attributes associated with the speech to generate modified speech comprises:modifying the one or more of the auditory comprehension attributes to simplify the speech without changing a meaning of the speech.

19. The method of claim 14, wherein modifying one or more of the auditory comprehension attributes associated with the speech to generate modified speech comprises:modifying the one or more of the auditory comprehension attributes to adjust a humor associated with the speech.Atty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC120. The method of claim 14, 15, 16, 17, 18, or 19, wherein modifying one or more of the auditory comprehension attributes associated with the speech to generate modified speech comprises:modifying the one or more of the auditory comprehension attributes to decrease a cognitive load associated with perception of the speech.

21. The method of claim 14, 15, 16, 17, 18, or 19, wherein modifying one or more of the auditory comprehension attributes associated with the speech to generate modified speech comprises:modifying the one or more of the auditory comprehension attributes to increase an engagement of the user.

22. The method of claim 14, 15, 16, 17, 18, or 19, wherein modifying one or more of the auditory comprehension attributes associated with the speech to generate modified speech comprises:modifying the one or more auditory comprehension attributes based on user information associated with the user.

23. The method of claim 14, 15, 16, 17, 18, or 19, wherein modifying one or more of the auditory comprehension attributes associated with the speech to generate modified speech comprises:modifying the one or more auditory comprehension attributes based on environmental information associated with an auditory environment of the user.

24. The use of the method of claim 14, 15, 16, 17, 18, 19, 20, 21, 22, or 23, in a hearing aid, an implantable auditory prosthesis, a sleep disorder device, a seizure device, a balance or movement disorder device, a tinnitus management device, or a visual device.

25. A system according to claim 14, 15, 16, 17, 18, 19, 20, 21, 22, or 23, wherein the system is a hearing aid system, a cochlear implant system, a sleep disorder system, a seizure system, a balance or movement disorder system, a tinnitus management system, or a visual system.Atty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC126. A system comprising:a memory, andat least one processor operable coupled to the memory, wherein the at least one processor is configured to:receive sound signals that include received speech, wherein the received speech comprises auditory comprehension attributes;during the converting, modify one or more of the auditory comprehension attributes associated with the received speech to generate modified speech having, relative to the received speech, at least one of a decreased associated cognitive load or an increased engagement level;using the modified speech to generate stimulation signals for delivery to a user; anddeliver the stimulation signals to the user.

27. The system of claim 26, wherein the stimulation signals are acoustic signals.

28. The system of claim 26, wherein the stimulation signals are electrical stimulation signals.

29. The system of claim 26, 27, or 28, wherein, when modifying one or more of the auditory comprehension attributes, the at least one processor is configured to:modify a wording or inflection of the received speech.

30. The system of claim 26, 27, or 28, wherein, when modifying one or more of the auditory comprehension attributes, the at least one processor is further configured to:simplify the received speech.

31. The system of claim 26, 27, or 28, wherein, when modifying one or more of the auditory comprehension attributes, the at least one processor is further configured to:modify the one or more of the auditory comprehension attributes associated with the received speech to generate modified speech based on user information or environmental information associated with an auditory environment of the user.Atty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC132. The system of claim 26, wherein the system is a hearing aid system, a cochlear implant system, a sleep disorder system, a seizure system, a balance or movement disorder system, a tinnitus management system, or a visual system.

33. One or more non-transitory computer readable storage media comprising instructions that, when executed by a processor, cause the processor to:receive sound signals that include received speech, wherein the received speech comprises auditory comprehension attributes;convert the sound signals to stimulation signals for delivery to a user;during the converting, modify one or more of the auditory comprehension attributes associated with the received speech to generate modified speech, wherein the modified speech has a same meaning as the received speech; anddeliver the stimulation signals to the user to evoke perception of the modified speech.

34. The one or more non-transitory computer readable storage media of claim 33, wherein, when modifying one or more of the auditory comprehension attributes associated with the received speech to generate modified speech, the processor further causes the processor to:modify a wording or inflection of the received speech.

35. The one or more non-transitory computer readable storage media of claim 33, wherein, when modifying one or more of the auditory comprehension attributes associated with the received speech to generate modified speech, the processor further causes the processor to:simplify the received speech.

36. The one or more non-transitory computer readable storage media of claim 33, wherein, when modifying one or more of the auditory comprehension attributes associated with the received speech to generate modified speech, the processor further causes the processor to:increase engagement associated with the received speech.Atty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC137. The one or more non-transitory computer readable storage media of claim 33, 34, 35, or 36, wherein, when modifying one or more of the auditory comprehension attributes associated with the received speech to generate modified speech, the processor further causes the processor to:modify the one or more of the auditory comprehension attributes associated with the received speech to generate modified speech based on user information or environmental information associated with an auditory environment of the user.

38. A method compri sing :receiving, at a device associated with a user, one or more audio signals that include speech;modifying, using a machine learning model, the speech based on an auditory cognitive capacity or preference associated with the user to produce modified speech; and delivering stimulation signals to the user to evoke perception of the modified speech.

39. The method of claim 38, wherein modifying the speech includes modifying a wording or inflection associated with the speech.

40. The method of claim 38, wherein modifying the speech includes modifying a context of the speech.

41. The method of claim 38, wherein modifying the speech includes simplifying the speech without changing a meaning of the speech.

42. The method of claim 38, wherein modifying the speech includes modifying a humor associated with the speech.

43. The method of claim 38, 39, 40, 41, or 42, wherein the machine learning model is a large language model (LLM).

44. The use of a device according to claim 38, 39, 40, 41, 42, or 43, in a cochlear implant, sleep disorder device, a seizure device, a balance or movement disorder device, a tinnitus management device, or a visual device.Atty. Docket No. 3065.0877i Client Ref. No. CID04114W0PC145. A system according to claim 38, 39, 40, 41, 42, or 43, wherein the system is a cochlear implant system, a sleep disorder system, a seizure system, a balance or movement disorder system, a tinnitus management system, or a visual system.

46. A method comprising:receiving, at a sensory device, spatial signals that include target content, wherein the target content comprises comprehension attributes;converting the spatial signals to stimulation signals for delivery to a user of the sensory device;during the converting, modifying one or more of the comprehension attributes associated with the target content to generate modified target content, wherein the modified target content has a same meaning as the modified target content; anddelivering the stimulation signals to the user to evoke perception of the modified target content.

47. The method of claim 46, wherein modifying one or more of the comprehension attributes comprises:modifying the one or more comprehension attributes based on user information associated with the user.

48. The method of claim 46, wherein modifying one or more of the comprehension attributes comprises:modifying the one or more comprehension attributes based on environmental information associated with a spatial environment of the user.

49. The use of the method of claim 46, 47, or 48, in a hearing aid, an implantable auditory prosthesis, a sleep disorder device, a seizure device, a balance or movement disorder device, a tinnitus management device, or a visual device.

50. A system according to method of claim 46, 47, or 48, wherein the system is a hearing aid system, a cochlear implant system, a sleep disorder system, a seizure system, a balance or movement disorder system, a tinnitus management system, or a visual system.

Images