Audio personalization through mental state detection

Abstract

Systems are provided for controlling audio content to a user by a personal audio device. A method includes: receiving sensor data from a sensor of the personal audio device; determining, by a processor, a mental state of the user based on the sensor data; determining, by the processor, at least one audio setting based on the mental state; and controlling, by the processor, audio signals of the personal audio device based on the at least one audio setting.

Description

Attorney Docket No.: 0597-000036-WO-POAAUDIO PERSONALIZATION THROUGH MENTAL STATE DETECTIONFIELD

[0001] The present disclosure relates to audio systems and more particularly to personal audio devices, and more particularly to providing personalization of personal audio devices based on mental state detection.BACKGROUND

[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0003] Hearing choices and preferences vary from individual to individual. Audio personalization is an important method to quantify and deploy these preferences. User’s typically will manually configure audio settings on their personal audio devices to satisfy their audio preferences. However, audio preferences can change over a short time span, for example, preferences can be based on the changing environment, or on the user’s changing mental state. Manually updating audio preferences every time change occurs to a new preference can be tedious and time consuming.

[0004] Accordingly, it is desirable to provide methods and systems for automatically personalizing audio systems for a user. Furthermore, other desirable features and characteristics of the present disclosure will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and the foregoing technical field and background.SUMMARY

[0005] This section provides a general summary of the disclosure and is not a comprehensive disclosure of its full scope or all of its features.

[0006] In one or more forms, the techniques described herein relate to methods and systems for controlling audio content to a user by a personal audio device. In one or more forms, a method includes: receiving sensor data from a sensor of the personal audio device; determining, by a processor, a mental state of the user based on the sensor data; determining, by the processor, at least one audio setting based on the mental state; and controlling, by the processor, audio signals of the personal audio device based on the at least one audio setting.Attorney Docket No.: 0597-000036-WO-POA

[0007] In one or more forms, the sensor is a body sensor.

[0008] In one or more forms, the sensor is a pulse wave sensor.

[0009] In one or more forms, the method includes receiving preference data associated with the mental state from a user, and wherein the determining the at least one audio setting is based on the preference data associated with the mental state.

[0010] In one or more forms, the method includes storing audio settings configuration methods and performing the audio settings configuration methods based on the mental state.

[0011] In one or more forms, the at least one audio setting includes an equalizer setting.

[0012] In one or more forms, the at least on audio setting includes a spatialization setting.

[0013] In one or more forms, the at least one audio setting includes a mode setting.

[0014] In one or more forms, the mode setting includes at least one of a fade in mode, and a fade out mode.

[0015] In one or more forms, the mode setting includes at least one of a stop mode and a playback mode.

[0016] In one or more forms, the at least one audio setting includes a speech enhancement setting.

[0017] In one or more forms, the mental state comprises at least one of a drowsy state, an asleep state, an attentive state, a productive state, a stressed state, a high cognitive load state, a low cognitive load state.

[0018] In other forms a system includes: one or more processors; and a computer-readable storage medium storing instructions which, when executed by the one or more processors, cause the one or more processors to: receive sensor data from a sensor of the personal audio device; determine a mental state of the user based on the sensor data; determine at least one audio setting based on the mental state; and control audio signals of the personal audio device based on the at least one audio setting.

[0019] In one or more forms, the sensor is a body sensor.

[0020] In one or more forms, the instructions cause the one or more processors to receive preference data associated with the mental state from a user,Attorney Docket No.: 0597-000036-WO-POA and to determine the at least one audio setting based on the preference data associated with the mental state.

[0021] In one or more forms, the instructions cause the one or more processors to store audio settings configuration methods and perform the audio settings configuration methods based on the mental state to determine the at least one audio setting.

[0022] In one or more forms, the at least one audio setting includes at least one of an equalizer setting, a spatialization setting, a mode setting, and a speech enhancement setting.

[0023] In one or more forms, the mental state comprises at least one of a drowsy state, an asleep state, an attentive state, a productive state, a stressed state, a high cognitive load state, a low cognitive load state.

[0024] In other forms, a non-transitory, tangible computer-readable storage device storing instructions which, when executed by one or more processors, cause the one or more processors to: receive sensor data from a sensor of a personal audio device; determine a mental state of a user based on the sensor data; determine at least one audio setting based on the mental state; and control audio signals of the personal audio device based on the at least one audio setting.

[0025] In one or more forms, the sensor comprises a body sensor and the mental state comprises at least one of a drowsy state, an asleep state, an attentive state, a productive state, a stressed state, a high cognitive load state, a low cognitive load state.

[0026] Further areas of applicability will become apparent from the description provided herein. It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.BRIEF DESCRIPTION OF DRAWINGS

[0027] In order that the disclosure may be well understood, there will now be described one or more embodiments thereof, given by way of example, reference being made to the accompanying drawings, in which:

[0028] FIG. 1 illustrates an audio system including a personalization system in accordance with one or more implementations of the present disclosure;Attorney Docket No.: 0597-000036-WO-POA

[0029] FIG. 2 is a dataflow diagram illustrating a personalization system of the audio system in accordance with one or more implementations of the present disclosure; and

[0030] FIGS 3, 4, and 5 are flowcharts illustrating personalization methods that may be performed by the personalization system of the audio system in accordance with one or more implementations of the present disclosure.

[0031] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.DETAILED DESCRIPTION

[0032] The following description is merely exemplary in nature and is not intended to limit the present disclosure, application, or uses. It should be understood that throughout the drawings, corresponding reference numerals indicate like or corresponding parts and features.

[0033] In one or more forms, the present disclosure provides for audio systems having a personalization system that automatically configures audios settings based on preferences associated with a detected mental state of the user.

[0034] With reference now to FIG. 1 , an audio system 100 is shown in accordance with one or more forms of the present disclosure. The audio system 100 includes, for example, an audio device 102 that includes or communicates with one or more sensors 104 and a controller 106. The audio device 102 is configured with one or more drivers 108 that generate sound waves for listening to by a user. The audio system 100 is shown in FIG. 1 as a personal audio device, such as in-ear phones. As can be appreciated, the audio system 100 of the present disclosure can include other personal audio devices for presenting audio to a user including but not limited to, headphones, cell phones, tablets, television, speakers, or other portable personal devices configured to provide audio content.

[0035] In various forms, the one or more sensors 104 sense observable conditions initiated by the user. For example, the sensors 104b and 104c sense a user manipulation of a button, switch, or other input device. In various forms, at least one of the one or more sensors 104 includes a body sensor that senses observable conditions of the user’s body. In various forms the body sensor 104a can include, but is not limited to, a photoplethysmography (PPG) sensor or other pulse or heartrate sensor.Attorney Docket No.: 0597-000036-WO-POA

[0036] The controller 106 generates control signals to control the audio drivers 108 to generate the desired sound waves of the audio content. The controller 106 receives sensed data from the one or more sensors 104 and controls the sounds waves based thereon. In various forms, the controller 106 may be integrated within the audio device 102 or be integrated within a separate device and in communication (e.g., wirelessly or wired) with the audio device 102.

[0037] The controller 106 generally operates with any sort of conventional processing hardware, including, but not limited to, at least one processor 110, memory 112, an input / output device 114, a communication device 116, and an operating system 118. The processor 110 may be implemented using any suitable processing system, such as one or more processors, controllers, microprocessors, microcontrollers, processing cores and / or other computing resources spread across any number of distributed or integrated systems, including any number of “cloudbased” or other virtual systems.

[0038] The memory 112 represents any non-transitory short- or longterm storage or other computer-readable media capable of storing programming instructions for execution on the processor 110, including any sort of random access memory (RAM), read only memory (ROM), flash memory, magnetic or optical mass storage, and / or the like.

[0039] The operating system 118 includes computer-executable programming instructions, when read and executed by the processor 110, cause the processor 110 to operate basic functions such as scheduling tasks, executing applications, memory allocation, and controlling the input / output devices 114. The input / output devices 114 generally represents the interface(s) mass storage, data entry devices, and / or the audio actuators. The communication device 116 communicates, for example, audio content, to and / or from other systems according to one or more wired or wireless protocols.

[0040] In various forms, the memory 112 includes a database 120 that stores audio personalization information as disclosed herein. As can be appreciated, the memory 112 represents one suitable implementation of such computer-readable media, and alternatively or additionally, the processor 110 could receive and cooperate with external computer-readable media that is realized as a portable or mobile component or application platform, e.g., a portable hard drive, a USB flash drive, an optical disc, or the like. In various forms, the memory 112 may further storeAttorney Docket No.: 0597-000036-WO-POA a personalization system 122 configured to maintain the personalization information and configure audio settings in accordance with one or more forms of the present disclosure.

[0041] In accordance with one or more use cases of the audio system 100, a user operates a feature of the audio system 100 to listen to audio content. The personalization system 122 monitors the mental state of the user and adjusts the audio content based on defined methods and / or defined personal preferences associated with the mental state.

[0042] In one example, if the personalization system 122 identifies an individual’s descent to a low cognitive load state leading to a sleep state, the personalization system 122 modifies equalizer settings to ease the transition. In addition, once the personalization system 122 identifies that the individual is in a sleep state, the personalization system 122 pauses the audio rendering. This is particularly useful in cases of the audio content including audiobooks where a “bookmark” is important

[0043] In another example, depending on the user’s desired state of mind indicated by their personal preferences, the personalization system 122 modifies the equalizer settings or changes the degree of spatialization to ensure that the individual remains in the desired state. For example, the audio content may be rendered with a flat or default equalizer setting initially. But if the user is detected to be transitioning to an undesired state, the personalization system 122 modifies the equalizer settings by attenuating different frequency bands selectively to nudge the individual back to the desired state. In another example, for a focus mode, the individual may prefer music to be played in the background rather than the foreground

[0044] In still another example, the personalization system 122 identifies an increase in mental load due to poor signal quality and dynamically turns on or increases a level of speech enhancement to reduce the cognitive load. This can be applied for both recorded / broadcast content (passive consumption) or for phone calls and other forms of two way communications. The personalization system 122 turns on automatic closed captioning based on this detection.

[0045] With reference now to FIG. 2, a dataflow diagram illustrates the personalization system 122 in accordance with various forms. As can be appreciated, various exemplary forms of the personalization system 122, according to the present disclosure, may include any number of modules and / or sub-modules embedded withinAttorney Docket No.: 0597-000036-WO-POA the controller 106. In various exemplary forms, the modules and sub-modules shown in FIG. 2 may be combined and / or further partitioned to enable personalization of the audio system 100 based on sensed body data. In various forms, the personalization system 122 includes a personal preferences configuration module 202, a mental state detection module 204, a settings determination module 206, an audio control module 208, a personal preferences datastore 210, and a configuration methods datastore 212.

[0046] The personal preferences configuration module 202 is configured to obtain and organize audio preferences of the user and store the audio preferences in the personal preferences datastore 210. For example, the personal preferences configuration module 202 generates output data 214 (e.g., either audio or visual) requesting input by the user of one or more preferences. In response, the personal preferences configuration module 202 receives as input preference data 216. The preference data 216 may be entered, for example, by a user interacting with one or more sensors 104b, 104c or other inputs associated with the controller 106. The preference data 216 may include a mode of operation (e.g., a foreground mode, a background mode, a stop mode, a pause mode, etc.), an equalizer attribute of an audio signal such as, but not limited to, a volume, a tone, a frequency, etc., and / or a particular mental state (e.g., a drowsy state, a sleep state, an attentive state, a productive state, a stressed state, a high cognitive load state, a low cognitive load state, etc.).

[0047] In various other forms, the preference data 216 may be obtained without a request initiated by the output data 214. For example, the preference data 216 may be learned based on a user’s interaction with input devices to adjust settings in various scenarios of the feature provided by the audio system 100.

[0048] The personal preferences configuration module 202 associates the preferences with a feature of the audio system 100 and stores the associated data in the personal preferences datastore 210. For example, a data entry can be arranged and stored as feature data, mental state data, preference data. As can be appreciated, the features and possible preferences may vary depending upon the features and settings offered by the audio system 100.

[0049] The mental state detection module 204 is configured to determine the user’s mental state and to generate mental state data 218 based thereon. For example, the mental state detection module 204 receives as input sensor data 220Attorney Docket No.: 0597-000036-WO-POA generated by, for example, one or more of the body sensors 104a of the audio system 100. The mental state detection module 204 processes the sensor data 218 to determine a current mental state of the user. In various forms, the sensor data 218 can indicate a heartrate or pulse or other heartbeat pattern of the user. In various forms, any number of thresholds, ranges, or patterns can be associated with the various mental states including, but not limited to, a drowsy state, a sleep state, an attentive state, a productive state, a stressed state, a high cognitive load state, a low cognitive load state, etc.

[0050] The mental state detection module 204 compares the sensor data 218 to the defined thresholds, ranges, or patterns to determine the current mental state. The mental state detection module 204 generates the mental state data 220 to indicate the current mental state. As can be appreciated, other methods of determining the current mental state may be implemented in various forms. For example, machine learning models may be used to evaluate a combination of different sensor data in order to predict the mental state of the user in various forms.

[0051] The audio settings determination module 206 is configured to determine audio settings of the audio system based on the mental state data 220 and to generate audio settings data 222 based thereon. For example, the audio settings determination module 206 receives as input the mental state data 220. The audio settings determination module 206 retrieves personal preferences data 224 from the personal preferences datastore 210 and configuration methods data 226 from the configuration methods datastore 212 based on the mental state data 220. The audio settings determination module 206 generates setting values based on the personal preferences associated with the determined mental state and / or based on defined configuration methods associated with the current feature and determined mental state. The audio settings determination module 206 generates the audio settings data 222 based on the determined values.

[0052] The audio control module 208 controls the audio of the audio system 100 based on the audio settings data 222. For example, the audio control module 208 determines control signal data 228 based on the audio settings data 222. The control signal data 228 controls a mode of operation of the audio system (e.g., change from mode to mode, pausing a current mode, stopping a current mode, starting a mode, etc.), and / or controls an equalizer setting of an audio signal generated while in the mode of operation.Attorney Docket No.: 0597-000036-WO-POA

[0053] With reference now to FIGS. 3, 4, and 5 and with continued reference to FIGS. 1-2, flowcharts illustrate example processes 300, 400, 500 for controlling an audio system 100 based on a detected mental state and / or personal preferences of the user in accordance with various forms. As can be appreciated in light of the disclosure, the order of operations performed by the processes 300, 400, 500 is not limited to the sequential execution as illustrated in FIGS. 3, 4, and 5 but may be performed in one or more varying orders as applicable and in accordance with the present disclosure. In various forms, the processes 300, 400, 500 can be scheduled to run based on one or more predetermined events or run automatically based on an occurrence of one or more events.

[0054] With particular reference to FIG. 3, a flowchart illustrates a process 300 for controlling the audio system 100 based on a mental state in accordance with one or more forms. The process 300 may be performed by the audio system 100 and the personalization system 122 therein in accordance with one or more forms.

[0055] The process 300 may begin at 305. It is determined whether a mental state feature is enabled at 210. When a mental state feature is enabled at 210, the personal preferences are associated with the different mental states and features of the audio system 100 based on the preference data 216 (e.g., either automatically or in response to output data 214 requesting the preferences) and stored in the personal preferences datastore 210 at 312.

[0056] Thereafter, sensor data 218 is monitored at 314 and a mental state is determined at 316. Audio settings data 222 is determined based on the mental state data 220 and the configuration methods data 226 at 318. Thereafter, control signal data 228 is generated to control the audio based on the audio settings data 222 at 320. The process 300 continues so long as feature with mental state detection is enabled at 310. Once the feature is turned off or disabled at 310, the process 300 may end at 322.

[0057] With particular reference to FIG. 4, a flowchart illustrates a process 400 for controlling audio settings of an audio playback feature of the audio system 100 as defined by configuration methods data 226 in accordance with one or more forms. The process 400 may be performed by the audio system 100 and the personalization system 122 therein in accordance with one or more forms.Attorney Docket No.: 0597-000036-WO-POA

[0058] The process 400 may begin at 405. The sensor data 218 is received at 410 and evaluated at 412. When a value of the sensor data 218 is determined to indicate a mental state of drowsiness at 412, then a playback recording of the audio signals is started at 414. The audio settings are adjusted to fade out (e.g., decrease volume of the audio signals) at 416. The duration of the fade out is limited. For example, if a timer is met (e.g., 300 seconds from the moment of detection), the audio signals are stopped at 420. The process 400 proceeds to step 410.

[0059] If, however, the timer is not met at 418, sensor data 218 is received at 420 and evaluated at 422. When a value of the sensor data 218 does not indicate a mental state of drowsiness at 422 (e.g., the user is transitioning out of a drowsy state), then the audio settings are adjusted to fade in (e.g., increase volume of the audio signals) at 424. The process 400 may end at 432.

[0060] If at 412, a value of the sensor data 218 does not indicate a state of drowsiness and it is determined that the user is transitioning from a sleep state or a drowsy state at 425, the user is prompted to playback the recorded audio at 426. If the user does not respond with approval of the playback, the process 400 may end at 432.

[0061] If, however, the user responds with an approval of the playback at 428, the recorded audio is played back at 430. For example, the playback is performed at the beginning of the nearest logical block of playable content from the detected playback point (the beginning of a sentence, a paragraph of an audiobook, the beginning of a song in an album, etc.). Thereafter, the process 400 may end at 432.

[0062] If at 412, a value of the sensor data 218 does not indicate a state of drowsiness and it is determined that the user is not transitioning from a sleep state or a drowsy state at 425, the process 400 may end at 432.

[0063] With reference now to FIG. 5 and with continued reference to FIGS. 1-3, a flowchart illustrates a process 500 for controlling settings of an audio playback feature of the audio system 100 as defined by configuration methods data 226 in accordance with one or more forms. The process 500 may be performed by the audio system 100 and the personalization system 122 therein in accordance with one or more forms.

[0064] The process 500 may begin at 505. The sensor data 218 is received at 510 and evaluated at 512. When a value of the sensor data 218 isAttorney Docket No.: 0597-000036-WO-POA determined to indicate a mental state of drowsiness at 512, then in order to prevent the user from falling asleep, an additional audio track containing noise or distortion of the signal is added to the audio signals. For example, the audio settings are adjusted to fade in (e.g., increase volume or add additional signals) at 514.

[0065] The duration of the fade in is limited. For example, if a timer is met at 516 (e.g., 60 seconds from the moment of detection), the fade in function is fully active, (e.g., the noise sample overlaps the audio file being played back or the volume has been reached), a waring mode may be entered at 518. Thereafter process 500 may end at 520.

[0066] If, however, the timer is not met at 516, sensor data 218 is received at 510 and evaluated at 512. When a value of the sensor data 218 is determined to indicate a mental state of other that than drowsiness8 or asleep at 512, then the audio settings are adjusted to fade out (e.g., decrease volume or remove additional signals) at 522. Similarly, a timer is used to limit the fade out. For example, once a timer is met at 524, the process 500 may end at 520.

[0067] As used herein, the phrase at least one of A, B, and C should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR, and should not be construed to mean “at least one of A, at least one of B, and at least one of C.”

[0068] In this application, the term “controller” and / or “module” may refer to, be part of, or include: an Application Specific Integrated Circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components (e.g., op amp circuit integrator as part of the heat flux data module) that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip.

[0069] The term memory is a subset of the term computer-readable medium. The term computer-readable medium, as used herein, does not encompass transitory electrical or electromagnetic signals propagating through a medium (such as on a carrier wave); the term computer-readable medium may therefore be considered tangible and non-transitory. Non-limiting examples of a non-transitory, tangible computer-readable medium are nonvolatile memory circuits (such as a flashAttorney Docket No.: 0597-000036-WO-POA memory circuit, an erasable programmable read-only memory circuit, or a mask readonly circuit), volatile memory circuits (such as a static random access memory circuit or a dynamic random access memory circuit), magnetic storage media (such as an analog or digital magnetic tape or a hard disk drive), and optical storage media (such as a CD, a DVD, or a Blu-ray Disc).

[0070] The apparatuses and processes described in this application may be partially or fully implemented by a special purpose computer created by configuring a general-purpose computer to execute one or more particular functions embodied in computer programs. The functional blocks, flowchart components, and other elements described above may serve as software specifications, which can be translated into the computer programs by the routine work of a skilled technician or programmer.

[0071] The description of the disclosure is merely exemplary in nature and, thus, variations that do not depart from the substance of the disclosure are intended to be within the scope of the disclosure. Such variations are not to be regarded as a departure from the spirit and scope of the disclosure.

Claims

Attorney Docket No.: 0597-000036-WO-POACLAIMSWhat is claimed is:1 . A method of controlling audio content to a user by a personal audio device, comprising: receiving sensor data from a sensor of the personal audio device; determining, by a processor, a mental state of the user based on the sensor data; determining, by the processor, at least one audio setting based on the mental state; and controlling, by the processor, audio signals of the personal audio device based on the at least one audio setting.

2. The method of claim 1 , wherein the sensor is a body sensor.

3. The method of claim 2, wherein the sensor is a pulse wave sensor.

4. The method of claim 1 , further comprising receiving preference data associated with the mental state from a user, and wherein the determining the at least one audio setting is based on the preference data associated with the mental state.

5. The method of claim 1 , further comprising storing audio settings configuration methods and performing the audio settings configuration methods based on the mental state.

6. The method of claim 1 , wherein the at least one audio setting includes an equalizer setting.

7. The method of claim 1 , wherein the at least on audio setting includes a spatialization setting.

8. The method of claim 1 , wherein the at least one audio setting includes a mode setting.Attorney Docket No.: 0597-000036-WO-POA9. The method of claim 8, wherein the mode setting includes at least one of a fade in mode, and a fade out mode.

10. The method of claim 8, wherein the mode setting includes at least one of a stop mode and a playback mode.11 . The method of claim 1 , wherein the at least one audio setting includes a speech enhancement setting.

12. The method of claim 1 , wherein the mental state comprises at least one of a drowsy state, an asleep state, an attentive state, a productive state, a stressed state, a high cognitive load state, a low cognitive load state.

13. A system for controlling audio content to a user by a personal audio device, comprising: one or more processors; and a computer-readable storage medium storing instructions which, when executed by the one or more processors, cause the one or more processors to: receive sensor data from a sensor of the personal audio device; determine a mental state of the user based on the sensor data; determine at least one audio setting based on the mental state; and control audio signals of the personal audio device based on the at least one audio setting.

14. The system of claim 13, wherein the sensor is a body sensor.

15. The system of claim 13, wherein the instructions cause the one or more processors to receive preference data associated with the mental state from a user, and to determine the at least one audio setting based on the preference data associated with the mental state.

16. The system of claim 13, wherein the instructions cause the one or more processors to store audio settings configuration methods and perform the audioAttorney Docket No.: 0597-000036-WO-POA settings configuration methods based on the mental state to determine the at least one audio setting.

17. The system of claim 13, wherein the at least one audio setting includes at least one of an equalizer setting, a spatialization setting, a mode setting, and a speech enhancement setting.

18. The system of claim 13, wherein the mental state comprises at least one of a drowsy state, an asleep state, an attentive state, a productive state, a stressed state, a high cognitive load state, a low cognitive load state.

19. A non-transitory, tangible computer-readable storage device storing instructions which, when executed by one or more processors, cause the one or more processors to: receive sensor data from a sensor of a personal audio device; determine a mental state of a user based on the sensor data; determine at least one audio setting based on the mental state; and control audio signals of the personal audio device based on the at least one audio setting.

20. The non-transitory, tangible computer-readable storage device of claim 19, wherein the sensor comprises a body sensor and the mental state comprises at least one of a drowsy state, an asleep state, an attentive state, a productive state, a stressed state, a high cognitive load state, a low cognitive load state.

Images