Determination of craniofacial gestures of a wearer of an on-head wearable

Abstract

Techniques and apparatuses are described that perform a determination of craniofacial gestures of a wearer of an on-head wearable. In an example aspect, an intelligent personal assistant device 110 can proactively engage with users based on their craniofacial gestures detected through an on-head wearable device 108, such as earbuds. Using various sensors, the wearable device monitors combinations of head position and facial expressions to anticipate when the wearer might need assistance. The wearable device's sensors monitor head and facial positions and movements, and when they detect a gesture that matches one associated with assistant engagement, they notify the intelligent personal assistant 110 to interact with the wearer. This system allows the assistant to initiate contact at potentially helpful moments rather than waiting for direct commands. Additionally, various craniofacial gestures can trigger various other actions through the device.

Description

DETERMINATION OF CRANIOFACIAL GESTURES OF A WEARER OF AN ON- HEAD WEARABLEBACKGROUND

[0001] Users typically start interacting with an intelligent personal assistant through specific vocal prompts, such as saying ‘OK Google®” to begin an engagement with Google® Assistant™ virtual personal assistant on an Android® platform. This vocal prompt is called a “wake word” and represents one of the various approaches to intelligent personal assistant interaction initiation. Typically, intelligent personal assistant interaction initiation involves users actively indicating their intent to begin an interaction with an intelligent personal assistant.SUMMARY

[0002] Techniques and apparatuses are described that implement a determination of craniofacial gestures of a wearer of an on-head wearable. In an example aspect of the technology described herein, an intelligent personal assistant device can proactively engage with users based on their craniofacial gestures detected through an on-head wearable device, such as earbuds. Using various sensors, the wearable device monitors combinations of head position and facial expressions to anticipate when the wearer might need assistance. The wearable device’s sensors monitor head and facial positions and movements, and when they detect a gesture that matches one associated with assistant engagement, they notify the intelligent personal assistant to interact with the wearer. This system allows the assistant to initiate contact at potentially helpful moments rather than waiting for direct commands. Additionally, various craniofacial gestures can trigger various other actions through the device.

[0003] Aspects described below include a method performed by an on-head wearable for the determination of craniofacial gestures of a wearer. The method includes the reception of a first signal by at least a first sensor of the wearable on-head device. The first signal is indicative of the position of the head of a wearer of the on-head wearable device. The method also includes a the reception of a second signal by at least one second sensor of the on-head wearable device. The second signal is received by the face of the wearer and is indicative of a facial affect display of the face of the wearer. The method also includes determining at least one craniofacial gesture of the wearer based on the first and second signals received.

[0004] Aspects described below also include an apparatus comprising an on-head wearable device comprising a first sensor and a second sensor. The on-head wearable device configured to perform any of the described methods.

[0005] Aspects described below include a computer-readable storage medium comprising computer-executable instructions that, responsive to execution by a processor, cause an on-head wearable device to perform any one of the described methods.

[0006] Aspects described below also include a system with means for performing the determination of craniofacial gestures of a wearer of an on-head wearable.BRIEF DESCRIPTION OF DRAWINGS

[0007] Apparatuses for and techniques for the determination of craniofacial gestures of a wearer of an on-head wearable are described with reference to the following drawings. The same numbers are used throughout the drawings to reference like features and components:FIG. 1 illustrates an example environment in which the determination of craniofacial gestures of a wearer of an on-head w earable can be implemented;FIG. 2 illustrates an example implementation of an intelligent personal assistant that can implement aspects of the determination of craniofacial gestures of a wearer of an on-head wearable;FIG. 3 illustrates an example implementation of an on-head wearable device that can implement aspects of the detennination of craniofacial gestures of a wearer of the on-head earable;FIG. 4 illustrates other example implementations of a computing device that can implement aspects of the determination of craniofacial gestures of a wearer of an on-head wearable;FIG. 5 illustrates examples of head positions and movements that may be indicated by signals received by a head-position sensor of a computing device that can implement aspects of the determination of craniofacial gestures of a w earer of an on-head wearable;FIG. 6 illustrates examples of eye positions of facial affect displays that may be indicated by signals received by a facial-affect-display sensor of a computing device that can implement aspects of the determination of craniofacial gestures of a wearer of an on-head wearable;FIG. 7 illustrates more examples of eye positions and movements of facial affect displays that may be indicated by signals received by a facial-affect-display sensor of a computing device that can implement aspects of the detennination of craniofacial gestures of a w earer of an on-head wearable;FIG. 8 illustrates examples of facial affect displays that may be indicated by signals received by a facial-affect-display sensor of a computing device that can implement aspects of the determination of craniofacial gestures of a w earer of an on-head wearable;FIG. 9 illustrates examples of combinations of head position and facial affect displays that may be indicated by signals received by head-position and facial-affect-display sensors of a computing device that can implement aspects of the determination of craniofacial gestures of a wearer of an on-head wearable;FIG. 10 is an example schematic of a computing device that can implement aspects of the determination of craniofacial gestures of a wearer of an on-head wearable;FIG. 11 illustrates an example method for performing an aspect of the determination of craniofacial gestures of a wearer of an on-head wearable;FIG. 12 illustrates an example method for performing an aspect of the determination of craniofacial gestures of a wearer of an on-head wearable;FIG. 13 illustrates an example computing system embodying, or in which techniques may be implemented that enable use of, the determination of craniofacial gestures of a w earer of an on-head wearable.DETAILED DESCRIPTION

[0008] An intelligent personal assistant is an artificial-intelligence-powered system that interacts naturally with users. Often, users make requests of intelligent personal assistants using natural language. The assistants may respond to the users’ requests by performing tasks such as answering questions, managing schedules, and controlling connected devices. The intelligent personal assistants are often integrated into smartphones, smart speakers, and other such devices that help users with tasks through voice or text commands.

[0009] Typically, a user actively initiates an engagement with an intelligent personal assistant by using a “wake word” to indicate their intent to begin an interaction. In some instances, the user may touch a device to start an interaction with the intelligent personal assistant. Unfortunately, the typical interaction-initiation approaches have limitations. For example, the user must be trained to know the required trigger to initiate an interaction with a nearby intelligent personal assistant, such as a necessary- “wake word” or the right button to press.

[0010] Furthermore, these interaction-initiation approaches are not actually universal. Instead, they are limited to proprietary brands or products. For example, a “wake word” for brand X of an assistant typically differs from the “wake word” for other brand assistants. This brand-specific universality can lead to problems when more than one intelligent personal assistant is present within earshot of a user. In such instances, the “wake word” may initiate interaction with a various assistant than the user intended. Furthermore, the user-assistant interaction seems more artificial by limiting users to a pre-defined “wake word” or button press sequence to engage with an intelligent personal assistant.

[0011] To address this challenge, techniques are described that implement a determination of craniofacial gestures of a wearer of an on-head wearable. In some aspects of the technology described herein, an intelligent personal assistant may initiate engagement with a user based on detecting their state of mind through craniofacial gestures — combinations of head position and facial expressions. In this way, the wearer’s need for assistance by an intelligent personal assistant is anticipated. For example, a person may tilt their head left while looking left, attempting to recall something. In response, the intelligent personal assistant may ask the person if it can assist. In some aspects of the technology described herein, sensors of an on-head wearable device, like earbuds, monitor the wearer’s head position and facial expressions to detect these gestures. When a detected gesture matches one affiliated with assistant engagement, the intelligent personal assistant is notified to engage with the wearer. Various determined craniofacial gestures can also trigger other actions.Operating Environment

[0012] FIG. 1 illustrates an example environment 100 in which the determination of craniofacial gestures of a wearer of an on-head wearable can be implemented. As depicted, the example environment includes two scenes: 100-1 and 100-2.

[0013] Scenes 100-1 include a wearer 102, who is a person seated casually doing nothing in particular, and a close-up insert of the head 104 of the wearer. Wearer 102 is not speaking and not engaging with any computing and smart devices. Wearer 102 has a neutral facial expression, eyes 106 forward and level, and a level and forward-looking head 104. Wearer 102 is wearing a computing device called, herein, an on-head wearable device 108. As depicted, the on-head w earable device 108 is a pair of earbuds.

[0014] An intelligent personal assistant device 110 is resting unused and unengaged nearby, which is depicted as a smartphone in Fig. 1. For example, the smartphone implements an intelligent personal assistant application thereon or accesses such a software sendee via network communications. Examples of such applications or software services include Google® Assistant™ virtual personal assistant, Amazon® Alexa® cloud-based virtual assistant, or personal digital assistant utilizing Gemini™ large language model & API.

[0015] Scene 100-2 is the same as Scene 100-1, but later in time. Thus, Scene 100-2 includes the same items as in Scene 100-1, but again later in time. Scene 100-2 includes wearer 102, but he has moved since scene 100-1. Wearer 102 in Scene 100-2 is wearing the same on-head wearable device 108 and the same intelligent personal assistant device 110 is nearby.

[0016] Compared to Scene 100-1, wearer 102 in Scene 100-2 has changed their state of mind. In Scene 100-1, wearer 102 had no particular state of mind revealed by their face or head movement or position. However, that has changed in Scene 100-2. The state of mind of w earer 102 in Scene 100-2 is that of thinking. More particularly, this person is attempting to remember something. This is indicated by their craniofacial gestures, which are a combination of head position / movement and facial affect displays / movement. As shown best in the close-up insert of Scene 100-2, wearer 102 has their head 104 slightly tilted to their right and their eyes 106 are looking up and to the right.

[0017] Scene 100-2 also includes a detailed callout 120 of the on-head wearable device 108 that reveals some of the high-level components thereof. As depicted, the on-head wearable device 108 includes a head-position sensor 122, a facial-affect-display sensor 124, one or more processors 126, a computer-readable medium (CRM) 128 (which includes memory media and storage media), and a communication (“comm’') interface 130.

[0018] As configured, the head-position sensor 122 receives a signal that indicates the position of the head of a wearer (e.g., head 104 of wearer 102) of the on-head wearable device 108 (e.g., earbuds). Herein, the head-position sensor 122 may be called a first sensor and the signal that it receives may be called a first signal.

[0019] As configured, the facial-affect-display sensor 124 receives a signal that indicates the facial-affect-display of a wearer (e.g., wearer 102) of the on-head wearable device 108 (e.g., earbuds). A facial affect display includes voluntary or involuntary facial expressions. Herein, the facial-affect-display sensor 124 may be called a second sensor and the signal that it receives may be called a second signal.

[0020] Communication interface 130 may be employed to communicate with the intelligent personal assistant device 110. For example, the on-head wearable device 108 may notify the intelligent personal assistant device 110 to initiate engagement with the wearer based upon a determined craniofacial gesture that the wearer is expressing. In determining craniofacial gesture, the actual words or verbal content spoken by the wearer are not taken into consideration. The verbal content is specifically excluded from the craniofacial gesture determination process. However, the facial movements of the w earer’s mouth during speech can be considered as part of their facial affect display, and therefore, these movements may be included as a component of the determined craniofacial gesture.

[0021] In response to the notification from the on-head wearable device 108, the intelligent personal assistant device 110 may, for example, audibly ask the wearer, “How can I help?” As indicated at 112, this is accomplished in Scene 100-2 by the intelligent personal assistant device110 transmiting a digital audio message “How can I help?” to the on-head wearable device 108, which generates the sound through its speakers to wearer 102. The intelligent personal assistant device 110 is further described with respect to FIG. 2. The on-head wearable device 108 is further described with respect to FIG. 3.

[0022] As depicted in FIG. 1, the on-head wearable device 108 and the intelligent personal assistant device 110 are logically and physically distinct and separate computing devices. However, in other instances, the two physically separate devices may be considered the same logical computing system. In other instances still, the intelligent personal assistant device 110 may be logically or physically integrated as part of the on-head wearable device 108.

[0023] FIG. 2 illustrates an example of the intelligent personal assistant device 110. The intelligent personal assistant device 110 is illustrated with various non-limiting example devices, including a smartphone 110-1, a tablet 110-2, a laptop 110-3, a desktop computer 110-4, a computing watch 110-5, smart glasses 110-6, a gaming system 110-7, a smart speaker 110-8. and a vehicle 110-9. Other devices may also be used, such as a home service device, a kitchen appliance, a television, a smart thermostat, a baby monitor, a Wi-FiIMrouter, a drone, a trackpad, a drawing pad, a netbook, an e-reader, a home automation and control system, a wall display, and another home appliance. Note that the intelligent personal assistant device 110 can be w earable, non-wearable but mobile, or relatively immobile (e.g., desktops and appliances).

[0024] The intelligent personal assistant device 110 includes one or more computer processors 202 and at least one computer-readable medium 204, which includes memory media and storage media. Applications and / or an operating system (not shown) embodied as computer-readable instructions on the computer-readable medium 204 can be executed by the computer processor 202 to provide some of the functionalities described herein. The computer-readable medium 204 also includes an intelligent personal assistant application 206, which may use information provided by the on-head wearable device 108 to perform an action. Examples of actions can include responding to a request for engagement with a user, sending a message, initiating communication, performing a search, and providing results thereof.

[0025] The intelligent personal assistant device 110 can also include a network interface 208 for communicating data over wired, wireless, or optical networks. For example, the network interface 208 may communicate data over a local-area-network (LAN), a wireless local-area-network (WLAN), a personal-area-network (PAN), a wire-area-network (WAN), an intranet, the Internet, a peer-to-peer netw ork, point-to-point network, a mesh network, Bluetooth®, and the like.

[0026] The intelligent personal assistant device 110 may also include an output system 210, which may include a display for presenting visual content and / or speakers for generating sound. Forexample, in response to a notification from on-head wearable device 108, the intelligent personal assistant device 110 may, for example, audibly ask the wearer, “How can I be of assistance?” through the speakers of the output system 210. In other instances, the intelligent personal assistant 110 may indicate (e.g., with an audible tone and / or blinking light) that it is active and ready for user engagement.

[0027] In other aspects of the technology described herein, some components and / or functionality of the intelligent personal assistant 110 may be distributed across communicatively coupled computing devices. For example, the intelligent personal assistant device 110 may be implemented, at least in part, on the “cloud.”Determination of Craniofacial Gestures of a Wearer of an On-head Wearable

[0028] FIG. 3 illustrates the on-head wearable device 108. The on-head wearable device 108 is illustrated with various non-limiting example devices including earbuds 108-1, a helmet 108-2, a hat 108-3, goggles 108-4, headphones 108-5, aheadset 108-6, smart glasses 108-7, avirtual reality headset 108-8, and a hairclip 108-9. Other devices may also be used, such as contact lenses, hearing aids, eye patches, headgear, head-wom microphones, augmented reality glasses, headmounted displays, headbands, hair pins, scarf, turbans, tiaras, crowns, bandanas, barrettes, veils, or the like. As implemented, the on-head wearable device 108 is head-wom and, thus, wearable.

[0029] As depicted, the on-head wearable device 108 includes or is associated with, the headposition sensor 122, the facial-affect-display sensor 124, one or more computer processors 126, computer-readable media 128, a craniofacial gesture determination application 320, the communication interface 130, and an output system 322. As depicted, the craniofacial gesture determination application 320 includes a craniofacial gesture (CG) determiner 310, a craniofacial-gesture library 312, a match discemer 314, an action unit 316, and apersonalizer 318.

[0030] As configured, the head-position sensor 122 receives a signal 330 that indicates the position of the head of a wearer (e g., wearer 102 of FIG. 1) of the on-head wearable device 108. This signal 330 herein may be called head-position indicative because it indicates static position information, such as the present position of the wearer’s head. In addition, the head-position indicative signal 330 may also indicate dynamic information, such as head motion. Herein, the head-position sensor 122 may be called a first sensor, and the signal that it receives may be called a first signal.

[0031] In one or more aspects of the technology described herein, the head-position sensor 122 may be implemented as an inertial measurement unit (IMU), which measures movement and orientation using a combination of accelerometers and gy roscopes. In some instances, the head-position sensor 122 receives measurement signals from one or more accelerometers and / or measurement signals from one or more gyroscopes. The head-position indicative signal 330 may be based on one or more of these received signals or a combination thereof. Thus, for some instances, the combined measurements of the accelerometers and gyroscopes may be described herein as a signal received by the head-position sensor 122.

[0032] Since the 1MU is part of the on-head wearable device 108 and, as the name implies, the device is worn on the head, the IMU measures the movement and orientation (e.g., position) of the head of the wearer. The accelerometer measures linear head movement (such as moving forward or backward), while the gyroscope measures rotational head movement (such as turning your head). By combining these measurements, the IMU can provide real-time data about precisely how someone’s head is positioned and moving in three-dimensional space.

[0033] As configured, the facial-affect-display sensor 124 receives a signal 332 that indicates the facial-affect-display of a wearer (e.g., wearer 102) of the on-head wearable device 108. This signal 332 herein may be called a facial-affect-display indicative because it indicates static position information, such as the facial muscles that are presently active. In addition, the facial-affect-display indicative signal 332 also indicates dynamic information, such as changes and motion of facial muscles. Herein, the facial-affect-display sensor 124 may be called a second sensor and the signal that it receives may be called a second signal.

[0034] A facial affect display includes involuntary movements that often reveals a person’s state of mind through facial muscle activity. When people are actively thinking, problem-solving, recalling, or the like, they often exhibit distinctive movements of their facial muscles or combinations of such movements, such as upward glances, furrowed brows, or squinted eyes. A person attempting to recall a memory is often accompanied by upward and / or sideways eye movements and slightly parted lips. Concentration may show through pressed lips and narrowed eyes, while creative thinking often involves unfocused gazing and relaxed facial features.

[0035] People may compress their lips and tense the forehead muscles during mental exertion. Their eyebrows may shift asymmetrically, and their lips may part slightly in movements of confusion. It is common for a person’s eyes to make subtle movements that track imagined visual experiences. When recalling emotional memories, their faces briefly mirror those past feelings through micro-expressions.

[0036] As used herein, a facial affect display covers voluntary or involuntary’ facial expressions. A voluntary facial expression is a mostly deliberate, conscious manipulation of facial muscles to convey a specific message or emotion. For example, people often force a smile for photographs. In contrast, an involuntary facial expression is automatic, often unconscious, facial musclemovements that often reveal a person’s true state of mind. The involuntary facial expressions often reveal the authentic, uncontrolled internal state of mind.

[0037] In one or more aspects of the technology' described herein, the facial-affect-display sensor 124 may be implemented as an electromyograph (EMG), which measures electrical activity in response to a nerve’s stimulation of muscles. Thus, an EMG measures muscle activity or movement. And the EMG does this using conductive contacts that touch a person’s skin.

[0038] Since the EMG is part of the on-head wearable device 108 and the device is worn on the head, the EMG measures the movement of muscles in the face of the wearer. As shown in FIG. 1, the on-head wearable device 108 is a pair of earbuds. One or both earbuds may have conductive skin contacts to touch the wearer’s skin. From these contacts, the EMG may measure movements of muscles in the face, which include eye movements. EMG can detect, for example, activation patterns from key facial muscles like the zygomaticus major (e.g., smiling) and corrugator supercilii (e.g., frowning) while simultaneously monitoring eye-related behaviors like blinks, saccades, and gaze shifts through the orbicularis oculi muscle group. In this way, the EMG may provide real-time data about a person’s facial affect display.

[0039] In one or more aspects of the technology described herein, the facial-affect-display sensor 124 may be implemented as an audioplethysmography (APG), which is an active acoustic method capable of sensing subtle changes observable at a user's outer and middle ear. An APG sensor involves transmitting and receiving acoustic signals that at least partially propagate within a person’s ear canal.

[0040] By transmitting and receiving acoustic signals, the APG sensor can recognize changes in an acoustic circuit (which includes the wearable sealing the ear canal) to receive a signal that is indicative of facial affect displays. The APG sensor may include at least one speaker, and one microphone oriented towards the ear canal to detect a change that occurs in the physical structure of the ear of the wearer. For example, changes to the physical structure include a change in the geometric shape of the ear canal and / or a change in the volume of the ear canal. This change can be caused, at least in part, by subtle blood vessel deformations in the ear canal caused by the person’s heart pumping. Other changes can also be caused by the person’s breathing, movement of the person’s jaw, eyes, and / or other facial movements made by the person.

[0041] The craniofacial determiner 310 obtains the head-position indicative signal 330 received by the head-position sensor 122 and the facial-affect-display indicative signal 332 received by the facial-affect-display sensor 124. Craniofacial determiner 310 determines what craniofacial gesture of the wearer is correlated with the combination of the received head-position indicative signal 330 and the received facial-affect-display indicative signal 332.

[0042] A craniofacial gesture is a non-verbal form of expression, such as voluntary or involuntary' expression. A craniofacial gesture is a combination of head position or movement and facial affect display. In some instances, a craniofacial gesture is fully or mostly involuntary', and a person has no intent to communicate with others. Regardless, such a craniofacial gesture may be unintentionally communicative. For example, raising eyebrows may show surprise, or wrinkling a nose may show disgust. In other instances, a craniofacial gesture is voluntary and may be intended to communicate with others. For example, a head nods up and down may be intended to say “yes” or express agreement.

[0043] A craniofacial gesture often reveals a person's state of mind, which includes their present state of cognitive processing and emotional states. Herein, cognitive processing states refer to how the brain processes and handles information from external and internal sources. For example, this may include attention, perception, memory, language processing, visual or auditory processing, and executive functions. Emotional states include emotions being felt by a person, such as fear, anger, joy, and sadness. Herein, the emotional states of particular interest are those that affect cognitive processing states. Emotional states can influence attention, memory, decision-making, and other aspects of cognitive processing states.

[0044] In some implementations, a dataset (e.g., table) of gesture-signals correlations may be constructed. For example, test subjects may wear the on-head wearable device 108 (or an equivalent) and be taken through a series of craniofacial gestures in a laboratory setting. The resulting dataset will contain a multitude of signal measurements and their corresponding craniofacial gestures. In other or additional implementations, a machine-learned (ML) model may be trained, under supervision, on one or more datasets of gesture-signals correlations to hone the correlations.

[0045] The craniofacial-gesture library 312 is a data library that may include one or more datasets of gesture-signals correlations. In some implementations, the craniofacial-gesture library' 312 may include an ML model trained on such datasets. Some of the craniofacial gestures in the library' are a set of triggering craniofacial gestures. Each gesture in this set has a triggering action associated with it. When the wearer's craniofacial gesture matches a triggering craniofacial gesture, the triggering action associated with the triggering craniofacial gesture is triggered and thus performed.

[0046] The craniofacial-gesture library 312 may include associations between triggering craniofacial gestures and the cognitive process state of the wearer. For example, the wearer may have a “thinking” or “remembering” cognitive process state associated with a triggering craniofacial gesture.

[0047] The match discemer 314 obtains a set of the triggering craniofacial gestures (which may be part of the craniofacial -gesture library 312). Each triggering craniofacial gesture has a triggering action affiliated therewith. Match discemer 314 discerns whether the craniofacial gesture of the wearer (as determined by craniofacial detenniner 310) matches one or more of the triggering craniofacial gestures of the set. When multiple candidates match within a defined tolerance, match discemer 314 may pick an optimal or best candidate or, alternatively, a random candidate.

[0048] If there is a match, action unit 316 initiates the triggering action affiliated with the matched triggering craniofacial gesture. In some instances, action unit 316 performs the triggering action, and in other instances, action unit 316 requests another computing device (e.g., intelligent personal assistant 110) to perform the action. Examples of triggering actions can include sending a notification, requesting the intelligent personal assistant device 110 to initiate engagement with the wearer, sending a message, requesting a message to be sent, initiating communication, requesting communication, requesting a search, dispatching an alert, requesting communication, requesting device activation, requesting device interaction, requesting feedback, and soliciting a response, transmitting a signal, data, files, images, videos, audio, a database, and / or Internet, and triggering an interaction.

[0049] Personalizer 318 gathers personalized entries for one or more personalized datasets of gesture-signals correlations. The personalized dataset includes entries based on data gathered from a particular wearer (such as wearer 102) wearing and using the on-head wearable device 108 (or an equivalent) in actual daily use. The wearer provides feedback to improve the identification of relevant craniofacial gestures of the wearer and provide personalized triggering actions for such gestures. In other or additional implementations, the ML model may be further trained, under supervision based on personalized feedback, on one or more personalized datasets of gesturesignals correlations to hone the correlations.

[0050] The computer processor 126 may include one or more computer processors. Computer-readable media 128 may include one or more computer-readable media. Such media may include memory media or storage media. Applications (e.g., craniofacial gesture determination application 320) and / or an operating system (not shown) implemented as computer-readable instructions on the computer-readable media 128 can be executed by the computer processors 126 to provide some or all of the functionalities described herein, such as some or all of the functions of the craniofacial determiner 310, the match discemer 314, the action unit 316, the personalizer 318. In addition, the computer-readable media 128 may store data structures such as the craniofacial-gesture library 312. As shown in Fig. 3, the computer-readable instructions and the data structuresof the craniofacial determiner 310, the craniofacial-gesture library 312, the match discemer 314, the action unit 316, and the personalizer 318 reside in the computer-readable media 128. However, in other implementations, the functional components of the craniofacial determiner 310, the match discemer 314, the action unit 316, and the personalizer 318 may reside elsewhere.

[0051] The on-head wearable device 108 can use the communication interface 130 for communicating data over wired, wireless, or optical networks. By way of example and not limitation, the communication interface 130 may communicate data over a local-area-network (LAN), a wireless local-area-network (WLAN), a personal-area-network (PAN), a wide-area-network (WAN), an intranet, the Internet, a peer-to-peer network, point-to-point network, or a mesh network.

[0052] The on-head wearable device 108 can use the output system 322, which may include a display for presenting visual content and speakers for generating sound. For example, example earbuds 108-1 will have speakers for sound generation. The on-head wearable device 108 may notify the intelligent personal assistant device 110 to initiate engagement with the wearer. In so doing, the intelligent personal assistant device 110 may transmit a digital audio signal to the wearable device 108 on-head. In response, the on-head wearable device 108 generates audio that asks the wearer, "How can I assist you?” through the speakers of the output system 322.

[0053] As depicted herein, the on-head wearable device 108 and the intelligent personal assistant device 110 are physically and separate computing devices. However, other configurations may be employed in other instances. They may, as shown, be implemented as separate devices, both physically and in how they function. Alternatively, while remaining as two distinct hardware units, they can operate together as a single computing system. In some configurations, the intelligent personal assistant device 110 may be fully integrated into the on-head w earable device 108, either through software integration and / or by physically embedding its components.

[0054] FIG. 4 illustrates examples of other embodiments of an on-head w earable device. As shown in scene 400-1, wearer 102 is wearing the on-head wearable device 108-3, which is a hat. As shown in scene 400-2, wearer 102 is wearing the on-head wearable device 108-5, which are over-the-ear headphones. As shown in scene 400-3, wearer 102 is wearing the on-head wearable device 108-7, wftich is a smartglass device, which may employ an optical sensor as part of the facial-affect-display sensor. The optical sensor can track eye movement effectively.

[0055] FIG. 5 illustrates several non-exhaustive examples 500 of a person exhibiting various static head positions and / or dynamic head positions (e.g., head movements), which can be sensed based on the received head-position indicative signal 330 of the head-position sensor 122. The examples include head left lateral flexion 502 (e.g., slightly tilted left head position), head right lateralflexion 504 (e.g., slightly tilted right head position), head extension 506 (e.g., slightly tilted up head position), head flexion 508 (e.g., slightly tilted down head position), sightly tilted right and rotated right head position 510, movement of head from level to tilt right 512, and movement of head shaking side to side 514. Other examples of static and dynamic head positions include head rotation, left rotation, right rotation, nodding, rolling, swiveling, bobbing, quick jerks, and smooth head-tracking movements.

[0056] FIGs. 6-8 illustrates several non-exhaustive examples 600 of a person exhibiting various facial affect displays, which includes voluntary and / or involuntary facial expressions. Facial affect displays include, for example, eye positions, eye movements, facial muscle contractions, and facial muscle movements. While the content of the wearer’s speech is not sensed or considered, their facial movements during speech can be sensed and considered as part of their facial affect display.

[0057] Facial affect displays include a variety of observable (e.g., by the facial-affect-display sensor 124) facial muscle movements and positions that convey information about the state of mind of a person. These include various eye positions, movements, and various contractions and movements of the facial muscles. When detected by a facial-affect-display sensor, these displays often reveal multiple aspects of the wearer’s state of mind, including their emotional state, social interaction and conversational cues, and cognitive processing. The facial-affect-display sensor can detect eye movements and other facial indicators that provide insight into how a person feels, thinks, and engages with others.

[0058] As illustrated in FIGs. 6-8, the state of mind of a person is indicated, at least in part, by the facial-affect-display signal received by the facial-affect-display sensor. Examples of the state of mind so indicated may include looking up, looking down, looking left, looking right, looking up and to the right, looking up and the left, looking down and to the right, looking down and the right, eyes closed, eyes closing, eyes opening, eyes blinking, blinking frequency, eyes rolling, saccadic eye movement, pursed lips, lip biting, mouth turned up, mouth turned down, smile, one side of mouth raised, open mouth, raised and arched eyebrows, lowered and knit together eyebrows, eyebrows drawn up in the inner comers, thinking, recalling remembering, recalling, imagining, happy, sad, angry, surprised, fearful, disgusted, confused, or combinations thereof.

[0059] FIG. 6 illustrates several non-exhaustive examples 600 of a person exhibiting various eye positions, which can be sensed based on the received facial-affect-display indicative signal 332 of the facial-affect-display sensor 124. The examples include eyes looking right 602, eyes looking left 604, eyes looking up 606, eyes looking down 608, eyes looking up and to the right 610, eyeslooking up and to the left 612, eyes looking down and to the right 614, eyes looking down and to the right 616.

[0060] FIG. 7-1 illustrates several non-exhaustive examples 700-1 of a person exhibiting various eye motions, which can be sensed based on the received facial-affect-display indicative signal 332 of the facial-affect-display sensor 124. The examples include eyes closed 702, eyes closing 704, eyes blinking 706, eyes moving center to left 708, and eyes moving from up and to left to down and to the right 710.

[0061] FIG. 7-2 illustrates several additional non-exhaustive examples 700-2 of a person exhibiting various eye motions, which can be sensed based on the received facial-affect-display indicative signal 332 of the facial-affect-display sensor 124. The examples include eyes rolling 712 and saccadic eye movement 714. Saccadic eye movement often reveals the state of mind of a person. These rapid eye shifts reflect attention patterns. Regular movement often indicates focused engagement, but erratic movement patterns may suggest distraction or anxiety. During increased cognitive toad or complex problem solving (which are examples of cognitive processing states), saccade frequency increases. Memory recall produces distinctive saccadic patterns.

[0062] FIG. 8 illustrates several non-exhaustive examples 800 of zoomed-in portions of a person’ s face that exhibit various voluntary or involuntary facial expressions, which can be sensed based on the received facial-affect-display indicative signal 332 of the facial-affect-display sensor 124. The examples include a person with pursed lips 802, a person biting their lip 804, a mouth of a person turned up 806, a mouth of a person turned down 808, a smiling person 809, a person with one side of their mouth raised 810, an open mouth of a person 812, the raised and arched eyebrows of a person 814. the lowered and knit together eyebrows of a person 816. and a person with eyebrows drawn up in the inner comers 818.

[0063] FIG. 9 illustrates several non-exhaustive examples 900 of craniofacial gestures, w hich can be sensed based on a combination of the received head-position indicative signal 330 of the headposition sensor 122 and the received facial-affect-display indicative signal 332 of the facial-affect-display sensor 124. The examples include a head is slightly tilted to the right and eyes looking to the right 902, a head is slightly tilted to the right and eyes looking up and to the right 904, a head is slightly tilted to the right and eyes looking down and to the right 906, a head is slightly tilt to the left and eyes looking to the left 908, a head is slightly tilt to the left and eyes looking up and to the left 910, and a head is slightly tilt to the left and eyes looking down and to the left 912.

[0064] FIG. 10 is a schematic diagram of a craniofacial gesture training system 1000, which includes one or more computing devices suitable to perform one or more aspects of the technology described herein. As depicted, the craniofacial gesture training system 1000 includes a head-position signal training corpus 1002, a facial-affect-display signal training corpus 1004, a craniofacial gesture training corpus 1006, a classifier 1008, a trainer 1010, and a personalization / calibration dataset 1012. A training corpus is a collection of digital assets and associated metadata that is used to train a ML model or other similar models, such as those employing deep learning, neural networks, natural language processing, computer vision, statistical learning, reinforcement learning, and / or expert systems.

[0065] In particular, the craniofacial gesture training system 1000 performs training, calibration, and / or personalization of the entries in the craniofacial gesture library 312 of the on-head wearable device 108. That library includes craniofacial gestures and their correlations to the combinations of the head-position indicative signal 330 received by the head-position sensor 122 and facial-affect-display indictive signal 332 received by the facial-affect-display sensor 124. Those gesturesignals correlations are assigned (e.g., classified), at least in part, by the craniofacial gesture training system 1000.

[0066] The head-position signal training corpus 1002 is a collection of head-position indicative signals received by the head-position sensor 122 of the on-head wearable device 108. Similarly, the facial-affect-display signal training corpus 1004 is a collection of facial-affect-display indictive signals received by the facial-affect-display sensor 124 of the on-head wearable device 108. Likewise, the craniofacial gesture training corpus 1006 is a collection of labeled craniofacial gestures that a wearer of the on-head wearable device 108 might make.

[0067] In some implementations, the collected signals and craniofacial gestures of the training corpra are synchronized. In some instances, this data may be acquired in a laboratory setting by having test subj ects wear the on-head wearable device 108 (or an equivalent) and be taken through a series of craniofacial gestures in a laboratory setting. The resulting dataset will contain a multitude of signal measurements and their corresponding craniofacial gestures.

[0068] The classifier 1008 receives data from the head-position signal training corpus 1002, facial-affect-display signal training corpus 1004, and craniofacial gesture training corpus 1006 to classify and / or label the data. In particular, classified combinations of signals from the headposition signal training corpus 1002 and facial-affect-display signal training corpus 1004 with labeled or categorized craniofacial gestures of the craniofacial gesture training corpus 1006.

[0069] The trainer 1010 may assist the classifier 1008 in being better at classifications. This may be done, for example, through supervised learning approaches. Such approaches may range from manual to fully automated labeling / classification. The trainer 1010 also helps personalize / calibrate the on-head wearable device 108 for a wearer based onpersonalization / calibration dataset 1012 produced by a personalization or calibration process performed by the personalizer 318 of the device 118.

[0070] The personalization / calibration dataset 1012 may include one or more personalized datasets of gesture-signals correlations generated by the personalizer 318. The personalization / calibration dataset 1012 includes entries based on data gathered from a particular wearer (such as wearer 102) wearing and using the on-head wearable device 108 (or an equivalent) in actual daily use. The wearer provides feedback to improve the identification of relevant craniofacial gestures of the wearer and provide personalized triggering actions for such gestures.

[0071] In some implementations, the craniofacial gesture library 312 may be dataset (e.g.. table) of gesture-signals correlations. In other instances, the craniofacial gesture library 312 may be implemented as a ML model and / or using other approaches such as deep learning, neural networks, natural language processing, computer vision, statistical learning, reinforcement learning, and / or expert systems.Example Methods

[0072] FIGs. 11 to 12 depict example methods 1100 and 1200 for implementing aspects of the determination of craniofacial gestures of the wearer of an on-head wearable. Methods 1100 and 1200 are shown as sets of operations (or acts) performed but not necessarily limited to the order or combinations in which the operations are shown herein. Further, any of one or more of the operations may be repeated, combined, reorganized, or linked to provide a wide array of additional and / or alternate methods. In portions of the following discussion, reference may be made to the environment 100 of FIG. 1, and entities detailed in FIGs. 2 and 3, reference to which is made for example only. The techniques are not limited to performance by one entity or multiple entities operating on one device.

[0073] At 1102 in FIG. 11 , a first signal is received by at least a first sensor of an on-head wearable device. The first signal indicates the position of the head of the wearer of the on-head wearable device. For example, the head-position sensor 122 receives head-position indicative signal 330 that indicates the head position or movement of a wearer of the on-head wearable device 108. The on-head wearable device 108 may include a processor 126, computer-readable media 128, the first sensor 122, and the second sensor 124.

[0074] Examples of some of the head positions or movements indicated by this signal are shown in FIGs. 1, 4, 5, and 9. In some instances, the received first signal can indicate static head positions and dynamic head positions (e.g., head movements). Examples of sensed head positions andmovements that the head-position sensor 122 may sense include flexion, extension, left and right lateral flexion, rotation, shaking, nodding, rolling, swiveling, bobbing, and the like.

[0075] At 1104 in FIG. 11, a second signal is received by at least a second sensor of the on-head wearable device. The second signal indicates the facial-affect-display of the face of the wearer of the on-head wearable device. For example, the facial-affect-display sensor 124 receives a facial-affect-display indicative signal 332 that indicates facial muscle activations (including position and movement) of a wearer of the on-head wearable device 108. Examples of some of the facial muscle movements or positions indicated by this signal are shown in FIGs. 1, 4, 6, 7-1, 7-2, 8, and 9.

[0076] Facial affect displays, which includes voluntary and / or involuntary facial expressions. Facial affect displays include, for example, eye positions, eye movements, facial muscle contractions, and facial muscle movements. Examples of facial affect displays of the wearer indicated by the facial-affect-display sensor include, for example, eye movements, emotional state displays, social displays, conversational signals, and cognitive process state displays.

[0077] Examples of the facial-affect-displays indicated, at least in part, by the signal sensed by the facial-affect-display sensor include eye movements (e.g., looking up, down, left, right, or diagonal combinations; closing, opening, blinking, rolling, and saccadic movements), mouth positions (e.g., pursed lips, lip biting, upturned or downtumed mouth, smiling, asymmetrical mouth raising, open mouth), eyebrow configurations (e.g.. raised and arched, lowered and knitted, inner comers drawn up), and cognitive-emotional states (e.g., thinking, recalling, remembering, imagining), as well as basic emotions (e.g., happiness, sadness, anger, surprise, fear, disgust, confusion) or combinations thereof.

[0078] At 1106 in FIG. 11, a determination is made of at least one craniofacial gesture of the wearer based on a correlation of the received first and second signals. The craniofacial determiner 310 of the on-head wearable device 108 obtains the head-position indicative signal 330 received by the head-position sensor 122 and the facial-affect-display indicative signal 332 received by the facial-affect-display sensor 124. Craniofacial determiner 310 determines what craniofacial gesture of the wearer is correlated with the combination of the received head-position indicative signal 330 and the received facial-affect-display indicative signal 332. Examples of some craniofacial gestures that the wearer may be determined to are shown in FIGs. 1 and 4-9.

[0079] In aspects of the technology described herein, nothing the wearer says (e.g., their verbal content) is considered in determining craniofacial gestures. Thus, as depicted, the detennined craniofacial gesture excludes verbal content expressed by the wearer. However, their mouth movements while speaking may be considered to be part of their facial-affect-display; thus, partof the determined craniofacial gesture determination. However, in some implementations, the verbal content of a wearer’s speech may be considered when determining craniofacial gestures.

[0080] At 1202 in FIG. 12, a set of triggering craniofacial gestures is obtained. The match discemer 314 of the on-head wearable device 108 obtains a set of the triggering craniofacial gestures from, for example, the craniofacial -gesture library 312.

[0081] A triggering craniofacial gesture is one with an assigned or labeled triggering action affiliated therewith. An example of a triggering action includes notifying the intelligent personal assistant device 110 to initiate an engagement with the wearer 102. The gesture-action affiliation may be assigned or classified as part of the training performed by the craniofacial gesture training system 1000 and / or part of the personalization / calibration performed, at least in part, by the personalizer 318. Match discemer 314 discerns whether the craniofacial gesture of the w earer (as determined by craniofacial determiner 310) matches one or more of the triggering craniofacial gestures of the set.

[0082] At 1204 in FIG. 12, the determined craniofacial gesture of the wearer 1206 is obtained from operation 1106 of method 1100 of FIG. 11. Based on the obtained set of the triggering craniofacial gestures and the determined craniofacial gesture of the wearer (as determined by craniofacial determiner 310), the match discemer 314 of the on-head wearable device 108 discerns whether matches one or more of the triggering craniofacial gestures of the set. When multiple candidates match within a defined tolerance, match discemer 314 may pick an optimal or best candidate or, alternatively, a random candidate.

[0083] At 1206 in FIG. 12, w hen there is a match, the triggering action affiliated with the matched triggering craniofacial gesture is initiated and / or performed by, for example, the on-head wearable device 108, the intelligent personal assistant device 110, or another computing device in communication with the wearable device. In response to a match, action unit 316 initiates the triggering action affiliated with the matched triggering craniofacial gesture.

[0084] In some instances, action unit 316 performs the triggering action or directs other components of the on-head wearable device 108 to perform them. In other instances, action unit 316 delegates actions to external computing devices, like the intelligent personal assistant 110. These triggering actions include a wide range of functions, such as sending a notification, requesting the intelligent personal assistant device 110 engage with the wearer, sending or requesting messages, initiating or requesting communication, and requesting a launch of a search of data, files, images, videos, audio, a database, and / or Internet.

[0085] The craniofacial-gesture library 312 may maintain associations between specific craniofacial gestures and the wearer’s cognitive-processing state. These associations help identifywhen the wearer is engaged in mental processes, such as making certain craniofacial gestures that typically indicate deep in thought or actively trying to recall information.Example Computing System

[0086] FIG. 13 illustrates vanous components of an example computing system 1300 that can be implemented as any type of client, server, and / or computing device as described with reference to the previous FIGs. 1, 2, 3, and 10 to implement aspects of the determination of craniofacial gestures of a wearer of an on-head wearable.

[0087] The computing system 1300 includes communication devices 1302 that enable wired and / or wireless communication of device data 1304 (e.g., received data, data that is being received, data scheduled for broadcast, or data packets of the data). The device data 1304 or other device content can include configuration settings of the device, media content stored on the device, and / or information associated with a user of the device. Media content stored on the computing system 1300 can include any type of audio, video, and / or image data. The computing system 1300 includes one or more data inputs 1306 via which any type of data, media content, and / or inputs can be received, such as signals from sensors (e.g., head-position sensor 122 and facial-affect-display sensor 124), human utterances, user-selectable inputs (explicit or implicit), messages, music, television media content, recorded video content, and any other type of audio, video, and / or image data received from any content and / or data source.

[0088] The computing system 1300 also includes communication interfaces 1308, which can be implemented as any one or more of a serial and / or parallel interface, a wireless interface, any type of network interface, a modem, and as any other type of communication interface. The communication interfaces 1308 provides a connection and / or communication links between the computing system 1300 and a communication network by which other electronic, computing, and communication devices communicate data with the computing system 1300.

[0089] The computing system 1300 may include the on-head wearable device 108 or other components thereof. In some instances, the computing system 1300 is in communication with the on-head wearable device 108.

[0090] The computing system 1300 includes one or more processors 1310 (e.g., any of microprocessors, controllers, and the like), which process various computer-executable instructions to control the operation of the computing system 1300. Alternatively, or in addition, the computing system 1300 can be implemented with any one or combination of hardware, firmware, or fixed logic circuitry that is implemented in connection with processing and control circuits, which are generally identified at 1312. Although not shown, the computing system 1300can include a system bus or data transfer system that couples the various components within the device. A system bus can include any one or combination of various bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus that utilizes any of a variety of bus architectures.

[0091] The computing system 1300 also includes a computer-readable medium 1314. such as one or more memory devices that enable persistent and / or non-transitory data storage — in contrast to mere signal transmission — examples of which include random access memory' (RAM), non-volatile memory' (e g., any one or more of a read-only memory' (ROM), flash memory, EPROM. EEPROM, etc.), and a disk storage device. The disk storage device may be implemented as any type of magnetic or optical storage device, such as a hard disk drive, a recordable and / or rewriteable compact disc (CD), any type of a digital versatile disc (DVD), and the like. The computing system 1300 can also include a mass storage medium device (storage medium) 1316.

[0092] The computer-readable medium 1314 provides data storage mechanisms to store the device data 1304, as well as various device applications 1318 and any other types of information and / or data related to operational aspects of the computing system 1300. For example, an operating system 1320 can be maintained as a computer application with the computer-readable medium 1314 and executed on the processors 1310. The device applications 1318 may include a device manager, such as any form of a control application, software application, signal-processing and control module, code that is native to a particular device, a hardware abstraction layer for a particular device, and so on.

[0093] The device applications 1318 also include any system components, engines, or managers to implement the determination of craniofacial gestures of a wearer of an on-head wearable. In this example, device applications 1318 include the intelligent personal assistant application 206 of FIG. 2 or the craniofacial gesture (CG) determination application 320 of FIG. 3.Conclusion

[0094] This description is intended to be illustrative and not restrictive. While the dimensions and types of materials described herein are intended to be illustrative, they are by no means limiting and are examples of embodiments. In the following claims, the terms ‘’first,” ‘’second,” “top,” “bottom,” and the like are used merely as labels and are not intended to impose numerical or positional requirements on their objects. As used herein, an element or step recited in the singular and preceded by the word “a” or “an” should be understood as not excluding the plural of such elements or steps, unless such exclusion is explicitly stated. Additionally, the phrase “at least oneof A and B” and the phrase “A and / or B” should each be understood to mean “only A, only B, or both A and B.”

[0095] Moreover, unless explicitly stated to the contrary , embodiments “comprising’" or “having” an element or a plurality of elements with a particular property may include additional elements not having that property. And when broadly descriptive adverbs such as “substantially” and “generally” are used herein to modify an adjective, these adverbs mean “mostly,” “mainly,” “for the most part,” “to a significant extent,” “to a large degree,” and / or “at least 51% to 99% out of a possible extent of 100%,” and do not necessarily mean “perfectly,” “completely,” “strictly,” “entirely.” or “100%”. Additionally, the word “proximate” may be used herein to describe the location of an object or portion thereof concerning another object or portion thereof and / or to describe the positional relationship of two objects or their respective portions thereof concerning each other and may mean “near,” “adjacent,” “close to, “ “close by, “ “at,” or the like. The phrase “approximately equal to,” as used herein, may mean one or more of "‘exactly equal to. “nearly equal to,” “equal to somewhere between 90% and 110% of,” or the like.

[0096] Although techniques using, and apparatuses including, the determination of craniofacial gestures of a wearer of an on-head wearable have been described in language specific to features and / or methods, it is to be understood that the subject of the appended claims is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as example implementations of the determination of craniofacial gestures of a wearer of an on-head wearable.

[0097] Some Examples are described below.

[0098] Example 1: A method comprising:receiving a first signal, by at least a first sensor of an on-head wearable device, indicative of a position of a head of a wearer of the on-head w earable device;receiving a second signal, by at least one second sensor of the on-head w earable device and from a face of the wearer, the second signal being indicative of a facial affect display of the face of the wearer; anddetermining at least one craniofacial gesture of the wearer based on the received first and second signals.

[0099] Example 2: The method of example 1, wherein the detennining of at least one craniofacial gesture is based on a correlation of the received first and second signals.

[0100] Example 3: The method of any previous example, wherein the facial affect display of the face of the wearer includes involuntary facial expressions.

[0101] Example 4: The method of any previous example, wherein the determined craniofacial gesture excludes verbal content expressed by the wearer.

[0102] Example 5: The method of any previous example, wherein the first sensor includes an inertial measurement unit (IMU) and the second sensor is selected from a group consisting of an electromyograph (EMG) and an audioplethysmograph (APG).

[0103] Example 6: The method of any previous example further comprising:obtaining a set of triggering craniofacial gestures, wherein each triggering craniofacial gesture of the set is affiliated with a triggering action;discerning that the determined at least one craniofacial gesture matches a triggering craniofacial gesture of the set; andin response to the discernment, performing the triggering action affiliated with the matched triggering craniofacial gesture.

[0104] Example 7 : The method of example 6, wherein the matched triggering craniofacial gesture is associated with a cognitive process state of the wearer.

[0105] Example 8: The method any one of examples 6 and 7. wherein the affiliated triggering action performed is selected from a group consisting of sending a notification, requesting engagement from an intelligent personal assistant, sending a message, initiating communication, performing a search of data, and a combination thereof.

[0106] Example 9: The method of any previous example, wherein the position of the head of the wearer indicated by the first signal is selected from a group consisting of static head positions and dynamic head positions.

[0107] Example 10: The method of any previous example, wherein the position of the head of the wearer indicated by the first signal is selected from a group consisting of flexion, extension, left lateral flexion, rotation, right lateral flexion, left rotation, right rotation, nodding, shaking, rolling, quick jerks, smooth head tracking movements, and combinations thereof.

[0108] Example 11 : The method of any previous example, wherein the facial affect display of the wearer indicated by the second signal includes eye movements.

[0109] Example 12: The method of any previous example, wherein the facial affect display of the wearer indicated by the second signal is selected from a group consisting of eye movements, basic emotion displays, social displays, conversational signals, and brain processes displays.

[0110] Example 13: The method of any previous example, wherein the facial affect display of the wearer indicated by the second signal is selected from a group consisting of looking up, looking down, looking left, looking right, looking up and to the right, looking up and the left, looking down and to the right, looking down and the right, eyes closed, eyes closing, eyes opening, eyes blinking, blinking frequency, eyes rolling, saccadic eye movement, pursed lips, lip biting, mouth turned up, mouth turned down, smile, one side of mouth raised, open mouth, raised and arched eyebrows, lowered and knit together eyebrows, eyebrows drawn up in the inner comers, thinking, recalling remembering, recalling, imagining, happy, sad, angry, surprised, fearful, disgusted, confused, and combinations thereof.

[0111] Example 14: The method of any previous example, wherein the on-head wearable device includes a processor, computer-readable media, the first sensor, and the second sensor.

[0112] Example 15: The method of any previous example, wherein the on-head wearable device is selected from a group consisting of eyeglasses, contact lens, hearing aid, eye patch, helmet, hat, headgear, goggles, hard hat, headphone, earbuds, headset, smart glasses, virtual reality headset, augmented reality glasses, head-mounted display, headband, hair clip, hair pin, scarf, turban, tiara, crown, bandana, barrettes, veils, and a combination thereof.

[0113] Example 16: An on-head wearable device comprising:at least one first sensor;at least one second sensor; andat least one processor, the device configured to perform, using the at least one first sensor, the at least one second sensor, and at least one processor, any one of the methods of examples 1-15.

[0114] Example 17: A computer-readable storage medium comprising instructions that, responsive to execution by a processor, cause an on-head wearable device to perform any one of the methods of examples 1-15.

Claims

CLAIMSWhat is claimed is:

1. A method comprising:receiving a first signal, by at least a first sensor of an on-head wearable device, indicative of a position of a head of a wearer of the on-head wearable device;receiving a second signal, by at least one second sensor of the on-head wearable device and from a face of the wearer, the second signal being indicative of a facial affect display of the face of the wearer; anddetermining at least one craniofacial gesture of the wearer based on the received first and second signals.

2. The method of claim 1, wherein the determining of at least one craniofacial gesture is based on a correlation of the received first and second signals.

3. The method of any previous claim, wherein the facial affect display of the face of the wearer includes involuntary facial expressions.

4. The method of any previous claim, wherein the determined craniofacial gesture excludes verbal content expressed by the wearer.

5. The method of any previous claim, wherein the first sensor includes an inertial measurement unit (IMU) and the second sensor is selected from a group consisting of an electromyograph (EMG) and an audiopl ethysmograph (APG).

6. The method of any previous claim further comprising:obtaining a set of triggering craniofacial gestures, wherein each triggering craniofacial gesture of the set is affiliated with a triggering action;discerning that the determined at least one craniofacial gesture matches a triggering craniofacial gesture of the set; andin response to the discernment, performing the triggering action affiliated with the matched triggering craniofacial gesture.

7. The method of claim 6, wherein the matched triggering craniofacial gesture is associated with a cognitive process state of the wearer.

8. The method of any one of claims 6 and 7, wherein the affiliated triggering action performed is selected from a group consisting of sending a notification, requesting engagement from an intelligent personal assistant, sending a message, initiating communication, performing a search of data, and a combination thereof.

9. The method of any previous claim, wherein the position of the head of the wearer indicated by the first signal is selected from a group consisting of static head positions and dynamic head positions.

10. The method of any previous claim, wherein the position of the head of the wearer indicated by the first signal is selected from a group consisting of flexion, extension, left lateral flexion, rotation, right lateral flexion, left rotation, right rotation, nodding, shaking, rolling, quick jerks, smooth head tracking movements, and combinations thereof.

11. The method of any previous claim, wherein the facial affect display of the wearer indicated by the second signal includes eye movements.

12. The method of any previous claim, wherein the facial affect display of the wearer indicated by the second signal is selected from a group consisting of eye movements, basic emotion displays, social displays, conversational signals, and brain processes displays.

13. The method of any previous claim, wherein the facial affect display of the wearer indicated by the second signal is selected from a group consisting of looking up, looking down, looking left, looking right, looking up and to the right, looking up and the left, looking down and to the right, looking down and the right, eyes closed, eyes closing, eyes opening, eyes blinking, blinking frequency, eyes rolling, saccadic eye movement, pursed lips, lip biting, mouth turned up, mouth turned down, smile, one side of mouth raised, open mouth, raised and arched eyebrows, lowered and knit together eyebrows, eyebrows drawn up in the inner comers, thinking, recalling remembering, recalling, imagining, happy, sad, angry, surprised, fearful, disgusted, confused, and combinations thereof.

14. The method of any previous claim, wherein the on-head wearable device includes a processor, computer-readable media, the first sensor, and the second sensor.

15. The method of any previous claim, wherein the on-head wearable device is selected from a group consisting of eyeglasses, contact lens, hearing aid, eye patch, helmet, hat, headgear, goggles, hard hat, headphone, earbuds, headset, smart glasses, virtual reality headset, augmented reality glasses, head-mounted display, headband, hair clip, hair pin, scarf, turban, tiara, crown, bandana, barrettes, veils, and a combination thereof.

16. An on-head wearable device comprising:at least one first sensor;at least one second sensor; andat least one processor, the device configured to perform, using the at least one first sensor, the at least one second sensor, and at least one processor, any one of the methods of claims 1-15.

17. A computer-readable storage medium comprising instructions that, responsive to execution by a processor, cause an on-head wearable device to perform any one of the methods of claims 1-15.

Images