Monitoring cardiac health using video data

Abstract

A method for monitoring cardiac health of a user of a mobile computing device includes activating an imaging device associated with the mobile computing device in response to a trigger indicating a presence of the user within a field of view of the imaging device. The method also includes obtaining, via the imaging device, video data of a face of the user for a time period. Further, the method includes determining, via a machine-learned model programmed in a processor of the mobile computing device, one or more cardiac metrics based on, at least, the video data. The one or more cardiac metrics includes, at least, a heart rate of the user. Furthermore, the method includes determining one or more cardiac biomarkers based, at least in part, on the one or more cardiac metrics. The one or more cardiac biomarkers includes, at least, a resting heart rate of the user. In addition, the method includes causing a display of the mobile computing device to display, at least, the one or more cardiac biomarkers in response to a user input.

Description

MONITORING CARDIAC HEALTH USING VIDEO DATAFIELD

[0001] The present disclosure relates generally to monitoring cardiac health of a user using a mobile computing device. More particularly, the present disclosure relates to passively collecting video data from one or more imaging devices of the mobile computing device and determining cardiac metrics from the collected data.BACKGROUND

[0002] Advances in sensors and wearable technologies have made it increasingly possible for individuals to collect data about themselves with the goal of self-knowledge through personal data. For example, photoplethysmography (PPG) sensors may be included in wearable devices for determining cardiac metrics. However, PPG signals from sensors in wearable devices may be sensitive to device fit. Other techniques for monitoring cardiac metrics include electrocardiogram (ECG) techniques. However, ECG sensors for wearable devices require users to actively perform a measurement. Moreover, use of wearable devices is not as widespread as smartphone use such that fewer users may be able to determine personal cardiac metrics from collected data.

[0003] Furthermore, gaining self-knowledge can be more challenging than only a simple task of data collection. For example, human cardiac health may be influenced by a plurality of factors. As such, cardiac health management may be delayed until after an onset of noticeable symptoms and / or reaching a certain age. Due to this reactive approach, individuals may lack knowledge and motivation to adopt and / or maintain behaviors associated with better cardiac health.

[0004] Accordingly, the present disclosure is directed to a system and method for passively collecting video data with user consent from one or more imaging devices of the mobile computing device, which is used to determine cardiac metrics for a user.SUMMARY

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

[0006] In an aspect, the present disclosure is directed to a method for monitoring cardiac health of a user of a mobile computing device. The method includes activating an imaging device associated with the mobile computing device in response to a trigger indicating a presence of the user within a field of view of the imaging device. The method also includes obtaining, via the imaging device, video data of a face of the user for a time period. Further, the method includes determining, via a machine-learned model programmed in a processor of the mobile computing device, one or more cardiac metrics based on, at least, the video data. The one or more cardiac metrics include, at least, a heart rate of the user. Furthermore, the method includes determining one or more cardiac biomarkers based, at least in part, on the one or more cardiac metrics. The one or more cardiac biomarkers include, at least, a resting heart rate of the user. In addition, the method includes causing a display of the mobile computing device to display, at least, the one or more cardiac biomarkers in response to a user input.

[0007] In another aspect, the present disclosure is directed to a mobile computing device including one or more processors and one or more computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing device to perform operations. The operations include activating an imaging device associated with the mobile computing device in response a trigger indicating a presence of a user within a field of view of the imaging device. The operations also include obtaining, via the imaging device, video data of a face of the user for a time period. Further, the operations include determining, via a machine-learned model programmed in a processor of the mobile computing device, one or more cardiac metrics based on, at least, the video data. The one or more cardiac metrics include, at least, a heart rate of the user. Furthermore, the method includes determining one or more cardiac biomarkers based, at least in part, on the one or more cardiac metrics. The one or more cardiac biomarkers include, at least, a resting heart rate of the user. In addition, the method includes causing a display of the mobile computing device to display, at least, the one or more cardiac biomarkers in response to a user input.

[0008] Other example aspects of the present disclosure are directed to other systems, methods, apparatuses, tangible non-transitory computer-readable media, and devices for performing functions described herein. These and other features, aspects, and advantages of various implementations will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated inand constitute a part of this specification, illustrate implementations of the present disclosure and, together with the description, help explain the related principles.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] A full and enabling disclosure of the present invention, including the best mode thereof, directed to one of ordinary skill in the art, is set forth in the specification, which makes reference to the appended figures, in which:

[0010] FIG. 1 illustrates a schematic diagram of an example computing environment according to embodiments of the present disclosure;

[0011] FIG. 2 A illustrates a schematic diagram of an example mobile computing device according to embodiments of the present disclosure;

[0012] FIG. 2B illustrates a schematic diagram of an example server system according to embodiments of the present disclosure;

[0013] FIG. 2C illustrates a schematic diagram of an example mobile computing device capturing video of a user according to embodiments of the present disclosure;

[0014] FIG. 3 illustrates an example flow chart implemented by a cardiac metric module according to embodiments of the present disclosure;

[0015] FIG. 4 illustrates a flow diagram of a computer-implemented method according to one or more embodiments of the present disclosure;

[0016] FIG. 5 illustrates a flowchart diagram depicting an example method for training a machine-learned model according to embodiments of the present disclosure; and

[0017] FIG. 6 illustrates a block diagram of an example processing flow for using machine-learned model(s) to process input(s) to generate output(s) according to embodiments of the present disclosure.DETAILED DESCRIPTION

[0018] Reference now will be made in detail to embodiments, one or more examples of which are illustrated in the drawings. Each example is provided by way of explanation of the embodiments, not limitation of the present disclosure. In fact, it will be apparent to those skilled in the art that various modifications and variations can be made to the embodiments without departing from the scope or spirit of the present disclosure. For instance, features illustrated or described as part of one embodiment can be used with another embodiment toyield a still further embodiment. Thus, it is intended that aspects of the present disclosure cover such modifications and variations.Overview

[0019] Repeated use of reference characters and / or numerals in the present specification and / or figures is intended to represent the same or analogous features, elements, or operations of the present disclosure. Repeated description of reference characters and / or numerals that are repeated in the present specification is omitted for brevity.

[0020] As referred to herein, the terms “includes” and “including” are intended to be inclusive in a manner similar to the term “comprising.” As referenced herein, the terms “or” and “and / or” are generally intended to be inclusive, that is (i.e.), “A or B” or “A and / or B” are each intended to mean “A or B or both.” As referred to herein, the terms “first,” “second,” “third,” and so on, can be used interchangeably to distinguish one component or entity from another and are not intended to signify location, functionality, or importance of the individual components or entities. As referenced herein, the terms “couple,” “couples,” “coupled,” and / or “coupling” refer to chemical coupling (e.g., chemical bonding), communicative coupling, electrical and / or electromagnetic coupling (e.g., capacitive coupling, inductive coupling, direct and / or connected coupling, etc.), mechanical coupling, operative coupling, optical coupling, and / or physical coupling.

[0021] As referenced herein, the term “system” can refer to hardware (e.g., application specific hardware), computer logic that executes on a general-purpose processor (e.g., a central processing unit (CPU)), and / or some combination thereof. In some embodiments, a “system” described herein can be implemented in hardware, application specific circuits, firmware, and / or software controlling a general -purpose processor. In some embodiments, a “system” described herein can be implemented as program code files stored on a storage device, loaded into a memory, and executed by a processor, and / or can be provided from computer program products, for example, computer-executable instructions that are stored in a tangible computer-readable storage medium (e.g., random-access memory (RAM), hard disk, optical media, magnetic media).

[0022] Generally, the present disclosure is directed to a computing device configured to passively monitor cardiac biomarkers of a user. According to example embodiments of the present disclosure, an imaging device of a mobile computing device (e.g., a laptop, a tablet, a smartphone, etc.) can be activated in response to a trigger indicating a presence of the user within a field of view of the imaging device. Further, the mobile computing device canobtain video data of a face of a user during a time period. Furthermore, the mobile computing device can determine, via a machine-learned model of the mobile computing device, one or more cardiac metrics relating to the first time period based on, at least, the video data. The cardiac metric(s) include, at least, a heart rate. Moreover, the mobile computing device can determine one or more cardiac biomarkers based, at least in part, on the cardiac metric(s). The cardiac biomarker(s) include, at least, a resting heart rate. In addition, the mobile computing device can cause a display of the mobile computing device to display, at least, the cardiac marker(s) in response to a user input. As discussed herein, a cardiac metric refers to a quantitative measurement related to performance and / or function of a user’s cardiovascular system, and a cardiac biomarker refers to information indicating cardiovascular disease, cardiovascular damage, and / or the presence of other health related conditions (such as diabetes, hypertension, etc.), or general cardiovascular fitness.

[0023] Systems and methods in accordance with example embodiments of the present disclosure provide a number of technical effects and benefits. In particular, the systems and methods can include a computing device such as a battery-operated mobile computing device with reduced computing capacity (e.g., relative to larger computing devices such as servers, desktops, etc.) configured to passively determine cardiac biomarkers of a user. As one example of a technical effect, passive data collection during normal activities for the user can allow for determining cardiac metrics from data that is associated with normal user activities as opposed to data collected during standardized user activities, which can thereby lead to improved assessment of the user’s cardiac health. Further, by leveraging a mobile computing device to obtain video data for determining the cardiac metrics, the mobile computing device can permit collection of video data in various circumstances during the user’s day, which can allow for a better understanding of circumstances that impact a user’s cardiac health.Moreover, the computing device can utilize only local computational resources for passively monitoring the user’s cardiac health, which can assist in ensuring that the collected data relating to the user is safeguarded.

[0024] Further, a user may be provided with privacy-related controls allowing the user to make an election as to both if and when systems, programs, or features described herein may enable collection of health-related data and / or user information (e.g., information about a user’s social network, social actions, or activities, profession, a user’s preferences, or a user’s current location), and if the user is sent content or communications that may be of a sensitive or private nature from a server. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed.For example, a user’s identity may be treated so that no personally identifiable information can be determined for the user, or a user’s geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over what information is collected about the user, how that information is used, and what information is provided to the user. To that end, any information collected as described herein relating to the user (e.g., personal medical data, health conditions, etc.) is capable of being kept private and confidential and not being improperly used or published.

[0025] Moreover, one or more security measures can be implemented to ensure that the demographic data and / or physiological data of the user is safeguarded. For example, passcode or fingerprint authentication may be used to control access to the demographic data and the physiological or otherwise personal data of the user. Further, such data of the user can be stored in a privacy enhancing manner and not shared without the express consent of the user. For example, such data can be encrypted to secure the data from unauthorized access.

[0026] Example aspects of the present disclosure provide several technical effects, benefits, and / or improvements in computing technology.Example Devices and Systems

[0027] Referring now to the drawings, FIG. 1 illustrates a schematic diagram of a computing environment 100 according to one or more example embodiments of the present disclosure. As shown, the computing environment 100 may be implemented as a client server computing environment, including one or more server computing systems 112 and one or more mobile computing devices 200. Although the mobile computing device 200 is depicted as a smartphone in the embodiment illustrated in FIG. 1, it should be understood that the present disclosure is not so limiting. For example, the mobile computing device according to example embodiments may include, for example, a personal digital assistant (PDA), a tablet, a personal computer, a laptop computer, a smart television, a video game console, a wearable computing device, and / or another electronic and / or computing device that can collect image data of a user.

[0028] With reference to the example embodiment described above and depicted in FIG. 1, the mobile computing device 200 is configured to passively monitor cardiac health of a user. As such, in certain embodiments, the mobile computing device 200 can be capable of and / or configured to collect video data of a user when the mobile computing device 200 isbeing unlocked and / or to determine cardiac metric(s) using such data. Accordingly, in an embodiment, the mobile computing device 200 may be configured to determine cardiac biomarker(s) based on the cardiac metric(s), which will be described further below. The cardiac metric(s) can include, but are not limited to, heart rate (HR), average heart rate, maximum heart rate, heart rate recovery, and blood pressure. Although the cardiac biomarker is described herein as being a resting heart rate (RHR), it should be understood that the present disclosure is not so limiting. For example, the cardiac biomarker may include, but is not limited to, heart rate variability (HRV), blood pressure variability (BPV), or any other suitable biomarker (i.e., indicator of cardiac health issues) that can be derived from the cardiac metric(s).

[0029] Referring particularly to FIG. 2A, the mobile computing device 200 according to one or more embodiments of the present disclosure is illustrated. In general, the mobile computing device 200 is described herein with reference to a smartphone. However, it should be appreciated that the mobile computing device 200 may be implemented as any suitable mobile device, including, but not limited to, a smartwatch, a laptop, a tablet, a desktop computer, or any other computing device that is configured to perform and / or implement the response leveraging principles and features disclosed herein.

[0030] The mobile computing device 200 may include one or more processor(s) 210. In an embodiment, the processor(s) 210 can be configured to execute computer-readable instructions that, when executed, cause mobile computing device 200 to perform one or more operations. In an embodiment, the processor(s) 210 can be configured to execute operational code (e.g., instructions, processing threads, software) for mobile computing device 200 such as, for instance, firmware or the like. In the example embodiment depicted in FIG. 1, the processor(s) 210 can each be a central processing unit (CPU), microprocessor, microcontroller, integrated circuit (e.g., an application-specific integrated circuit (ASIC)), and / or another type of processing device. In an embodiment, the processor(s) 210 can be coupled to (e.g., electrically, communicatively, physically, operatively) to one or more components of the mobile computing device 200 such that the processor(s) 210 can facilitate one or more operations in accordance with the embodiments described herein.

[0031] In an embodiment, as shown in FIG. 2 A, the computer-readable instructions and / or operational code that can be executed by the processor(s) 210 can be stored in a memory device 212 of the mobile computing device 200. The memory device 212 can store computer-readable and / or computer executable entities (e.g., data, information, applications, models, algorithms) that can be created, modified, accessed, read, retrieved, and / or executedby each of the processor(s) 210. For example, a sequence processing model can be stored in the memory device 212 of the mobile computing device 200. In some embodiments, the memory device 212 can constitute, include, be coupled to (e.g., operatively), and / or otherwise be associated with a computing system and / or media such as, for example, one or more computer-readable media, volatile memory, non-volatile memory, random-access memory (RAM), read only memory (ROM), hard drives, flash drives, and / or other memory devices. In these or other embodiments, such one or more computer-readable media can include, constitute, be coupled to (e.g., operatively), and / or otherwise be associated with one or more non-transitory computer-readable media.

[0032] The mobile computing device 200 may also include the display 214. The display 214 may include any type of electronic display or screen known in the art. For example, in some embodiments, the display 214 may include a liquid crystal display (LCD) or organic light emitting diode (OLED) display such as, for instance, a transmissive LCD display or a transmissive OLED display. Further, the display 214 can be configured to provide brightness, contrast, and / or color saturation features according to display settings that can be maintained by the processor(s) 210. In some embodiments, the display 214 may include a touchscreen such as, for instance, a capacitive touchscreen. For example, in these embodiments, the display 214 may include a surface capacitive touchscreen or a projective capacitive touch screen that can be configured to respond to contact with electrical chargeholding members or tools, such as a human finger.

[0033] In further embodiments, the mobile computing device 200 may also include at least one additional human-machine interface (HMI) 220 configured to allow the processor(s) 210 to receive conventional inputs from a user. As such, the HMI 220 can enable a user to interact with the mobile computing device 200. In some embodiments, the HMI 220 can include one or more interfaces that provide information to a user, such as a display screen, a speaker, etc., and can also include one or more interfaces that allow a user to interact with information displayed on the screen, such as including a touch-screen component, a mouse component, a keyboard component, a stylus component, and the like. In some embodiments, the HMI 220 can receive information from a user. For example, the HMI 220 can receive user inputs (e.g., via sensors detecting a user pressing a virtual button on a touchscreen, via the mouse component receiving a user input specifying selection of information displayed on the screen, via the keyboard component receiving a user input specifying alphanumeric information, etc.) specifying information to the mobile computing device 200.

[0034] The mobile computing device 200 may also include one or more imaging devices 222, such as a camera (e.g., visible spectrum camera, infrared camera, hyperspectral camera, etc.). In addition, the imaging device(s) 222 may include an imaging sensor (e.g., a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD)) to capture, detect, or recognize a user’s behavior, figure, expression, etc. The video data can include individual image frames captured by the imaging device(s) 222. The image frames can be expressed in any number of different color spaces (e.g., greyscale, RGB, CMYK, etc.). As another example, the image frames can be generated by a Light Detection and Ranging (“LIDAR”) system or a Radio Detection and Ranging (“RADAR”) system. In other embodiments, the imaging device 222 may be separate from the mobile computing device 200 and in communication with the mobile computing device 200 (e.g., via wired or wireless connections).

[0035] The mobile computing device 200 may also include one or more sensors 224. For example, the sensor(s) 224 may include an inertial measurement unit (IMU) which includes one or more accelerometers and / or one or more gyroscopes. The accelerometer(s) may be used to capture motion information with respect to the mobile computing device 200. The gyroscope(s) may also be used additionally or alternatively to capture motion information with respect to the mobile computing device 200. For example, the IMU may be configured as a six-axis or six-dimensional inertial measurement unit (e.g., a tri-axial accelerometer and a tri-axial gyroscope). The motion information obtained via the IMU may be associated with the user when the mobile computing device 200 is worn or carried by the user. The IMU may further include one or more magnetometers, one or more barometers, and / or any other suitable sensors.

[0036] The mobile computing device 200 may also include one or more power components 216, such as a battery operable to be recharged through conventional plug-in approaches, or through other approaches such as capacitive charging through proximity with a power mat or other such device.

[0037] The mobile computing device 200 may also include one or more wireless components 218 operable to allow the processor(s) 210 to communicate with one or more electronic devices within a communication range of a particular wireless channel. The wireless channel can be any appropriate channel used to enable devices to communicate wirelessly, such as Bluetooth, cellular, NFC, Ultra-Wideband (UWB), or Wi-Fi channels. It should be understood that the mobile computing device 200 can have one or more conventional wired communications connections as known in the art.

[0038] The mobile computing device 200 may also include a cardiac monitoring module 300 and / or other modules and / or data that can be used to facilitate one or more operations described herein. The cardiac monitoring module 300 may include one or more hardware and / or software components and / or features that can be configured to perform cardiac monitoring of the user 250, as described further below.

[0039] In an embodiment, the communication between the server computing system 112 and the mobile computing device 200 can be facilitated by the network(s) 106. Any number of client computing devices and / or server computing devices can be included in the client-server environment and communicate over a network. The network(s) 106 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof. In general, communication between the computing devices can be carried via a network interface using any type of wired and / or wireless connection, using a variety of communication protocols (e.g., TCP / IP, HTTP, RTP, RTCP, etc.), encodings or formats (e.g., HTML, XML, etc.), and / or protection schemes (e.g., VPN, secure HTTP, SSL, etc ).

[0040] Referring now to FIG. 2B, the mobile computing device 200 can transmit the cardiac metrics of the user to the server computing system 112 (e.g., via the network(s) 106 of FIG. 1). In this embodiment, the server system 112 can use the received data to update a user profile for the user that can be stored in a database 114 (e.g., a log) of a memory 116 of the server system 112. In some embodiments, the server computing system 112 can transmit a machine-learned model, as will be described further below, to the mobile computing device 200 (e.g., via the network(s) 106 of FIG. 1) so as to permit the mobile computing device 200 to determine the cardiac metrics of the user. In such embodiments, the server system 112 may train the machine-learned model and provide the trained machine-learned model to the mobile computing device 200.

[0041] In some embodiments, the server computing system 112 can be implemented on one or more standalone data processing apparatuses or a distributed network of computers. In some embodiments, the server computing system 112 can employ various virtual devices and / or services of third-party service providers (e.g., third-party cloud service providers) to provide the underlying computing resources and / or infrastructure resources of the server computing system 112. In some embodiments, the server computing system 112 can include, but is not limited to, a handheld computer, a tablet computer, a laptop computer, a desktop computer, or a combination of any two or more of these data processing devices or other data processing devices.

[0042] The server computing system 112 can include one or more processors 118 such as, for instance, one or more CPUs. In these or other embodiments, the server computing system 112 can include one or more network interfaces 120 that can include, for example, an input / output (I / O) interface to the mobile computing device 200. In some embodiments, the server computing system 112 can include one or more communication buses for interconnecting these components.

[0043] The memory 116 according to example embodiments can include high-speed random-access memory such as, for instance, DRAM, SRAM, DDR RAM, or other randomaccess solid-state memory devices; and, optionally, can include non-volatile memory such as, for example, one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid state storage devices. The memory 116, optionally, can include one or more storage devices that can be remotely located from the processor(s) 118 (e.g., processing unit(s)). Further, the memory 116, or alternatively the non-volatile memory within the memory 116, can include a non-transitory computer readable storage medium. In some embodiments, the memory 116, or the non-transitory computer readable storage medium of the memory 116, can store one or more programs, modules, and data structures. In these embodiments, such programs, modules, and data structures can include, but not be limited to, one or more of an operating system that can include procedures for handling various basic system services and for performing hardware dependent tasks.

[0044] Referring now to FIG. 2C, a schematic diagram of the mobile computing device 200 capturing data of a user 250 according to example embodiments of the present disclosure is illustrated. In an embodiment, as shown, video data may be obtained by the mobile computing device 200 via the imaging device 222 thereof. For example, the imaging device 222 of the mobile computing device 200 has a field of view 223. The mobile computing device 200 can be positioned such that a face 252 of the user 250 is within the field of view 223 of the imaging device 222. Video data of user 250 can be captured and analyzed as described herein to determine cardiac metrics of the user 250.

[0045] For example, the imaging device 222 can be activated to obtain video data of the face 252 of the user 250 in response to a trigger indicating a presence of the user within the field of view 223 of the imaging device 222. The trigger may include, but is not limited to, sensor 224 data (e.g., obtained via the IMU) indicating movement of the mobile computing device 200, a user input (e.g., received via the HMI 220) indicating the user isinteracting with the mobile computing device 200, and / or any other suitable indication that the user is present within a field of view of the imaging device 222.

[0046] As an example, the trigger may occur when the mobile computing device 200 is being unlocked to permit the user 250 to access the mobile computing device 200 and features thereof. In such an example, the user 250 may hold the mobile computing device 200 such that the face 252 is within the field of view 223 of the imaging device 222 when the user 250 is attempting to unlock the mobile computing device 200. In some embodiments, the user 250 may attempt to unlock the mobile computing device 200 via facial recognition security features. As such, video data may be obtained while facial recognition security features are performed to determine whether to unlock the mobile computing device 200. In other embodiments, the user 250 may attempt to unlock the mobile computing device 200 by providing a user input, such as a passcode, touching a fingerprint scanner, etc.

[0047] Although the trigger is described herein as occurring during an unlocking of the mobile computing device 200, it should be understood that the present disclosure is not so limiting. For example, the trigger may occur during any suitable event that periodically occurs so as to permit passive collection of video data of a face of the user 250. That is, the imaging device 222 can be activated to obtain the video data in response to any periodic event such that a user input actively selecting video data collection for cardiac monitoring is not required.

[0048] Moreover, the imaging device 222 can obtain the video data of the face 252 of the user 250 for a time period. The imaging device 222 may be deactivated in response to expiration of the time period. A duration of the time period may, for example, be a fixed duration. In such an example, the duration may be determined empirically (e.g., based on testing to determine a minimum duration of the video data that permits cardiac monitoring according to the features and operations described herein). As another example, the duration of the time period may be a variable duration based on data collected while the video data is being obtained. For example, sensor 224 data may indicate (e.g., via data collected by the accelerometer and / or gyroscope) that movement of the mobile computing device 200 is greater than a threshold amount (e.g., determined based in image stabilization capabilities of the mobile computing device 200) while the video data is being obtained. As another example, the mobile computing device 200 can be configured to determine (e.g., based on utilizing image processing techniques to analyze the video data) that lighting around the user 250 lacks sufficient brightness to sufficiently illuminate the face 252 of the user 250 for cardiac monitoring according to the features and operations disclosed herein. As yet anotherexample, the mobile computing device 200 can be configured to detect a presence or an absence of the user 250 within the field of view 223 of the imaging device 222 (e.g., based on a face detector, as described below). For instance, the mobile computing device 200 may detect (e.g., via sensor 224 data, a user input, etc.) the trigger despite the user being absent of the field of view 223 of the imaging device 222 (e.g., the mobile computing device 200 may be inaccessible to the user 250, such as when the mobile computing device 200 is stored in luggage or jacket pockets, and the trigger may be detected based on movement of the mobile computing device 200 or contact with another object stored alongside the mobile computing device 200).

[0049] In some embodiments, the duration of the timer may be increased based on the collected data so as to permit stabilization of the mobile computing device 200, the user 250 to enter the field of view 223 of the imaging device 222, and / or improved lighting conditions around the user 250. In other embodiments, the duration of the timer may be decreased based on the collected data so as to deactivate the imaging device 222 (e.g., due to the absence of the user 250 from the field of view 223, excessive movement by the user, the absence of the user from the field of view 223, and / or the lighting around the user 250 lacking sufficient brightness). In such embodiments, the duration of the timer may be reduced to zero such that activation of the imaging device 222 is prevented until the collected data indicates improved conditions for obtaining video data 305 for cardiac monitoring. The duration of the timer may also be dependent on confidence scores associated with respective historical cardiac metrics 320 (as described further below). For example, the duration of the timer may be decreased when a predetermined number of consecutively determined confidence scores are less than a threshold score.

[0050] In some embodiments, data associated with the presence of the user 250, movement of the user 250 and / or the lighting around the user 250 can be collected while the video data is being obtained. In such embodiments, the mobile computing device 200 can be configured to provide the video data for cardiac monitoring, as described further below, based on the collected data. For example, the sensor 224 data may indicate (e.g., via data collected by the accelerometer and / or gyroscope) that the user 250 participated in an activity that is likely to cause elevated heart rates (e.g., running, climbing stairs, and / or other exercise activities) immediately prior to the trigger. As another example, the mobile computing device 200 can be configured to determine (e.g., based on utilizing image processing techniques to analyze the video data) that lighting around the user 250 lacks sufficient brightness to sufficiently illuminate the face 252 of the user 250 for cardiac monitoringaccording to the features and operations disclosed herein. As yet another example, the mobile computing device 200 can be configured to detect the presence of the user 250 within the field of view 223 of the imaging device 222, as mentioned above. When the collected data indicates an absence of the user 250 from the field of view 223, insufficiently bright lighting around the user 250 and / or performance of heart rate elevating activities by the user 250, the video data can be prevented from being provided for cardiac monitoring.Conversely, when the collected data indicates the presence of the user 250 within the field of view 223, sufficiently bright lighting around the user 250 and non-performance of heart rate elevating activities, the video data can be provided for cardiac monitoring, as described further below.

[0051] Referring now to FIG. 3, the cardiac monitoring module 300 is configured to, with consent of the user 250, determine one or more cardiac biomarkers 330 of the user 250. As set forth above, the cardiac biomarker(s) 330 can include at least a resting heart rate. The cardiac monitoring module 300 may be configured to receive one or more user inputs (e.g., via the HMI 220) consenting to cardiac monitoring. In such an example, the cardiac monitoring module 300 can be configured to display, via the display 214, an enablement screen including a virtual button, or the like, that the user 250 can select (e.g., via a user input) to enable or disable cardiac monitoring. As such, when disabled, the cardiac monitoring module 300 may be prevented from obtaining the video data 305.

[0052] In some embodiments, the cardiac monitoring module 300 can be configured to display, via the display 214, a condition screen including one or more virtual buttons, text fields, etc. configured to permit the user 250 to specify (e.g., via another user input) restrictions (e.g., timed-based, location-based, activity-based, etc.) on when video data 305 is not permitted to be obtained. Furthermore, the cardiac monitoring module 300 can be configured to provide one or more visual, audio, and / or haptic outputs (e.g., activating a speaker to output a sound, activating a light on the mobile computing device 200, displaying a message via the display 214, vibrating the mobile computing device 200 for a specified duration, etc.) configured to indicate to the user 250 that the video data 305 is being obtained. When enabled, the cardiac monitoring module 300 can be configured to obtain the video data 305 based on the specified restrictions and / or to provide the visual, audio, and / or haptic output(s) while obtaining the video data 305.

[0053] The cardiac monitoring module 300 may include a pre-processor 310 that can receive video data 305 of the face 252 of the user 250. As discussed above, the video data of the face 252 of the user 250 can be obtained using the imaging device 222 of the mobilecomputing device 200 in response to the trigger. In embodiments, the pre-processor 310 can be a software program loaded in memory and executed by a processor 210 included in the mobile computing device 200.

[0054] In some embodiments, the pre-processor 310 may include the face detector. The face detector can be a computer-implemented component that is configured to detect human faces in frames of video data 305 (e.g., specifically the face and not other body portions). Thus, given an input frame of video data 305, the face detector can output one or more face detections, where each face detection indicates a respective detected face location of a respective detected face in the input frame. The face detector may be a machine-learned model trained to perform human face detection operations using machine learning techniques.

[0055] The machine-learned model can be or include one or multiple machine-learned models or model components. Example machine-learned models can include neural networks (e.g., deep neural networks) or a generative model (e.g., large language models (LLM), non-linear models or linear models, decision tree based models, support vector machines, hidden Markov models, Bayesian networks, and / or k-means clustering models, etc.). Example machine-learned models can also use other architectures in lieu of or in addition to those models specifically mentioned herein.

[0056] Neural networks, such as those described herein, can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, and / or other forms of neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models). In another embodiment, the machine learning models described herein may include a rule-based approach, wherein actions are chosen based on a predetermined set of if-then rules or mathematical expressions with pre-defined parameters.

[0057] The location of the face can be indicated in a number of different ways. As one example, a face detection can indicate the location of the detected face as a bounding shape. The bounding shape can be a two-dimensional shape. The bounding shape can identify a set of pixels (or voxels) in the frame that correspond to the location of the face within the frame. The bounding shape can be a polygonal shape (e.g., a box or cube), a curved shape (e.g., circle or oval), or an arbitrary shape (e.g., a segmentation mask that identifies a specific set of pixels or voxels that encompass the detected face). In some embodiments, the bounding shape can be scaled (e.g., increased by a given factor) so as to ensure the entire face is captured within the bounding shape.

[0058] As another example, in addition or alternatively to the bounding shape, the face detection can indicate the location of the corresponding detected face through pose data describing a pose of the face. For example, the pose data can identify the location of one or more facial landmarks of the detected face. For example, the facial landmark(s) can include eye landmarks, nose landmarks, eyebrows landmarks, lip landmarks, chin landmarks, or other facial landmarks. In some embodiments, the pose data descriptive of the locations of the landmarks simply includes a number of points that correspond to the location of the landmarks. In other embodiments, the pose data descriptive of the locations of the landmarks includes a connected representation of the face in which the various landmarks (e.g., all lip landmarks) are connected via edges. In some embodiments, the pose data can include a centroid corresponding to a location that is an average location of the landmarks. In addition to landmark locations, the facial pose data can also describe various other characteristics of the detected face such as tilt, yaw, and / or roll angles of the face relative to a coordinate structure associated with the input frame. The angles can be computed, for example, based on the locations of the landmarks relative to each other.

[0059] Still referring to FIG. 3, upon detecting the face 252 of the user 250 in each frame of the video data 305, the pre-processor 310 can be configured to perform one or more pre-processes on the video data 305 so as to configure the video data 305 for processing by a machine-learned model 315. As an example, the pre-process(es) can include video stabilization in which the centroid of the facial landmarks for the face 252 of the user 250 is aligned across the frames of the video data 305. As another example, in addition or alternatively to the video stabilization, the pre-process(es) can include cropping the frames of the video data 305 (e.g., according to known image cropping techniques) to remove pixels located outside of the bounding shape of the face 252 of the user 250. As yet another example, in addition or alternatively to the video stabilization and cropping, the pre- processes) can include frame rate adjustments (e.g., according to known frame rate adjustment techniques) such that the video data 305 has a same frame rate (i.e., frames per second) as historical video data (e.g., stored in the memory device 212) utilized for cardiac monitoring of the user 250. Further, as another example, in addition or alternatively to the video stabilization, cropping, and frame rate adjustment, the pre-process(es) can include frame differencing in which pixel values are compared between sequential frames of the video data 305 to determine movement between the sequential frames (e.g., caused by blood pulsing through the face 252 of the user 250).

[0060] Performing the pre-process(es) on the video data 305 can configure the video data 305 for processing by the machine-learned model 315 and provides technical benefits for performing cardiac monitoring of the user 250. For example, performing video stabilization can increase a likelihood that pixels corresponding to a particular facial landmark align with each other in sequential frames by accounting for movement of the face 252 of the user 250 relative to the imaging device 222, which can improve detection of blood flow through the face 252 of the user 250 via the frame differencing. As another example, cropping the frames to include the bounding shape and adjusting the frame rates to be the same across all obtained video data 305 can reduce a size of the video data 305 and reduce frame rate deviations between various video data 305, which can assist in reducing computational resource consumption by the machine-learned model 315 for processing the video data 305 and thereby permitting the mobile computing device 200 to perform the cardiac monitoring of the user 250 using only local computational resources.

[0061] Still referring to FIG. 3, the cardiac monitoring module 300 inputs, at least, the pre-processed video data 305 into the machine-learned model 315. The machine-learned model 315 can be stored by the mobile computing device 200 (e.g., in the memory device 212 thereof). As such, in some embodiments, the mobile computing device 200 can perform the cardiac monitoring of the user 250 without offloading to a processor external to the mobile computing device 200. Performing the cardiac monitoring of the user 250 without offloading to a processor external to the mobile computing device 200 can conserve computational resources and reduce an amount of time for performing the cardiac monitoring of the user 250. Moreover, data of the user 250 can be further safeguarded by preventing transmission of the data related to performing the cardiac monitoring of the user 250 to an external processor. In other embodiments, at least some of the cardiac monitoring may be performed by the server computing system 112 and / or some other external processor.

[0062] The cardiac monitoring module 300 receives one or more cardiac metrics 320 as a prediction output by the machine-learned model 315. In addition, the cardiac monitoring module 300 may receive one or more confidence scores associated with the received cardiac metric(s) 320 as an output by the machine-learned model 315. As set forth above, the cardiac metric(s) 320 can include, but are not limited to, a heart rate. The machine-learned model 315 can be trained to accept the pre-processed video data 305 as input and to generate an output of the cardiac metric(s) 320 and corresponding confidence score(s) based on such inputs. For example, the machine-learned model 315 can be trained to analyze the pre- processed video data 305 to utilize remote photoplethysmography (rPPG) techniques toanalyze light from around the user 250 reflected off the face 252 of the user 250 so as to detect blood volume changes. The machine-learned model 315 can be further trained to predict the heart rate of the user 250 from the detected blood volume changes.

[0063] In some embodiments, the cardiac monitoring module 300 may further input the collected data associated with the movement of the user 250 and / or the light around the user 250 to the machine-learned model 315. That is, the machine-learned model 315 may be further trained to generate the output of the cardiac metric(s) 320 and corresponding confidence score(s) based additionally on such inputs. The machine-learned model 315 may be trained with ground truth data (i.e., data about a real-world condition or state). The ground truth data may include video data of various users in response to various triggers and annotated with various cardiac metrics. In such an example, the cardiac metric(s) may be collected via one or more sensors or devices worn by the various users while the ground truth data is collected. Training the machine-learned model 315 can include updating weights and biases via suitable techniques, such as back-propagation.

[0064] Further, the cardiac monitoring module 300 can be configured to categorize the cardiac metric(s) 320 based, at least in part on, the confidence score(s) associated with the respective cardiac metric(s) 320. For example, the cardiac monitoring module 300 can compare the confidence score(s) to a threshold score. If the confidence score is greater than or equal to the threshold score, then the cardiac monitoring module 300 can determine a category for the cardiac metric 320 associated with the confidence score is “valid.” If the confidence score is less than the threshold score, then the cardiac monitoring module 300 can determine a category for the cardiac metric 320 associated with the confidence score is “invalid.” The threshold score can be determined empirically (e.g., based on testing to determine a minimum confidence score that permits cardiac biomarkers to be determined within a given margin of error based on the cardiac metric associated with the minimum confidence score). The threshold score can be a fixed score, for example. As another example, the threshold score can be a variable score (e.g., depending on the data associated with the movement of the user 250 and / or the lighting around the user 250).

[0065] Referring still to FIG. 3, the cardiac monitoring module 300 can be configured to provide the cardiac metric(s) 320 to an aggregator 325 based on the category of the cardiac metric(s) 320. When the cardiac metric(s) 320 is(are) categorized as “valid,” the cardiac monitoring module 300 can provide the cardiac metric(s) 320 as input to the aggregator 325. When the cardiac metric(s) 320 is(are) categorized as “invalid,” the cardiac monitoring module 300 can prevent the cardiac metric(s) 320 from being provided input to theaggregator 325. The aggregator 325 can be configured to aggregate the valid cardiac metric(s) 320 and historical cardiac metric(s). That is, the aggregator 325 can accept the valid cardiac metric(s) 320 and the historical cardiac metric(s) as inputs and can generate an aggregated output based on such inputs. In some embodiments, the aggregator 325 can arrange the valid cardiac metric(s) 320 and the historical cardiac metric(s) in ascending order and select a given percentile (e.g., 10thpercentile) of the arranged cardiac metric(s) as the aggregate output. In other embodiments, the aggregator 325 can determine the aggregate output as a function of the valid cardiac metric(s) 320, the historical cardiac metric(s), the data associated with the movement of the user 250 and / or the lighting around the user 250 and / or any other suitable data related to performing cardiac monitoring of the user 250.

[0066] In an embodiment, the cardiac monitoring module 300 can maintain a look-up table, or the like, that associates the cardiac metric(s) 320 output by the machine-learned model 315 with a timestamp. As such, the cardiac monitoring module 300 can maintain the historical cardiac metric(s) of the user 250. The cardiac monitoring module 300 can be configured to access the look-up table to select the historical cardiac metric(s) stored over a first period of time (e.g., 1 day, 1 week, 1 month, etc.). As one example, the first period of time may specify the twenty four (24) hours preceding the timestamp associated with the valid cardiac metric(s) 320. As another example, the first period of time may specify an entire span of one day (e.g., beginning at midnight and ending at 11 :59pm).

[0067] Further, in embodiments, the cardiac monitoring module 300 can be configured to provide the aggregate output to a processing algorithm. The processing algorithm can be configured to accept a plurality of aggregate outputs over a second period of time and to generate a predicted output of the cardiac biomarker(s) 330 for the first period of time. For instance, aggregated outputs of predicted daily cardiac metric(s) 320 (e.g., a heart rate) obtained over 1 week may be input to the processing algorithm that outputs daily cardiac biomarker(s) 330 (e.g., resting heart rate) for the user 250. The second period of time may be greater than the first period of time. In other embodiments, the cardiac monitoring module 300 can be configured to provide the aggregate output as the cardiac biomarker(s) 330. The processing algorithm may be a Kalman filter or any other suitable type of processing algorithm.

[0068] Furthermore, in an embodiment, the cardiac monitoring module 300 can be configured to cause the display 214 to display the cardiac biomarker(s) 330 and / or the cardiac metric(s) 320 to the user 250 in response to a user input. For example, the user input selecting the cardiac biomarker(s) 330 and / or the cardiac metric(s) 320 may be detected bythe display 214 and / or at least one additional HMI 220. As such, the display 214 may be configured to display the cardiac biomarker(s) 330 and / or the cardiac metric(s) 320.

[0069] Moreover, the cardiac monitoring module 300 may be configured to provide recommendations to the user 250 regarding activities for the user 250 based on the cardiac biomarker(s) 330. For example, in response to the user input selecting the cardiac biomarker(s) 330 and / or the cardiac metric(s) 320, the display 214 may be further configured to display one or more recommendations. The recommendations may be manually selected by the user 250 and may include a measurable amount (e.g., how often and how much), relevancy (e.g., likelihood to improve one or more flexibility metrics), and / or a defined time for a recommended activity. The recommendations may be determined based on one or more machine-learned models, a look-up table, or the like, that associates various activities with various improvements in the biomarker(s) 330.

[0070] For example, in an embodiment, the cardiac monitoring module 300 may include one or more machine-learned models that can include at least one recommendation generation model. In such embodiments, the recommendation generation model(s) is configured to generate one or more recommendations regarding activities for the user 250 using machine learning. For example, in an embodiment, the machine-learned model(s) may be trained to output the recommendation(s) in response to receiving the cardiac biomarker(s) 330 and / or the cardiac metric(s) 320. In such embodiments, the recommendation(s) can include natural language recommendations of one or more activities that can be performed by the user to improve the cardiac biomarker(s) 330.

[0071] The recommendation can be generated based on the cardiac biomarker(s) 330, the cardiac metric(s) 320, and / or any other relevant information. The recommendation can correspond to at least one remedial action to improve the cardiac biomarker(s) 330. As an example, responsive to determining a relatively high resting heart rate for the particular user, the recommendation can include an indication that the user’s resting heart rate should be decreased. In some implementations, the remedial action can include dietary suggestions (e.g., eat more or less particular foods, consume X calories, eat a low carb diet, etc.,), exercise suggestions (e.g., perform a particular work out, achieve a certain number of steps per day, etc.), or any other lifestyle change suggestions (e.g., sleep for Y hours per night). For instance, a particular activity can be known (e.g., based on academic studies and / or historical data captured by the cardiac monitoring module 300) to be particularly effective at improving the cardiac biomarker(s) 330. As such, the recommendation can include the particular activity to target an improvement in the cardiac biomarker(s) 330. In this way, theeffectiveness of the recommendations in improving the user’s cardiac biomarker(s) 330 can be improved.

[0072] In some implementations, the recommendations can be tailored to the user 250. For instance, it may be determined that an improvement in weight loss would make a large improvement to the user’s cardiac biomarker(s) 330 relative to a similar level of improvement in sleep duration (or in other words, it may be determined that the user’s cardiac biomarker(s) is(are) more sensitive to changes in weight loss). As an example, assume that, for the user, it is determined that an improvement in sleep duration by 10% would not result in a change in the cardiac biomarker(s) 330, whereas an increase in weight loss by 10% would improve the user’s cardiac biomarker(s) 330. Responsive to this determination, the recommendation can additionally or alternatively include an indication that the user should increase weight loss rather than increase sleep duration (or at least prioritize improvement in the weight loss over improvement in sleep duration).

[0073] Additionally, or alternatively, the recommendations can be generated based on past behavior of the particular user and / or one or more other users. For instance, based on historical data associated with the particular user (and / or one or more other users), it can be determined that the particular user (and / or one or more other users) has previously been more likely to perform remedial actions of a particular type (at least relative to remedial actions of another type). As an example, assume that the particular user has previously been presented with a recommendation including a first exercise and a recommendation including a second exercise. Responsive to determining that the user followed the recommendation including the second exercise and did not follow the recommendation including a first exercise, the recommendations generated subsequently can be more likely to include second exercises rather than first exercises. As mentioned, this past behavioral information can also be aggregated across one or more other users. The one or more other users can be selected from a larger group of users based on one or more attributes of the particular user (e.g., age, sex, fitness profile, etc.). In this way, the effectiveness of the recommendations in modifying the behavior of users can improve over time.Example Methods

[0074] FIG. 4 illustrates a flow diagram of a computer-implemented method 400 according to one or more example embodiments of the present disclosure. The computer- implemented method 400 can be implemented using, for instance, the mobile computingdevice 200 described above with reference to the example embodiments depicted in FIGS. 1- 3. The example embodiment illustrated in FIG. 4 depicts operations performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that various operations or steps of the computer- implemented method 400 or any of the other methods disclosed herein can be adapted, modified, rearranged, performed simultaneously, include operations not illustrated, and / or modified in various ways without deviating from the scope of the present disclosure.

[0075] As shown at (402), the computer-implemented method 400 may include activating an imaging device 222 of a mobile computing device 200 operatively coupled to one or more processors (e.g., the processor(s) 210) in response to a trigger. As mentioned, the trigger can be the mobile computing device 200 being unlocked (e.g., via facial recognition security features) so as to permit a user 250 access to the mobile computing device 200.

[0076] As shown at (404), the computer-implemented method 400 may include obtaining, via the imaging device 222 of the mobile computing device 200 operatively coupled to one or more processors (e.g., the processor(s) 210), video data 305 of a face 252 of the user 250 for a time period.

[0077] As shown at (406), the computer-implemented method 400 may include determining, via a machine-learned model 315 of the mobile computing device 200 operatively coupled to one or more processors (e.g., the processor(s) 210), cardiac metric(s) 320 based on, at least, the video data 305. As mentioned, the cardiac metric(s) 320 include(s), at least, a heart rate of the user 250. Moreover, in embodiments, the computer- implemented method 400 can include determining, via a machine-learned model 315 of the mobile computing device 200 operatively coupled to one or more processors (e.g., the processor(s) 210), one or more confidence scores associated with the respective cardiac metric(s) 320. Further, the computer-implemented method 400 may include categorizing, via the mobile computing device 200 operatively coupled to one or more processors (e.g., the processor(s) 210), the cardiac metric(s) 320 based on the confidence score(s) associated with the respective cardiac metric(s) 320. Furthermore, the computer-implemented method 400 may include pre-processing, via the mobile computing device 200 operatively coupled to one or more processors (e.g., the processor(s) 210), the video data 305 prior to providing the video data 305 as input the machine-learned model 315. In addition, the computer- implemented method 400 may include determining, via the mobile computing device 200 operatively coupled to one or more processors (e.g., the processor(s) 210), whether to providethe video data 305 as input to the machine-learned model 315 based on collected data associated with at least one of movement of the user and or lighting around the user.

[0078] As shown at (408), the computer-implemented method 400 may include determining, via the mobile computing device 200 operatively coupled to one or more processors (e.g., the processor(s) 210), cardiac biomarker(s) 330 based, at least in part, on the cardiac metric(s) 320. As mentioned, the cardiac biomarker(s) 330 include(s), at least, a resting heart rate. Furthermore, the computer-implemented method 400 may include aggregating, via the mobile computing device 200 operatively coupled to one or more processors (e.g., the processor(s) 210), the cardiac metric(s) 320 and one or more historical cardiac metrics based on the categorization of the cardiac metric(s) 320. Moreover, the computer-implemented method 400 may include applying, via the mobile computing device 200 operatively coupled to one or more processors (e.g., the processor(s) 210), a processing algorithm (e.g., a Kalman filter) to an aggregated output of the cardiac metric(s) 320 and one or more historical cardiac metrics to determine the cardiac biomarker(s) 330.

[0079] As shown at (410), the computer-implemented method 400 may include causing, by the mobile computing device 200 operatively coupled to one or more processors (e.g., the processor(s) 210), a display 214 of the mobile computing device 200 to display, at least, the cardiac biomarker(s) 330 in response to a user input. Moreover, the computer- implemented method 400 may include causing, via the mobile computing device 200 operatively coupled to one or more processors (e.g., the processor(s) 210), the display 214 to display one or more recommendations regarding an activity the user 250 can perform to improve the cardiac biomarker(s) 330.

[0080] FIG. 5 depicts a flowchart of a method 500 for training one or more machine- learned models according to aspects of the present disclosure. One or more portion(s) of example method 500 can be implemented by a computing system that includes one or more computing devices such as, for example, computing systems described with reference to the other figures. Each respective portion of example method 500 can be performed by any (or any combination) of one or more computing devices. Moreover, one or more portion(s) of example method 500 can be implemented on the hardware components of the device(s) described herein, for example, to train one or more systems or models. FIG. 5 depicts elements performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, combined, or modified in various ways without deviating from the scope of thepresent disclosure. FIG. 5 is described with reference to elements / terms described with respect to other systems and figures for exemplary illustrated purposes and is not meant to be limiting. One or more portions of example method 500 can be performed additionally, or alternatively, by other systems.

[0081] As shown at (502), the method 500 can include obtaining a training instance. A set of training data can include a plurality of training instances divided between multiple datasets (e.g., a training dataset, a validation dataset, or testing dataset). Example data types for the training instance and various tasks associated therewith are described throughout the present disclosure.

[0082] As shown at 504, the method 500 can include processing, using one or more machine-learned models, the training instance to generate an output. The output can be directly obtained from the one or more machine-learned models or can be a downstream result of a chain of processing operations that includes an output of the one or more machine- learned models.

[0083] As shown at 506, the method 500 can include receiving an evaluation signal associated with the output. The evaluation signal can be obtained using a loss function.Various determinations of loss can be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, or various other loss functions. The evaluation signal can be computed using known ground-truth labels (e.g., supervised learning), or predicted or estimated labels (e.g., semi- or self-supervised learning). The reward can be computed using a machine-learned reward model configured to generate rewards based on output(s) received.

[0084] As shown at 508, the method 500 can include updating the machine-learned model using the evaluation signal. For example, values for parameters of the machine- learned model(s) can be learned, in some embodiments, using various training or learning techniques, such as, for example, backwards propagation. For example, the evaluation signal can be backpropagated from the output (or another source of the evaluation signal) through the machine-learned model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the evaluation signal with respect to the parameter value(s)). For example, system(s) containing one or more machine-learned models can be trained in an end-to-end manner. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations. In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. Example method 500 can include implementing a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

[0085] In some implementations, the method 500 can be implemented for training a machine-learned model from an initialized state to a fully trained state (e.g., when the model exhibits a desired performance profile, such as based on accuracy, precision, recall, etc.).

[0086] In some implementations, the method 500 can be implemented for particular stages of a training procedure. For instance, in some implementations, example method 500 can be implemented for fine-tuning a machine-learned model. Fine-tuning can include, for instance, smaller-scale training on higher-quality (e.g., labeled, curated, etc.) data. Fine- tuning can affect all or a portion of the parameters of a machine-learned model. For example, various portions of the machine-learned model can be “frozen” for certain training stages. For example, parameters associated with an embedding space can be “frozen” during finetuning (e.g., to retain information learned from a broader domain(s) than present in the finetuning dataset(s)). An example fine-tuning approach includes reinforcement learning. Reinforcement learning can be based on user feedback on model performance during use.Example Machine-Learned Models

[0087] FIG. 6 is a block diagram of an example processing flow for using machine- learned model(s) 1 to process input(s) 2 to generate output(s) 3.

[0088] Machine-learned model(s) 1 can be or include one or multiple machine- learned models or model components. Example machine-learned models can include neural networks (e.g., deep neural networks). Example machine-learned models can include nonlinear models or linear models. Example machine-learned models can use other architectures in lieu of or in addition to neural networks. Example machine-learned models can include decision tree based models, support vector machines, hidden Markov models, Bayesian networks, linear regression models, k-means clustering models, etc.

[0089] Example neural networks can include feed-forward neural networks, recurrent neural networks (RNNs), including long short-term memory (LSTM) based recurrent neural networks, convolutional neural networks (CNNs), diffusion models, generative-adversarial networks, or other forms of neural networks. Example neural networks can be deep neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multiheaded self-attention models.

[0090] Machine-learned model(s) 1 can include a single or multiple instances of the same model configured to operate on data from input(s) 2. Machine-learned model(s) 1 caninclude an ensemble of different models that can cooperatively interact to process data from input(s) 2. For example, machine-learned model(s) 1 can employ a mixture-of-experts structure. See, e.g., Zhou et al., Mixture-of-Experts with Expert Choice Routing, ARXIV:2202.09368V2 (Oct. 14, 2022).

[0091] Input(s) 2 can generally include or otherwise represent various types of data. Input(s) 2 can include one type or many different types of data. Output(s) 3 can be data of the same type(s) or of different types of data as compared to input(s) 2. Output(s) 3 can include one type or many different types of data.

[0092] Example data types for input(s) 2 or output(s) 3 include natural language text data, software code data (e.g., source code, object code, machine code, or any other form of computer-readable instructions or programming languages), machine code data (e.g., binary code, assembly code, or other forms of machine-readable instructions that can be executed directly by a computer's central processing unit), assembly code data (e.g., low-level programming languages that use symbolic representations of machine code instructions to program a processing unit), genetic data or other chemical or biochemical data, image data, audio data, audiovisual data, haptic data, biometric data, medical data, financial data, statistical data, geographical data, astronomical data, historical data, sensor data generally (e.g., digital or analog values, such as voltage or other absolute or relative level measurement values from a real or artificial input, such as from an audio sensor, light sensor, displacement sensor, etc.), and the like. Data can be raw or processed and can be in any format or schema.

[0093] In multimodal inputs 2 or outputs 3, example combinations of data types include image data and audio data, image data and biometric data, sensor data and medical data, etc. It is to be understood that any combination of data types in an input 2 or an output 3 can be present.

[0094] An example input 2 can include one or multiple data types, such as the example data types noted above. An example output 3 can include one or multiple data types, such as the example data types noted above. The data type(s) of input 2 can be the same as or different from the data type(s) of output 3. It is to be understood that the example data types noted above are provided for illustrative purposes only. Data types contemplated within the scope of the present disclosure are not limited to those examples noted above.Additional Disclosure

[0095] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken andinformation sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0096] While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure cover such alterations, variations, and equivalents.

[0097] Aspects of the disclosure have been described in terms of illustrative embodiments thereof. Any and all features in the following claims can be combined or rearranged in any way possible, including combinations of claims not explicitly enumerated in combination together, as the example claim dependencies listed herein should not be read as limiting the scope of possible combinations of features disclosed herein. Accordingly, the scope of the present disclosure is by way of example rather than by way of limitation, and the subject disclosure does not preclude inclusion of such modifications, variations or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. Moreover, terms are described herein using lists of example elements joined by conjunctions such as “and,” “or,” “but,” etc. It should be understood that such conjunctions are provided for explanatory purposes only. Clauses and other sequences of items joined by a particular conjunction such as “or,” for example, can refer to “and / or,” “at least one of’, “any combination of’ example elements listed therein, etc. Terms such as “based on” should be understood as “based at least in part on.”

[0098] The term “can” should be understood as referring to a possibility of a feature in various implementations and not as prescribing an ability that is necessarily present in every implementation. For example, the phrase “X can perform Y” should be understood asindicating that, in various implementations, X has the potential to be configured to perform Y, and not as indicating that in every instance X must always be able to perform Y. It should be understood that, in various implementations, X might be unable to perform Y and remain within the scope of the present disclosure.

[0099] The term “may” should be understood as referring to a possibility of a feature in various implementations and not as prescribing an ability that is necessarily present in every implementation. For example, the phrase “X may perform Y” should be understood as indicating that, in various implementations, X has the potential to be configured to perform Y, and not as indicating that in every instance X must always be able to perform Y. It should be understood that, in various implementations, X might be unable to perform Y and remain within the scope of the present disclosure.

Claims

WHAT IS CLAIMED IS:

1. A method for monitoring cardiac health of a user of a mobile computing device, the method comprising: activating an imaging device associated with the mobile computing device in response to a trigger indicating a presence of the user within a field of view of the imaging device; obtaining, via the imaging device, video data of a face of the user for a time period; determining, via a machine-learned model programmed in a processor of the mobile computing device, one or more cardiac metrics based on, at least, the video data, the one or more cardiac metrics comprising, at least, a heart rate of the user; determining one or more cardiac biomarkers based, at least in part, on the one or more cardiac metrics, the one or more cardiac biomarkers comprising, at least, a resting heart rate of the user; and causing a display of the mobile computing device to display, at least, the one or more cardiac biomarkers in response to a user input.

2. The method of claim 1, further comprising deactivating the imaging device in response to expiration of the time period.

3. The method of claim 1, further comprising determining, via the mobile computing device the one or more cardiac biomarkers without offloading to an external processor external to the mobile computing device.

4. The method of claim 1, further comprising determining, via the machine- learned model programmed in the processor of the mobile computing device, one or more confidence scores associated with the respective one or more cardiac metrics.

5. The method of claim 4, wherein determining the one or more cardiac biomarkers based, at least in part, on the one or more cardiac metrics further comprises: determining a category for the one or more cardiac metrics based, at least in part, on the one or more confidence scores associated with the respective one or more cardiac metrics; anddetermining the one or more cardiac biomarkers by aggregating the one or more cardiac metrics and one or more historical cardiac metrics based on the category.

6. The method of claim 5, wherein determining the one or more cardiac biomarkers by aggregating the one or more cardiac metrics and the one or more historical cardiac metrics based on the category further comprises: determining the one or more cardiac biomarkers by applying a processing algorithm to an aggregate output of the one or more cardiac metrics and the one or more historical cardiac metrics.

7. The method of claim 1, wherein determining, via the machine-learned model programmed in the processor of the mobile computing device, the one or more cardiac metrics based on, at least, the video data further comprises: collecting data associated with at least one of the presence of the user within the field of view of the imaging device, movement of the user, and lighting around the user; and providing the video data as an input to the machine-learned model based on the collected data.

8. The method of claim 1, wherein determining, via the machine-learned model of the mobile computing device, the one or more cardiac metrics based on, at least, the video data further comprises: processing, via a face detector of the mobile computing device, the video data to align a centroid of a plurality of facial landmarks relative to one or more frames of the video data; and providing the processed video data as an input to the machine-learned model.

9. The method of claim 1, wherein the trigger occurs when the mobile computing device is being unlocked so as to permit the user to access the mobile computing device.

10. The method of claim 1, wherein the mobile computing device comprises at least one of a smartphone, a tablet, a wearable computing device, or a laptop computer.

11. A mobile computing device: one or more processors; andone or more computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing device to perform operations, the operations comprising: activating an imaging device associated with the mobile computing device in response a trigger indicating a presence of a user within a field of view of the imaging device; obtaining, via the imaging device, video data of a face of the user for a time period; determining, via a machine-learned model programmed in a processor of the mobile computing device, one or more cardiac metrics based on, at least, the video data, the one or more cardiac metrics comprising, at least, a heart rate of the user; determining one or more cardiac biomarkers based, at least in part, on the one or more cardiac metrics, the one or more cardiac biomarkers comprising, at least, a resting heart rate of the user; and causing a display of the mobile computing device to display, at least, the one or more cardiac biomarkers in response to a user input.

12. The mobile computing device of claim 11, wherein the operations further comprise deactivating the imaging device in response to expiration of the time period.

13. The mobile computing device of claim 11, wherein the mobile computing device is configured such that the one or more cardiac biomarkers are determined without offloading to any processor external to the mobile computing device.

14. The mobile computing device of claim 11, wherein determining, via the machine-learned model of the mobile computing device, the one or more cardiac metrics based on, at least, the video data further comprises: determining, via the machine-learned model of the mobile computing device, one or more confidence scores associated with the respective one or more cardiac metrics.

15. The mobile computing device of claim 14, wherein determining the one or more cardiac biomarkers based, at least in part, on the one or more cardiac metrics further comprises:determining a category for the one or more cardiac metrics based, at least in part, on the one or more confidence scores associated with the respective one or more cardiac metrics; and determining the one or more cardiac biomarkers by aggregating the one or more cardiac metrics and one or more historical cardiac metrics based on the category.

16. The mobile computing device of claim 15, wherein determining the one or more cardiac biomarkers by aggregating the one or more cardiac metrics and the one or more historical cardiac metrics based on the category further comprises: determining the one or more cardiac biomarkers by applying a processing algorithm to an aggregate output of the one or more cardiac metrics and the one or more historical cardiac metrics.

17. The mobile computing device of claim 11, wherein determining, via the machine-learned model of the mobile computing device, the one or more cardiac metrics based on, at least, the video data further comprises: collecting data associated with at least one of the presence of the user within the field of view of the imaging device, movement of the user, and lighting around the user; and providing the video data as input to the machine-learned model based on the collected data.

18. The mobile computing device of claim 11, wherein determining, via the machine-learned model of the mobile computing device, the one or more cardiac metrics based on, at least, the video data further comprises: processing, via a face detector of the mobile computing device, the video data to align a centroid of a plurality of facial landmarks relative to one or more frames of the video data; and providing the processed video data as input to the machine-learned model.

19. The mobile computing device of claim 11, wherein the trigger occurs when the mobile computing device is being unlocked so as to permit the user to access the mobile computing device.

20. The mobile computing device of claim 11, wherein the mobile computing device comprises at least one of a smartphone, a wearable computing device, a tablet, or a laptop computer.

Images