Systems and methods for analyzing patient gait using fixed edge identification during virtual clinic / remote programming session for neuromodulation

Abstract

A method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system is provided. The method comprises receiving video data from a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session, identifying, from the received video data, at least one fixed edge in an environment surrounding the patient, determining, based on the at least one identified fixed edge, an amount of camera tilt of the PC device relative to the patient, normalizing anatomical landmark data of the patient gait data by applying a transformation to the anatomical landmark data to compensate for the determined amount of camera tilt, and providing the normalized anatomical landmark data to one or more trained computation models to classify one or more patient gait characteristics.

Description

(32736-2156)SYSTEMS AND METHODS FOR ANALYZING PATIENT GAIT USING FIXED EDGE IDENTIFICATION DURING VIRTUAL CLINIC / REMOTE PROGRAMMING SESSION FOR NEUROMODULATIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 739,741, filed on December 30, 2024, U.S. Provisional Patent Application No. 63 / 739,773, filed on December 30, 2024, U.S. Patent Application No.63 / 739,804, filed on December 30, 2024, U.S. Patent Application No. 63 / 739,822, filed on December 30, 2024, U.S. Patent Application No. 63 / 739,838, filed on December 30, 2024, and U.S. Patent Application No. 63 / 739,904, filed on December 30, 2024, the entire disclosure of each of which is hereby incorporated by reference herein.TECHNICAL FIELD

[0002] The present application is generally directed to providing digital health services to patients.BACKGROUND

[0003] Implantable medical devices have changed how medical care is provided to patients having a variety of chronic illnesses and disorders. For example, implantable cardiac devices improve cardiac function in patients with heart disease by improving quality of life and reducing mortality rates. Respective types of implantable neurostimulators provide a reduction in pain for chronic pain patients and reduce motor difficulties in patients with Parkinson’s disease and other movement disorders. A variety of other medical devices are proposed and are in development to treat other disorders in a wide range of patients.

[0004] Many implantable medical devices and other personal medical devices are programmed by a physician or other clinician to optimize the therapy provided by a respective device to an individual patient. Typically, the programming occurs using short-range communication links (e.g., inductive wireless telemetry) in an in-person or in-clinic setting. Since such communications typically require close immediate contact, there is only an extremely small(32736-2156)I ikelihood of a third-party establishing a communication session with the patient’s implanted device without the patient’s knowledge.

[0005] Remote patient care is a healthcare delivery method that aims to use technology to provide patient health services outside of a traditional clinical setting (e.g., in a doctor’s office or a patient’s home). It is widely expected that remote patient care may increase access to care and decrease healthcare delivery costs.SUMMARY

[0006] The present application is generally directed to providing digital health services to patients.

[0007] In one aspect, a method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system is provided. The method comprises receiving video data from a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session, identifying, from the received video data, at least one fixed edge in an environment surrounding the patient, determining, based on the at least one identified fixed edge, an amount of camera tilt of the PC device relative to the patient, normalizing anatomical landmark data of the patient gait data by applying a transformation to the anatomical landmark data to compensate for the determined amount of camera tilt, and providing the normalized anatomical landmark data to one or more trained computation models to classify one or more patient gait characteristics.

[0008] The fixed edge may provide a static frame of reference or a perspective frame of reference. In some examples, the fixed edge may be a horizontal or vertical edge. For example, the horizontal edge may correspond to doors or carpets. The horizontal edge may provide a static frame of reference. The vertical edge may be a vertical door line. The vertical edge may provide a perspective frame of reference. Repeating edges, such as a carpet with a repeating pattern and / or ceiling tiles may be used to determine depth.

[0009] In some examples, the center of the hip can be used as a reference for the landmark points, since this has minimal movement in a gait cycle. The reference can be used to normalize landmark data (e.g., coordinates).(32736-2156)

[0010] The normalization of landmark data can allow the landmark points in each frame to be scaled so that it is normalized to compensate for the amount of camera tilt.

[0011] In another aspect, a method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system is provided. The method comprises receiving video data from a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session, identifying, from the received video data, a fixed frame of reference as the patient moves closer to a camera of the PC device, performing, using the identified fixed frame of reference, a depth invariance transformation on anatomical landmark data of the patient gait data to generate transformed anatomical landmark data, and providing the transformed anatomical landmark data to one or more trained computation models to facilitate classifying one or more patient gait characteristics.

[0012] In some examples, the fixed frame of reference is a location on the patient, such as a point with minimal movement during a gait cycle. For example, this may correspond to a center of a hip of the patient since this has minimal movement during a gait cycle.

[0013] In some examples, performing the depth invariance transformation comprises normalizing anatomical landmark data based on a reference distance. The reference distance may be associated with the fixed frame of reference (e.g., from a landmark point to the fixed frame of reference, such as the center of the hip).

[0014] In another aspect, a method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system is provided. The method comprises receiving video data from a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session, analyzing the received video data to detect a depth of the patient relative to a camera of the PC device, performing one or more rescaling operations on one or more frames of the received video data, identifying anatomical landmark data of the patient from the one or more rescaled frames, and providing the identified anatomical landmark(32736-2156)data to one or more trained computation models to facilitate classifying one or more patient gait characteristics.

[0015] In some examples, the rescaling operations are based on the depth of the patient detected from analyzing the video data.

[0016] In some examples, performing one or more rescaling operations comprises performing one or more rescaling operations using a first neural network.

[0017] In some examples, detecting the depth of the patient in the video data comprises detecting the patient using a first neural network.

[0018] In some examples, identifying anatomical landmark data comprises identifying anatomical landmark data using a second neural network different from the first neural network.

[0019] In another aspect, a method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system is provided. The method comprises receiving video data form a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session, processing anatomical landmark data in the patient gait data by identifying a fixed frame of reference of the patient’s anatomy proximate to a hip of the patient as the patient moves closer to a camera of the PC device, further processing the anatomical landmark data to generate a plurality of distance metrics relative to the identified fixed frame of reference, and providing the plurality of distance metrics to one or more trained computation models to facilitate classifying one or more patient gait characteristics.

[0020] In some examples, the fixed frame of reference corresponds to the center of the hip of the patient.

[0021] In some examples, the distance metrics include a tilt angle. In some examples, providing the plurality of distance metrics to one or more trained computation models comprises providing the tilt angle to the one or more trained computation models to identify temporal distortions in patient movement.

[0022] In another aspect, a method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing session is provided. The method comprises receiving initial video(32736-2156)data from a patient controller (PC) device of the patient, the initial video data including initial patient gait data of the patient, the PC device comprising a camera comprising a plurality of image acquisition sensors, the initial video data acquired using a first image acquisition sensor of the plurality of image acquisition sensors, analyzing the initial video data to determine an image quality parameter, based on the determined image quality parameter, instructing the PC device to acquire subsequent video data using a second image acquisition sensor of the plurality of image acquisition sensors, identifying anatomical landmark data of the patient from the subsequent video data, and providing the identified anatomical landmark data to one or more trained computation models to facilitate classifying one or more patient gait characteristics.

[0023] In some examples, the image quality parameter may be based on aperture size or focal length. For example, the second image acquisition camera may have a higher or lower aperture size or focal length. This may be useful especially when the patient is walking towards or away from the camera.

[0024] In another aspect, a method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system is provided. The method comprises receiving video data from a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session, identifying anatomical landmark data from the patient gait data, generating, based on the identified anatomical landmark data, movement data indicative of movement of the patient, performing dynamic time warping operations on the movement data to generate metrics indicating a degree of asymmetry of matching warped points within the time series, and providing the generated metrics to one or more trained computation models to facilitate classifying one or more patient gait characteristics.

[0025] In some examples, dynamic time warming operations may measure similarity between two temporal sequences. For example, this may include computing the distance between anatomical positions, or kinematic or position metrics with reference to landmark points on a patient, for multiple time series. The distance calculation may be modified by applying time and / or phase compensation for the respective time series.(32736-2156)

[0026] In another aspect, a method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system is provided. The method comprises receiving video data from a patient controller (PC) device of the patient, the video data including patient gait data and patient facial data of the patient acquired by the PC device during the VC session; identifying first anatomical landmark data from the patient gait data; generating, based on the first anatomical landmark data, movement data indicative of movement of the patient; identifying second anatomical landmark data from the patient facial data; generating, based on the second anatomical landmark data, expression data indicative of one or more facial expressions of the patient; correlating repetition of one or more the one or more facial expressions to relative time positions over multiple gait cycles of the patient using the movement data and the expression data; and classifying or quantifying one or more patient gait characteristics using the correlated repetition.

[0027] Correlating facial expressions with the movement of the patient can provide an indication of patient gait characteristics, for example associating expressions indicative of pain with the movement.

[0028] In another aspect, a method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system is provided. The method comprises receiving video data and audio data from a patient controller (PC) device of the patient, the video data including patient gait data; identifying anatomical landmark data from the patient gait data; generating, based on the first anatomical landmark data, movement data indicative of movement of the patient; processing the audio data to identify frequency content associated the patient interaction with the floor; correlating patient interaction with the floor to the patient gait data over multiple gait cycles of the patient; and classifying or quantifying one or more patient gait characteristics using the correlated patient interaction.

[0029] The present disclosure also includes systems configured to perform the methods disclosed herein. For example, the system may comprise a clinician programmer (CP) device. The system may additionally or alternatively comprise a VC cloud computing system. The CP device and / or the VC cloud computing system may be configured to perform the methods. The system may also comprise(32736-2156)a patient controller (PC) device. The system may also comprise an implantable medical device (IMD).

[0030] In another aspect, a system for conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services is provided. The system may comprise a VC cloud computing system. The system is configured to receive video data from a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session, and identify, from the received video data, at least one fixed edge in an environment surrounding the patient. The system is configured to determine, based on the at least one identified fixed edge, an amount of camera tilt of the PC device relative to the patient and normalize anatomical landmark data of the patient gait data by applying a transformation to the anatomical landmark data to compensate for the determined amount of camera tilt. The system is configured to provide the normalized anatomical landmark data to one or more trained computation models to facilitate classifying or quantifying one or more patient gait characteristics.

[0031] In another aspect, a system for conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services is provided. The system may comprise a VC cloud computing system. The system is configured to receive video data from a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session, identify, from the received video data, a fixed frame of reference as the patient moves closer to a camera of the PC device, perform, using the identified fixed frame of reference, a depth invariance transformation on anatomical landmark data of the patient gait data to generate transformed anatomical landmark data, and provide the transformed anatomical landmark data to one or more trained computation models to facilitate classifying one or more patient gait characteristics.

[0032] In another aspect, a system for conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services is provided. The system may comprise a VC cloud computing system. The system is configured to receive video data from a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC(32736-2156)device during the VC session, analyze the received video data to detect a depth of the patient relative to a camera of the PC device, perform one or more rescaling operations on one or more frames of the received video data, identify anatomical landmark data of the patient from the one or more rescaled frames, and provide the identified anatomical landmark data to one or more trained computation models to facilitate classifying one or more patient gait characteristics.

[0033] In another aspect, a system for conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services is provided. The system may comprise a VC cloud computing system. The system is configured to receive video data form a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session, process anatomical landmark data in the patient gait data by identifying a fixed frame of reference of the patient’s anatomy proximate to a hip of the patient as the patient moves closer to a camera of the PC device, further process the anatomical landmark data to generate a plurality of distance metrics relative to the identified fixed frame of reference, and provide the plurality of distance metrics to one or more trained computation models to facilitate classifying one or more patient gait characteristics.

[0034] In another aspect, a system for conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services is provided. The system may comprise a VC cloud computing system. The system is configured to receive initial video data from a patient controller (PC) device of the patient, the initial video data including initial patient gait data of the patient, the PC device comprising a camera comprising a plurality of image acquisition sensors, the initial video data acquired using a first image acquisition sensor of the plurality of image acquisition sensors, analyze the initial video data to determine an image quality parameter, based on the determined image quality parameter, instruct the PC device to acquire subsequent video data using a second image acquisition sensor of the plurality of image acquisition sensors, identify anatomical landmark data of the patient from the subsequent video data, and provide the identified anatomical landmark data to one or more trained computation models to facilitate classifying one or more patient gait characteristics.(32736-2156)

[0035] In another aspect, a system for conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services is provided. The system may comprise a VC cloud computing system. The system is configured to receive video data from a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session, identify anatomical landmark data from the patient gait data, generate, based on the identified anatomical landmark data, movement data indicative of movement of the patient, perform dynamic time warping operations on the movement data to generate metrics indicating a degree of asymmetry of matching warped points within the time series, and provide the generated metrics to one or more trained computation models to facilitate classifying one or more patient gait characteristics.

[0036] In another aspect, a system for conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services is provided. The system may comprise a VC cloud computing system. The system is configured to receive video data from a patient controller (PC) device of the patient, the video data including patient gait data and patient facial data of the patient acquired by the PC device during the VC session, identify first anatomical landmark data from the patient gait data, generate, based on the first anatomical landmark data, movement data indicative of movement of the patient, identify second anatomical landmark data from the patient facial data, generate, based on the second anatomical landmark data, expression data indicative of one or more facial expressions of the patient, correlate repetition of one or more the one or more facial expressions to relative time positions over multiple gait cycles of the patient using the movement data and the expression data, and classify or quantify one or more patient gait characteristics using the correlated repetition.

[0037] In another aspect, a system for conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services is provided. The system may comprise a VC cloud computing system. The system is configured to receive video data and audio data from a patient controller (PC) device of the patient, the video data including patient gait data, identify anatomical landmark data from the patient gait data, generate, based on the first anatomical landmark data, movement data indicative of movement of the patient, process the(32736-2156)audio data to identify frequency content associated the patient interaction with the floor, correlate patient interaction with the floor to the patient gait data over multiple gait cycles of the patient, and classify or quantify one or more patient gait characteristics using the correlated patient interaction.

[0038] In another aspect, a system comprising a patient controller device, a clinician programmer device, one or more cloud servers, and an implantable medical device to remotely programming the implantable medical device is provided, wherein the implantable medical device is configured to facilitate therapy to a patient. The system is configured to implement: establishing a first communication between a patient controller, PC, device and the implantable medical device, wherein the implantable medical device is configured to facilitate therapy to the patient according to one or more programmable parameters, the PC device communicates signals to the implantable medical device to set or modify the one or more programmable parameters, and the PC device comprises a video camera; establishing a video connection between the PC device and a clinician programmer, CP, device of a clinician for a remote programming session in a second communication that includes an audio / video, A / V session; communicating a value for a respective programmable parameter of the medical device from the CP device to the PC device during the remote programming session; and modifying, by the PC device, the respective programming parameter of the implantable medical device according to the communicated value from the CP device during the remote programming session, wherein the patient controller is configured to process video data from the video camera to generate landmark data indicative of anatomical locations of the patient, communicate the landmark data to one or more of the PC device and one or more cloud servers in a separate data format from video data in the A / V session.

[0039] In some examples applicable to all aspects, the patient gait characteristics may indicate gait abnormalities, such as lack of heel strike, stooped posture, and / or movement speed. In some examples, the patient gait characteristics may indicate gait imbalance or asymmetry.

[0040] In some examples applicable to all aspects, an indication of the patient gait characteristics may be output, such as to the clinician programmer device. For example, the indication may identify a particular symptom such as gait(32736-2156)abnormality or gait imbalance. In some examples, the indication may comprise a metric associated with the patient’s gait. The indication may be displayed on the CP device.

[0041] Features described in relation to one aspect may be applied to other aspects and vice versa. Features of one aspect may be combined with other aspects. For example, if may be beneficial to combine one or more of the features including: performing compensation of camera tilt, performing depth invariance transformation, performing rescaling based on depth of the patient, generating distance metrics relative to a fixed frame of reference, switching image acquisition sensor based on image quality parameter, performing dynamic time warping operations on movement data, correlating facial expressions and movement data, and / or correlating patient interaction with the floor from audio data with patient gait data from video data. These various features may be used in various combinations to provide improved assessment and analysis of patient gait characteristics. It is therefore contemplated that multiple operations described herein may be performed based on at least the video data received that includes patient gait data.

[0042] Features described in relation to method aspects may be applied to system aspects and vice versa. In particular, the system may be configured to perform the methods described herein, which includes the system being configured to perform features that are described in the context of features of the methods.BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Embodiments of the present disclosure are illustrated by way of example, and not by way of limitation, in the Figures of the accompanying drawings in which like references indicate similar elements. It should be noted that different references to "an" or "one" embodiment in this disclosure are not necessarily to the same embodiment, and such references may mean at least one. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to effectuate such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.(32736-2156)

[0044] The accompanying drawings are incorporated into and form a part of the specification to illustrate one or more exemplary embodiments of the present disclosure. Various advantages and features of the disclosure will be understood from the following Detailed Description taken in connection with the appended claims and with reference to the attached drawing Figures in which:

[0045] FIG. 1A depicts an example architecture of a system configured to support remote patient therapy as part of an integrated remote care service session in a virtual clinic environment that may be deployed in a cloud-centric digital health implementation according to one or more embodiments of the present patent disclosure;

[0046] FIG. 1 B depicts an example network environment wherein the remote care service architecture of FIG. 1 A may be implemented according to a representative embodiment;

[0047] FIG. 2 depicts a flowchart illustrative of blocks, steps and / or acts that may be (re)combined in one or more arrangements with or without additional flowcharts of the present disclosure for facilitating remote care therapy in a secure network environment for purposes of some embodiments;

[0048] FIG. 3 depicts a flowchart illustrative of blocks, steps and / or acts that may be implemented for establishing a communication session with an implantable medical device;

[0049] FIGS. 4A and 4B depicts flowcharts illustrative of a remote care scenario involving an example digital health network architecture wherein an integrated remote care session may be established between a patient and a clinician operating respective controller devices for purposes of some embodiments of the present disclosure;

[0050] FIGS. 5A and 5B depict representations of an example user interface and associated dialog boxes provided with a clinician programmer device for selecting different therapy applications and / or service modes for purposes of some embodiments of the present disclosure;

[0051] FIG. 6 depicts a representation of an example user interface provided with a clinician programmer device for facilitating controls with respect to an AV communication session and a remote therapy session in an integrated(32736-2156)remote care service application for purposes of some embodiments of the present disclosure;

[0052] FIG. 7 depicts a block diagram of an external device that may be configured as a clinician programmer device, a patient controller device or an authorized third-party device operative in a digital health network architecture for purposes of some embodiments of the present disclosure;

[0053] FIG. 8 depicts a block diagram illustrating additional details of a patient controller device operative in a digital health network architecture for purposes of some embodiments of the present disclosure;

[0054] FIG. 9 depicts a block diagram illustrating additional details of a clinician programmer device operative in a digital health network architecture for purposes of some embodiments of the present disclosure;

[0055] FIG. 10 depicts a block diagram of an IMD and associated system that may be configured for facilitating a remote care therapy application and / or a local therapy session for purposes of an example embodiment of the present disclosure;

[0056] FIGS. 11 A-11 H depict representations of an example user interface and associated dialog boxes provided with a patient controller device for selecting different therapy applications and / or service modes and for facilitating controls with respect to an AV communication session and a remote therapy session in an integrated remote care service application for purposes of an embodiment of the present disclosure;

[0057] FIG. 12 depicts an example cloud-centric digital healthcare network architecture including one or more virtual clinic platforms, patient report processing platforms and remote data logging platforms for purposes of some example embodiments of the present disclosure;

[0058] FIG. 13 a flowchart depicting operations that may occur during a virtual clinic or remote programming session according to embodiments of the present disclosure;

[0059] FIG. 14 is a flowchart depicting an exemplary flow of operations for creating a predictive disorder model using ML / AI processing according to some embodiments;(32736-2156)

[0060] FIG. 15 depicts aspects of ML / AI learning data sets and predictive models according to embodiments;

[0061] FIG. 16 is a flowchart depicting exemplary operations for processing video data to support provision of neurostimulation therapies to patient according to some embodiments;

[0062] FIG. 17A-17B depict sets of key landmark points that may be utilized for facial expression analysis and pose analysis according to some embodiments;

[0063] FIG. 18A-18D show screenshots illustrating exemplary aspects of user interfaces utilized to control and view information during a virtual or remote session conducted according to some embodiments;

[0064] FIG. 19 is a flowchart depicting exemplary operations for conducting a virtual clinic / remote programming session according to some embodiments;

[0065] FIG. 20 is a flowchart depicting exemplary operations for conducting a physical therapy session in accordance with some embodiments;

[0066] FIG. 21 is a screenshot of an exemplary graphical user interface for presenting performance data associated with a patient;

[0067] FIG. 22A is an example natural image;

[0068] FIG. 22B is an example segmented image generated from the natural image of FIG. 22A with multiple natural edges identified by arrows;

[0069] FIG. 23A is a flowchart of an example method for processing tilt and contrastand providing user feedback;

[0070] FIG. 23B is a continuation of the flowchart of FIG. 23A;

[0071] FIG. 24A is a diagram showing an example frontal view gait analysis;

[0072] FIG. 24B is a diagram showing an example side view gait analysis;

[0073] FIG. 25A is a diagram illustrating an example gait analysis for a patient that is far away from the camera;

[0074] FIG. 25B is a diagram illustrating an example gait analysis fora patient that is close to the camera;

[0075] FIG. 26A is a diagram of an example gait analysis using both the gait analysis of FIG. 25A and the gait analysis of FIG. 25B;

[0076] FIG. 27A is a flowchart of an example method for rescaling an image and detecting landmarks;

[0077] FIG. 27B is a continuation of the flowchart of FIG. 27A;(32736-2156)

[0078] FIG. 28 is a perspective view of an example patient device that may be used for dynamic camera selection;

[0079] FIG. 29A is a flowchart of an example method that may be used for dynamic camera sensor selection;

[0080] FIG. 29B is a continuation of the flowchart of FIG. 29;

[0081] FIG. 30A shows example diagrams illustrating an angle of a patient’s torso when taking a first step;

[0082] FIG. 30B shows example diagrams illustrating a change in the angle of the patient’s torso when taking a second step;

[0083] FIG. 31A is a pre-therapy gait asymmetry plot of a hip line angle of the patient as the patient is walking towards the camera;

[0084] FIG. 31 B is a pre-therapy gait asymmetry plot of a shoulder-hip midpoint line angle as the patient is moving towards the camera;

[0085] FIG. 32A is a post-therapy gait asymmetry plot of the hip line angle of the patient as the patient is walking towards the camera;

[0086] FIG. 32B is a post-therapy gait asymmetry plot of the shoulder-hip mid-point line angle as the patient is walking towards the camera;

[0087] FIG. 33 is a flowchart of an example method of analyzing gait using a buffer queue;

[0088] FIG. 34 is a diagram of an example gait analysis for a patient using dynamic time warping, illustrating landmark detection of the patient’s left and right wrists;

[0089] FIG. 35 illustrates an example plot of movement of the patient’s left and right wrists identified by an example gait analysis;

[0090] FIG. 36 illustrates an example dynamic time warping distance between the patient’s wrists using the data of the plot of FIG. 35;

[0091] FIG. 37 illustrates the correlation between movement of the patient’s wrist and the dynamic time warping distance plot of FIG. 36;

[0092] FIG. 38 illustrates a plot of an example dynamic time warping distance using a full cycle approach;

[0093] FIG. 39 illustrates a plot of an example dynamic time warping distance using a moving leg approach;(32736-2156)

[0094] FIG. 40 is a flowchart of an example method for normalization of a human frame across ambulatory cycles specific for gait analysis;

[0095] FIG. 41 A illustrates an image frame of captured video data overlaid with landmark identification, showing the patient far away from the camera;

[0096] FIG. 41 B illustrates an image frame of the captured video data overlaid with landmark identification, showing the patient close to the camera;

[0097] FIG. 42 illustrates a scatter plot of positions of identified landmarks during the patient walking cycle, showing a fixed frame of reference based on a hip width of the patient;

[0098] FIG. 43A illustrates a pre-therapy scatter plot of positions of identified landmarks during the patient walking cycle;

[0099] FIG. 43B illustrates a post-therapy scatter plot of positions of identified landmarks during the patient walking cycle;

[0100] FIG. 44A is a flowchart of an example method for correlating patient grimaces to gait asymmetry;

[0101] FIG. 44B is a continuation of the flowchart of FIG. 44A;

[0102] FIG. 45A is an example spectrograph of a patient walking cycle of a patient without Parkinson’s disease;

[0103] FIG. 45B is an example spectrograph of a patient walking cycle of a patient with Parkinson’s disease;

[0104] FIG. 46A is an example plot of pitch contour based on the spectrograph of FIG. 45A;

[0105] FIG. 46B is an example plot of pitch contour based on the spectrograph of FIG. 45B;

[0106] FIG. 47A is an example plot of intensity curve based on the spectrograph of FIG. 45A;

[0107] FIG. 47B is an example plot of intensity curve based on the spectrograph of FIG. 45B;

[0108] FIG. 48A is an example plot of spectral centroid based on the spectrograph of FIG. 45A;

[0109] FIG. 48B is an example plot of spectral centroid based on the spectrograph of FIG. 45B;(32736-2156)

[0110] FIG. 49A is an example plot of spectral bandwidth based on the spectrograph of FIG. 45A;

[0111] FIG. 49B is an example plot of spectral bandwidth based on the spectrograph of FIG. 45B;

[0112] FIG. 50A is an example plot of spectral skewness based on the spectrograph of FIG. 45A;

[0113] FIG. 50B is an example plot of spectral skewness based on the spectrograph of FIG. 45B;

[0114] FIG. 51 A is an example plot of spectral kurtosis based on the spectrograph of FIG. 45A;

[0115] FIG. 51 B is an example plot of spectral kurtosis based on the spectrograph of FIG. 45B;

[0116] FIG. 52A is an example plot of the third spectral moment based on the spectrograph of FIG. 45A;

[0117] FIG. 52B is an example plot of the third spectral moment based on the spectrograph of FIG. 45B;

[0118] FIG. 53A is an example plot of the fourth spectral moment based on the spectrograph of FIG. 45A;

[0119] FIG. 53B is an example plot of the fourth spectral moment based on the spectrograph of FIG. 45B;

[0120] FIG 54 is a flowchart of an example method for identifying gait asymmetry using spectrogram data;

[0121] FIG 55A is a flowchart of another example method for identifying gait asymmetry using spectrogram data;

[0122] FIG. 55B is a continuation of the flowchart of FIG. 55A;

[0123] FIG. 56 is a diagram of a first option for an example method of dynamic switching;

[0124] FIG. 57 is a diagram of a second option for the example method of dynamic switching; and

[0125] FIG. 58 is a schematic diagram of an example method for dynamic switching.

[0126] The foregoing has outlined rather broadly the features and technical advantages of the present invention in order that the detailed description of the(32736-2156)invention that follows may be better understood. Additional features and advantages of the invention will be described hereinafter which form the subject of the claims of the invention. It should be appreciated by those skilled in the art that the conception and specific embodiment disclosed may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present invention. It should also be realized by those skilled in the art that such equivalent constructions do not depart from the spirit and scope of the invention as set forth in the appended claims. The novel features which are believed to be characteristic of the invention, both as to its organization and method of operation, together with further objects and advantages will be better understood from the following description when considered in connection with the accompanying figures. It is to be expressly understood, however, that each of the figures is provided for the purpose of illustration and description only and is not intended as a definition of the limits of the present invention.DETAILED DESCRIPTION

[0127] In the description herein for embodiments of the present disclosure, numerous specific details are provided, such as examples of circuits, devices, components and / or methods, to provide a thorough understanding of embodiments of the present disclosure. One skilled in the relevant art will recognize, however, that an embodiment of the disclosure can be practiced without one or more of the specific details, or with other apparatuses, systems, assemblies, methods, components, materials, parts, and / or the like set forth in reference to other embodiments herein. In other instances, well-known structures, materials, or operations are not specifically shown or described in detail to avoid obscuring aspects of embodiments of the present disclosure. Accordingly, it will be appreciated by one skilled in the art that the embodiments of the present disclosure may be practiced without such specific components. It should be further recognized that those of ordinary skill in the art, with the aid of the Detailed Description set forth herein and taking reference to the accompanying drawings, will be able to make and use one or more embodiments without undueexperimentation.(32736-2156)

[0128] Additionally, terms such as “coupled” and “connected,” along with their derivatives, may be used in the following description, claims, or both. It should be understood that these terms are not necessarily intended as synonyms for each other. “Coupled” may be used to indicate that two or more elements, which may or may not be in direct physical or electrical contact with each other, co-operate or interact with each other. “Connected” may be used to indicate the establishment of communication, i. e. , a communicative relationship, between two or more elements that are coupled with each other. Further, in one or more example embodiments set forth herein, generally speaking, an electrical element, component or module may be configured to perform a function if the element may be programmed for performing or otherwise structurally arranged to perform that function.

[0129] Example embodiments described herein relate to aspects of implementations of an integrated digital health network architecture that may be effectuated as a convergence of various technologies involving diverse end user devices and computing platforms, heterogeneous network connectivity environments, agile software as a medical device (SaMD) deployments, data analytics and machine learning, secure cloud-centric infrastructures for supporting remote healthcare, etc. Some embodiments may be configured to support various types of healthcare solutions including but not limited to remote patient monitoring, integrated session management for providing telehealth applications as well as remote care therapy applications, personalized therapy based on advanced analytics of patient and clinician data, remote trialing of neuromodulation therapies, e.g., pain management / amelioration solutions, and the like. Whereas some example embodiments may be particularly set forth with respect to implantable pulse generator (IPG) or neuromodulator systems for providing therapy to a desired area of a body or tissue based on a suitable stimulation therapy application, such as spinal cord stimulation (SCS) systems or other neuromodulation systems, it should be understood that example embodiments disclosed herein are not limited thereto but have broad applicability. Some example remote care therapy applications may therefore involve different types of implantable devices such as neuromuscular stimulation systems and sensors, dorsal root ganglion (DRG) stimulation systems, deep brain stimulation systems, cochlear implants, retinal implants, implantable cardiac rhythm management(32736-2156)devices, implantable cardioverter defibrillators, pacemakers, and the like, as well as implantable drug delivery / infusion systems, implantable devices configured to effectuate real-time measurement / monitoring of one or more physiological functions of a patient’s body (i.e., patient physiometry), including various implantable biomedical sensors and sensing systems. Further, whereas some example embodiments of remote care therapy applications may involve implantable devices, additional and / or alternative embodiments may involve external personal devices and / or noninvasive / minimally invasive (NIMI) devices, e.g., wearable biomedical devices, transcutaneous / subcutaneous devices, etc., that may be configured to provide therapy to the patients analogous to the implantable devices. Accordingly, all such devices may be broadly referred to as “personal medical devices,” “personal biomedical instrumentation,” or terms of similar import, at least for purposes of some example embodiments of the present disclosure.

[0130] As used herein, a network element, platform or node may be comprised of one or more pieces of network equipment, including hardware and software that communicatively interconnects other equipment on a network (e.g., other network elements, end stations, etc.), and is adapted to host one or more applications or services, more specifically healthcare applications and services, with respect to a plurality of end users, e.g., patients, clinicians, respective authorized agents, and associated client devices as well as other endpoints such as medical- and / or health-oriented Internet of Medical Things (loMT) devices / sensors and / or other Industrial loT-based entities. As such, some network elements may be operatively disposed in a cellular wireless or satellite telecommunications network, or a broadband wireline network, whereas other network elements may be disposed in a public packet-switched network infrastructure (e.g., the Internet or worldwide web, also sometimes referred to as the “cloud”), private packet-switched network infrastructures such as Intranets and enterprise networks, as well as service provider network infrastructures, any of which may span or involve a variety of access networks, backhaul and core networks in a hierarchical arrangement. In still further arrangements, one or more network elements may be disposed in cloud-based platforms or datacenters having suitable equipment running virtualized functions or applications, which may(32736-2156)be configured for purposes of facilitating patient monitoring, remote therapy, other telehealth / telemedicine applications, etc. for purposes of one or more example embodiments set forth hereinbelow.

[0131] One or more embodiments of the present patent disclosure may be implemented using different combinations of software, firmware, and / or hardware. Thus, one or more of the techniques shown in the Figures (e.g., flowcharts) may be implemented using code and data stored and executed on one or more electronic devices or nodes (e.g., a subscriber client device or end station, a network element, etc.). Such electronic devices may store and communicate (internally and / or with other electronic devices over a network) code and data using computer-readable media, such as non-transitory computer-readable storage media (e.g., magnetic disks, optical disks, random access memory, readonly memory, flash memory devices, phase-change memory, etc.), transitory computer-readable transmission media (e.g., electrical, optical, acoustical or other form of propagated signals - such as carrier waves, infrared signals, digital signals), etc. In addition, such network elements may typically include a set of one or more processors coupled to one or more other components, such as one or more storage devices (e.g., non-transitory machine-readable storage media) as well as storage database(s), user input / output devices (e.g., a keyboard, a touch screen, a pointing device, and / or a display), and network connections for effectuating signaling and / or bearer media transmission.

[0132] Without limitation, an example cloud-centric digital healthcare network architecture involving various network-based components, subsystems, service nodes etc., as well as myriad end user deployments concerning patients, clinicians and authorized third-party agents is illustrated in FIG. 12 wherein one or more embodiments of the present patent disclosure may be practiced. In one arrangement, example architecture 1260 may include one or more virtual clinic platforms 1214, remote data logging platforms 1216, patient / clinician report processing platforms 1218, as well as data analytics platforms 1220 and security platforms 1222, at least some of which may be configured and / or deployed as an integrated digital health infrastructure 1212 for effectuating some example embodiments of the present disclosure. One or more pools of patients having a myriad of health conditions and / or receiving assorted treatments, who may be(32736-2156)geographically distributed in various locations, areas, regions, etc., are collectively shown at reference numeral 1204, wherein individual patients may be provided with one or more suitable IMDs / IPGs, NIMI devices, other personal biomedical instrumentation, etc., depending on respective patients’ health conditions and / or treatments. A plurality of clinician programmer devices 1208, patient controller devices 1210, and authorized third-party devices 1211 associated with respective users (e.g., clinicians, medical professionals, patients and authorized agents thereof) may be deployed as external devices 1206 that may be configured to interact with patients’ IMDs and / or NIMI devices for effectuating therapy, monitoring, data logging, secure file transfer, etc., via local communication paths or over network-based remote communication paths established in conjunction with the digital health infrastructure network 1212.

[0133] Clinician controller device 1208 may permit programming of IPG 170 to provide a number of different stimulation patterns or therapies to the patient as appropriate for a given patient and / or disorder. Examples of different stimulation therapies include conventional tonic stimulation (continuous train of stimulation pulses at a fixed rate), BurstDR stimulation (burst of pulses repeated at a high rate interspersed with quiescent periods with or without duty cycling), “high frequency” stimulation (e.g., a continuous train of stimulation pulses at 10,000 Hz), noise stimulation (series of stimulation pulses with randomized pulse characteristics such as pulse amplitude to achieve a desired frequency domain profile). Any suitable stimulation pattern or combination thereof can be provided by IPG 170 according to some embodiments. Controller device 1208 communicates the stimulation parameters and / or a series of pulse characteristics defining the pulse series to be applied to the patient to IPG 170 to generate the desired stimulation therapy.

[0134] IPG 170 may be adapted to apply a variety of neurostimulation therapies while controller device 1208 may send signals to IPG 170 related to such therapies. Examples of suitable therapies include tonic stimulation (in which a fixed frequency pulse train) is generated, burst stimulation (in which bursts of multiple high frequency pulses) are generated which in turn are separated by quiescent periods, “high frequency” stimulation, multi-frequency stimulation, and noise stimulation. Descriptions of respective neurostimulation therapies are provided in the following publications: (1) Schu S., Slotty P.J., Bara G., von Knop M., Edgar D.,(32736-2156)Vesper J. A Prospective, Randomised, Double-blind, Placebo-controlled Study to Examine the Effectiveness of Burst Spinal Cord Stimulation Patterns for the Treatment of Failed Back Surgery Syndrome. Neuromodulation 2014; 17: 443-450; (2) Al-Kaisy A1 , Van Buyten JP, Smet I, Palmisani S, Pang D, Smith T. 2014. Sustained effectiveness of 10 kHz high-frequency spinal cord stimulation for patients with chronic, low back pain: 24-month results of a prospective multicenter study. Pain Med. 2014 Mar;15(3):347-54; and (3) Sweet, Badjatiya, Tan D1, Miller. Paresthesia-Free High-Density Spinal Cord Stimulation for Postlaminectomy Syndrome in a Prescreened Population: A Prospective Case Series.Neuromodulation. 2016 Apr;19(3):260-7. Noise stimulation is described in U.S. Patent No. US8682441B2. Burst stimulation is described in U.S. Patent No.8,224,453 and U.S. Published Application No. 20060095088. A “coordinated reset” pulse pattern is applied to neuronal subpopulation / target sites to desynchronize neural activity in the subpopulations. Coordinated reset stimulation is described, for example, by Peter A. Tass etal in COORDINATED RESET HAS SUSTAINED AFTER EFFECTS IN PARKINSONIAN MONKEYS, Annals of Neurology, Volume 72, Issue 5, pages 816-820, November 2012, which is incorporated herein by reference. The electrical pulses in a coordinated reset pattern are generated in bursts of pulses with respective bursts being applied to tissue of the patient using different electrodes in a time-offset manner. The time-offset is selected such that the phase of the neural-subpopulations are reset in a substantially equidistant phase-offset manner. By resetting neuronal subpopulations in this manner, the population will transition to a desynchronized state by the interconnectivity between the neurons in the overall neuronal population. All of these references are incorporated herein by reference.

[0135] In one arrangement, example architecture 1260 may encompass a hierarchical / heterogeneous network arrangement comprised of one or more fronthaul radio access network (RAN) portions or layers, one or more backhaul portions or layers, and one or more core network portions or layers, each of which may in turn include appropriate telecommunications infrastructure elements, components, etc., cooperatively configured for effectuating a digital healthcare ecosystem involving patients’ IMDs and / or NIMI devices 1204, external devices 1206, and one or more components of the digital health infrastructure network(32736-2156)1212, wherein at least a portion of the components of the infrastructure network 1212 may be operative as a cloud-based system for purposes of some embodiments herein. Further, at least a portion of the components of the digital health infrastructure network 1212 operating as a system 1200, one or more patients’ IMDs and / or NIMI devices 1204, and one or more external devices 1206 may be configured to execute suitable medical / health software applications in a cooperative fashion, e.g., in a server-client relationship, for effectuating various aspects of remote patient monitoring, telemedicine / telehealth applications, remote care therapy, etc. Without limitation, example embodiments of the present disclosure may relate to one or more aspects set forth immediately below.

[0136] In some example arrangements, a virtual clinic may be configured to provide patients and / or clinicians the ability to perform remote therapies using a secure telehealth session. To enhance clinician interaction and evaluation of a patient during a secure telehealth session, example embodiments herein may be configured to provide specific user interface (Ul) layouts and controls for clinician programmer devices and / or patient controller devices for facilitating real-time kinematic and / or auditory data analysis, which may be augmented with suitable artificial intelligence (Al) and / or machine learning (ML) techniques (e.g., neural networks, etc.) in some arrangements. Further, some example embodiments with respect to these aspects may involve providing kinematic III settings that enable different types of overlays, e.g., with or without a pictorial representation of the patient. Some example embodiments may be configured to enable one or more of the following features and functionalities: (i) separate or combined audio and / or peripheral sensor streams; (ii) capture of assessments from separate or different combinations of body features such as, e.g., limbs, hands, face, etc.; (iii) replay of another clinician’s video including the patient’s kinematic analysis (e.g., a secondary video stream with patient data), and the like.

[0137] In some arrangements, video-based real-time kinematic analysis may employ statistical methods within a pipeline of modeling techniques to track inter-frame correlation of images. To maintain real-time operation within a fixed compute environment such as an edge device (e.g., a patient controller device, a clinician programmer device, etc.), the inference latency needs to be less than certain thresholds in order to achieve reliable and / or acceptable performance.(32736-2156)Some example embodiments herein may therefore relate to providing a scheme for improving the accuracy of real-time kinematic and / or auditory analysis based on context-aware dynamic (re)configu ration of a neural network model or engine trained for facilitating real-time kinematic and / or auditory data analysis.

[0138] In some arrangements involving neurostimulation therapy, different stimulation settings and / or programs may be configured for providing varied levels of comfort to the patients, wherein respective patients may likely need to change individual settings depending on a number of factors, e.g., time of day, type(s) and / or level(s) of activities or tasks being engaged by the patients, etc. Further, continued use of a stimulation program or setting over an extended period of time could result in habituation that may reduce the benefits of therapy. Some example embodiments herein may therefore relate to a system and method for providing recommendations / reconfigurations of program settings based on the patient’s usage of the IMD and clinical observations / recommendations, e.g., to facilitate context-sensitive selection of neuromodulation programs / settings.

[0139] In some arrangements, video-, audio-, and / or sensing-based analytics and associated ML-based techniques effectuated using one or more constituent components of the digital health infrastructure 1212 may provide valuable insights with respect to the diagnosis / or prognosis of individual patients, especially those having certain neurological disorders. Some of the neurological symptoms may depend on the context, e.g., time of day, activity type, psychological / emotional conditions of the patient, etc., such that a generalized ML model may not be sufficiently accurate for predictive purposes in a particular setting. Some example embodiments herein may therefore relate to a scheme for rapidly collecting relevant patient data and analyzing / manipulating the data for generating suitable training datasets with respect to select ML-based models using an accelerated inference approach.

[0140] Because the video / audio data collected and used for training, validating and testing AI / ML-based models can contain various pieces of personal identification / identifiable information (PH) indicators associated with patients, it is appropriate to protect the privacy of the patient data by providing a suitable end-to-end security architecture. In some related arrangements, one or more constituent components of the digital health infrastructure 1212 may therefore be configured to(32736-2156)facilitate secure transfer with respect to the patient data collected for purposes of data analytics and associated ML-based techniques as set forth herein. Some example embodiments in this regard may be configured to provide a scheme for de-identifying data associated with neuromodulation patients, e.g., either in realtime or in a post-processed environment, that still allows implementing ML-based techniques and data sharing in a secure cooperative arrangement Still further embodiments may relate to facilitating an improved method of removal of identifiable information from video / audio data streams generated pursuant to a therapy session, e.g., a remote therapy session.

[0141] In relation to certain aspects of telemedicine, it is recognized that monitoring body rigidity may be an important factor in certain types of motor / neurological disorders, e.g., Parkinson's disease (PD). Rigidity may be defined as an involuntary increase in muscle tone in certain portions of the patient body, e.g., generally affecting arms, neck, leg, hip or trunk of the patient. Rigidity can be classified as lead-pipe when the movement is smooth and consistent or cog-wheeling when it is ratchet-type. Cog-wheeling generally occurs when rigidity is superimposed on tremor. Rigidity may be measured in-clinic with the clinician physically manipulating the patient to assess signs and symptoms. In some experimental setups, rigidity can also be measured with an apparatus to detect the displacement for applied force. However, in a telehealth scenario, there is a need to accurately and reliably measure rigidity in a remote setting. Some example embodiments herein may therefore relate to a system and method for facilitating remote-based assessment of rigidity that may involve combining signals from sensors and video analytics obtained via a secure remote session with the patient.

[0142] In still further aspects relating to remote patient monitoring, some example embodiments may be configured to effectuate a closed-loop, sensorbased Al-driven exercise training platform that may be implemented by way of an integrative telehealth application. Such embodiments advantageously leverage the principle that exercise regimen involving balance training as part of a physical routine can provide additional benefits for patients with balance / gait-related disorders on top of the benefits exercising itself already brings. Whereas exercises that involve balance training may be taught by a tea cher / instru ctor in a face-to-face setting, where the teacher can manipulate the trainee’s gesture in addition to(32736-2156)offering visual and verbal instructions, example embodiments herein may involve a remote learning and real-time patient monitoring session for facilitating an Al-driven, network-based remote exercising arrangement. Additionally, where patients with movement disorders such as PD often report difficulties with everyday tasks such as buttoning, brushing, writing, etc., example embodiments may be configured to provided individualized training tailored to the patient based on Al integration in order to enable the experience of a personal trainer with focused attention and real-time corrective measures for gesture training.

[0143] Additional details with respect to the various constituent components of the digital health infrastructure 1212, example external devices 1206 comprising clinician programmer devices 1208, patient controller devices 1210 and / or third-party devices 1211, as well as various interactions involving the network-based entities and the end points (also referred to as edge devices) will be set forth immediately below in order to provide an example architectural framework wherein one or more of the foregoing embodiments may be implemented and / or augmented according to the teachings herein.

[0144] Turning to FIG. 1A, depicted therein is an example architecture of a system configured to support remote patient therapy as part of an integrated remote care service session in a virtual clinic environment that may be deployed in a cloud-centric digital health implementation for purposes of an embodiment of the present patent disclosure. As used herein, a “remote care system” may describe a healthcare delivery system configured to support a remote care service over a network in a communication session between a patient and a clinician wherein telehealth or telemedicine applications involving remote medical consultations as well as therapy applications involving remote programming of the patient’s IMD may be launched via a unified application interface facilitated by one or more network entities (e.g., as a virtual clinic platform). In some arrangements, a remote care system may also include a remote patient monitoring system and / or a remote healthcare provisioning system without the involvement of a clinician. In still further arrangements, a remote care system may include one or more Al-based expert systems or agents, e.g., involving supervised learning, that may be deployed in a network to provide or otherwise augment the capabilities of a system to effectuate enhanced healthcare solutions relating to diagnosis, remote learning, therapy(32736-2156)selection, as well as facilitate network-based solutions for enhancing patients’ overall well-being. In some aspects, remote care therapy may involve any care, programming, or therapy instructions that may be provided by a doctor, a medical professional or a healthcare provider, and / or their respective authorized agents, collectively referred to as a “clinician”, using a suitable clinician device, with respect to the patient’s IMD, wherein such therapy instructions may be mediated, proxied or otherwise relayed by way of a controller device associated with the patient. As illustrated, example remote care system 100A may include a plurality of patient controller devices exemplified by patient controller device 150 and a plurality of clinician programmer devices exemplified by a clinician programmer device 180 (also referred to as a clinician programmer or clinician device) that may interact with a network-based infrastructure via respective communication interfaces. Example patient and clinician devices may each include a corresponding remote care service application module, e.g., a patient controller application 152 and a clinician program mer / controller application 182, executed on a suitable hardware / software platform for supporting a remote care service that may be managed by a network entity 155. In some embodiments, example network entity 155 may comprise a distributed datacenter or cloud-based service infrastructure (e.g., disposed in a public cloud, a private cloud, or a hybrid cloud, involving at least a portion of the Internet) operative to host a remote care session management service 157. In one arrangement, patient controller application 152 and clinician programmer application 182 may each include a respective remote session manager 154, 184 configured to effectuate or otherwise support a corresponding communication interface 160, 190 with network entity 155 using any known or heretofore unknown communication protocols and / or technologies. In one arrangement, interfaces 160, 190 are each operative to support an audio / video or audiovisual (AV) channel or session 163A, 163B and a remote therapy channel or session 165A, 165B, respectively, with an AV communication service 161 A and a remote therapy session service 161 B of the remote care session management service 157 as part of a common bi-directional remote care session 159, 199 established therewith. In one arrangement, patient controller application 152 and clinician programmer application 182 may each further include or otherwise support suitable graphical user interfaces (GUIs) and associated(32736-2156)controls 156, 186, as well as corresponding AV managers 158, 188, each of which may be interfaced with respective remote session managers 154, 184 for purposes of one or more embodiments of the present disclosure as will be set forth in additional detail further below. Remote care session manager 154 of the patient controller application 152 and remote care session manager 184 of the clinician programmer application 182 may each also be interfaced with a corresponding data logging manager 162, 186 for purposes of still further embodiments of the present disclosure. In one arrangement, remote care session manager 154 of patient controller application 152 is further interfaced with a security manager 164, which may be configured to facilitate secure or trusted communication relationships with the network entity 155. Likewise, remote care session manager 184 of clinician programmer application 182 may also be interfaced with a security manager 188 that may be configured to facilitate secure or trusted communication relationships with the network entity 155. Each security manager 164, 188 may be interfaced with a corresponding therapy communication manager 166, 190 with respect to facilitating secure therapy communications between the clinician programmer device 180 and the patient controller device 150. Therapy communication manager 166 of the patient controller application 152 may also interface with a local communication module 168 operative to effectuate secure communications with the patient’s IPG / IMD 170 using a suitable short-range communications technology or protocol. In still further arrangements, security managers 164, 188 of patient controller and clinician programmer applications 152, 182 may be configured to interface with the remote care session management service 157 to establish trusted relationships between patient controller device 150, clinician programmer device 180 and IPG / IMD 170 based on the exchange of a variety of parameters, e.g., trusted indicia, cryptographic keys and credentials, etc.

[0145] In one arrangement, the integrated remote care session management service 157 may include a session data management module 171 , an AV session recording service module 175 and a registration service module 183, as well as suitable database modules 173, 185 for storing session data and user registration data, respectively. In some arrangements, at least part of the session data may include user-characterized data relating to AV data, therapy(32736-2156)settings data, network contextual data, and the like, for purposes of still further embodiments of the present patent disclosure.

[0146] Skilled artisans will realize that example remote care system architecture 100A set forth above may be advantageously configured to provide both telehealth medical consultations as well as therapy instructions over a communications network while the patient and the clinician / provider are not in close proximity of each other (e.g., not engaged in an in-person office visit or consultation). Accordingly, in some embodiments, a remote care service of the present disclosure may form an integrated healthcare delivery service effectuated via a common application user interface that not only allows healthcare professionals to use electronic communications to evaluate and diagnose patients remotely but also facilitates remote programming of the patient’s IPG / IMD for providing appropriate therapy, thereby enhancing efficiency as well as scalability of a delivery model. Additionally, example remote care system architecture 100A may be configured to effectuate various other aspects relating to remote learning, remote patient monitoring, etc., as noted above. Further, an implementation of example remote care system architecture 100A may involve various types of network environments deployed over varying coverage areas, e.g., homogenous networks, heterogeneous networks, hybrid networks, etc., which may be configured or otherwise leveraged to provide patients with relatively quick and convenient access to diversified medical expertise that may be geographically distributed over large areas or regions, preferably via secure communications channels in some example embodiments as will be set forth in detail further below.

[0147] FIG. 1B depicts an example network environment 100B wherein the remote care service architecture of FIG. 1 A may be implemented according to some embodiments. Illustratively, example network environment 100B may comprise any combination or sub-combination of a public packet-switched network infrastructure (e.g., the Internet or worldwide web, also sometimes referred to as the “cloud”, as noted above), private packet-switched network infrastructures such as Intranets and enterprise networks, health service provider network infrastructures, and the like, any of which may span or involve a variety of access networks, backhaul and core networks in an end-to-end network architecture arrangement between one or more patients, e.g., patient(s) 102, and one or more(32736-2156)authorized clinicians, healthcare professionals, or agents thereof, e.g., generally represented as caregiver(s) or clinician(s) 138. Example patient(s) 102, each having one or more suitable implantable and / or NIMI devices, e.g., IMD 103, may be provided with a variety of corresponding external devices for controlling, programming, otherwise (re)configuring the functionality of respective implantable device(s) 103, as is known in the art. Such external devices associated with patient(s) 102, referred to herein as patient devices 104, which are representative of patient controller device 150 shown in FIG. 1A, may comprise a variety of user equipment (UE) devices, tethered or untethered, that may be configured to engage in remote care sessions involving telehealth and / or therapy sessions according to some embodiments described below. By way of example, patient devices 104 may comprise commercial off-the-shelf (COTS) equipment or proprietary portable medical / healthcare devices (non-COTS), which may be configured to execute a therapy / digital healthcare application program or“app”, wherein various types of communications relating to control, therapy / diagnostics, and / or device file management may be effectuated for purposes of some embodiments. Accordingly, example patient devices 104 may include, in addition to proprietary medical devices, devices such as smartphones, tablets or phablets, laptops / desktops, handheld / palmtop computers, wearable devices such as smart glasses and smart watches, personal digital assistant (PDA) devices, smart digital assistant devices, etc., any of which may operate in association with one or more virtual assistants, smart home / office appliances, smart TVs, external / auxiliary AV equipment, virtual reality (VR), mixed reality (MR) or augmented reality (AR) devices, and the like, which are generally exemplified by wearable device(s) 106, smartphone(s) 108, tablet(s) / phablet(s) 110, computer(s) 112, and AV equipment 114. As such, example patient devices 104 may include various types of communications circuitry or interfaces to effectuate wired or wireless communications, short-range and long-range radio frequency (RF) communications, magnetic field communications, etc., using any combination of technologies, protocols, and the like, with external networked elements and / or respective implantable devices 103 corresponding to patient(s) 102. With respect to networked communications, patient devices 104 may be configured, independently or in association with one or more dig ital / virtual assistants, smart home / premises appliances and / or home(32736-2156)networks, to effectuate mobile communications using technologies such as Global System for Mobile Communications (GSM) radio access network (GRAN) technology, Enhanced Data Rates for Global System for Mobile Communications (GSM) Evolution (EDGE) network (GERAN) technology, 4G Long Term Evolution (LTE) technology, Fixed Wireless technology, 5thGeneration Partnership Project (5GPP or5G) technology, Integrated Digital Enhanced Network (IDEN) technology, WiMAX technology, various flavors of Code Division Multiple Access (CDMA) technology, heterogeneous access network technology, Universal Mobile Telecommunications System (UMTS) technology, Universal Terrestrial Radio Access Network (UTRAN) technology, All-1 P Next Generation Network (NGN) technology, as well as technologies based on various flavors of IEEE 802.11 protocols (e.g., WiFi), and other access point (AP)-based technologies and microcell-based technologies involving small cells, femtocells, picocells, etc.Further, some embodiments of patient devices 104 may also include interface circuitry for effectuating network connectivity via satellite communications. Where tethered UE devices are provided as patient devices 104, networked communications may also involve broadband edge network infrastructures based on various flavors of Digital Subscriber Line (DSL) architectures and / or Data Over Cable Service Interface Specification (DOCSIS)-compliant Cable Modem Termination System (CMTS) network architectures (e.g., involving hybrid fibercoaxial (HFC) physical connectivity). Accordingly, by way of illustration, an edge / access network portion 119A is exemplified with elements such as WiFi / AP node(s) 116-1, macro-cell node(s) 116-2 such as eNB nodes, gNB nodes, etc., microcell nodes 116-3 (e.g., including micro remote radio units or RRUs, etc.) and DSL / CMTS node(s) 116-4.

[0148] In similar fashion, clinicians and / or clinician agents 138 may be provided with a variety of external devices for controlling, programming, otherwise (re)configuring or providing therapy operations with respect to one or more patients 102 mediated via respective implantable device(s) 103, in a local therapy session and / or telehealth / remote therapy session, depending on implementation and use case scenarios. External devices associated with clinicians / agents 138, referred to herein as clinician devices 130, which are representative of clinician programmer device 180 shown in FIG. 1 A, may comprise a variety of UE devices, tethered or(32736-2156)untethered, similar to patient devices 104, that may be configured to engage in telehealth and / or remote care therapy sessions as will be set forth in detail further below. Clinician devices 130 may therefore also include non-COTS devices as well as COTS devices generally exemplified by wearable device(s) 131, smartphone(s) 132, tablet(s) / phablet(s) 134, computer(s) 136 and external / auxiliary AV equipment 137, any of which may operate in association with one or more virtual assistants, smart home / office appliances, VR / AR / MR devices, and the like. Further, example clinician devices 130 may also include various types of network communications circuitry or interfaces similar to that of personal devices 104, which may be configured to operate with a broad range of technologies as set forth above.Accordingly, an edge / access network portion 119B is exemplified as having elements such as WiFi / AP node(s) 128-1 , macro / microcell node(s) 128-2 and 128-3 (e.g., including micro remote radio units or RRUs, base stations, eNB / gNB nodes, etc.) and DSL / CMTS node(s) 128-4. It should therefore be appreciated that edge / access network portions 119A, 119B may include all or any subset of wireless / wireline communication infrastructures, technologies and protocols for effectuating data communications with respect to an example embodiment of the present disclosure.

[0149] In one arrangement, a plurality of network elements or nodes may be provided for facilitating an integrated remote care therapy service involving one or more clinicians 138 and one or more patients 102, wherein such elements are hosted or otherwise operated by various stakeholders in a service deployment scenario depending on implementation, e.g., including one or more public clouds, private clouds, or any combination thereof as previously noted. According to some example embodiments, a remote care session management node or platform 120 may be provided, generally representative of the network entity 157 shown in FIG.1 A, preferably disposed as a cloud-based element coupled to network 118, that is operative in association with a secure communications credentials management node 122 and a device management node 124, to facilitate a virtual clinic platform whereby a clinician may advantageously engage in a telehealth session and / or a remote care therapy session with a particular patient via a common application interface and associated AV and therapy controls, as will be described further below.(32736-2156)

[0150] FIG. 2 depicts a flowchart for establishing a remote programing or virtual clinic session according to known processes. Additional details regarding establishment of remote programming or virtual clinic sessions may be found in U.S. Patent No. 10,124,177 which is incorporated herein by reference. Although some details are described herein regarding establishment of a virtual clinic / remote programming session, any suitable methods may be employed according to other embodiments. At block 202, the patent controller device connects to the patient’s medical device. For example, the patent controller device may establish a BLUETOOTH communication session with the patient’s implantable pulse generator. At block 204, the patient launches the patient controller app on the patient controller device. At block 206, the patient starts a virtual clinic check in process by selecting a suitable GUI component of the patient controller app. In block 208, the patient may provide patient credentials. At block 210, the clinician launches the clinician programmer app on the clinician programmer device. In block 212, the clinician provides credentials. At block 214, the clinician checks into the virtual clinic to communicate with the patient. At block 216, the virtual clinic infrastructure establishes a secure connection between the patient controller app and clinician programmer app to conduct communications. Known cybersecurity features may be applied to establish the secure connection including using PKI processes, encryption processes, authentication processes, etc. Biometric data and other data may also be employed to enhance the secure nature of the communication session by validating user identities and authorization. Upon establishment, the communications may include audio and video communications between the patient and the clinician. Also, the clinician may conduct remote programming of the patient’s medical device during the session while communicating with the patient.

[0151] FIG. 3 depicts a flowchart illustrative of known blocks, steps and / or acts that may be implemented for establishing a communication session with an implantable medical device. Additional details regarding establishing a communication session with an implantable medical device may be found in U.S. Patent No. 11,007,370 which is incorporated herein by reference. Although example operations are described in FIG. 3, any suitable methods of securing communication between a PC device and a patient medical device may be(32736-2156)employed as appropriate for a given patient therapy. At block 302, a bonding procedure is initiated to establish a trusted relationship between the implantable medical device or other medical device of the patient and a patient controller device. The bonding procedure may be initiated by using a magnet to activate a Hall sensor in the medical device. Alternatively, near field communication (e.g., inductive coupling) may be employed to initiate the bonding procedure. The use of a magnetic or inductive coupling provides a degree of physical access control to limit the possibility of unauthorized devices from communicating with the patient’s medical device. That is, a device that attempts to obtain authorization to communicate must be brought into physical proximity with the patient at a time that is controlled by the patient thereby reducing the possibility of unauthorized devices from improperly gaining the ability to communicate with the patient’s device. At block 304, a communication session is established between the patient’s medical device and the patient’s controller (PC) device. For example, a BLUETOOTH communication session may be established. At block 306, authentication operations are conducted. For example, known credentials, PKI, encryption, and other cybersecurity operations may be applied between the patient’s medical device and the PC device to determine whether the PC device should be allowed to conduct communications with the patient’s medical device. At block 308, encryption key data may be exchanged between devices for future communications. At block 310, other PC identifiers or data may be stored in IMD to control future communications. Upon establishment of a trusted relationship between the patient’s medica device and a PC device, the PC device may be used to conduct remote programming session. The remote programming sessions may also be subjected to cybersecurity methods such as the use of credentials, PKI, encryption, etc.

[0152] FIGS. 4A and 4B depicts flowcharts illustrative of a remote care scenario involving an example digital health network architecture wherein an integrated remote care session may be established between a patient and a clinician operating respective controller devices that support suitable graphical user interfaces (GUIs) for facilitating a therapy session, an audio / visual (AV) communication session, or a combination of both, for purposes of some example embodiments of the present disclosure. As will be set forth further below, patient(32736-2156)controller and / or clinician programmer devices may be provided with appropriate application software to effectuate suitable GUIs on respective devices for facilitating a remote care session including a secure AV session / channel and a therapy session / channel as part of a common application interface that can support telehealth / telemedicine applications, remote monitoring, remote therapy, remote learning, data logging, etc. Process flow 400A of FIG. 4A may commence with a patient launching an integrated digital health application executing on the patient controller / device to initiate a secure communications channel with a remote clinician (block 402), e.g., by selecting a “Remote Care” option from a pull-down menu, clicking on an icon on the Ul display screen, or via a voice command, etc. In one embodiment, the patient may be ushered into a virtual waiting room, which may be realized in a Ul screen window of the patient / clinician device (block 404). At block 406, the clinician responds to the waiting patient, e.g., via a secure AV communication channel of the remote care session. At block 408, one or more physiological / biological data of the patient (stored or real-time) may be provided to the clinician via secure communications. In some embodiments, one or more digital keys of the clinician and / or the patient may be employed to secure communications. At block 410, the clinician evaluates the patient in view of the physiological / biological data, telemedicine / video consultation, audio / visual cues and signals regarding patient’s facial expressions, hand movement / tremors, walking, gait, ambulatory status / stability, and other characteristics to arrive at appropriate medical assessment. Depending on such telehealth consultation / evaluation, the clinician may remotely adjust stimulation therapy settings for secure transmission to the patient device, which may be securely transmitted via encrypted communications. In a further scenario, a remote clinician proxy or agent may be executed at or in association with the patient controller / device upon launching a remote session, wherein the proxy / agent is operative to effectuate or otherwise mediate the transmission of any therapy settings to the patient’s IMD, either in real-time or at some point in the future depending upon programmatic control. After completing the requisite therapy and consultative communications, the remote care session may be terminated, e.g., either by the clinician and / or the patient, as set forth at block 412.(32736-2156)

[0153] Process flow 400B of FIG. 4B is illustrative of an embodiment of a high level scheme for delivering healthcare to a patient via an integrated remote care session. At block 422, a remote care session between a controller device associated with the patient and a programmer device associated with a clinician may be established, wherein the clinician and the patient are remotely located with respect to each other and the remote care session includes an AV communication session controlled by one or more A / V controls provided at the patient controller device and the clinician programmer device. At block 424, various telehealth consultation services may be provided to the patient by the clinician based on interacting with the patient via the AV communication channel of the remote care session as previously noted. Responsive to determining that the patient requires remote therapy, one or more remote programming instructions may be provided to the patient's IMD via a remote therapy session or channel of the remote care session with the patient controller device while the AV communication session is maintained (block 426).

[0154] Skilled artisans will recognize that some of the blocks, steps and / or acts set forth above may take place at different entities and / or different times (i.e. , asynchronously), and possibly with intervening gaps of time and / or at different locations. Further, some of the foregoing blocks, steps and / or acts may be executed as a process involving just a single entity (e.g., a patient controller device, a clinician programmer device, ora remote session manager operating as a virtual clinic, etc.), or multiple entities, e.g., as a cooperative interaction among any combination of the end point devices and the network entities. Still further, it should be appreciated that example process flows may be interleaved with one or more sub-processes comprising other IMD<=>patient or IMD<=>clinician interactions (e.g., local therapy sessions) as well as virtual clinic<=>patient or virtual clinic<=>clinician interactions (e.g., remote patient monitoring, patient / clinician data logging, remote learning, rigidity assessment, context-aware kinematic and auditory analysis, etc., as will be set forth further below).Accordingly, skilled artisanswill recognize that example process flows may be altered, modified, augmented or otherwise reconfigured for purposes of some embodiments herein.(32736-2156)

[0155] In one implementation, an example remote care session may be established between the patient controller device and the clinician programmer device after the patient has activated a suitable GUI control provided as part of a GUI associated with the patient controller device and the clinician has activated a corresponding GUI control provided as part of a virtual waiting room displayed on a GUI associated with the clinician programmer device. In another arrangement, remote programming instructions may be provided to the patient's IMD via the remote therapy session only after verifying that remote care therapy programming with the patient's IMD is compliant with regulatory requirements of one or more applicable local, regional, national, supranational governmental bodies, nongovernmental agencies, and international health organizations. In a still further variation, various levels of remote control of a patient's controller and its hardware by a clinician programmer device may be provided. For example, suitable GUI controls may be provided at the clinician programmer device for remotely controlling a camera component or an auxiliary AV device associated with the patient controller device by interacting with a display of the patient's image on the screen of the clinician programmer device, e.g., by pinching, swiping, etc., to pan to and / or zoom on different parts of the patient in order to obtain high resolution images. Additional embodiments and / or further details regarding some of the foregoing variations with respect to providing remote care therapy via a virtual clinic may be found in the following U.S. patent applications, publications and / or patents: (i) U.S. Patent Application Publication No. 2020 / 0398062, entitled “SYSTEM, METHOD AND ARCHITECTURE FOR FACILITATING REMOTE PATIENT CARE”; (ii) U.S. Patent Application Publication No. 2020 / 0402656, entitled “Ul DESIGN FOR PATIENT AND CLINICIAN CONTROLLER DEVICES OPERATIVE IN A REMOTE CARE ARCHITECTURE”; (iii) U.S. Patent Application Publication No. 2020 / 0402674, entitled “SYSTEM AND METHOD FOR MODULATING THERAPY IN A REMOTE CARE ARCHITECTURE”; and (iv) U.S. Patent Application Publication No. 2020 / 0398063, entitled “DATA LABELING SYSTEM AND METHOD OPERATIVE WITH PATIENT AND CLINICIAN CONTROLLER DEVICES DISPOSED IN A REMOTE CARE ARCHITECTURE”, each of which is hereby incorporated by reference herein.(32736-2156)

[0156] FIGS. 5A and 5B depict representations of an example user interface and associated dialog boxes or windows provided with a clinician programmer device, e.g., as a touch screen display of device 180 exemplified in FIG. 1 A, for selecting different therapy applications and / or service modes in an integrated remote care service application for purposes of some example embodiments of the present disclosure. In one arrangement, example GUI(s) of the clinician device may be optimized or resized to provide a maximum display window for the presentation of a patient’s image during remote therapy while allowing the presentation of appropriate remote care therapy session and setting controls as well as AV communication session controls such that high quality video / image information may be advantageously obtained by the clinician, which can help better evaluate the patient’s response(s) to the applied / modified therapy settings and / or the clinician's verbal, textual, and / or visual requests to perform certain tasks as part of remote monitoring by the clinician. Accordingly, in some example embodiments, the clinician device may be provided with one or more non-transitory tangible computer-readable media or modules having program code stored thereon for execution on the clinician device as part of or in association with a clinician programmer application for facilitating remote therapy and telehealth delivery in an integrated session having a common application interface that effectuates an optimized GUI display within the form factor constraints of the device. In one arrangement, a code portion may be provided for displaying a virtual waiting room identifying one or more patients, each having at least one IMD / NIMI device configured to facilitate a therapy, wherein the virtual waiting room is operative to accept input by the clinician to select a patient to engage in a remote care session with a patient controller device of the selected patient. A code portion may be provided for displaying one or more audio controls and one or more video controls for facilitating an AV communication session associated with the remote care session after the remote care session is established between the patient controller device and the clinician programmer device. Various AV session controls may be represented as suitable icons, pictograms, etc. as part of a GUI display of the at the clinician programmer device, roughly similar to the GUI presentation at a patient controller device as will be set forth below. Further, example video controls may be configured to effectuate a first display window (i.e. , a clinician image(32736-2156)window) and a second display window (i.e., a patient image window) on the GUI display for respectively presenting an image of the clinician and an image of the patient. A code portion may be provided for displaying one or more remote care therapy session and setting controls, wherein the one or more remote care therapy setting controls are operative to facilitate one or more adjustments with respect to the patient's IMD settings in order to provide appropriate therapy to the patient as part of a remote therapy component of the remote care session. Preferably, the code portion may be configured to provide the AV communication session controls as well as the remote care therapy session and setting controls in a consolidated manner so as to facilitate the display thereof in a minimized overlay panel presented on the GUI screen while maximizing the second display window such that an enlarged presentation of the patient's image is effectuated during the remote care session. In some embodiments, the remote care therapy setting controls may be configured to expand into additional graphical controls for further refining one or more IMD settings depending on the implementation and / or type(s) of therapy applications the clinician programmer device is configured with. For example, such remote care therapy setting controls may comprise icons or pictograms corresponding to, without limitation, one or more of a pulse amplitude setting control, a pulse width setting control, a pulse frequency setting control, a pulse delay control, a pulse repetition parameter setting control, a biphasic pulse selection control, a monophasic pulse section control, a tonic stimulation selection control, a burst stimulation selection control, a lead selection control, an electrode selection control, and a "Stop Stimulation" control, etc., at least some of which may be presented in a set of hierarchical or nested pull-down menus or display windows. In still further embodiments, a code portion may be provided for displaying one or more data labeling buttons as part of the GUI display of the clinician programmer device, similar to the GUI embodiments of the patient controller device described above, wherein the one or more data labeling buttons are operative to accept input by the clinician corresponding to a subjective characterization of AV quality, therapy response capture, and other aspects of therapy programming during the remote care session.(32736-2156)

[0157] GUI screen 500A depicted in FIG. 5A is representative of a "login" screen that may be presented at the clinician device upon launching the clinician programmer application for facilitating a clinician to select a service mode, e.g., a remote care service mode or an in-office care service mode. A "Patient Room" selector menu option 502 may be operative to present a "generator" window 505 that includes an "In-Office" patient option 506 or a "Remote" patient option 508, wherein the activation or selection of the Remote patient option 508 effectuates one or more windows or dialog boxes for facilitating user login, registration, authentication / authorization and other security credentialing services, as exemplified by windows 510A, 510B. Upon validation, the clinician may be presented with a virtual waiting room 518 identifying one or more remote patients as exemplified in GUI screen 500B of FIG. 5B. Each remote patient may be identified by one or more identifiers and / or indicia, including, without limitation, personal identifiers, respective IMD identifiers, therapy identifiers, etc., subject to applicable privacy and healthcare laws, statutes, regulations, and the like.Accordingly, in some embodiments such identification indicia may comprise, inter alia, patient names, images, thumbnail photos, IMD serial numbers, etc., collectively referred to as Patient ID (PID) information, as illustrated by PID-1 520-1 and PID-2520-2. In some embodiments, a time indicator may be associated with each remote patient, indicating how long a remote patient has been "waiting" (e.g., the time elapsed since launching a remote care session from his / her controller device). In some embodiments, a priority indicator may also be associated with remote patients, wherein different priorities may be assigned by an intervening human and / or AI / ML-based digital agent. Furthermore, patients may have different types of IMDs to effectuate different therapies and a patient may have more than one IMD in some cases. An example embodiment of virtual waiting room 518 may therefore include a display of any combination of remote patients and their respective IMDs by way of suitably distinguishable PIDs having various pieces of information, wherein the PIDs may be individually selectable by the clinician for establishing a remote care session that may include remote therapy programming or just telehealth consultations.(32736-2156)

[0158] FIG. 6 depicts a representation of an example user interface of a clinician programmer device with additional details for facilitating graphic controls with respect to an AV communication session and a remote therapy session in an integrated remote care service application for purposes of some embodiments of the present disclosure. As illustrated, GUI screen 600 is representative of a display screen that may be presented at the clinician device after establishing that remote therapy programming is to be effectuated for a selected remote patient. In accordance with some of the embodiments set forth herein, GUI screen 600 may be arranged so that the patient's video image is presented in an optimized or resized / oversized display window 602 while the clinician's video image is presented in a smaller display window 604 along with a compact control icon panel 606 to maximize the level of detail / resolution obtained in the patient's image.Furthermore, the smaller clinician image window 604 may be moved around the Ul screen by "dragging" the image around the viewing area of the patient's image window 602 to allow more control of the positioning of the display windows so that the patient's image view is unimpeded and / or optimized at a highest possible resolution. It will be appreciated that such high level video quality is particularly advantageous in obtaining more reliable cues with respect to the patient's facial expressions, moods, gestures, eye / iris movements, lip movements, hand movements tremors, jerks, twitches, spasms, contractions, or gait, etc., that may be useful in diagnosing various types of motor / neurological disorders, e.g., Parkinson's disease. As will be seen further below, a remote data logging platform may be configured to store the AV data of the sessions for facilitating model building and training by appropriate AI / ML-based expert systems or digital assistants for purposes of further embodiments of the present patent disclosure.

[0159] Control panel window 606 may include a sub-panel of icons for AV and / or remote care session controls, e.g., as exemplified by sub-panel 607 in addition to a plurality of icons representing remote therapy setting controls, e.g., pulse amplitude control 608, pulse width control 610, pulse frequency control 612, increment / decrement control 614 that may be used in conjunction with one or more therapy setting controls, along with a lead selection indication icon 619. Skilled artisans will recognize that the exact manner in which a control panel window may be arranged as part of a consolidated GUI display depends on the therapy(32736-2156)application, IMD deployment (e.g., the number of leads, electrodes per lead, electrode configuration, etc.), and the like, as well as the particular therapy settings. Additional control icons relating to stimulation session control, e.g., Stop Stimulation icon 609, as well as any other icons relating to the remote care session such as lead / electrode selection 613, may be presented as minimized sub-panels adjacent to the control panel window 606 so as not to compromise the display area associated with the patient' image display 602.

[0160] In some embodiments, a code portion may be provided as part of the clinician programmer application to effectuate the transitioning of GUI screen 600 to or from a different sizing (e.g., resizing) in order to facilitate more expanded, icon-rich GUI screen in a different display mode. For example, a client device GUI screen may be configured such that the clinician's and patient's video images are presented in smaller windows, respectively, with most of the rest of the display region being populated by various icons, windows, pull-down menus, dialog boxes, etc., for presenting available programming options, lead selection options, therapy setting options, electrode selection options, and the like, in a more elaborate manner. In some embodiments, the video Ul panels and related controls associated with clinician / patient video image windows may be moved around the GUI screen by "dragging" the images around the display area. Still further, the positioning of the video Ul panels and related controls associated with clinician / patient video image windows may be stored as a user preference for a future Ul setup or configuration that can be instantiated or initialized when the controller application is launched. As can be appreciated, it is contemplated that a clinician device may be configured to be able to toggle between multiple GUI display modes by pressing or otherwise activating zoom / collapse buttons that may be provided on respective screens.

[0161] In some further embodiments, a clinician device may be provided with additional functionality when utilizing or operating in the resized display GUI screen mode. By way of a suitable inputting mechanism at the clinician device, e.g., by pressing or double-tapping a particular portion of the patient's image, or by scrolling a cursor or a pointing device to a particular potion of the patient's image, etc., the clinician can remotely control the AV functionality of the patient controller device, e.g., a built-in camera or an auxiliary AV device such as AV equipment, in(32736-2156)order to zoom in on and / or pan to specific portions of the patient's body in order to obtain close-up images that can enable better diagnostic assessment by the clinician. In such embodiments, zooming or enlarging of a portion of the patient's image, e.g., eye portion, may be effectuated by either actual zooming, i.e., physical / optical zooming of the camera hardware, or by way of digital zooming (i.e., by way of image processing).

[0162] In some embodiments, both optical and digital zooming of a patient's image may be employed. In still further embodiments, the patient controller device and / or associated AV equipment may be panned and / or tilted to different portions of the patient's body to observe various motor responses and / or conditions while different programming settings may be effectuated in a remote therapy session, e.g., shaking and tremors, slowed movement or bradykinesia, balance difficulties and eventual problems standing up, stiffness in limbs, shuffling when walking, dragging one or both feet when walking, having little or no facial expressions, drooling, muscle freezing, difficulty with tasks that are repetitive in nature (like tapping fingers or clapping hands or writing), difficulty in performing everyday activities like buttoning clothes, brushing teeth, styling hair, etc.

[0163] In still further embodiments, separate remote therapy session intervention controls (e.g., pause and resume controls) may be provided in addition to stimulation start and termination controls, which may be operative independent of or in conjunction with AV communication session controls, in a manner similar to example patient controller GUI embodiments set forth hereinbelow. Still further, data labeling buttons or controls may also be provided in a separate overlay or window of GUI screen 600 (not shown in FIG. 6) to allow or otherwise enable the clinician to provide different types of data labels for the AV data and therapy settings data for purposes of some embodiments of the present patent disclosure.

[0164] FIG. 7 depicts a block diagram of a generalized external edge device operative in a digital health network architecture for purposes of some embodiments of the present disclosure. For example, depending on configuration and / or modality, external device 700 may be representative of a clinician programmer device, a patient controller device, or a delegated device operated by an agent of a patient or a clinician having subordinate levels of privilege authorization with respect to remote therapy and monitoring operations. Further,(32736-2156)external device 700 may be a COTS device or non-COTS device as previously noted. Still further, external device 700 may be a device that is controlled and managed in a centralized enterprise device management system (EDMS), also referred to as a mobile / medical device management system (MDMS), which may be associated with the manufacturer of the IMDs and associated therapy application components in some embodiments (e.g., as an intranet implementation, an extranet implementation, or internet-based cloud implementation, etc.), in order to ensure that only appropriately managed / provisioned devices and users are allowed to engage in communications with IMDs with respect to monitoring the devices and / or providing therapy to patients using approved therapy applications. Still further, external device 700 may be a device that is not controlled and managed in such a device management system. Accordingly, it will be realized that external device 700 may comprise a device that may be configured in a variety of ways depending on how its functional modality is implemented in a particular deployment.

[0165] Example external device 700 may include one or more processors 702, communication circuitry 718 and one or more memory modules 710, operative in association with one or more OS platforms 704 and one or more software applications 708-1 to 708-K depending on configuration, cumulatively referred to as software environment 706, and any other hardware / software / firmware modules, all being powered by a power supply 722, e.g., battery. Example software environment 706 and / or memory 710 may include one or more persistent memory modules comprising program code or instructions for controlling overall operations of the device, inter alia. Example OS platforms may include embedded real-time OS systems, and may be selected from, without limitation, iOS, Android, Chrome OS, Blackberry OS, Fire OS, Ubuntu, Sailfish OS, Windows, Kai OS, eCos, LynxOS, QNX, RTLinux, Symbian OS, VxWorks, Windows CE, MontaVista Linux, and the like. In some embodiments, at least a portion of the software applications may include code or program instructions operative as one or more medical / digital health applications for effectuating or facilitating one or more therapy applications, remote monitoring / testing operations, data capture and logging operations, trial therapy applications, etc. Such applications may be provided as a single integrated app having various modules(32736-2156)that may be selected and executed via suitable drop-down menus in some embodiments. However, various aspects of the edge device digital healthcare functionalities may also be provided as individual apps that may be downloaded from one or more sources such as device manufactures, third-party developers, etc. By way of illustration, application 708-1 is exemplified as digital healthcare app configured to interoperate with program code stored in memory 710 to execute various operations relative to device registration, mode selection, remote / test / trial programming, therapy selection, security applications, and provisioning, etc., as part of a device controller application.

[0166] In some embodiments of external device 700, memory modules 710 may include a non-volatile storage area or module configured to store relevant patient data, therapy settings, and the like. Memory modules 710 may further include a secure storage area 712 to store a device identifier (e.g., a serial number) of device 700 used during therapy sessions (e.g., local therapy programming or remote therapy programming). Also, memory modules 710 may include a secure storage area 714 for storing security credential information, e.g., one or more cryptographic keys or key pairs, signed digital certificates, etc. In some arrangements, such security credential information may be specifically operative in association with approved / provisioned software applications, e.g., therapy / test application 708-1, which may be obtained during provisioning. Also, a non-volatile storage area 716 may be provided for storing provisioning data, validation data, settings data, metadata etc. Communication circuitry 718 may include appropriate hardware, software and interfaces to facilitate wireless and / or wireline communications, e.g., inductive communications, wireless telemetry or M2M communications, etc. to effectuate IMD communications, as well as networked communications with cellular telephony networks, local area networks (LANs), wide area networks (WANs), packet-switched data networks, etc., based on a variety of access technologies and communication protocols, which may be controlled by the digital healthcare application 708-1 depending on implementation.

[0167] For example, application 708-1 may include code or program instructions configured to effectuate wireless telemetry and authentication with an IMD / NIMI device using a suitable M2M communication protocol stack which may(32736-2156)be mediated via virtual / digital assistant technologies in some arrangements. By way of illustration, one or more bi-directional communication links with a device may be effectuated via a wireless personal area network (WPAN) using a standard wireless protocol such as Bluetooth Low Energy (BLE), Bluetooth, Wireless USB, Zigbee, Near-Field Communications (NFC), WiFi (e.g., IEEE 802.11 suite of protocols), Infrared Wireless, and the like. In some arrangements, bi-directional communication links may also be established using magnetic induction techniques rather than radio waves, e.g., via an induction wireless mechanism. Alternatively and / or additionally, communication links may be effectuated in accordance with certain healthcare-specific communications services including, Medical Implant Communication Service (MICS), Wireless Medical Telemetry Service (MTS), Medical Device Radiocommunications Service (MDRS), Medical Data Service (MDS), etc. Accordingly, regardless of which type(s) of communication technology being used, external device 700 may be provided with one or more communication protocol stacks 744 operative with hardware, software and firmware (e.g., forming suitable communication circuitry including transceiver circuitry and antenna circuitry where necessary, which may be collectively exemplified as communication circuitry 718 as previously noted) for effectuating appropriate short-range and long-range communication links for purposes of some example embodiments herein.

[0168] External device 700 may also include appropriate audio / video controls 720 as well as suitable display(s) (e.g., touch screen), camera(s), microphone, and other user interfaces (e.g., GUIs) 742, which may be utilized for purposes of some example embodiments of the present disclosure, e.g., facilitating user input, initiating IMD / network communications, mode selection, therapy selection, etc., which may depend on the aspect(s) of a particular digital healthcare application being implemented.

[0169] FIG. 8 depicts a block diagram illustrating additional details pertaining to a patient controller device operative in a digital health network architecture for purposes of some embodiments of the present disclosure.Example patient controller device 800 may be particularly configured for securely packaging and transmitting patient data to an external entity, e.g., a clinician programmer device and / or a network entity disposed in the digital health network(32736-2156)in order to facilitate remote monitoring, AI / ML model training, and the like.Consistent with the description provided above with respect to a generalized edge device, patient controller device 800 may be provided with a patient controller application 802 configured to run in association with a suitable device hardware / software environment 850 effectuated by one or more processor and memory modules 806, one or more OS platforms 808, and one or more persistent memory modules 816 comprising program code or instructions for controlling overall operations of the device, inter alia. Example OS platforms may include a variety of embedded real-time OS systems as noted previously. In one implementation, a secure file system 810 that can only be accessed by the patient controller application 802 may be provided, wherein one or more patient data files 812 may be stored in a packaged encrypted form for secure transmission for purposes of some embodiments herein. Also, patient controller application 802 may include a therapy manager 824 operative to facilitate remote and / or nonremote therapy applications and related communications using one or more communication interfaces, e.g., interface 834 with an IPG / IMD 804 and network communications interface 836 with a network entity, as previously noted. A logging manager 830 associated with therapy manager 824 may be provided for logging data. A security manager 828 associated with therapy manager 824 may be provided for facilitating secure or trusted communications with a network entity in some embodiments. A therapy communication manager 832 may be provided for facilitating secure therapy communications between patient controller 800 and a clinician programmer (not shown in this FIG.). Therapy communication manager 832 may also be interfaced with local communication interface 834 to effectuate secure communications with the patient’s IPG / IMD 804 using a suitable short-range communications technology or protocol as noted previously.

[0170] In still further arrangements, suitable software / firmware modules 820 may be provided as part of patient controller application 802 to effectuate appropriate user interfaces and controls, e.g., A / V GUIs, in association with an audio / video manager 822 for facilitating therapy / diagnostics control, file management, and / or other input / output (I / O) functions. Additionally, patient controller 800 may include an encryption module 814 operative independently and / or in association or otherwise integrated with patient controller application 802(32736-2156)for dynamically encrypting a patient data file, e.g., on a line-by-line basis during runtime, using any known or heretofore unknown symmetric and / or asymmetric cryptography schemes, such as the Advanced Encryption Standard (AES) scheme, the Rivest-Shamir-Adleman (RSA) scheme, Elliptic Curve Cryptography (ECC), etc.

[0171] FIG. 9 depicts a block diagram illustrating additional details pertaining to a clinician programmer device operative in a digital health network architecture for purposes of some embodiments of the present disclosure. Similar to the example patient controller device 800 described above, example clinician programmer 900 may be particularly configured for facilitating secure transmission of patient data to an external entity, e.g., another clinician programmer device and / or a network entity disposed in the digital healthcare network in order to facilitate remote monitoring, AI / ML model training, and the like. A clinician programmer application 902 may be configured to run in association with a suitable device hardware / software environment 950 effectuated by one or more processor and memory modules 304, one or more OS platforms 906, and one or more persistent memory modules 914 comprising program code or instructions for controlling overall operations of the device, inter alia. As before, example OS platforms may include a variety of embedded real-time OS systems according to some embodiments. Further, a secure file system 908 may be provided in clinician programmer 900 that can only be accessed by the clinician programmer application 902, wherein one or more patient data files 310 (e.g., corresponding to one or more patients) may be stored in a packaged encrypted form, respectively, for purposes of some embodiments herein. In one implementation, clinician programmer application 902 may include a therapy manager 926 operative to facilitate remote and / or non-remote therapy applications and related communications using one or more communication interfaces, e.g., interface 924. For example, interface 924 may be configured to communicate with an IMD (not shown in this FIG.) using various short-range communication links with respect to in-person or in-clinic therapy according to some embodiments as previously noted. Likewise, example interface 924 may be configured to provide connectivity with wide-area networks for facilitating remote programming of an IMD and / or a telehealth session in some scenarios. A logging manager 928 associated with(32736-2156)therapy manager 924 may be provided for logging data for respective patients. A security manager 930 associated with therapy manager 926 may be provided for facilitating secure or trusted communications with a network entity in some embodiments. A therapy communication manager 932 may be provided for facilitating secure therapy communications between clinician programmer 900 and a patient controller (not shown in this FIG.). Suitable software / firmware modules 920 may be provided as part of clinician programmer application 902 to effectuate appropriate user interfaces and controls, e.g., A / V GUIs, in association with an audio / video manager 922 for facilitating therapy / diagnostics control, file management, and / or other I / O functions as noted previously. Further, clinician programmer 900 may include an encryption module 912 similar to that of patient controller 800, wherein the encryption module 912 is operative in association and / or otherwise integrated with clinician programmer application 902 for encrypting a patient data file, e.g., dynamically on a line-by-line basis, during runtime using suitable techniques.

[0172] FIG. 10 depicts a block diagram of an IMD and associated system that may be configured for facilitating a remote care therapy application and / or a local therapy session for purposes of an example embodiment of the present disclosure. In general, therapy system 1000 may be adapted to generate electrical pulses to stimulate spinal cord tissue, peripheral nerve tissue, deep brain tissue, DRG tissue, cortical tissue, cardiac tissue, digestive tissue, pelvic floor tissue, or any other suitable biological tissue of interest within a patients body, using an IMD ora trial IMD as previously noted. In one example embodiment, IMD 1002 may be implemented as having a metallic housing or can that encloses a controller / processing block or module 1012, pulse generating circuitry including or associated with one or more stimulation engines 1010, a charging coil 1016, a power supply or battery 1018, a far-field and / or near field communication block or module 1024, battery charging circuitry 1022, switching circuitry 1020, sensing circuitry 1026, a memory module 1014, and the like. IMD 1002 may include a diagnostic circuit module associated with a sensing module 1026 adapted to effectuate various diagnostics with respect to the state / condition of one or more stimulation electrodes and sensing electrodes of an implantable lead system as well as other bio / physiological sensors integrated or otherwise operative with IMD(32736-2156)1002. Control ler / processor module 1012 typically includes a microcontroller or other suitable processor for controlling the various other components of IMD 1302. Software / firmware code, including digital healthcare application and encryption functionality, may be stored in memory 1014 of IMD 1002, and / or may be integrated with controller / processor module 1012. Other application-specific software code as well as associated storage components (not particularly shown in this FIG.) for execution by the microcontroller or processor 1012 and / or other programmable logic blocks may be provided to control the various components of the device for purposes of an embodiment of the present patent disclosure. As such, example IMD 1002 may be adapted to generate stimulation pulses according to known or heretofore known stimulation settings, programs, etc.

[0173] In one arrangement, IMD 1002 may be coupled (via a “header” as is known in the art, not shown in this FIG.) to a lead system having a lead connector 1008 for coupling a first component 1006A emanating from IMD 1002 with a second component 1006B that includes a plurality of electrodes 1004-1 to 1004-N, which may be positioned proximate to the patient tissue. Although a single lead system 1006A / 1006B is exemplified, it should be appreciated that an example lead system may include more than one lead, each having a respective number of electrodes for providing therapy according to configurable settings. For example, a therapy program may include one or more lead / electrode selection settings, one or more sets of stimulation parameters corresponding to different lead / electrode combinations, respectively, such as pulse amplitude, stimulation level, pulse width, pulse frequency or inter-pulse period, pulse repetition parameter (e.g., number of times for a given pulse to be repeated for respective stimulation sets or “stimsets” during the execution of a program), etc. Additional therapy settings data may comprise electrode configuration data for delivery of electrical pulses (e.g., as cathodic nodes, anodic nodes, or configured as inactive nodes, etc.), stimulation pattern identification (e.g., tonic stimulation, burst stimulation, noise stimulation, biphasic stimulation, monophasic stimulation, and / or the like), etc. Still further, therapy programming data may be accompanied with respective metadata and / or any other relevant data or indicia.(32736-2156)

[0174] As noted previously, external device 1030 may be deployed for use with IMD 1002 for therapy application, management and monitoring purposes, e.g., either as a patient controller device ora clinician programmer device. In general, electrical pulses are generated by the pulse generating circuitry 1010 under the control of processing block 1012, and are provided to the switching circuitry 1020 that is operative to selectively connect to electrical outputs of IMD 1002, wherein one or more stimulation electrodes 1004-1 to 1004-N per each lead 1006A / B may be energized according to a therapy protocol, e.g., by the patient or patient’s agent (via a local session) and / or a clinician (via a local or remote session) using corresponding external device 1030. Also, external device 1030 may be implemented to charge / recharge the battery 1018 of IPG / IMD 1002 (although a separate recharging device could alternatively be employed), to access memory 1012 / 1014, and / orto program or reprogram IMD 1002 with respect to one or more stimulation set parameters including pulsing specifications while implanted within the patient. In alternative embodiments, however, separate programmer devices may be employed for charging and / or programming the IMD device 1002 device and / or any programmable components thereof. Software stored within a non-transitory memory of the external device 1030 may be executed by a processor to control the various operations of the external device 1030, including facilitating encryption of patient data logged in or by IMD 1002 and extracted therefrom. A connector or "wand" 1034 may be electrically coupled to the external device 430 through suitable electrical connectors (not specifically shown), which may be electrically connected to a telemetry component 1032 (e.g., inductor coil, RF transceiver, etc.) at the distal end of wand 1034 through respective communication links that allow bi-directional communication with IMD 1002. Alternatively, there may be no separate or additional external communication / telemetry components provided with external device 1030 in an example embodiment that uses BLE or the like for facilitating bi-directional communications with IMD 1002.

[0175] In a setting involving in-clinic or in-person operations, a user (e.g., a doctor, a medical technician, or the patient) may initiate communication with IMD 1002. External device 1030 preferably provides one or more user interfaces 1036 (e.g., touch screen, keyboard, mouse, buttons, scroll wheels or rollers, or the like), allowing the user to operate IMD 1002. External device 1030 may be controlled by(32736-2156)the user through user interface 1036, allowing the user to interact with IMD 1002, whereby operations involving therapy application / programming, coordination of patient data security including encryption, trial IMD data report processing, etc., may be effectuated.

[0176] FIGS. 11 A-11 H depict representations of an example user interface and associated dialog boxes or windows provided with a patient controller device for selecting different therapy applications and / or service modes and for facilitating controls with respect to an AV communication session as well as a remote therapy session in an integrated remote care service application for purposes of an embodiment of the present disclosure. In some example implementations, a patient controller device, e.g. device 150 shown in FIG. 1A, may be provided with one or more non-transitory tangible computer-readable media or modules having program code stored thereon for execution on the patient controller device as part of or in association with a patient controller application, e.g., application 152, for facilitating remote therapy and telehealth applications in an integrated session having a common application interface. A code portion may be provided for displaying a mode selector icon on a GUI display screen of the patient controller device, wherein the mode selector icon is operative for accepting input by the patient to launch a remote care session with a clinician having a clinician programmer device. A code portion may be provided for displaying one or more audio controls and one or more video controls for facilitating an AV communication session or channel associated with the remote care session after the remote care session is established between the patient controller device and the clinician programmer device. Such AV controls may be represented as suitable icons, pictograms, and the like, e.g., a video / camera icon for controlling a video channel, a microphone icon for controlling an audio channel, a speaker icon for volume control, as well as control icons operative with respect to picture-in-picture (PIP) display regions, and the like. For example, video controls may be operative to effectuate a first display window and a second display window on the GUI display for respectively presenting an image of the clinician and an image of the patient in a PI P display mode. Yet another code portion may be provided for displaying one or more remote care therapy session controls in an overlay panel presented on the GUI display, wherein the one or more remote care therapy session controls are(32736-2156)operative with respect to starting and ending a remote care therapy session by the patient as well as facilitating a temporary intervention or interruption of the therapy session while the AV communication session is maintained. As noted above, an example remote care therapy session may involve providing one or more programming instructions to the patient's IMD as part of the remote care session, and temporary intervention of the remote therapy may only suspend the remote programming of the patient’s IMD although the AV communication session between the patient and the clinician remains active. In further embodiments, one or more code portions may be provided with the patient controller application to effectuate tactile controls with respect to different portions, fields, regions or X-Y coordinates of an active GUI display window that may be configured to interact with the functionality of the AV controls and / or therapy session controls. In still further embodiments, one or more code portions may be provided with the patient controller application to effectuate one or more data labeling buttons, icons, pictograms, etc., as part of the GUI display of the patient controller device, wherein the one or more data labeling buttons are operative to accept input by the patient corresponding to a subjective characterization of audio and / or video quality of the AV communications and / or other aspects of the therapy by the patient during the remote care session. In still further embodiments, one or more code portions may be provided with the patient controller application to facilitate patient input / feedback with respect to a trial therapy or treatment involving an IMD or a NIMI device, which may be augmented with one or more data labeling buttons, icons, pictograms, etc., wherein the patient input / feedback data may be provided to a network-based AI / ML model for facilitating intelligent decision-making with respect to whether the IMD / NIMI device should be deployed in a more permanent manner (e.g., implantation) and / or whether a particular therapy setting or a set of settings, including context-sensitive therapy program selection, may need to be optimized or otherwise reconfigured.

[0177] As illustrated, FIGS. 11 A and 11 B depict example GUI screens 1100A and 1100B of a patient controller device that allow user input with respect to various mode settings / selections, including the activation and deactivation of allowing a remote control programming (i.e., therapy) session to be conducted, e. g., in trial therapy mode, etc. GUI display screen 1100A includes a mode selector(32736-2156)1102 that may be activated to show various mode settings which in turn may be selected, enabled or otherwise activated by using associated tactile controls. For example, modes such as “Airplane Ready” 1104A, “Surgery Mode” 1106A, “MRI Mode” 1108A, “Remote Control Mode” 1110A, “Trial Mode” 1112A, each having a corresponding swipe button 1104B-1112B are depicted. GUI screen 1100B illustrates a display that may be effectuated upon selecting or allowing Remote Control 1120 wherein a Remote Care Mode 1122A may be selected or enabled for activating remote therapy using a corresponding swipe button 1122B. A patient may therefore selectively permit the activation of remote therapy (i.e. , remote programming of the IMD), whereby if activated and connected, a clinician can securely change or modify the therapy settings of the patient’s IMD by effectuating appropriate therapy setting controls and associated GUIs provided at a controller device as previously set forth.

[0178] FIG. 11 C depicts an example GUI display screen 1100C of the patient controller device during a remote care session, wherein an image of the selected clinician 1142 and an image of the patient 1144 may be presented in a PIP display region. In one display mode, the patient’s image 1144 may be presented as a smaller offset or overlay image and the clinician’s image 1142 may be presented as a main, larger image. In some embodiments, the patient image window 1144 may be moved around the Ul screen by "dragging" the image around the viewing window allocated to the clinician image 1142. An image swap control 1146 may be provided to swap the PIP display regions in another display mode, whereby the patient’s image 1144 may be presented as the main, larger image whereas the clinician’s image 1142 may be presented in a smaller overlay window.

[0179] In some embodiments, a control panel 1140 may also be presented as part of the GUI screen 1100C, wherein various AV communication session controls and remote therapy session controls may be displayed as suitable icons, pictograms, etc., in a consolidated GUI display as noted above. A video session icon 1130 may be activated / enabled or deactivated / disabled to selectively turn on or off the video channel of the session. A microphone icon 1134 may be activated / enabled or deactivated / disabled to selectively turn on or off the audio channel of the session. A pause / resume icon 1132 may be activated / enabled or deactivated / disabled to selectively pause or suspend, or resume the remote(32736-2156)therapy session involving remote programming of the patient’s IMD or any other remote digital healthcare application executing on the patient controller. In some implementations, activating or deactivating the video session icon 1130 may also be configured to turn on or off the remote therapy session. In some implementations, separate remote therapy session controls (e.g., start control, end control, etc. in addition to pause and resume controls) may be provided that are operative independent of the AV communication session controls. Still further, data labeling buttons may also be provided in a separate overlay or window of the GUI screen 1100C (not shown in this FIG.) to allow or otherwise enable the patent to input a subjective characterization of the AV data and therapy experience data as noted previously.

[0180] In a further embodiment of a digital health network architecture of the present patent disclosure, a digital health “app” may be installed on or downloaded to a patient controller device, e.g., patient controller device 1210 shown in FIG. 12, to permit a patient to report therapy outcomes for clinician review and analysis. An example of an existing digital health app that is available to patients in certain jurisdictions for reporting therapy outcomes to neurostimulation (including spinal cord stimulation) is the MYPATH TM app from Abbott (Plano, TX). The digital health app may use the network communications capabilities of patient controller device 1210 to communicate patient-reported data to patient report processing platform 1218. The clinician of a given patient may review the patient-reported data stored on platform 1218 to determine whether the therapy is working as expected and whether the patient requires reprogramming to optimize therapy. In some cases, the patient may provide the patient-reported data during a “trial” period which is used to evaluate the effectiveness of the therapy for the patient from a temporary external system before surgical implantation of the IMD. If the trial is successful, surgical implantation of the IMD may occur. Additionally or alternatively, the patient may provide the outcome data after surgical implantation of the IMD to allow monitoring of the patient’s condition and response to the therapy to continue on an ongoing basis.

[0181] Turning to FIG. 11 D, depicted therein is an example GUI display screen 1100D of a patient controller device for facilitating reporting of patient outcomes (either during an initial trial period to evaluate therapy or during long-(32736-2156)term use of the therapy). The GUI display for reporting patient outcomes may be provided as a single app or may be integrated with other digital health apps (such as a remote programming / virtual clinic app and / or the patient controller app for controlling operations of the IMD or minimally invasive device). Example GUI display screen 1100F shown in FIG. 11 F is illustrative of a menu panel for use by the patient to input data related to the patient’s condition related to the therapy provided by the patient’s IMD or minimally invasive device. By way of example, a Pain Location Selection menu 1160, a Baseline Pain Level menu 1162, a Baseline Well-being menu 1164, a Baseline Survey menu 1166 and a Preferences and Reminders menu 1168 are illustrated. In one arrangement, Pain Location Selection menu 1160 is operative to enable the patient to select / identify one or more locations on the patient's body where pain is located or perceived, e.g., left foot, right foot, ankle(s), knee(s), leg(s), pelvis, groin, hip, abdomen, upper / lower back, hand(s), arm(s), etc.

[0182] In some example arrangements, baseline data regarding pain levels (e.g., as a whole and / or for identified bodily regions), sense of well-being, measurements of physiologic and behavioral markers may be established for the patients, wherein each patient may select a varying trial period, e.g., each day, each week, 2 weeks, etc. Patients may answer a plurality of questions with respect to each baseline, wherein the answers may be alphanumeric input (e.g., on a scale of 0 to 10), graphic input, orA / V input, or any combination thereof (as shown in GUI 1100E and GUI 1100H in FIGS. 11 E and 11 H respectively as examples). One or more questionnaires 1170, 1172 may be provided as part of a GUI display screen 1100G for purposes of obtaining patient input(s), as exemplified in FIG. 11G, at least some of which may be presented in a set of hierarchical or nested pull-down menus or dialog boxes.

[0183] In some example arrangements, various pieces of data and information from the end points disposed in a digital healthcare network architecture, e.g., architecture 1260 shown in FIG. 12, may also be transmitted to one or more cloud-centric platforms without end user involvement, e.g., as background data collection processes, in addition to user-initiated secure data transfer operations.(32736-2156)

[0184] As previously noted, one or more remote data logging platforms 1216 of system 1200 (shown in FIG. 12) may be configured to obtain, receive or otherwise retrieve data from patient controller devices, clinician programmer devices and other authorized third-party devices. On an individual patient level and on a patient population basis, patient aggregate data 1250 is available for processing, analysis, and review to optimize patient outcomes for individual patients, fora patient population as a whole, and for relevant patient subpopulations of patients.

[0185] Patient aggregate data (PAD) 1250 may include basic patient data including patient name, age, and demographic information, etc. PAD 1250 may also include information typically contained in a patient’s medical file such as medical history, diagnosis, results from medical testing, medical images, etc. The data may be inputted directly into system 1200 by a clinician or medical professional. Alternatively, this data may be imported from digital health records of patients from one or more health care providers or institutions.

[0186] As previously discussed, a patient may employ a patient controller “app” on the patient’s smartphone or other electronic device to control the operations of the patient’s IMD or minimally invasive device. For example, for spinal cord stimulation or dorsal root stimulation, the patient may use the patient controller app to turn the therapy on and off, switch between therapy programs, and / or adjust stimulation amplitude, frequency, pulse width, and / or duty cycle, among other operations. The patient controller app is adapted to log such events (“Device Use / Events Data”) and communicate the events to system 1200 to maintain a therapy history for the patient for review by the patient’s cl inician(s) to evaluate and / or optimize the patient’s therapy as appropriate.

[0187] PAD 1250 may include “Patient Self-Report Data” obtained using a digital health care app operating on patient controller devices 1210. The patient self-report data may include patient reported levels of pain, patient well-being scores, emotional states, activity levels, and / or any other relevant patient reported information. The data may be obtained using the MYPATH app from Abbott as one example.(32736-2156)

[0188] PAD 1250 may include sensor data. For example, IM Ds of patients may include integrated sensors that sense or detect physiological activity or other patient states. Example sensor data from IMDs may include dated related to evoked compound action potentials (ECAPs), local field potentials, EEG activity, patient heart rate or other cardiac activity, patient respiratory activity, metabolic activity, blood glucose levels, and / or any other suitable physiological activity. The integrated sensors may include position sensing circuits and / or accelerometers to monitor physical activity of the patient. Data captured using such sensors can be communicated from the medical devices to patient controller devices and then stored within patient / clinician data logging and monitoring platform 1216. Patients may also possess wearable devices (see, e.g., device 106 in FIG. 1 B) such as health monitoring products (heart rate monitors, fitness tracking devices, smartwatches, etc.). Any data available from wearable devices may be likewise communicated to monitoring platform 1216.

[0189] As previously discussed, patients may interact with clinicians using remote programming / virtual clinic capabilities of system 1200. The video data captured during virtual clinic and / or remote programming sessions may be archived by platform 1214. The video from these sessions may be subjected to automated video analysis (contemporaneously with the sessions or afterwards) to extract relevant patient metrics. PAD data 1250 may include video analytic data for individual patients, patient sub-populations, and the overall patient population for each supported therapy.

[0190] The data may comprise various data logs that capture patientclinician interactions (“Remote Programming Event Data” in PAD 1250), e.g., individual patients’ therapy / program settings data in virtual clinic and / or in-clinic settings, patients’ interactions with remote learning resources, physiological / behavioral data, daily activity data, and the like. Clinicians may include clinician reported information such as patient evaluations, diagnoses, etc. in PAD 1250 via platform 1216 in some embodiments. Depending on implementation, the data may be transmitted to the network entities via push mechanisms, pull mechanisms, hybrid push / pull mechanisms, event-driven or trigger-based data transfer operations, and the like.(32736-2156)

[0191] In some example arrangements, data obtained via remote monitoring, background process(es), baseline queries and / or user-initiated data transfer mechanisms may be (pre)processed or otherwise conditioned in order to generate appropriate datasets that may be used fortraining, validating and testing one or more AI / ML-based models or engines for purposes of some embodiments. In some example embodiments, patient input data may be securely transmitted to the cloud-centric digital healthcare infrastructure wherein appropriate AI / ML-based modeling techniques may be executed for evaluating the progress of the therapy trial, predicting efficacy outcomes, providing / recommending updated settings, etc.

[0192] In one implementation, “Big Data” analytics may be employed as part of a data analytics platform, e.g., platform 1220, of a cloud-centric digital health infrastructure 1212. In the context of an example implementation of the digital health infrastructure 1212, “Big Data” may be used as a term fora collection of datasets so large and complex that it becomes virtually impossible to process using conventional database management tools or traditional data processing applications. Challenges involving “Big Data” may include capture, curation, storage, search, sharing, transfer, analysis, and visualization, etc. Because “Big Data” available with respect to patients’ health data, physiological / behavioral data, sensor data gathered from patients and respective ambient surroundings, daily activity data, therapy settings data, health data collected from clinicians, etc. can be on the order of several terabytes to petabytes to exabytes or more, it becomes exceedingly difficult to work with using most relational database management systems for optimizing, ranking and indexing search results in typical environments. Accordingly, example AI / ML processes may be implemented in a "massively parallel processing” (MPP) architecture with software running on tens, hundreds, or even thousands of servers. It should be understood that what is considered "Big Data" may vary depending on the capabilities of the datacenter organization or service provider managing the databases, and on the capabilities of the applications that are traditionally used to process and analyze the dataset(s) for optimizing ML model reliability. In one example implementation, databases may be implemented in an open-source software framework such as, e.g., Apache Hadoop, that is optimized for storage and large-scale processing of datasets on clusters of commodity hardware. In a Hadoop-based implementation, the software(32736-2156)framework may comprise a common set of libraries and utilities needed by other modules, a distributed file system (DFS) that stores data on commodity machines configured to provide a high aggregate bandwidth across the cluster, a resourcemanagement platform responsible for managing compute resources in the clusters and using them for scheduling of AI / ML model execution, and a MapReduce-based programming model for large scale data processing.

[0193] In one implementation, data analytics platform 1220 may be configured to effectuate various AI / ML-based models or decision engines for purposes of some example embodiments of the present patent disclosure that may involve techniques such as support vector machines (SVMs) or support vector networks (SVNs), pattern recognition, fuzzy logic, neural networks (e.g., ANNs / CNNs), recurrent learning, and the like, as well as unsupervised learning techniques involving untagged data. For example, an SVM / SVN may be provided as a supervised learning model with associated learning algorithms that analyze data and recognize patterns that may be used for multivariate classification, cluster analysis, regression analysis, and similar techniques. Given example training datasets (e.g., a training dataset developed from a preprocessed database or imported from some other previously developed databases), each marked as belonging to one or more categories, an SVM / SVN training methodology may be configured to build a model that assigns new examples into one category or another, making it a non-probabilistic binary linear classifier in a binary classification scheme. An SVM model may be considered as a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible (i.e., maximal separation). New examples are then mapped into that same space and predicted to belong to a category based on which side of the gap they fall on. In addition to performing linear classification, SVMs can also be configured to perform a nonlinear classification using what may be referred to as the “kernel trick”, implicitly mapping their inputs into high-dimensional feature spaces. In a multiclass SVM, classification may typically be reduced (i.e., “decomposed”) to a plurality of multiple binary classification schemes. Typical approaches to decompose a single multiclass scheme may include, e.g., (i) one-versus-all classifications; (ii) one-(32736-2156)versus-one pair-wise classifications; (iii) directed acyclic graphs; and (iv) errorcorrecting output codes.

[0194] In some arrangements, supervised learning may comprise a type of machine leaning that involves generating a predictive model or engine based on decision trees built from a training sample to go from observations about a plurality of features or attributes and separating the members of the training sample in an optimal manner according to one or more predefined indicators. Tree models where a target variable can take a discrete set of values are referred to as classification trees, with terminal nodes or leaves representing class labels and nodal branches representing conjunctions of features that lead to the class labels. Decision trees where the target variable can take on continuous values are referred to as regression trees. In some other arrangements, an embodiment of the present patent disclosure may advantageously employ supervised learning that involves ensemble techniques where more than one decision tree (typically, a large set of decision trees) are constructed. In one variation, a boosted tree technique may be employed by incrementally building an ensemble by training each tree instance to emphasize the training instances previously mis-modeled or mis-classified. In another variation, bootstrap aggregated (i.e., "bagged") tree technique may be employed that builds multiple decision trees by repeatedly resampling training data with or without replacement of a randomly selected feature or attribute operating as a predictive classifier. Accordingly, some example embodiments of the present patent disclosure may involve a Gradient Boosted Tree (GBT) ensemble of a plurality of regression trees and / or a Random Forest (RF) ensemble of a plurality of classification trees, e g., in pain score classification and modeling.

[0195] Depending on implementation, various types of data (pre)processing operations may be effectuated with respect to the myriad pieces of raw data collected for / from the subject populations, e.g., patients, clinicians, etc., including but not limited to sub-sampling, data coding / transformation, data conversion, scaling or normalization, data labeling, and the like, prior to forming one or more appropriate datasets, which may be provided as an input to a training module, a validation / testing module, or as an input to a trained decision engine for facilitating prediction outcomes. In some arrangements, example data signal (pre)processing(32736-2156)methodologies may account for varying time resolutions of data (e.g., averaging a data signal over a predetermined timeframe, e.g., every 10 minutes, for all data variables), missing values in data signals, imbalances in data signals, etc., wherein techniques such as spline interpolation method, synthetic minority over-sampling technique (SMOTE), and the like may be implemented.

[0196] In some embodiments, sensor data, video data, and / or audio data are analyzed during a virtual clinic or remote programming session to determine rigidity of a patient. For example and referring to FIG. 13, a flowchart illustrating exemplary operations that may occur during a virtual clinic or remote programming session according to embodiments of the present disclosure is shown. It is noted that the exemplary operations shown in and described with respect to FIG. 13 may employ various types of infrastructure, such as the exemplary infrastructure shown in FIG. 12, or other infrastructure, such as suitable patient device(s), clinician device(s), one or more servers, cloud computing resources, or other types of computing devices / resources suitable for facilitating the exemplary operations described in more detail with reference to FIG. 13 below.

[0197] In the exemplary flow of FIG. 13, a patient may be instructed to conduct one or more tasks for determining characteristics of the patient's condition. In some embodiments, the operations of FIG. 13 are conducted concurrently with a virtual clinic and / or remote programming session between a patient and a clinician. In some embodiments, the operations may occur in an automated basis. For example, the patient may execute an app on the patient's device to access an “avatar” clinician which instructs the patient to perform the tasks for the patient evaluation. In a patient evaluation session, the avatar clinician provides an avatar user interface (Ul) to aid a patient on positioning the camera during the digital exam. The avatar III may provide an example of the exam (e.g., walking, finger tapping, or performance of any suitable tasks), a target box with color (e.g. about the hands, limbs, torso, head, or other feature to be digitally analyzed), and auditory feedback where to move in front of the camera (e.g., for a gait exam). The avatar III may employ kinematic Al to detect body parts and provide instructions to position for best camera viewing for the digital exam.(32736-2156)

[0198] The avatar digital feature of the patient app periodically walks the patient through standard Parkinson's (movement disorder) exam), while using kinematics, and auditory analysis to compare the patient's results from previous. The avatar patient feature schedules, notifies, and guides the patient through a Parkinson's weekly exam (or other suitable time period). Kinematics using the camera captures the patients tests as the avatar clinician in the app instructs the patient how to perform the tasks for the automated exam. The results are captured, and overtime a timeline of progression is created and available for the clinician to review. Further, the patient's clinician is notified if results of an exam fall below limits set.

[0199] Exemplary tasks may include flexion tasks, tension tasks, or other types of tasks that may be used to evaluate the rigidity of the patient. The one or more tasks may be utilized to capture information that may be utilized to determine characteristics of the patient from which the patient's condition may be diagnosed. Additionally or alternatively, the characteristics may also be utilized to configure a neurostimulation therapy to address the patient's condition. In some aspects, various types of devices may be employed to gather data during patient performance of one or more tasks. For example, the devices may include sensors (e.g., accelerometers, gyroscopes, inertial measurement units (IMlls), electro (EMG) sensors, ultrasound sensors, video, and the like). Exemplary sensors that can be employed to obtain data to assist patient evaluation based on patient movement (e.g., during performance of the one or more tasks) are described in the publication “Quantification of Hand Motor Symptoms in Parkinson's Disease: A Proof-of-Principle Study Using Inertial and Force Sensors”, Ann Biomed Eng. 2017; 45(10): 2423-2436, by Josien C. van den Noort et al., the content of which is incorporated herein by reference.

[0200] In some embodiments, a clinician may select one or more tasks for performance by the patient for the digital exam or evaluation. For example, the clinician may select the exams that are appropriate for the patient at a given time. An avatar feature in the clinician and patient apps will then guide the patient and clinician through the patient tasks for the digital exam. While the patient tasks are performed, the clinician programmer app and / or one or more server or other(32736-2156)computing platforms performs kinematics and auditory real-time analysis for display to the clinician(s).

[0201] In addition or alternative to capturing sensor data as the patient performs the one or more tasks, one or more cameras may be utilized to capture media content (e.g., image data, video data, etc.) of the patient performing the one or more tasks. In some aspects, the sensor data and media content obtained as the patient performs the one or more tasks may be utilized to diagnose the patient's condition during a virtual clinic and / or a remote programming session. In some aspects, a specific region-based analysis (auto or user-driven) may be conducted to adjust focus within a video field-of view, such as to focus the media content on a particular region of the patient's body (e.g., the hands, legs, feet, upper body, lower body, etc.). The region of interest of the patient's body may be associated with or determined based on the diagnosed condition(s). To illustrate, a region of interest for a patient diagnosed or suspected of suffering from tremors may be the patient's hand, and the region of interest for a patient diagnosed or suspected of suffering from Parkinson's disease may be the patient's feet and / or legs (e.g., to perform gait analysis, etc.). It is noted that the exemplary regions of the patient's body and conditions described above have been provided for purposes of illustration, rather than by way of limitation. Therefore, it should be understood that the diagnostic and analysis techniques disclosed herein may be readily applied to other conditions and / or regions of interest on a patient's body.

[0202] The exemplary flow shown in FIG. 13 may be used to provide realtime feedback on the patient's condition. This allows changes to be quantified in real-time and associated with neuromodulation or other therapies, such as by quantifying changes to measure features or characteristics overtime. This can be achieved through high-definition capture and comparison of multiple frames of media content (e.g., multiple sequential images or frames of video content).Additionally, multiple types of captured data may be utilized to provide multiple independent data points or to a combination of different data points for analysis / evaluation (e.g., a combination of captured data associated with vascular change data coupled with rigidity data). In aspects where media content is utilized for analysis and evaluation, multiple types of cameras may be used (e.g., media(32736-2156)content may be captured using one or more imaging cameras, video cameras, and / or thermal-based cameras, such as infrared cameras) during a session.

[0203] As shown in step 1301, the one or more tasks performed by the patient may include one or more flexion / extension tasks. As the patient performs the task(s) the system captures patient data using one or more sensors (e.g., IMU sensors, EMG sensors, ultrasound sensors, video camera, and / or the like). In some aspects, the patient data or a portion thereof may be captured by patient devices, such as the patient devices 104 described and illustrated with reference to FIG. 1B.

[0204] The one or more tasks performed by the patient may additionally or alternatively include: one or more opposite hand drawing / writing tasks, at step 1302; one or more walking task(s), at step 1303; one or more standing and / or balancing tasks, at step 1304; and one or more finger-tapping tasks, as step 1305. In a patient digital examination or evaluation, one or more suitable metrics may be provided to the clinician when a patient performs Bradykinesia finger tapping exam. The application uses kinematics to detect the patient's hand and finger, and then monitors (distance between the two fingers, and rate of tap). The application provides the clinician a real-time chart showing finger distance overtime. This chart will let the clinician see the finger tap movement, and if it slowed up over time.

[0205] Any additional tasks may be included as appropriate. For example, the patient may be instructed to conduct a grip sensing test. For example, using the capacitive touch capacity of a smartphone, the patient's app may monitor the reported area (based on force) from the smartphone's detection of user touch upon the touch screen. Pushing your thumb on the display, the app may monitor the capacitive area reported from the touch panel matrix. Alternatively, a grip sensor accessory may be employed. As the patient grips the device harder more of your hand be detected along the side of the smart device. Such sensing may be employed to detect motor functions and possibly be used to calculate a rigidity score.

[0206] As described above with reference to step 1301, various sensors and devices may be utilized to capture patient data as the patient performs the one or more tasks. For example, patient data may be captured as the patient performs the(32736-2156)tasks associated with steps 1302-1305 using finger cap accessories, IMU sensors, EMG electrode(s), EEG electrodes, gyroscopes, accelerometers, cameras, temperature sensors, heart rate sensors, blood pressure sensors, other types of sensors, or combinations thereof (e.g., a combination involving finger cap accessories, IMU sensors, EMG electrode(s), a combination involving finger cap accessories, IMU sensors, EMG and EEG electrode(s), video camera, and so on).

[0207] In step 1306, a set of rigidity related metrics are determined. In some aspects, the rigidity related metrics may be determined by one or more servers, such as a server of the virtual clinic / remote programming platform 1214 of FIG. 12 . Each metric may be obtained by processing respective sets of patient data captured during performance of one or more of the various patient tasks described above. In some embodiments, the one or more servers may evaluate the patient data to determine characteristics associated with the patient's performance of the task(s), such as a minimum jerk trajectory, a smoothness of motion, a cocontraction profile, a gait profile, an arm-swing profile, a balance / sway, a range of motion, completeness of movement, or other characteristics, and the metrics may be determined based on the characteristics.

[0208] While quantifying metrics around or based on patient movement, stiffness measures or metrics may be calculated. For example, metrics such as the minimum jerk trajectory may be used to compute the smoothness of the movement. In a healthy patient, the movement trajectory has a smooth bell-shaped curve with minimal jerk, which may be calculated based on the time derivative of acceleration (e.g., based on accelerometer data). When a patient has tremor or rigidity, it is expected that the movement trajectory deviates from the minimum jerk trajectory (e.g., a jerk trajectory of a healthy patient) and the acceleration may be unsteady. As another non-limiting example, IMU sensors with enough sampling rate can detect the smoothness of motion based on high frame per second (FPS) video recordings. In addition, in a healthy subject, at the initiation of and during the movement, the co-contraction of the agonist and antagonist muscles decreases. However, in a patient with rigidity, the co-contraction levels are higher at the baseline and the decrease of the co-contraction may not be large enough at the initiation of movement. Muscle tone recordings, such as may be observed using(32736-2156)EMG signals / sensors, of agonist and antagonist muscles may be used along other sensors for the analysis of the co-contraction profile during movement.

[0209] Other measures, such as comparison of movement metrics between the right and left sides (ex: frequency of steps, step interval, or length of arm swing) can be used as a proxy for evaluating or quantifying additional aspects of the patient's condition, such as asymmetry. Metrics associated with balance can be determined by quantifying sway when the patient is performing the one or more tasks (e.g., the standing / balance task(s) at step 1304). Additionally, range-of-motion and completeness of movement can be detected when the patient is performing the one or more tasks (e.g., the flexion / extension and / or the walking tasks, at steps 1301 and 1303, respectively). Identification of any inability for fullextension, lack of motion in trunk, abnormal arm-swing, gait abnormalities (e.g., as detected from lack of heel strike, stooped posture, movement speed, etc.), or other types of analysis may also be performed. Other inputs may include proxies for facial expressions as rigidity may also correlate with blank affect.

[0210] At step 1307, the respective metrics are used to compute a rigidity score or other relevant patient scores or measures indicative of the patient's condition. In some embodiments, the score(s) may be calculated by one or more servers, such as servers of the virtual clinic / remote programming platform 1214 of FIG. 12 . The calculation of the metrics and / or the rigidity score may be assisted using ML / AI algorithms. In aspects, the ML / AI algorithms may be trained based on previously captured patient data (e.g., historical patient data captured and processed in accordance with the concepts disclosed herein, such as by the system 1200 of FIG. 12 ).

[0211] In some embodiments, a patient-specific method of computing a rigidity score is performed. For example, Parkinson's Disease (PD) is a heterogeneous disorder, and a score that is patient-specific may be more informative for patients and clinicians to understand individual patient progress (e.g., progression of the disease and / or improvement of the patient's symptoms as a result of treatment). Therefore, characterizing noise present in the signal data when the patient is not rigid may be performed to define a floor value or noise signature that may be used for further computations and operations. To illustrate, the noise signature may be used to characterize the rigidity profile for an individual(32736-2156)patient (e.g., individual patients may have different noise signatures). When the patient's measures are outside the noise limits, it may serve as an indication of the presence of rigidity (e.g., rigidity for the specific patient being evaluated).

[0212] The above-described techniques may enable the presence of rigidity to be transformed into a binary outcome (e.g., rigidity is present or not present on an individual patient basis) that may be further enhanced with a quantitative score computed from the signals, patient data, and metrics described above. The ML / AI algorithms may be utilized to create or provide a predictive model that integrates the signals and patient data collected from sensors and video during performance of the one or more tasks by the patient. The model may output a value that provides an objective or sensor-driven score. A subjective or evaluator-driven score can also be determined based on clinician assessment and patient-reported outcomes (PROs). The PROs can also be patient-specific and enhanced by the use of chatbots to ensure that all information is being collected without an undue burden on patients. The combined objective and subjective scores may represent the overall rigidity score for the patient and provide a holistic assessment of the patient's symptoms and progress.

[0213] In some embodiments, other relevant patient scores may be provided. For example, tremor and / or bradykinesia scores may be provided.Alternative, a “pain score” may be provided to the clinician that is related to a computed level of pain of the patient. The patient score(s) may include the clinician a real-time chart showing finger distance over time (for the finger tapping task). This chart will let the clinician see the finger tap movement, and if it slowed up over time.

[0214] At step 1308, the patient score is provided to the clinician. In some embodiments, the rigidity score may be provided to the clinician during a virtual clinic / remote programming session, such as may be performed using the system 1200 of FIG. 12 . At step 1309, the patient's neurostimulation parameters (e.g., deep brain stimulation parameters, spinal cord stimulation parameters, and / or the like) may be modified and used to update the patient's neurostimulation system. As non-limiting examples, stimulation parameters may include amplitude, frequency, pulse width, and the like. It is noted that in some aspects the scores calculated based on the ML / AI algorithms or model(s) may be utilized to automatically adjust(32736-2156)the patient's neurostimulation parameters, rather than providing the score(s) to a clinician and having the clinician modify the neurostimulation parameters.

[0215] Referring to FIG. 14 , a flow diagram depicting an exemplary flow of operations for creating a predictive disorder model using ML / AI processing according to some embodiments is shown. Additionally, FIG. 15 depicts ML / AI learning data sets 1500 that may be used for creating a predictive disorder model, such as a model created according to the exemplary operations of FIG. 14 . It is noted that the operations illustrated and described with reference to FIG. 14 may be utilized with data (e.g., the data captured using the flow of FIG. 13 ) captured using a virtual clinic / remote programming system (e.g., the virtual clinic / remote programming platform 1214 of FIG. 12 or other systems).

[0216] At step 1401, data from neurostimulation patients is captured. In some aspects, the captured data may include data captured from patients in a “stimulation-off” state. Patients may be considered in the “stimulation-off” state when the neurostimulation systems of the respective patients are not providing electrical stimulation to target neural sites. Referring briefly to FIG. 15 , in some aspects, the captured data may include “context” data and “sensor” data. The context data may include various types of data, such as geographic data, ethnicity data, disease stage data, existing prescriptions (if any) for each patient, neurostimulation therapy settings for each patient, timestamps (e.g., time of day when therapy was provided, etc.), and macro-modality data. It is noted that the exemplary types of context data shown in FIG. 15 are provided for purposes of illustration, rather than by way of limitation and that the context data may include other types of data or combinations thereof. The sensor data, which may be captured using a variety of sensors or devices, such as peripheral sensors, implanted sensors, or other types of sensors, as described with reference to FIG.13 . The sensor data may also include image content, video content, audio data content (e.g., captured using camera and microphone components of user devices or other types of devices). It is noted that the sensor data may include data captured using any suitable sensor, device, or component, and that peripheral and implanted sensors are shown in FIG. 15 for purposes of illustration, rather than by way of limitation. For example, the sensor data may also include data from(32736-2156)wearable devices, such as biometric or other types of data captured by a “smart” watch, a health monitoring device, or other types of consumer electronic devices.

[0217] Referring back to FIG. 14 , at step 1402, the captured data while the patients are in the “stimulation-off” may be recorded or stored to one or more databases, such as one or more databases of a server platform (e.g., the virtual clinic / remote programming platform 1214 of FIG. 12 ). At step 1403, the recorded data is labeled. To illustrate, the data captured while the patient is in the “stimulation-off” state may be labeled with information that indicates the data is representative of a disorder state (e.g., patient biometrics, measurements, and the like were captured in the absence of neurostimulation therapy). The labeled data may be utilized to train the ML / AI algorithms to recognize the state of a patient's condition or disorder.

[0218] Similarly to steps 14001-1403, data is captured, at step 1404, from patients in a “stimulation-on” state (e.g., while the patients are receiving neurostimulation therapy). It is noted that the receiving of neurostimulation therapy may be continuous (e.g., stimulation pulses may be delivered to target tissue of the patient continuously during the data capturing at step 1404) or periodic (e.g., stimulation pulses may be delivered to target tissue of the patient for a period of time and not delivered for a period of during the data capturing at step 1404). At step 1405, the captured data, which may include image data, audio data, video data, sensor data, other types of data, or combinations thereof, is recorded to one or more databases. At step 1406, the data captured while the patient is in the “stimulation-on” state may be labeled with information that indicates the data is representative of a state of the patient's condition during providing of the neurostimulation therapy (e.g., patient biometrics, measurements, and the like were captured in the presence of the neurostimulation therapy). The labeled data may be utilized to train the ML / AI algorithms to recognize the state of a patient's condition or disorder when neurostimulation therapy is provided or the impact of the neurostimulation therapy on the patient's condition or disorder. At step 1407, the state / context data (e.g., the state / context data of FIG. 15) may be processed or conditioned (e.g., using generalized Baye conditionalization, Jeffrey conditionalization, and / or any other suitable methods). The conditioning may improve model accuracy.(32736-2156)

[0219] At step 1408, ML / AI processing is applied to at least a portion of the captured data (e.g., a portion of the data captured while the patients are in the “stimulation-off” and “stimulation-on” states) to develop a predictive disorder model. In some aspects, the predictive disorder model may be configured to classify disorders of patients (e.g., health state, disorder state, disorder level, etc.). The model(s) may utilize various classification techniques, such as random forest classification, Naive Bayes classification, k-means clustering, genetic algorithms, neural networks, reinforcement learning strategies (e.g., Q-learning, Temporal Difference learning, Markov decision processes, etc.), and / or any suitable ML / AI methods.

[0220] The predictive disorder model may include various disorder states. For example, for a movement disorder such as Parkinson's Disease, the predictive disorder model may include model components such as tremor, rigidity, facial drop, balance, hallucinations, and / or any other relevant disorder symptoms. As illustrated in FIG. 15 , the model components may also be employed to create an overall disorder classification or classification model, such as “PD level” that reflects an ensemble of the selected model components. It is noted that while FIG.15 illustrates aspects of a model for a movement disorder, such description has been provided for purposes of illustration, rather than by way of limitation and predictive models of the present disclosure may utilize additional model data for analysis of other disorders, such as chronic pain disorders.

[0221] At step 1408, the predictive model is used to evaluate patients and ascertain the condition(s) of their neurological disorders. As described above, the predictive models may be trained with datasets associated with patients in the “stimulation-off” and “stimulation-on” states. As such, the predictive models may be utilized to evaluate the condition of a patient's disorder or condition based on data captured while the neurostimulation stimulation therapy (or other type of therapy) is or is not being provided. The outputs of the predictive model may be utilized (e.g., by a clinician or automated tools) to assist with programming of neurostimulation parameters (e.g., adjust the parameters, conditions for triggering delivery of neurostimulation, etc.), augmenting virtual clinic / remote programming sessions, and / or any other suitable patient evaluation processes.(32736-2156)

[0222] As discussed herein, video data of patients may be processed to support providing neurostimulation therapies according to some embodiments. The processing of the video data may occur substantially in real-time (e.g., during a virtual clinic / remote programming session). Alternatively, the processing of video data may occur on previously recorded video. The processing of the video data may be used to assist evaluation of one or more symptoms of the patient.Additionally or alternatively, the processing of the video data may be utilized to build a ML / AI model for use in neurostimulation therapies.

[0223] Referring to FIG. 16 , a flowchart depicting exemplary operations for processing video data to support provision of neurostimulation therapies to patient according to some embodiments is shown. At step 1601 , video data of the patient is captured. The video data may be captured, for example, during a virtual clinic / remote programming session according to some embodiments. The video data may be captured from a user device, such as patient controller (e.g., patient controller 150 of FIG. 1A, patient controller devices 1210 of FIG. 12 , or other types of devices). In step 1602, the video data is processed to extract features associated with the patient. To illustrate, the extracted features may be associated with the patient's facial expression, posture, limb activity, or other anatomical considerations. In some aspects, the features may be extracted using landmark recognition software. For example, if the video analysis is conducting an analysis of facial expression, suitable software libraries include the FACE MARK facial landmark API available from OpenCV, the CLM-framework (also known as the Cambridge Face Tracker), FACE++ Web API, CLOUD VISION API (Google) as examples. When the video analysis is conducted based on posture, limb activity, or other anatomical considerations that involve a large amount of the patient's body, other landmark software packages may be employed such as MEDIAPIPE Pose (Google) and OPENPOSE library (OpenCV). MediaPipe also provides a number of software API packages for video analysis including FACE MESH and POSE packages for detecting facial landmarks and body landmarks respectively.

[0224] In step 1603, key landmark points are selected from the points or features generated during step 1602. The key landmark points may be selected to identify relevant characteristics related to neurological disorder symptoms. For example, facial expression and body posture can be related to pain and / or motor(32736-2156)symptoms of chronic neurological disorders. For example, FIGS. 17Aand 17B depict sets 1700 and 1750, respectively, of key landmark points that may be utilized for facial expression analysis and pose analysis according to some embodiments.

[0225] In step 1604, regional area metrics are calculated from the key landmark points. The regional area metrics are indicative of the area bounded by key points. For example, in FIG. 17A, areas ALEFT and ARIGHT are shown, and in FIG. 17B areas A1-A4 are shown. As discussed herein, the calculated areas may be compared to generate a representation of balance in expression and / or pose which is related to symptoms of neurological disorders. To illustrate, when a patient is comfortable and happy, the shape of the face is balanced and exhibits a relatively high degree of symmetry. However, when a patient is subjectively experiencing pain, their facial expression will often contort and lose the balance or symmetry in the shape of the face. Similarly, a loss of balance or symmetry can be detected in the pose of a patient when the patient is experiencing pain or is subject to certain symptoms of motor disorders (e.g., Parkinson's Disease). In some embodiments, the patient may be requested to perform one or more physical task (e.g., as described with reference to FIG. 13 ) while the data is captured and / or the analysis occurs. For example, the patient may be requested to walk a short distance within the view of the patient's camera. The processing and analysis of the patient's pose or posture during this task may occur to classify the patient's disorder state.

[0226] In step 1605, one or more metrics may be calculated based on the set of landmark points or features and / or regional areas. For example, the one or more metrics may include ratios, such as a balance ratio, a cross ratio, and / or other relevant ratios derived from the landmark points and / or regional areas. In some aspects, the ratios and / or metrics may be calculated for each video frame or for a relevant fraction or set of video frames. For example, one frame every 0.1 or 0.2 seconds may be selected for analysis even though the frame rate of the video signal may be higher. The selection of the frame rate analysis may be varied depending on the quality of the video signal and any other relevant factor.(32736-2156)

[0227] To illustrate, analysis of a patient's facial expression using the set of key landmark points 1700 of FIG. 17A, the two areas (Aieft and Aright) may be defined based on the respective key landmark points as shown in FIG. 17A. The balance ratio may be defined as: balance ratio = Aleft! Aright.

[0228] For analysis of facial expression using the set of key landmark points 1700, a distance (Dvertjcal) between point 10 and point 9 and a distance (DhorjzontaI) between point 4 and point 5 may be determined. Subsequently, DvertjC(Jand Dhortzontalmay be used to calculate a face ratio. The face ratio may be expressed as: face ratio = DverticalI Dhonzontal.

[0229] For analysis of facial expression using set of key landmark points 1700, a distance DcrosA) between point 11 and point 14, and a distance ( / >im) between point 12 and point 13 may be calculated and used to determine a cross ratio. The cross ratio may be defined as: Cross ratio = Dcmssl / Dcross2.

[0230] With reference to FIG. 17B, analysis of patient pose or posture using the set of key landmark points 1750, let Diett be defined as the distance between point 1 and point 7, and Dright be defined as the distance between point 2 and point 8. Using Dieftand Dright, a balance ratio may be determined. The balance ratio may be defined as: Balance ratio = Diett / Dright.

[0231] For analysis of patient pose or posture using the set of key landmark points 1750, let Dcrossi is defined as the distance between point 1 and point 8, and Dcross2 be defined as the distance between point 2 and point 7. The cross ratio may then be defined as: cross ratio = Dcrossi / Dcross2.

[0232] For analysis of patient pose or posture using set of key landmark points 1750, let Dshouider-kneei be defined as the distance between point 1 and point 5 and let Dshouider-knee2 be defined as the distance between point 2 and point 6. A shoulder-knee ratio may then be calculated based on Dshouider-kneei and DShouider-knee2 . For example, the shoulder-knee ratio may be defined as :shoulder-knee ratio = Dshouider-kneei / Dshoulder-knee2.

[0233] For analysis of patient pose or posture using set of key landmark points 1750, let Deibow-footi be defined as the distance between point 9 and point 8, and let Deibow-foot2 be defined as the distance between point 10 and point 7. An(32736-2156)elbow-foot-cross ratio may be calculated based on Deibow-footi and Deibow-foot2 . For example, the elbow-foot-cross ratio may be defined as: elbow-foot-cross ratio = Delbow-footl I Delbow-foot2.

[0234] Referring back to FIG. 16 , at step 1606, a time domain sliding window is applied to the one or more metrics (e.g., the ratio data and / or other relevant metric data). In some embodiments, the sliding window may have a duration of one second. That is, the ratio and metric data from a given one second time window is selected for aggregate analysis. However, it is noted that sliding windows having a duration longer than one second or shorter than one second may be utilized.

[0235] In step 1608, a ML / AI model, such as the predictive model described above with reference to FIGS. 14 and 15 or another model, may be applied to media content of a particular sliding window. The model may be configured to process or apply a classification the media content (e.g., to frames of video content, etc.). In an aspect, only relevant portions of the media content (e.g., relevant video frames) within the sliding time window may be classified. For example, the relevant portions may include video frames determined to exhibit (e.g., by the model) a threshold probability of being correctly classified as either depicting a healthy / normal state ora symptom state (e.g., pain).

[0236] The model may be configured to calculate a probability of relevance for each portion of the media content (e.g., a probability that a classification for a particular frame of video depicts a healthy / normal state ora symptom state), and relevant portions of the media content may be identified based on a threshold probability. In an aspect, the threshold probability used to identify relevant portions of the media content may be a probability of at least 0.8 (e.g., media content having a probability of relevance 0.8, where 0.8=threshold probability). It is noted that a threshold probability of 0.8 has been described for purposes of illustration, rather than by way of limitation and that the threshold probability may be configured to a higher or lower value if desired. Moreover, in some aspects, different threshold probabilities may be used for different conditions / symptoms. Regardless of the particular value to which the threshold probability is set or the number of probability threshold used, it should be understood that the threshold probabilities may be changed overtime. For example, initially the threshold(32736-2156)probability may be configured to a first value (e.g., 0.65), but the model may become more accurate overtime (e.g., as additional training data is obtained and additional training of the model is performed). As the model's classification capabilities improve the threshold probability may be adjusted, thereby minimizing the likelihood that portions of the media content identified as relevant end up being unsuitable for use in evaluating the patient. It is noted that any portions of the media content having a threshold below the threshold probability may be ignored (e.g., because those portions of the media content may be associated with inaccurate or incorrect classifications, or may otherwise be unsuitable for further use in evaluating the state of the patient or the patient's condition).

[0237] At step 1609, the relevant portions of the media content identified in step 1608 may be used to calculate a disorder score. In an aspect, the disorder score may be calculated based on metrics derived from the relevant portions of the media content, such as the exemplary metrics described above with reference to step 1604. In some aspects, aggregate metrics (e.g., average, median, etc.) may be calculated based on the ratios / metrics of the relevant portions of the media content. The calculated ratio / metrics for the relevant portions of the media content within the sliding window may then be subjected to ML / AI processing or classification to calculate the disorder score, which may be attributed to or associated with the sliding window (and the relevant portions of the media content thereof).

[0238] In step 1609, the disorder score(s) is provided to the clinician. In some aspects, the disorder score(s) may be provided to the clinician via a graphical user interface, such as a graphical user interface of an application resident on a clinician programmer device. In some aspects, the graphical user interface may include one or more graphical user interface components that change color in a manner relevant to the patient state classification. For example, the patient video component may include a border component. The border component may include a “green” color for a patient normal state, a “red” color for a patient symptom present state (e.g., pain, rigidity, tremor, and / or the like is present), and a “neutral” color (e.g., gray) for intermediate or uncertain classifications. It is noted that the exemplary colors and associated meanings described above have been provided for purposes of illustration, rather than by(32736-2156)way of limitation and that other types of color schemes and indications may be utilized in accordance with the concepts disclosed herein.

[0239] In step 1610, the clinician ora computational therapy algorithm provides settings for the patient's neurostimulation therapy based on the processed video of the patient. In some aspects, the above-described process may be performed continuously or repeatedly during the virtual clinic / remote programming session. That is, as the clinician changes stimulation parameters, the indication of the patient state is updated as the video of the patient is streamed to the clinician for review during the virtual clinic / remote programming session.

[0240] As discussed herein, ML / AI models of neurological disorders may be constructed using a variety of data sets. The ML / AI models may be employed to automatically classify patient states to assist virtual clinic / remote programming sessions. A subjective or evaluator-driven score can also be determined based on clinician assessment and patient-reported outcomes (PROs). The PROs can also be patient-specific and enhanced by the use of chatbots to ensure that all information is being collected without an undue burden on patients. ML / AI models may be constructed using, in part, such data. Such data may be obtained prior to a virtual clinic / remote programming session, at its initiation, or during the session. For example, patient emotional / well-being data may be obtained at the beginning of a session to increase the accuracy of the ML / AI operations during the session. Other models may be constructed for use or selection by the clinician based on one or more of the data types described herein (e.g., video, audio, sensor, context, patient reported data, PROs data, and / or any other PAD as discussed herein). The clinician may select from available ML / AI models and / or the virtual clinic / remote programming infrastructure or CP app may automatically select the appropriate ML / AI model(s) based on available data. In some embodiments, the virtual clinic / remote programming infrastructure or CP app may select appropriate models for use during a remote-programming session based on latency or other context. For example, different models may be selected depending upon a task being performed by the patient. Audio only models may be applied at selected portions of the remote-programming session (e.g., during clinician interview of the patient) and other models at different times (e.g., during patient performance of physical tasks).(32736-2156)Also, certain models may activated / d eactivated based on available processing resources and latency constraints associated with A / V session.

[0241] The respective ML / AI models may be employed to automatically classify or quantify patient states to assist virtual clinic / remote programming sessions. Referring to FIG. 18A, a screenshot depicting exemplary user interface fora clinician programmer device for conducting virtual clinic / remote programming sessions is shown as a user interface 1800. The user interface 1800 may be adapted to provide analytics of the patient concurrently with presentation of video content of the patient as the patient performs one or more tasks (e.g., the one or more tasks of FIG. 13 or other tasks) to the clinician. As described above, the presentation of the video content may include presentation of pre-recorded video of the patient. Additionally or alternatively, the presentation of the video content may occur in real-time, such as via streaming the video content to the clinician programmer device during a live virtual clinic / remote programming session.

[0242] In some embodiments, the analytics include analytics associated with kinematics (e.g., data related to movement of the patient). In some embodiments, the analytics include auditory data from analysis of the patient's voice. It is noted that the exemplary analytics described above (e.g., kinematics and auditory analytics) have been provided for purposes of illustration, rather than by way of limitation and that additional types of analytics may be utilized by embodiments of the present disclosure. The user interface 1800 may include interactive GUI elements that enable the clinician to control selection of available ones of the analytics during presentation the media content. For example, the user interface 1800 may include pop-up control component 1801. The pop-up control component 1801 may allow the clinician to activate selected analytics, as well as configure parameters associated with the analytics. To illustrate, in FIG. 18A the pop-up control component 1801 includes interactive elements (e.g., check boxes) that enable the clinician to select specific kinematics, such as kinematics related to the patient's face, hands, arms, and legs. The pop-up control component 1801 also includes interactive elements to activate presentation of analytics associated with the patient's voice.(32736-2156)

[0243] As described above, the media content (e.g., video data) may be processed using feature recognition software to identify various features or points associated with the patient. The features may be displayed over the patient video during presentation of the media content in accordance with the analytics activated by the clinician. In some aspects, lines connecting different features (e.g., key landmark points) may also be displayed as appropriate (e.g., along the torso, arms, legs, fingers, etc.) as shown in interface 1800. Additionally, a mesh display generated based on facial features may be displayed over the patient video as shown in interface 1800. In some aspects, the mesh display may be generated using recognition software libraries, such as a library or libraries of the above-mentioned landmark recognition software.

[0244] In some embodiments, the presentation of video and / or audio may be anonymized. As shown in FIG. 18B, video of the patient may be replaced entirely by display of the landmark points, mesh, and frame components generated based on the features identified in the media content. It is noted that the exemplary landmark points, mesh, and frame components may be animated based on the kinematics analytics and / or the auditory analytics, thereby enabling the clinician to visualize movement of the patient as the patient performs the one or more tasks, such as the tasks described above with reference to FIG. 13 or other tasks. It is noted that in some aspects the user interface 1800 may also provide interactive elements that enable the clinician to toggle between the overlay view described with reference to FIG. 18A and the anonymous view described with reference to FIG. 18B. The ability to toggle between the two different views may enable the clinician to more easily see certain kinematics or focus on specific aspects of the displayed kinematics (e.g., to display kinematics of the patient's hands and arms during evaluation of tremors, or feet and legs during evaluation of Parkinson's, such as gait analysis).

[0245] Referring to FIG. 18C, a screenshot depicting aspects of a user interface which may be displayed on a patient device is shown as a user interface 1820. In some aspects, the user interface 1820 may be displayed on a patient controller device (e.g., the patient controller 150 of FIG. 1A, patient controller 800 of FIG. 8 , or patient controller 1210 of FIG. 12 ) or another patient device (e.g., one or more of the patient devices 104 of FIG. 1 B). The user interface 1820 may(32736-2156)include functionality for generating media content (e.g., video content, image content, and / or audio content) that may be used in connection with the various analytics described above with reference to FIGS. 18A and 18B.

[0246] As shown in FIG. 18C, the user interface 1820 may display the landmark graphical features corresponding to the kinematic components selected by the clinician, enabling the patient to view the same kinematics as the clinician. Additionally, user interface 1820 may be used to provide feedback or commands to the patient, such as instructions for the patient to move their face, hands, limbs, torso, etc. into an optimal position for generation of the selected analytics. For example, user interface 1820 includes GUI component 1821, shown as a box, around the patient's right hand. In some aspects, the GUI component 1821 may provide a visual indication regarding whether the patient or a portion of the patient's body, is in a desired or optimal visual location. For example, the GUI component 1821 may be colored “green” to indicate the patient is in the optimal location and colored “red” to indicate the need to reposition. Further, the GUI component 1821 may include one or more indicators, shown as an arrow in FIG.18C, to indicate a direction of movement to place the patient into the optimal location.

[0247] Referring to FIG. 18D, a screenshot depicting aspects of a user interface presented to a patient are shown as GUI component 1830. The GUI component 1830 may provide functionality for controlling how input from the patient during a virtual clinic / remote programming session is received. For example, GUI 1830 may include interactive elements (e.g., check boxes, radio buttons, etc.) that may be selected or activated by the patient to grant permission for audio communication, visual communication, or audio / visual communication during the session. Additionally, the GUI component 1830 may provide interactive elements enabling the patient to control whether auditory and / or video content may be recorded during the session.

[0248] The GUI component 1830 may also include interactive elements that enable the patient to control whether the kinematic analytics are displayed to the clinician in the anonymized or non-anonymized mode, including sub-segments. For example, the patient may select to anonymize all video data by checking the interactive element labeled “Video” underneath the first “Anonymize” header. In(32736-2156)such case, all available video options will be automatically checked. Alternatively, the patient may select specific video sub-components to be anonymized (e g., “Face”, “Hands”, “Arms”, “Legs,” “Torso” and “Background). Similarly, the patient may also choose to anonymize the voice communication (if desired). The patient may choose to allow the clinician to view a full video presentation (such as the patient view in FIG. 18C, a completely anonymized view such as the patient view in FIG. 18B, or any variation thereof). It is noted that the GUI component 1830 is provided by way of illustration, rather than by way of limitation and that the GUI 1820 may include other suitable interface configurations and components that provide functionality for allowing the patient to view and control aspects of audio and visual communications during a virtual clinic / remote programming session.

[0249] Referring to FIG. 19, a flowchart depicting exemplary operations for conducting a virtual clinic / remote programming session according to some embodiments is shown. In step 1901, a virtual clinic session is started (e.g., using the operations discussed herein). In step 1902, the patient's consent for AV communications is obtained. Obtaining the patient's consent may include obtaining user input for respective anonymizing options, as described above with reference to FIGS. 18C and 18D. In some aspects, the patient (or the clinician) may choose to remove the background from the video communications or any bodily region or area from the video (including, optionally, one or more of the options from FIG. 18D). In step 1903, the initial AV communication for the virtual clinic / remote programming session begins according to the patient AV selections. In step 1904, options for kinematic and / or auditory analysis are received from the clinician to assist patient evaluation, as described above with reference to FIGS. 18A and 18B. In step 1905, the video / audio stream is augmented with ML / AI assisted kinematic and / or auditory analytics (including, optionally, one or more of the analytics discussed herein). In step 1906, the clinician or a computational therapy algorithm provides settings for the patient's neurostimulation therapy based on the processed video of the patient.

[0250] As discussed herein, some embodiments the kinematic analysis may be conducted while a patient performs a functional task (e.g., walking to permit evaluation of the patient by the clinician). In some embodiments, a patient physical therapy application is provided to the patient to assist management of the patient's(32736-2156)neurological disorder. The physical therapy application may operate on a patient controller (e.g., the patient controller 150 of FIG. 1A, patient controller 800 of FIG.8 , or patient controller 1210 of FIG. 12 ) or another patient device (e.g., one or more of the patient devices 104 of FIG. 1 B). In some embodiments, virtual reality (VR) devices may be used to assist the physical therapy operations. The physical therapy application may provide guided instructions for the patient to conduct physical activities. The physical activities may include activities selected to improve the patient's health, improve motor function, accommodate the patient to changes in physical sensations during movement as a result of a neurostimulation therapy, and / or assist any suitable clinical improvement in the patient. Additionally, the physical therapy application may capture patient data while the patient performs respective activities according to the instructions of the app. For example, physiological data, movement data, and other data may be captured using one or more patient devices (e.g., one or more of the patient devices 104 of FIG. 1 B or another device). Additionally, video and / or audio data from the patient may be captured and analyzed as discussed herein. The captured data may be reviewed by the clinician to evaluate the patient's response and progress to neurostimulation therapy.

[0251] Referring to FIG. 20, a flowchart depicting exemplary operations for conducting a physical therapy session in accordance with some embodiments is shown. In step 2001 , the patient starts the physical therapy application on a patient controller (e.g., the patient controller 150 of FIG. 1A, patient controller 800 of FIG.8, or patient controller 1210 of FIG. 12 ) or another patient device (e.g., one or more of the patient devices 104 of FIG. 1 B). In some aspects, the physical therapy application may be the physical therapy application described above. In step 2002, patient activity is recorded using camera functionality and / or sensors of available patient devices. As explained above, media content may be recorded during the patient's performance of the task(s) or activities using camera functionality and / or a microphone functionality of the patient's patient controller device, computer, or other suitable device. In some aspects, one or more external sensors may additionally or alternatively be employed to gather relevant patient data. For example, insole pressure sensors for quantifying balance, pressure distribution, or other information may be utilized. Also, wearable accelerometers and / or(32736-2156)gyroscopes (e.g., embedded within bracelets, watches, gloves, shoes, and the like) may be used to capture information that may be used to quantify arm and leg movement, sleep, trajectory, and / or timing. In some aspects, the patients can opt to wear “markers” on key joints of their body and use an optic movement-tracking system (such as Kinect). In some embodiments, breathing sensors, such as strap-on chest pressure sensors may be employed to detect respiration activity. Also, any number of sensors for sensing cardiac related activity may be employed (e.g., heart rate / blood oxygen sensors, such as watches, or wearable EKG sensors to detect heart rate and oxygen level).

[0252] In step 2003, the patient is provided guided instructions for one or more physical tasks or activities to be completed by the patient. In some aspects, the guided instructions may include video and / or audio presentations. For example, a video or images and text may be displayed to the user to illustrate the types of tasks or activities the patient is to perform.

[0253] Patients with movement disorders such as Parkinson's disease often report difficulties with everyday tasks such as buttoning (e.g., a shirt), brushing, and / or writing. In some embodiments, the physical therapy application may be tailored to the specific condition or disorder of the patient in order to train the patient on activities impacted by their specific condition or disorder, which may provide substantial improvements in the patient's quality of life.

[0254] In some embodiments, the physical therapy application may provide instructions for suitable physical exercises, such as tai chi or mild to moderate effort treadmill training, aerobic training, and dance activities. Tai chi may be advantageous for patients with neurological disorders. There have also been clinical studies that investigated the benefit of exercising, particularly the ones that involve balance training, such as tai chi, for movement disorders (e.g., Parkinson's disease). For example, in a study (Tai Chi versus routine exercise in patients with early- or mild-stage Parkinson's disease: a retrospective cohort analysis, Braz J Med Biol Res. 2020; 53(2): e9171 ) that involved 500 people with mild-to-moderate Parkinson's disease, one group received tai chi training 80 minutes per day, three days per week, for two months. The other group received regular exercising (including treadmill training, aerobic training, and dance) for 90 minutes per day, three days per week, for two months. Participants in the tai chi group reported a(32736-2156)significantly reduced number of falls (average of 3.45 vs. 7.45 over the past six months), and many of them discontinued or reduced the use of other therapies, such as levodopa.

[0255] Similarly, in another study (Tai Chi and Postural Stability in Patients with Parkinson's Disease, N Engl J Med 2012; 366:511-519) that recruited 195 men and women with mild to moderate Parkinson's disease, subjects were randomly assigned to twice-weekly sessions of either tai chi, strength-building exercises, or stretching. After six months, those who did tai chi were about two times better than those in the resistance-training group and four times better than those in the stretching group in terms of balance. The tai chi group also had significantly fewer falls, and slower rates of decline in overall motor control. These studies demonstrate that exercise that involves balance training as part of a physical therapy routine can provide additional benefits for people with balance / gait-related disorders, such as Parkinson's disease, in addition to the benefits exercising itself already brings.

[0256] In some embodiments, the patient may be provided a VR or AR viewing device to augment the user experience for the presentation of the guided instructions (and feedback described herein). Although optional, being immersed in a VR / AR-based environment can often encourage the trainee to consistently exercise by providing additional visual and audio stimuli. In the case of tai chi training, a pre-recorded VR / AR teacher can be presented in front of the trainee for learning. The VR / AR teacher can also be a persona (or hologram) based on adaptable chatbots to personalize the therapy experience. The teacher's body can be superimposed onto the trainee's body so that the trainee can mimic the teacher's exact movements. Here, the trainee would match the ideal movement trajectory / posture outlined by the digital teacher. In this case, the gaze of the patient or a slight change of EMG may cue the movement initiation.

[0257] In another aspect of exercising with a virtual reality or augmented reality experience according to the present disclosure, the VR or AR detects the intention of the subject, filters out the tremor, and displays undisturbed arm or leg movement. The rationale is that a tremor in Parkinson's disease patients may be caused by overcompensation of the posture in the body control while the postural information is erroneous because of the malfunction of thalamic relay neurons. By(32736-2156)displaying the correct posture without the tremor using VR / AR, the patient may stop trying to compensate (or overcompensate) for the postural error, which hypothetically may reduce the occurrence of the tremor or the severity of the tremor. Because the tremor frequency is around 5 Hz, notch filtering may remove the tremor and show smooth motion.

[0258] In case of a freezing event, the display of the intended movement initiation may rescue a patient from freezing.

[0259] In some embodiments, a gesture training paradigm may also be implemented for patients who prefer gesture training over (or in addition to) balance and gait training. This training will leverage music or art “therapy.” In other embodiments, the gesture training paradigm can also be disguised as games instead of music training. In embodiments using a music-based therapy paradigm, the patient will learn to control the notes and the pitch of the music via various common gestures they would use in real life, such as brushing, writing, buttoning, finger movements, and the like. Each gesture can code a note, and the relative positioning of the two hands / arms can control pitch. This paradigm could look very similar to playing the theremin, and an Al may be implemented to rate the performance and offer haptic feedback (as discussed herein). Gloves or arm sleeves with actuators embedded at the interface may be employed for haptic feedback in some embodiments.

[0260] A less specific version of this gesture training can also be implemented as “3D painting”, where the patient is asked to paint with a VR device in a virtual 3D space. This would involve precise positioning of fine movements as well as localization in a 3D space. This can be done either via copying an existing 3D painting, or having the patient create their original painting. In either case, an Al can be used to judge the movement precision of the patients, instead of the paintings themselves.

[0261] In step 2004, feedback may be provided during patient performance of the task(s) or activities. Having this measurement in turn offers opportunities for the Al in the physical therapy application to offer feedback to the patient if the patient repetitively makes the wrong move or mistimes the move. The feedback can be achieved via actuators embedded into the fabrics of the patient's clothing, or special clothing articles (such as gloves, socks, shoes) that have embedded(32736-2156)actuators. For example, if the patient is supposed to shift his weight to the left foot but failed to do so, a vibration can take place on the left foot to remind the patient. The cadence and frequency of the feedback can be driven by the Al or set by the patient. Presumably, as the patient gets better via practicing, the haptic feedback can change to a different vibration pattern to signal more complex feedback, such as to accelerate movements, deaccelerate movements, or even signaling a “good job’’.

[0262] In some aspects, exercise programs (moves) may have different difficulty levels. For patients who have more severe disease conditions or faster progressive severity, the program may start with the easiest level(s) and work their way up to higher difficulties. The Al can also correspondingly offer more feedback and support as the exercise difficulties increase.

[0263] In the case of a fall, or any situation of crisis, the sensors be configured to detect the adverse event and offer an “emergency intervention” by either switching to the most efficacious setting, or calling a care coordinator.Similarly, if the sensors detect that the patient is engaging in more dangerous movements, strong vibrations can also be sent via haptic feedback to remind / prevent the patient from performing movements that are out of the patient's comfort zone.

[0264] In step 2005, the stimulation parameters of the patient's neurostimulation therapy may be titrated during the performance of the tasks or activities. For example, if a patient has an implanted device such as DBS, the device can interact with the exercise platform in a closed-loop manner. The efficacy of neurostimulation fora neurological disorder can be state-dependent, as exercising could potentially change the efficacy of certain programmed settings. In such a situation, the implanted device can be controlled (e.g., by the patient controller device) to make small adjustments to programmed parameters to “explore” the therapeutic state space when patient performs various exercises, and the data can be used as training data for a deep learning algorithm to predict which parameter set is best suited for each exercise for this particular patient, thereby enabling an “exercise mode” to be individually developed for each patient. This can also include explorations of known and / or novel stimulation waveforms and paradigms that could be better suited for the patient given a specific exercise.(32736-2156)

[0265] Additionally, based on how well the patient is progressing in their familiarity and control of their exercise, an Al can be trained to slowly decrease the current of the implanted device setting, and therefore offering less therapy. This will offer the patient more “challenge” in terms of not being able to control their movement due to movement disorder symptoms. This “rehab mode” can motivate the patient to be intentional with their exercise training. The percent deviation from the optimal programming parameter can be prescribed by a physician and be controlled within a configured safety limit based on the particular patient, such as to maintain the parameters within a therapeutic range. In this manner, a digital equivalent of physical therapy may be provided and enable patients to perform therapy tasks at increasing levels of difficulty (either due to the particular tasks selected, the speed at which the tasks are performed, or due to particular configurations of stimulation parameters).

[0266] Similarly, if the patient performs worse in the exercise over time, the Al can adjust to a more efficacious therapeutic amplitude or setting, to offer more assistance to the patients.

[0267] Further, exercise alters the plasticity of the brain, and therefore, with long term recordings of sensor data and neural data at the implanted devices, one can infer any long term changes in the patient's disease improvement / progression. A correlation can be helpful for the clinicians to understand the patient's disease states and can also instruct a change in the programming setting of the implanted device.

[0268] In step 2006, an analysis of patient video and / or sensor data is conducted to characterize the patient conducting. The analysis may include ML / AI processing as discussed herein. Kinematic data may be calculated or determined for the patient based on the video and sensor data using one or more of the techniques described herein.

[0269] In step 2007, the performance data and / or video data is provided to a clinician for review. For example and referring to FIG. 21 , a screenshot of an exemplary graphical user interface for presenting performance data associated with a patient is shown as a user interface 2021. The user interface 2021 may be configured to present video of the patient conducting a task or activity (e.g., tai chi) with kinematic analysis as discussed herein. The video may be provided to the(32736-2156)clinician after being previously recorded by the patient. Additionally or alternatively, the video may be provided to the clinician concurrently with the patient performance of the activity (e.g., streaming the video content during a virtual clinic / remote programming session). If deemed appropriate (e.g., by the clinician viewing the video of the patient or an AI / ML process), new or adjusted stimulation therapy settings may be provided for control the patient's neurostimulation system, in step 2008.

[0270] As discussed herein, virtual clinic (VC) technologies may significantly improve accessibility to care for patients. While remote therapy delivery implementations may be accurate and effective in certain settings, processing video and audio data can be challenging at times due to multiple factors. Additionally, while special purpose devices and equipment have been proposed to assist computerized-characterization of patient movement for neurostimulation patients, the features of such special purpose devices and equipment are not available in commercially available smartphones. Accordingly, Applicants have proposed VC kinematic analysis that achieve relatively high degree of accuracy when determining kinematic related features of patient movement with commercially available hardware (such as commercially available smartphones).

[0271] Video recordings and / or analysis through a virtual clinic app or clinician programmer may be challenging due to variations in recording angles and positioning of the camera, making it difficult to align video data collected across multiple patients / sessions. This also applies to kinematics Al vision systems, since there are multiple frames of reference (e.g., the patient’s motion in the room and (b) the camera positioning relative to vertical and horizonal axes). Accordingly, extraction of biomarkers that are real-time (statistical in nature) may be sensitive to variations in the frame of reference and such extraction may significantly impact computational model performance.

[0272] As discussed herein, kinematics Al systems process video frames may be recorded through one or more cameras of a smartphone, tablet, or other suitable device. Depending on the positioning of the camera (i.e., due to pitch, roll, and yaw), identification of different landmark key-points across patients and their relative relationships may vary in unexpected manners thereby making further analysis or modeling possibly less accurate as expected if an ideal, fixed frame of(32736-2156)reference was employed. Specifically, the use of a handheld device may cause variation in camera position and / or orientation during video capture. Further, common physical tests (e.g., walking, turning, standing, etc.) for diagnosis or evaluation of neurostimulation patients may change the spatial relationship between the camera(s) and the patient during video capture. To address these issues, one or more mechanisms are provided by Applicant to allow for tilt, distance, and / or other invariance transformation(s) of key points for kinematic analysis. In some embodiments, the following processing techniques are applied during video capture and kinematic analysis: (i) resolving tilt of the camera by examining natural edges of objects in each frame of the image, and / or (ii) using a center of the hip (which has relatively little movement) in a gait cycle to normalize coordinates (for example, based on the distance of the left shoulder to the right shoulder and the distance of the center of the hip to the center of the shoulder, the landmark in each frame can be scaled so that it is normalized (and reference to the hip center) as a patient moves closer to / furth er away from the camera).

[0273] Systems and methods for kinematic analysis such as analyzing patient gait using the techniques described herein may be implemented, for example, using any of the systems described above, such as, without limitation, the system 1200 (shown in FIG. 12), the remote care system 100A (shown in FIG.1A), the network environment 100B (shown in FIG. 1 B), the external device 700 (shown in FIG. 7), the patient controller device 800 (shown in FIG 8), the clinician programmer 900 (shown in FIG. 9), the therapy system 1000 (shown in FIG. 10), and / or the system 1200 including the digital health infrastructure network 1212 (shown in FIG. 12).

[0274] In a clinic setting, multiple natural edges corresponding to doors, carpets, etc. provide a horizon line which is a static frame of reference. For example, vertical door lines provide a perspective frame of reference. In another example, a carpet with a repeating pattern and / or ceiling tiles may be used to determine depth. With reference to FIGS. 22A and 22B, distance invariance transformation of key landmark points on a patient for kinematic analysis can be accomplished by resolving a tilt of the camera capturing the video data of the patient 102 during a virtual session by examining natural edges of objects in each image frame 2200 of the captured video data. In an example embodiment, using(32736-2156)the audio / video manager 822 of the patient controller 800, the software 820 captures video data using a camera of the patient controller 800. The frames and / or individual images 2200 of the captured video data are analyzed and key landmarks of the patient 102 and structures and / or objects in the environment surrounding the patient 102 are segmented out. In an example embodiment, the software 820 segments the images frames 2200 (shown in FIG. 22A) of the captured video data using edge detection, wherein in a clinic setting, multiple natural edges corresponding to doors, carpets, etc. provide a horizon line which is a static frame of reference when segmenting the video data. For example, vertical door lines 2202 provide a perspective frame of reference. In another example, carpet with a repeating pattern 2204 and / or ceiling tiles 2206 may be used to determine depth.

[0275] The natural edges 2208 (shown in FIG. 22B) are identifiable using any suitable edge detection transform that enables the system to function as described herein. For example, search-based methods to detect edges may first compute a measure of edge strength (e.g., a first-order derivative expression such as the gradient magnitude) and then search for local directional maxima of the gradient magnitude. The local directional maxima may be a computed estimate of the local orientation of the edge (e.g., the gradient direction). Also, zero-crossing edge detection searches for zero crossings in a second-order derivative expression computed from an image to find edges. The zero-crossings of the Laplacian or the zero-crossings of a non-linear differential expression may be employed. Also, a pre-processing step may be applied first such a smoothing stage with a suitable filter (e.g., Gaussian smoothing).

[0276] Upon identifying one or more suitable edge(s) in one or more initial frames in the video, then, the identified edges can be tracked from frame to frame. The identified edges may include one or more horizontal and / or vertical edges. Changes in x, y coordinates associated with the identified edge(s) from a previous frame to a current frame allow for correction of motion of a camera (e.g., a camera of a handheld smartphone) via a geometric transform or a perspective warp after homographic matching. Perspective warp algorithms are available in application programming interface (API) packages in various programming languages. In general, such perspective warp algorithms transform a source image (e.g., in this(32736-2156)case a respective video frame) into a destination image by applying matrix operations. In a clinical setting (such room or hallway in a medical setting), edges defined by the ceiling, the floor, one or more doors, one or more walls, and / or the like may be identified using suitable edge detection techniques.

[0277] Software 820 may analyze frames 2200 of the video data and identify and segment out the natural edges 2208, such as the vertical door lines 2202, the repeating pattern 2204 of the carpet, the ceiling tiles 2206, etc. In some embodiments, the segmentation process may isolate the identified natural edges 2208 from the surrounding environment and identify a location of each of the identified natural edges 2208 within a coordinate space to register the individual frames 2200 of the video data to one another. In this manner, the identified natural edges 2208 (such as horizontal and vertical edges) can be tracked from frame to frame.

[0278] Changes in x and y coordinates associated with the respective edges from a previous frame 2200 allow for correction of motion of a camera (e.g., a handheld camera) using a geometric transform or a perspective warp after homographic matching. The software 820 continuously analyzes the frames 2200 of the video data in real-time and identifies commonalities between the identified natural edges 2208 in the real-time frames 2200 of the video data. A distance between the camera and one or more of the identified natural edges 2208 is determined using, for example, a predetermined distance between the camera and at least one or more natural edges 2208, a known zoom level of the real-time video data captured by the camera, known dimensions of common doorframe sizes, ceiling tiles, etc., although the distance between the camera and the identified natural edges 2208 may be determined using any suitable means without departing from the scope of the disclosure, such as, without limitation, data obtained from a multi-lens camera (e.g., stereovision).

[0279] Referring to FIGS. 23A and 23B, a method of processing tilt and contrast of video data captured during a virtual clinic / remote programming session is illustrated and generally identified by reference numeral 2300. As the method begins 2302 and video image data is collected, all counters, such as the bad contrast counter and the heavy tilt counter, are reset 2304. The frames of the video image data are read and / or analyzed 2306 and an image segmentation is(32736-2156)generated 2308. As the frames are segmented, the contrast of the frames is detected, and it is determined if the contrast of the frames is good 2310 (e.g., a contrast metric satisfies a threshold value) or otherwise capable of being analyzed and detecting edges or other natural edges. If it is determined that the contrast of the frames is not good, the contrast of the frames is adjusted 2312, and the bad contrast counter is incremented 2314. If the bad contrast counter is incremented, it is determined 2316 if the bad contrast count is high or otherwise exceeds a predetermined threshold.

[0280] If it is determined that the bad contrast count does not exceed the predetermined threshold, the method returns to reading the frames 2306. If it is determined that the bad contrast count exceeds the predetermined threshold, a bad contrast warning is displayed 2318 and the method returns to reading the frames 2306. If it is determined that the contrast of the frames is good, a change in position of the lines detected within the frames are filtered 2320 within an x, y tolerance by degree and it is determined if enough lines have been identified in the frames 2322 by comparing the number of lines identified in the frames to a predetermined threshold.

[0281] If not enough lines are identified (the number of identified lines in the frames is less than the predetermined threshold), the heavy tilt counter is incremented 2324 and it is determined if the number of counts of the heavy tilt counter exceeds a predetermined threshold 2326. If the number of counts of the heavy tilt counter does not exceed the predetermined threshold, the method returns to reading the frames 2306. If the number of counts of the heavy tilt counter exceeds the predetermined threshold, a tilt warning is displayed 2328 and the method returns to reading the frames 2306. If it is determined that the number of lines identified in the frames exceeds the predetermined threshold, a homographic match is performed 2330, a perspective warp is performed 2332 on the segmented frames of the video data, and the counters are reset 2304.

[0282] With reference to FIGS. 24A and 24B, another important aspect of gait analysis is gait imbalance while a patient walks. In an example embodiment, video image data is be captured using a camera 2402 while the patient 102 is walking towards 2404 or away 2406 from the camera 2402 in an environment 2410 during a frontal view gait analysis (shown in FIG. 24A). A frontal view gait analysis(32736-2156)may be more intuitive and improves the likelihood that the patient 102 is within a field of view 2408 of the camera 2402 compared to analyzing gait from a different point of view. However, as the patient 102 moves significantly away 2406 from the camera 2402 and towards 2404 the camera 2402 and / or field of view 2408, challenges may arise with i) the scale of the patient 102 within the frame and ii) the size of the patient 102 being relatively small as compared to the patient being closer to the camera 2402 and causing deep neural networks (DNNs) to make false detections. In some embodiments, a side view gait analysis may be performed (shown in FIG. 24B), with the patient 102 walking within an environment 2410 from left to right 2412 or from right to left 2414 in front of the camera 2402 and the field of view 2408. However, in clinics (and at home), it may often be difficult to set up and find enough space to record an entire gait cycle using a side view.

[0283] With reference to FIGS. 25A, 25B, and 26A-26C, kinematics Al systems include video that may be recorded through an application of a camera of a smartphone, tablet, or other suitable device. Multiple algorithms and DNNs may be used in kinematics analysis. The performance of each DNN may vary depending on the size (e.g., in absolute pixels) of the patient 102 within the screen. For example, the patient 102 may be too far away from the camera 2402 to be detected by a first DNN (DNN1) (shown in FIG. 25A). However, the DNN1 may be able to detect the patient 102 if the patient 102 is close enough to the camera 2402 (shown in FIG. 25B).

[0284] To address at least this issue, a dynamic switched pipeline of ensemble models that enables high accuracy and adaptive latency of interference is provided. In an example embodiment, a second DNN (DNN2) may be used to detect and rescale a portion of an image 2506 (shown in FIG. 26A) of the patient 102 captured by the camera 2402 to develop a rescaled image 2508 (shown in FIG. 26B), and DNN1 may subsequently be used to generate landmarks 2510 for the patient 102 in the rescaled image 2508 (shown in FIG. 26C). The software 820 utilizes mechanisms for dynamically switching flow between multiple DNNs based on expected image accuracy and / or reduced latency. In some embodiments discussed herein, the software 820 may compare a number of person detection frames (PDFs) to a first threshold value Th1 and a second threshold value Th2(32736-2156)when determining if DNN1 should be utilized or if DNN2 should be utilized to detect the patient 102 within the image 2506 captured by the camera 2402 and / or when the image 2506 should be rescaled. As discussed herein, the PDF may refer to an ongoing counter that tracks the number of times (e.g., in successive frames) that the patient is detected. Alternatively, PDF may be implemented as any suitable counter to track detection of one or more predefined patient features relevant to kinematic analysis.

[0285] With reference to FIGS. 27A and 27B, a method for processing video of patient for kinematic analysis using multiple DNNs is performed and generally identified by reference numeral 2700. PDF is a dynamic counter as discussed and Th1 and Th2 are dynamic thresholds that are updated to enhance accuracy and latency. As the method begins 2702 and video image data is collected, and the frames of the captured video image data are read and / or analyzed 2704. It is determined 2706 if a number of counted PDFs exceeds a first threshold value Th1. If it is determined that the number of counted PDFs does not exceed the first threshold valve Th1 , DNN2 is used to detect landmarks 2708 within the image frame, and it is determined 2710 if landmarks have been detected in the image frame.

[0286] If landmarks have been detected in the image frame, the landmarks are processed 2712 and it is determined 2714 if the PDF counter is less than Th1 and greater than a second threshold value Th2. If the PDF counter is less than Th1 and greater than Th2, the PDF counter is reset 2716 and the method returns to reading the image frames 2704. If the PDF counter is not less than Th1 and / or is not greater than Th2, the method returns to reading the image frames 2704 without resetting the PDF counter.

[0287] If landmarks have not been detected, DNN1 is used to detect 2718 the patient within the image frame and it is determined 2720 if DNN1 has already processed the image frame. If DNN1 has already processed the image frame, the method returns to reading the images frames 2704. If DNN1 has not already processed the image frame, it is determined 2722 if the patient is detected within the image frame. If the patient has not been detected within the image frame, the method returns to reading the image frames 2704. If the patient has been detected within the image frame, Th1 for the PDF counter is modified and / or incremented(32736-2156)2724, the image frame is rescaled 2726 based on person bounding, and DNN2 is used to detect landmarks 2708, and the method continued as described above.

[0288] If it is determined that the number of counted PDFs exceeds the first threshold value Th1 , it is determined 2728 if the number of counted PDFs exceeds the second threshold value Th2. If the number of counted PDFs exceeds the second threshold value Th2, landmarks are detected 2708 by DNN2 and the method continues as described above. If the number of counted PDFs does not exceed the second threshold value Th2, it is determined if a patient has been detected 2722, and the method continues as described above.

[0289] Also, additionally or alternatively to employing multiple DNNs, some representative embodiments may employ multiple cameras during kinematic analysis. With reference to FIG. 28, the patient device 104 may include multiple camera sensors having different and / or varying apertures and focal lengths. For example, the patient device 104 may include a first camera 2802, a second camera 2804, and a third camera 2806. In some embodiments, the first camera 2802 may include a 77 mm focal length, have 3x optical zoom, and may include a f / 2.8 aperture, the second camera 2804 may include a 13 mm focal length and may include a f / 1.8 aperture, and the third camera 2806 may include a 26 mm focal length and a f / 1.5 aperture, although the patient device 104 may include any number and type of camera sensors without departing from the disclosure. In an example embodiment, the software 820 and / or the audio / video manager 822 may selectively change between and / or utilize one or more of the cameras 2802, 2804, and 2806.

[0290] With reference to FIGS. 29A and 29B, a method for dynamic camera sensor selection is illustrated and generally identified by reference numeral 2900. As the method begins 2902, a list of camera sensors and focal lengths associated with the patient device is generated 2904, and image frames are acquired 2906 from the active camera sensor.

[0291] A bounding box (BBox) surrounding the patient is generated 2908 and it is determined 2910 if the BBox is greater than a threshold percentage Th% of the patient device screen and if the patient is walking towards the active camera. If the BBox is greater than the Th% and the patient is walking towards the active camera, it is determined 2912 if a higher focal length feature is available on(32736-2156)the patient device. If no higher focal length feature is available, the method returns to acquiring image frames from the active sensor 2906.

[0292] If a higher focal length feature is available, the active sensor is changed 2914 to the camera sensor having a focal length feature and the method returns to acquiring image frames from the active sensor. If the BBox does not exceed Th% and the patient is not walking towards the active camera, it is determined 2916 if a person detection error exceeds a threshold number of pixels (e.g., percentage of the patient device screen size) and the patient is walking away from the active camera.

[0293] If the patient detection error does not exceed the threshold number of pixels and the patient is not walking away from the active camera, the method returns to acquiring image frames from the active sensor 2906. If the patient detection error does exceed the threshold number of pixels and the patient is walking away from the active camera, it is determined 2918 if a lower focal length feature is available on the patient device. If no lower focal length feature is available, the method returns to acquiring image frames from the active sensor 2906. If a lower focal length feature is available, the active sensor is changed 2914 to the camera sensor having a focal length feature and the method returns to acquiring image frames from the active sensor.

[0294] Turning to FIGS. 30A and 30B, as described hereinabove, video recordings through a virtual clinic app or a clinician programmer may be challenging due to variations in recording angles and positioning of the camera, making it difficult to align video data collected across multiple patients and / or sessions. This difficulty also applies to kinematics Al vision systems, since there are multiple frames of reference (e.g., the patient’s motion in the room and the camera positioning relative to vertical and horizontal axes). Extracting biomarkers and / or other landmarks in real time (e.g., statistical in nature) is sensitive to variations in the frame of reference, and can significantly impact model performance. Thus, some representative embodiments provide compensation of body motion as captured by a two-dimensional (2D) camera (e.g., not having a depth sensor) without requiring a complicated setup to reduce patient motion and / or camera positioning that may pose barriers for usage. Further, depending upon the positioning of the camera (e.g., due to pitch, roll, and yaw), there may be(32736-2156)different landmark key-points across patients, which can make further analysis and modeling challenging.

[0295] To address these issues, some representative embodiments provide a mechanism for distance invariant transformation for key points (e.g., landmark points) of a kinematics analysis. Specifically, a center of a hip of a patient generally has the least movement in a gait cycle, and is used to normalize coordinates. Based on the distance of a left shoulder to a right shoulder and a distance of a center of the hip to a center of the shoulder, the landmark in each frame may be i) scaled so it is normalized when a patient gets closer to / further away from the camera and ii) referenced to the hip center.

[0296] In some example embodiments, gait symmetry may be quantified using a room frame of reference. In this manner, once a frame of reference is acquired (e.g., post tilt correction), overall posture asymmetry may be evaluated by computing an angle 0i between a line 3002 generated from the torso 3004 of the patient 102 and a vertical line or plane 3006 within the captured image frame 3008. For example, as the patient 102 walks, the angle 0i varies in a first direction 01 as the patient steps forward with a first foot (shown in FIG. 30A) and in a second direction 02 as the patient steps forward with the opposite foot (shown in FIG. 30B).

[0297] FIGS. 31 A and 31 B illustrate pre-therapy patient gait asymmetry plots of the angle 0i over multiple image frames. For a symmetrical gait, the left and right angles 01 and 02 should be approximately the same. As shown in FIG. 31 A, pre-therapy, the hip line angle of the patient 102 moving towards the camera indicates a right inclination. FIG. 31 B illustrates the shoulder dip mid-point line angle of the patient 102, pre-therapy, moving towards the camera indicating a right inclination. The axial symmetry of the patient’s 102 gait correlates with gait postural symmetry.

[0298] FIGS. 32A and 32B illustrate end-of-therapy patient gait asymmetry plots of the 0i over multiple image frames. At the end-of-therapy, the patient 102 still inclines to the right some, but the gait has at least partially improved compared to the pre-therapy gait shown in FIGS. 31 A and 31 B. The temporal high frequency distortion (e.g., jitter) shown in FIGS. 32A and 32B may indicate pain (e.g., wincing) experienced by the patient 102 during the exercise.(32736-2156)

[0299] With additional reference to FIGS. 33-39, as described hereinabove, analyzing video data through a virtual clinic application or clinician programmer may be challenging, as the video data may include a relatively large amount of data (including data that may not be relevant to the medical condition being assessed). A kinematics Al system that assists physicians with objective evaluation of pain may require analysis of imbalance of gait while a patient walks towards a camera. This analysis may be a relatively sophisticated and computationally intensive task that, with existing computational rates, may be infeasibly to implement using a single neural network. For efficiency in a clinical setting, there is a need for such a system to perform analysis in real-time.Additionally, quantifying the asymmetry measurements may be challenging (e.g., as a time series of left and right landmarks are out of phase with one another). Analyzing the whole time series, however, accurate measurements are attainable.

[0300] A kinematics Al system provided herein includes an ensemble of neural networks and adaptive computation filters. The software 820 of the system identifies specific body parts to provide a near real-time gait analysis using video data acquired using a camera (e.g., frontal camera data as a patient walks towards the camera). The software 820 may also quantify pain experienced by the patient during movement by examining symmetry waveforms (e.g., high frequency components, jitter) over gait cycles.

[0301] With reference to FIG. 33, a method of analyzing gait using a kinematics Al system is illustrated and generally identified by reference numeral 3300. The software receives 3302 input video, such as video image frames captured by one or more cameras of the patient device of the patient walking towards or away from the camera. The software buffers 3304 the captured video data in a buffer queue which in some embodiments, may be about 1 / 3rdof the frame rate (e.g., frames per second) of the captured video data. The software extracts 3306 key-points and / or other landmarks of the patient, the surrounding environment, and / or both the patient and the surrounding environment using one or more of the methods and systems described herein, although any suitable method or system for extracting key-points may be used without departing from the scope of the disclosure. With key-points extracted from the captured video data, a direction of motion of the patient is identified 3308.(32736-2156)

[0302] For example, the software may identify whether the patient is walking towards the camera, away from the camera, turning left or right, is moving and / or is out of the field of view of the camera, and / or if the patient is truncated within the field of view of the camera. In some embodiments, if it is determined that the patient is partially or completely out of the field of view of the camera, the software may indicate that the patient is turning (e.g., left or right relative to the camera). In some embodiments, incorrect detections may be identified by updating the extracted key-points using previous image frames of the captured video data or by identifying and using silhouettes of the patient as a reference for identifying the patient’s direction of motion.

[0303] With the patient’s direction of motion identified, the software identifies whether a leg of the patient is moving and / or determines 3310 which leg of the patient is moving. For example, in a current image frame being analyzed, the software may generate and apply an average of slopes of the left ankle and the right ankle of the patient 102 from captured image frames stored in the buffer queue. In some embodiments, the leg of the patient that is identified as moving may be determined based on analysis of the captured image frames stored in the buffer queue where a majority of the capture image frames stored in the buffer queue indicate that a particular leg is moving. The software identifies 3312 features within the captured image frames, such as distances between keylandmarks, angles, step cycle parameters, dynamic time warping (DTW), patient silhouette area, etc., and / or combinations thereof. The software may only analyze features that are considered relevant to the analysis, which may be stored as data table or another suitable list of relevant features. In some embodiments, identified features of the patient may be separated based on the identified direction of motion of the patient and the identified moving leg of the patient. The software analyzes 3314 each identified feature based on a mean of values, a standard deviation, skewness, and / or kurtosis.

[0304] In some embodiments, dynamic time warping (DTW) may be used for gait symmetry analysis to perform a motion flow analysis of landmarks identified within an image frame 3402 corresponding to body extremities that move significantly during gait. DTW is an algorithm for measuring similarity between two temporal sequences and is available in programming language implementations(32736-2156)such as Python’s FastDTW. At a qualitative level, DTW involves computing the distance between two time series, A and B (e.g., anatomical positions or kinematic or position metrics defined by or calculated in reference to landmarks of a patient in some embodiments), by finding a path through a cost matrix that minimizes the cumulative distances between them. The DTW distance may be calculated in reference to a summation of the distance between points from series A and series B over all in an optimal alignment path. Formal definitions of DTW calculations are found in literature and may be used according to representative embodiments. Various of DTW calculations may be employed that are representative of a distance calculation between two time series of values indicative of patient data from patient video, where the distance calculation is modified by applying time and / or phase compensation for the respective time series datasets.

[0305] In some representative embodiments, DTW matches a time lag between waveforms for a left body part motion 3404 and a right side body part motion 3406, and estimates a distance between those body parts (e.g., wrists, shoulders, legs, etc.). In a perfectly symmetrical gait, generally, in the flow of datapoints, most normalization and scaling are symmetric between the left and the right body parts. Foran asymmetric gait, a DTW distance 3408 (shown in FIGS. 36 and 37) provides a metric of the degree of asymmetry of matching warped points within a time series, thereby providing a direct correlative measure of asymmetry. The more similarity between the left and right body parts will result in a lower amount of DTW distance 3408. In some embodiments, the DTW distance 3408 may be calculated using a full cycle approach and / or a moving leg approach as discussed herein.

[0306] With reference to FIG. 38, using the full cycle approach, a DTW distance calculation for each gait cycle of the patient 102 moving toward the camera or away from the camera motion is made using scaled X and scaled Y key-points e.g., the patient’s 102 wrists). The DTW distance 3808 is determined between left and right-side parameters 3804, 3806 in the cycle. The determined DTW distance is normalized for the cycle by frame rate per second (FPS) of the captured video data. In some embodiments, features of the DTW distances 3808 are extracted, such as a mean value across all cycles of captured video data for comparison. In one non-limiting embodiment, the DTW distance 3808 variation is(32736-2156)compared during therapy progression (e.g., over multiple sessions) to quantify improvement of gait.

[0307] With reference to FIG. 39, using the moving leg approach, a DTW distance calculation is made for each consecutive left and right leg pair using scaled X and scaled Y key-points. A DTW distance 3908 of left side parameters is determined from a left leg moving region 3904 and a DTW distance 3908 of right side parameters is determined from a right leg moving region 3906. The DTW distance 3908 values are normalized by the FPS of the captured video data. In some embodiments, features of the DTW distances 3908, such as a mean of each pair of DTS distances 3908, may be extracted as a final DTS distance 3908 value for the captured video data. In one non-limiting embodiment, DTW distance 3908 variation may be compared during therapy progression (e.g., over multiple sessions) to quantify improvement of gait.

[0308] With reference to FIGS. 40-44B, as described hereinabove, an important aspect of gait is an imbalance of gait while the patient walks. A side view of the patient allows visualization of the entire gait cycle and therefore, a strong asymmetry recording. The use of 2D cameras, however, poses a specific challenge with gait analysis as the physical dimensions and the frame of reference of the patient under analysis is constantly changing, which may cause a problem for standard discriminatory analysis techniques which tend to detect depth artifacts rather than gait artifacts. To address these issues, a system for normalization of a human frame across ambulatory cycles specific for gait analysis is provided, which includes a transformation that is invariant of depth and frame of reference.

[0309] For example, some representative embodiments herein provide for kinematics Al based analysis of gait asymmetry in real-time using a front facing camera. In some embodiments, normalized and scaled landmarks may be used to calculate the asymmetry of the body during a full cycle of movement and provide a gait asymmetry score. Additionally or alternatively, normalized and scaled landmarks may be used to calculate the asymmetry of the body during a moving leg (e.g., a stride) and provide a gait asymmetry score. Such score(s) may be used to alert the user about the abnormality of movement, to compare the effectiveness of stimulation on gait correction, to provide longitudinal quantification of gait improvement for a patient undergoing spinal cord stimulation, and / or to(32736-2156)identify when a spinal cord stimulation patient requires adjustment to therapy using gait asymmetry.

[0310] In some embodiments, suitable processing of data in the various frames to calculate kinematic scores may be applied to improve accuracy of the kinematic analysis by automatically discarding video frames exhibiting transient misclassification of landmarks from a neural network operating in real time, by automatically discarding frames with high occlusion, by automatically discarding frames where the patient is turning around, and / or by analyzing gait by examination of angles subtended across landmarks (e.g., should-knee-ankle) across various phases within a gait cycle.

[0311] With reference to FIG. 40, a method for normalization of a human frame across ambulatory cycles specific for gait analysis is illustrated and generally identified by reference numeral 4000. The software receives 4002 input video, such as video image frames captured by one or more cameras of the patient device of the patient walking towards or away from the camera. Key-points and / or landmarks are extracted 4004 from the captured image frames using any of the methods and systems described herein, although any suitable methods and systems may be used to extract key-points and / or landmarks without departing from the scope of the disclosure. Each image frame is normalized 4006 using a scale of a hip width of the patient and / or a shoulder to hip length of the patient referenced to the center of the patient’s hip.

[0312] With additional reference to FIGS. 41 A, 41 B, and 42, the software 820 identifies key-points and / or landmarks within a first captured image frame 4102 (shown in FIG. 41 A) and a second, subsequent captured image frame 4104 (shown in FIG. 41 B). In an example embodiment, the software 820 determines a first hip width 4106a and / or a first shoulder to hip length 4108a within the first captured image frame 4102 referenced to a center of the patient’s 102 hip. The software 820 determines a second hip width 4106b and / or a second shoulder to hip length 4108b within the second captured image frame 4104 referenced to a center of the patient’s 102 hip. Using the difference in width and / or length between the first hip width 4106a and the second hip width 4106b and / or the first shoulder to hip length 4108a and the second shoulder to hip length 4108b, a scaling factor is determined which is invariant of depth and frame of reference.(32736-2156)

[0313] For example, X and Y positions of key-points and / or landmarks identified within each image frame of the captured image frames may be plotted (shown in FIG. 42), such as, without limitation, a patient’s left shoulder 4202, right shoulder 4204, left hip 4206, right hip 4208, left knee 4210, right knee 4212, left ankle 4214, right ankle 4216, nose 4218, left elbow 4220, right elbow 4222, left wrist 4224, right wrist 4226, left eye 4228, right eye 4230, left ear 4232, right ear 4234, neck 4236, etc. Using the relative change in distances the identified keypoints in each image frame, the X-Y position of the left hip 4206 and the right hip 4208 are relatively stationary, providing a fixed frame of reference that is invariant of depth and frame of reference.

[0314] Turning to FIGS. 43A and 43B, an improvement in gait can be quantified by comparing plots of the X, Y positions of the key-points and / or landmarks identified within each image frame over multiple sessions. For example, a first plot 4300 (shown in FIG. 43A) of patient 102 gait can be generated pretherapy and a second plot 4302 (shown in FIG. 43B) of the patient 102 gait can be generated at any point during therapy or at the end of treatment. The X, Y positions of each of the key-points and / or landmarks shown in the first plot 4300 define a first set of clusters 4304 that indicates a spread of motion for each keypoint and / or landmark as the patient 102 walks towards or away from the camera pre-therapy. The first set of clusters 4304 define first distribution areas, bounding each cluster of the first sets of clusters 4304. The X, Y positions of each of the keypoints and / or landmarks shown in the second plot 4302 define a second set of clusters 4306 that indicates a spread of motion for each key-point and / or landmark as the patient 102 walks towards or away from the camera post therapy. The second set of clusters 4306 define second distribution areas, bounding each cluster of the second set of clusters 4306. The distribution areas for each key-point and / or landmark of the second set of clusters 4306 is compared to the distribution areas for each key-point and / or landmark of the first set of clusters 4304. In an example embodiment, the distribution areas of the second set of clusters 4306 is smaller than the distribution areas of the first set of clusters 4304, or grouped closer to a centerline of the patient 102, which is indicative of an improvement in gait. In some embodiments, the distribution areas of the key-points and / or landmarks on the patient’s 102 left side may be compared to the distribution areas(32736-2156)of the corresponding key-points and / or landmarks on the patient’s 102 right side to identify a gait imbalance. For example, distribution areas of the key-points and / or landmarks on the patient’s 102 left side that are equal to or substantially equal to the corresponding distribution area of the key-points and / or landmarks on the patient’s 102 right side indicates gait symmetry. In contrast, if the size of the distribution areas for the patient’s 102 left side and the patient’s right side are not equal, or extend further away from the centerline of the patient 102 on either the left side or the right side, such an imbalance may indicate gait asymmetry.

[0315] In some embodiments, the software 820 may pre-process the captured video data of the patient 102 walking in front of the camera by utilizing one or more sensors associated with the patient device 104 to identify magnetic north. The software 820 may fuse magnetic north into the captured video data in real-time for visual homography. In this manner, the captured video data may be stabilized even when the patient 102 is holding the patient device 104.

[0316] In some embodiments, patient 102 facial expressions may be analyzed to identify grimaces or other facial expressions indicative of pain during kinematic analysis (see, also, discussion facial landmarks of FIGS. 17A and 17B herein). For example, the captured video data of the patient 102 may be analyzed by the software 820 using the Facial Action Coding System (see Facial Action Coding System by Ekman, P., & Friesen, W. V. (1978)) to identify individual muscle movements (e.g., Action Units (A.U.)) indicating pain or discomfort.Combinations of A. U.S can be correlated to different emotions, such as grimaces. The A.U. data may be defined in respective templates correlating landmark data to facial expressions and such templates (see “blendshapes” below) may include composite expressions of multiple such A. U.S.

[0317] For example, non-linear combinations of one or more of A.U. 1 , 2, 3, 5, 11, 12, 20, 23-27, and 42-44 can be indicative of one or more types of grimaces. In one non-limiting embodiment, A.U.s 1 , 2, 3, 5, 11, 12, 20, 23-27, and 42-44 are assigned to a set of A.U.s, which may be stored on the memory 806. Different types of grimaces and different locations of grimaces correlate to respective combinations of A.U.s of the set of A.U.s, with specific combinations of A.U.s corresponding to a specific blendshape. In this manner, different blendshapes corresponding to each combination of A.U.s of the set of A.U.s can be identified. In(32736-2156)an example embodiment, movement of the set of key landmark points 1700 are compared to the A. ll.s of the set of A. U.S and any A. U.S corresponding to the movements are identified. The identified A. U.S are analyzed by the software 820, and it is determined if there are any blendshapes corresponding to one or more types and locations of grimaces present in the analyzed captured video data.

[0318] In some embodiments, identified grimaces can be correlated to positions of the patient’s 102 extremities as the patient 102 walks towards or away from the camera. For example, grimaces identified only while the patient 102 is moving their left leg, but not while the patient 102 is moving their right leg, may be indicative of asymmetrical pain, and therefore, may be indicative of gait asymmetry. In some embodiments, an intensity of a patient’s 102 grimace may be determined, which can be correlated to a pain intensity. If a grimace is identified on the patient’s 102 face while moving both legs or during a duration of the time walking towards the camera, a grimace having greater intensity during movement of the left leg, but not during movement of the right leg, may indicate pain asymmetry, and therefore, may indicate gait asymmetry.

[0319] With additional reference to FIGS. 44A and 44B, a method of correlating patient grimaces to gait asymmetry is illustrated and generally identified by reference numeral 4400. Video image data of the patient walking in front of a camera is collected and the frames of the captured video image data are read and / or analyzed 4402. Key-points and / or landmarks are identified 4404 within the captured image frames using, for example, any of the systems and methods described hereinabove. Movement of the position of the identified landmarks is monitored 4406 across a plurality of captured image frames in real-time.

[0320] The movement of the identified landmarks is compared 4408 to a predetermined set of Action Units, and it is determined 4410 if the movement of the identified landmarks correspond to one or more of the Action Units of the predetermined set of Action units. If the movement of the identified landmarks does not correspond to one or more of the Action Units, the method returns to reading 4402 the frames of the captured video image data. If the movement of the identified landmarks correspond to one or more of the Action Units, the identified action units are compared 4412 to predetermined combinations of Action Units corresponding to the type, the intensity, and / or the location of the movements to(32736-2156)determine 4414 if the identified Action Units correspond to one or more of the predetermined combinations of Action Units.

[0321] If the identified Action Units do not correspond to one or more of the predetermined combinations of Action Units, the method returns to reading 4402 the frames of the captured video image data. If it is determined that the identified Action Units correspond to one or more predetermined combinations of Action Units, the identified predefined combination of Action Units is correlated 4416 to one or more blendshapes that are representative of the identified predefined combination of Action Units. The location and / or the intensity of the identified grimace is determined 4418 and it is determined 4420 if a grimace is located on both sides of the patient’s face. If a grimace is only located on one side of the patient’s face, the location of the grimace (e.g., a left side or a right side of the patient’s face) is correlated 4422 to movement of the patient’s extremities, such as, without limitation, the patient’s legs, the patient’s wrists, etc., and a gait asymmetry is identified 4424.

[0322] If a grimace is located on both sides of the patient’s face, it is determined 4426 if an intensity of the grimace on one side of the patient’s face is higher than an intensity of the grimace on the other side of the patient’s face. If the intensity of the grimaces on both sides of the patient’s face are equal or are substantially equal, the method returns to reading 4402 the frames of the captured video data. If an intensity of the grimace on one side of the patient’s face is greater than an intensity of the grimace on the other side of the patient’s face, the location of the grimace having a higher intensity is correlated 4422 to movement of one or more of the patient’s extremities and a gait asymmetry is identified 4424 based on the grimace location and extremity movement.

[0323] In some embodiments, the software 820 may analyze audio data associated with the captured video data and identify auditory expressions as the patient 102 is walking in front of the camera. For example, if a “blurt” (a sudden, possibly non-verbal, auditory expression) or other increase in auditory signal (generally herein an auditory impulse) is detected within the audio data, the software 820 may correlate the timing of the blurt to movement of the patient’s 102 extremities. Based on the correlation of the identified blurt or auditory impulse and movement of the patient’s 102 extremities, such as, without limitation, the patient’s(32736-2156)legs, pain experienced by the patient 102 during movement of the correlated extremity can be identified and used to determine if a gait asymmetry exists. In some embodiments, the timing of the detected blurt or auditory impulse may be correlative to an intra-gait step and / or inter-cycle to identify the type of pain being experienced by the patient 102. The software 820 may analyze the audio data in lieu of, or in combination with, any of the systems and methods for determining gait asymmetry described herein.

[0324] In some embodiments, the software 820 may perform a spectrogram analysis of the audio data associated with the captured video data of the patient 102 walking in front of the camera to identify and / or enhance identification of gait asymmetry and / or Parkinson’s disease. In an example embodiment, the software 820 generates a spectrogram of the audio frequencies associated with the captured video data.

[0325] FIG. 45A illustrates one example of a spectrogram 4500 of a patient without gait asymmetry and / or without Parkinson’s disease. FIG. 45B illustrates one example of a spectrogram 4510 of a patient with gait asymmetry and / or with Parkinson’s disease. Footsteps can be visually identified in the spectrograms 4500 and 4510 as bright, vertical lines 4502 and can be identified in a spectral data representation as higher intensity spectral segments. Although one or more spectrograms, plots, or other data representations are visualized in respective figures for discussion, representative embodiments need not explicitly produce a visualization of spectral content and may only generate a data representation of the frequency content for analysis as discussed herein.

[0326] Normal (e.g., a non-Parkinson’s patient) footsteps produce short, high intensity sounds at regular and / or semi-regular time intervals. In contrast, for patients suffering from Parkinson’s disease, the patient 102 may shuffle or otherwise drag one or more of their feet while walking, which can be identified in the spectrograms 4500 and 4510 as bright, horizontal lines 4512 (e.g., a sound cause by a foot being dragged over a longer period of time than a normal footstep) in place of, or otherwise dominating the bright vertical lines 4502. The temporal characteristics of the footsteps can be correlated to movement of the patient’s 102 extremities identified using, for example, any of the systems and methods described herein.(32736-2156)

[0327] For example, the audio representation of the patient’s 102 foot striking the ground identified by the software 820 occurs at a particular point in time during the captured video data, and may be detected as part of a repeating pattern as the patient 102 walks in front of the camera. The software 820 may identify a repeating pattern of short duration, high intensity frequencies 4502 in the spectrogram 4500, 4510 at specific points in time, which may be correlated to the patient’s footsteps identified through analysis of the visual image frames of the captured video data. The software 820 may also identify elongated (e.g., long duration) frequencies 4512 having an intensity that is less than the high intensity frequencies 4502 in the spectrogram 4500, 4510, which may be indicative of shuffling and / or foot dragging (e.g., gait asymmetry). Correlating the audio and visual identifications of the patient’s steps may improve the accuracy of identification of gait asymmetry and / or may be used to train or otherwise improve the DNN’s described herein. In some embodiments, the audio data may be utilized on its own to identify the patient’s footsteps and determine gait asymmetry.

[0328] In some embodiments, the software 820 may generate and / or analyze a pitch contour graph 4600, 4610 (shown in FIGS. 46A and 46B) based on the generated spectrogram 4500. Changes in pitch can be used to identify how hard the patient’s foot strikes the ground while walking in front of the camera. For example, the pitch of each identified footstep may be different, such as pitch A and pitch B. Over the course of the captured video and audio data, a pattern of pitches A and B may be identified by the software 820. In this manner, if pitch B regularly follows pitch A, and vice versa, the pattern of differing pitches may be used to identify gait asymmetry. For a patient 102 without gait asymmetry, the repeating pattern may be in the form of a plurality of peaks 4522 (pitch A) and valleys 4524 (pitch B), each iteration of which having about the same peak pitch contour and duration (shown in FIG. 46A). In contrast, a patient with gait asymmetry the repeating pattern may be in the form of a plurality of plateaus 4526 (pitch A), extending over a longer duration than the peaks 4522, and valleys 4528 (pitch B). In some embodiments, the pattern may be irregular or may repeat at regular intervals.(32736-2156)

[0329] In some embodiments, the software 820 may generate and / or analyze an intensity curve graph 4700, 4710 (shown in FIGS. 47A and 47B) based on the generated spectrogram 4500. Similar to the analysis of the pitch contour graph 4600, 4610, differences or repeating patterns of different audio intensity over the course of the captured video and audio data may be used to identify gait imbalance. For example, fora patient 102 without gait asymmetry, the intensity curve 4700 may include more frequent, higher intensity peaks 4702 and for a patient 102 with gait asymmetry, the intensity curve 4710 may include less frequent, and lower intensity peaks 4712, or may be generally flat.

[0330] In some embodiments, the software 820 may analyze the audio data associated with the captured video data to identify asthenia experienced by the patient 102 while walking in front of the camera. In an example embodiment, weakness (e.g., asthenia) and / or strain experienced by the patient 102 while walking can be identified based on an analysis of the spectrogram 4500, 4510 generated from the audio data. For example, the patient 102 may alter their gait or otherwise “limp” due to pain experienced while taking a right step ora left step, or during both left and right steps. Several aspects of the captured audio signal may be used to determine and / or identify asthenia and gait asymmetry.

[0331] In an example embodiment illustrated in FIGS. 48A and 48B, the software 820 generates and plots a spectral centroid 4800 and 4810 based on the spectrogram 4500. The spectral centroid 4800, 4810 corresponds to the energy frequencies, which can be correlated to the patient’s 102 footsteps captured in the video and / or audio data of the patient 102 walking in front of the camera. For example, a higher centroid indicates that the sound has more energy in its higher frequencies, whereas a lower centroid indicates that the sound has more energy in its middle to lower frequencies. FIG. 48A illustrates a patient 102 without gait asymmetry, showing a repeating pattern of short, high spectral centroid peaks 4802 overtime. In contrast, and as illustrated in FIG. 48B, a spectral centroid curve for a patient 102 with gait asymmetry may include a repeating pattern of elongated, high spectral centroid plateaus 4812 overtime.

[0332] With reference to FIGS. 49A and 49B, the software 820 may generate and plot a spectral bandwidth 4900 and 4910 based on the spectrogram 4500. The spectral bandwidth 4900, 4910 corresponds to a frequency range-in- PCT15854WOO1(32736-2156)between a measured peak frequency and a predetermined difference in power, such as, for example, where power is half its maximum value. In one non-limiting embodiment, the spectral bandwidth 4900, 4910 indicates a 3 dB bandwidth over time. Higher, elongated spectral bandwidths 4912 may indicate shuffling orfoot dragging as the patient is walking, whereas lower, shorter spectral bandwidths 4902 may indicate normal gait patterns, without or with minimal foot dragging. For example, if the patient 102 is limping or otherwise has one foot in contact with the floor for a longer period of time than the other foot, the repeating pattern may illustrate a higher spectral bandwidth 4912 for the foot in contact with the floor for a longer period of time than the other foot. In this manner, gait asymmetry may be identified.

[0333] FIGS. 50A and 50B illustrates a plot of spectral skewness 5000 and 5010 generated by the software 820 based on the spectrogram 4500. Spectral skewness 5000, 5010 may indicate asymmetry of the audio signal’s frequency spectrum about its spectral centroid, such as, for example, the spectral centroid 4800, 4810 shown in FIGS. 48A and 48B. For example, a higher, elongated spectral skewness 5012 indicates that there are more higher frequencies in the audio signal captured as the patient 102 is walking than lower frequencies, and a lower, shorter duration spectral skewness 5002 indicates that there less higher frequencies in the audio signal than lower frequencies. The software 820 may utilize the determined spectral skewness 5000, 5010 in a similar manner to the spectral centroid 4800, 4810 and the spectral bandwidth 4900, 4910, where higher spectral skewness may indicate foot dragging and / or gait asymmetry.

[0334] Turning to FIGS. 51 A and 51 B, the software 820 may generate and plot spectral kurtosis 5100 and 5110 based on the spectrogram 4500. The spectral kurtosis 5100, 5110 may indicate an amount of the detected frequency spectrum’s power is concentrated at either high frequencies or low frequencies. For example, high, short duration spectral kurtosis 5102 may indicate energy concentrated in a few dominant frequencies having a large difference from the mean of the detected frequencies (e.g., leptokurtic) whereas lower, more elongated or more frequent spectral kurtosis 5112 may indicate energy that more evenly distributed across the detected frequencies (e.g., platykurtic). Spectral kurtosis 5100, 5110 may also help identify stationary and non-stationary audio signals, such as audio signals where(32736-2156)the frequency changes rapidly over a period of time (e.g., an unsteady signal). In this manner, high, short duration spectral kurtosis 5100 may be indicative of footsteps, which generate generally short, high energy signals, compared to a patient’s foot that is dragging or otherwise in contact with the ground over a longer period of time.

[0335] In some embodiments, a third spectral moment 5200 (shown in FIG.52A) and 5210 (shown in FIG. 52B) may be determined and used by the software 820 to determine gait asymmetry. The third spectral moment 5200, 5210 may indicate asymmetry of the audio signal’s frequency spectrum about its spectral centroid, such as, for example, the spectral centroid 4800, 4810 shown in FIGS.48A and 48B. For example, positive third spectral moments 5202 may indicate that there are more low frequencies in the audio signal captured as the patient 102 is walking than higher frequencies, and negative third spectral moments 5212 may indicate that there are more high frequencies in the audio signal than low frequencies. The software 820 may utilize the determined third spectral moment 5200, 5210 in a similar manner to the spectral centroid 4800, 4810, the spectral bandwidth 4900, 4910, and the spectral kurtosis 5000, 5010 where a negative third spectral moment may indicate foot dragging and / or gait asymmetry.

[0336] With reference to FIGS. 53A and 53B, the software 820 may generate and plot a fourth spectral moment 5300 and 5310 based on the spectrogram 4500. The fourth spectral moment 5300, 5310 is similar to the spectral kurtosis 5000, 5010, except a fourth spectral moment 5300 having a higher value 5302 may be indicative of more high frequencies or large variations of frequencies in the audio signal data, and a lower value 5312 may be indicative of a more even frequency distribution.

[0337] Turning to FIG. 54, a method of identifying gait asymmetry using a spectrogram is illustrated and generally identified by reference numeral 5400. Audio data is received 5402 from the video data captured while the patient is walking in front of the camera, and a spectrogram is generated 5404 from the received audio data. A pitch contour of the spectrogram is analyzed 5406 to identify one or more repeating pattern. Optionally, or in addition to analyzing the pitch contour, an intensity curve of the spectrogram may be analyzed 5408 to identify one or more repeating patterns. The one or more identified repeating(32736-2156)patterns are analyzed to determine 5410 if the one or more repeating patterns include constant values (e.g., no significant deviations). If the one or more identified repeating patterns include constant values, the method returns to receiving audio data 5402. If the one or more repeating patterns do not include constant values (e.g., values deviate from one portion of the repeating patter to another portion), the repeating pattern is correlated 5412 to the patient’s leg movements identified using, for example, any of the systems and methods described herein for analyzing video data, and it is determined if gait asymmetry exists 5414.

[0338] With reference to FIGS. 55A and 55B, a method of identifying gait asymmetry using spectrogram data is illustrated and generally identified by reference numeral 5500. Audio data is received 5502 from the video data captured while the patient is walking in front of the camera, and a spectrogram is generated 5504 from the received audio data. A spectral centroid based on the spectrogram is analyzed 5506 to identify if one or more repeating patterns are present in the audio data.

[0339] A spectral bandwidth based on the spectrogram may be analyzed 5508 in lieu of or in addition to the spectral centroid to determine if one or more repeating patterns are present in the audio data. A spectral skewness based on the spectrogram may be analyzed 5510 in addition to, or in lieu of, one or both of the spectral centroid or the spectral bandwidth to determine if one or more repeating patterns are present in the audio data. A spectral kurtosis based on the spectrogram may be analyzed 5512 in addition to, or in lieu of, any or all of the spectral centroid, the spectral bandwidth, or the spectral skewness to determine if one or more repeating patterns are present in the audio data. A third spectral moment based on the spectrogram may be analyzed 5514 in addition to, or in lieu of, any or all of the spectral centroid, the spectral bandwidth, the spectral skewness, or the spectral kurtosis.

[0340] A fourth spectral moment based on the spectrogram may be analyzed 5516 in addition to, or in lieu of, any or all of the spectral centroid, the spectral bandwidth, the spectral skewness, the spectral kurtosis, or the third spectral moment to determine if one or more repeating patterns are present in the audio data. It is determined 5518 if one or more of the repeating patterns has a(32736-2156)high frequency bias. If the one or more repeating patterns do not have a high frequency bias, the method returns to receiving 5502 audio data from the video data captured while the patient is walking. If the one or more repeating patterns do have a high frequency bias, the one or more repeating patterns are correlated 5520 to the patient’s leg movements identified using, for example, any of the systems and methods described herein for analyzing video data, and it is determined if gait asymmetry exists 5522.

[0341] With reference to FIGS. 56-58, processing video and audio data can be challenging at times due to multiple factors, such as the processing ability of patient devices 104 (e.g., patient controller device 800 shown in FIG. 8) and / or clinician devices 130 (e.g., clinician controller device 900 shown in FIG. 9) and the quality of the network connection, such as, without limitation, between the patient devices 104 and / or the clinician devices 130 and network 118 (shown in FIG. 1B). For example, during a virtual clinic and / or a remote programming session, poor network quality may cause transmission and / or processing delays, a loss of frames, and / or a reduction in frame resolution. Likewise, device performance and / or hardware capability can also cause delays, excess heat, and / or lagging. Such issues can make real-time analysis of the data captured during the virtual clinic and / or remote programming session challenging.

[0342] In some embodiments, dynamic switching may be used to perform the kinematic Al algorithms to address the above described network quality and device performance challenges. In an example embodiment, dynamic switching may be accomplished using two mechanisms or options to allow an optimized analysis of the data captured during the virtual clinic and / or remote programming session.

[0343] With reference to FIG. 56, in the first option 5600, the kinematics Al algorithms are performed locally in addition to rendering the raw captured video data. Overlayed video frames are transmitted to the receiving device, such as, for example, the patient device 104 and / or the clinician device 130. Landmark information may be transmitted over a chat feature and / or via a server channel, such as, for example, SignalR.(32736-2156)

[0344] With reference to FIG. 57, in the second option 5700, the raw video data is captured locally and transmitted to the receiving device, such as, for example, the patient device 104 and / or the clinician device 130. Kinematics Al algorithms and overlaying the received captured video frames with landmark information is performed on the received captured video data, as well as displaying the desired landmark information.

[0345] With additional reference to FIG. 58, dynamic switching is performed by analyzing the video data captured by the patient device 104 and comparing predetermined metrics identified in the captured video data to real-time benchmarks. The real-time benchmarks may include, without limitation, hardware capacity (e.g., GPU, RAM, CPU speeds, etc.), resource constraints (e.g., network bandwidth, cost, etc.), environmental conditions, device conditions (e.g., heating, battery life, etc.), and quality of service algorithms. Based on the comparison to the real-time benchmarks, the software 820 identifies the option 5600, 5700 that provides the best performance. Although some embodiments perform respective operations on the PC device, such operations may be performed on other devices such as the CP programmer device and / or on one or more cloud service platforms according to other embodiments.

[0347] These embodiments provide i) a method to normalize and scale the video acquired by the 2D camera to be depth invariant using body parts of the person in view, ii) a method to fix the frame of reference of analysis of motion to the skeletal frame of the person to remove motion artifacts introduced by nonlinear walk trajectory, iii) a method to compute gait asymmetry using distribution of relative coordinates of specific joints over gait cycles, iv) a method to compute relative motion of each joint with respect to a landmark with lower using a frontal video view, v) a method to assess lateral asymmetricity of specific joints during gait, vi) a method to assess longitudinal improvements in uniformity of joint motions during gait for patients with spinal cord stimulators, vii) a method to identify and correlate grimaces on a patient’s face to gait using blendshapes, viii) a method to identify and correlate auditory expressions or other auditory impulses by patients during the walking cycle, ix) a method of segmenting images captured by a camera to assess temporal and spatial attributes for gait asymmetry, x) a method for analyzing spectrograms to correlate audio with patient footfalls and identify gait(32736-2156)asymmetry, and xi) a method for fusing magnetic north to captured video data based on signals received from one or more sensors associated with the patient device.

[0348] Although the present invention and its advantages have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims. Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification. As one of ordinary skill in the art will readily appreciate from the disclosure of the present invention, processes, machines, manufacture, compositions of matter, means, methods, or steps, presently existing or later to be developed that perform substantially the same function or achieve substantially the same result as the corresponding embodiments described herein may be utilized according to the present invention. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.

[0349] Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification.

[0350] The invention is defined in the appended claims. A non-exhaustive list of aspects of the invention set out in the following clauses is useful for understanding the invention (any individual feature, step, or component of any of the clauses may be combined with one or more individual features, steps, or components of other clauses without limitation. None of such items should be construed as being required to be combined with other items unless explicitly stated as such by Applicant):

[0351] Clause 1. A method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system, the method comprising: receiving video data from a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session; identifying, from the received video data, at least one fixed edge in an environment surrounding the(32736-2156)patient; determining, based on the at least one identified fixed edge, an amount of camera tilt of the PC device relative to the patient; normalizing anatomical landmark data of the patient gait data by applying a transformation to the anatomical landmark data to compensate for the determined amount of camera tilt; and providing the normalized anatomical landmark data to one or more trained computation models to facilitate classifying or quantifying one or more patient gait characteristics.

[0352] Clause 2. The method of clause 1 , wherein identifying at least one fixed edge comprises identifying at least one vertical edge in the environment.

[0353] Clause 3. The method of clause 1 , wherein identifying at least one fixed edge comprises identifying at least one horizontal edge in the environment.

[0354] Clause 4. The method of clause 1 , wherein identifying at least one fixed edge comprises identifying at least one fixed edge using an edge detection transform.

[0355] Clause 5. The method of any preceding clause, wherein the anatomical landmark data includes a distance between a left shoulder and a right shoulder of the patient.

[0356] Clause 6. The method of any of clauses 1 through 5, wherein the anatomical landmark data includes a distance between i) a center of a hip of the patient and ii) a center of shoulders of the patient.

[0357] Clause 7. The method of any preceding clause, wherein providing the normalized anatomical landmark data comprises providing the normalized anatomical landmark data to one or more trained computation models to facilitate computing gait asymmetry for the patient.

[0358] Clause 8. The method of any of clauses 1 through 7, wherein providing the normalized anatomical landmark data comprises providing the normalized anatomical landmark data to one or more trained computation models to facilitate assessing an effectiveness of stimulation on gait correction for the patient.

[0359] Clause 9. The method of any preceding clause, further comprising causing the PC device to display a notification that instructs the patient to adjust the camera tilt of the PC device based on the determined camera tilt.(32736-2156)

[0360] Clause 10. The method of any preceding clause, further comprising causing the PC device to display a notification that instructs the patient to adjust illumination of the environment

[0361] Clause 11. A method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system, the method comprising: receiving video data from a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session; identifying, from the received video data, a fixed frame of reference as the patient moves closer to a camera of the PC device; performing, using the identified fixed frame of reference, a depth invariance transformation on anatomical landmark data of the patient gait data to generate transformed anatomical landmark data; and providing the transformed anatomical landmark data to one or more trained computation models to facilitate classifying or quantifying one or more patient gait characteristics.

[0362] Clause 12. The method of clause 11 , wherein the fixed frame of reference is associated with a center of a hip of the patient.

[0363] Clause 13. The method of clause 11 or 12, wherein performing a depth invariance transformation comprises normalizing at least one anatomical landmark point from the video data based on a reference distance.

[0364] Clause 14. The method of clause 13, wherein the reference distance is a hip width of a patient.

[0365] Clause 15. The method of clause 13, wherein the reference distance is a distance between i) a center of a hip of the patient and ii) a center of shoulders of the patient.

[0366] Clause 16. The method of any of clauses 11 through 15, wherein providing the transformed anatomical landmark data comprises providing the transformed anatomical landmark data to one or more trained computation models to facilitate computing gait asymmetry for the patient.

[0367] Clause 17. The method of any of clauses 11 through 15, wherein providing the transformed anatomical landmark data comprises providing the transformed anatomical landmark data to one or more trained computation models to facilitate assessing an effectiveness of stimulation on gait correction for the patient.(32736-2156)

[0368] Clause 18. The method of any of clauses 11 through 17, further comprising generating a scatter plot based on the transformed anatomical landmark data.

[0369] Clause 19. The method of any of clauses 11 through 18, wherein the camera of the PC device is a two-dimensional camera.

[0370] Clause 20. The method of clause 19, wherein providing the transformed anatomical landmark data comprises providing the transformed anatomical landmark data to one or more trained computation models to facilitate computing lateral asymmetry of specific joints of the patient.

[0371] Clause 21. A method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system, the method comprising: receiving video data from a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session; analyzing the received video data to detect a depth of the patient relative to a camera of the PC device; performing one or more rescaling operations on one or more frames of the received video data; identifying anatomical landmark data of the patient from the one or more rescaled frames; and providing the identified anatomical landmark data to one or more trained computation models to facilitate classifying or quantifying one or more patient gait characteristics.

[0372] Clause 22. The method of clause 21 wherein detecting the depth of the patient in the video data comprises detecting the patient in the video data using the first deep neural network.

[0373] Clause 23. The method of clause 22, wherein performing one or more rescaling operations comprises detecting the patient in the video data using the first deep neural network.

[0374] Clause 24. The method of clause 22, wherein identifying anatomical landmark data comprises identifying anatomical landmark data using a second deep neural network different from the first deep neural network.

[0375] Clause 25. The method of any of clauses 21 through 24, wherein identifying anatomical landmark data comprises identifying anatomical landmark data based on a detected latency parameter.(32736-2156)

[0376] Clause 26. The method of any of clauses 21 through 24, wherein identifying anatomical landmark data comprises identifying anatomical landmark data based on a detected accuracy parameter.

[0377] Clause 27. The method of any of clauses 21 through 26, wherein providing the identified anatomical landmark data comprises providing the identified anatomical landmark data to one or more trained computation models to facilitate computing gait asymmetry for the patient.

[0378] Clause 28. The method of any of clauses 21 through 27, wherein providing the identified anatomical landmark data comprises providing the identified anatomical landmark data to one or more trained computation models to facilitate assessing an effectiveness of stimulation on gait correction for the patient.

[0379] Clause 29. The method of any of clauses 21 through 28, wherein the video data is acquired while the patient walks towards the camera of the PC device.

[0380] Clause 30. The method of any of clauses 21 through 29, wherein the identified anatomical landmark data includes at least one of a hip location and a shoulder location of the patient.

[0381] Clause 31. A method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system, the method comprising: receiving video data from a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session; processing anatomical landmark data in the patient gait data by identifying a fixed frame of reference of the patient's anatomy proximate to a hip of the patient as the patient moves closer to a camera of the PC device; further processing the anatomical landmark data to generate a plurality of distance metrics relative to the identified fixed frame of reference; and providing the plurality of distance metrics to one or more trained computation models to facilitate classifying or quantifying one or more patient gait characteristics.

[0382] Clause 32. The method of clause 31 , wherein the fixed frame of reference is associated with a center of the hip of the patient.

[0383] Clause 33. The method of clause 31 , wherein the plurality of distance metrics include a tilt angle.(32736-2156)

[0384] Clause 34. The method of clause 33, wherein the tilt angle is defined by i) a vertical line and ii) a line between a center of the patient’s hip and a center of the patient’s shoulders.

[0385] Clause 35. The method of clause 34, wherein the vertical line is determined based on vertical edges detected in the video data.

[0386] Clause 36. The method of any of clauses 33 through 35, wherein providing the plurality of distance metrics to one or more trained computation models comprises providing the tile angle to the one or more trained computation models to identify temporal distortions in patient movement that are indicative of pain.

[0387] Clause 37. The method of any of clauses 31 through 36, wherein the camera of the PC device is a two-dimensional camera.

[0388] Clause 38. The method of any of clauses 31 through 37, wherein providing the plurality of distance metrics comprises providing the plurality of distance metrics to one or more trained computation models to facilitate computing gait asymmetry for the patient.

[0389] Clause 39. The method of any of clauses 31 through 37, wherein providing the plurality of distance metrics comprises providing the plurality of distance metrics to one or more trained computation models to facilitate assessing an effectiveness of stimulation on gait correction for the patient.

[0390] Clause 40. The method of any of clauses 31 through 39, wherein the video data is acquired while the patient walks towards the camera of the PC device.

[0391] Clause 41. A method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system, the method comprising: receiving initial video data from a patient controller (PC) device of the patient, the initial video data including initial patient gait data of the patient, the PC device comprising a camera comprising a plurality of image acquisition sensors, the initial video data acquired using a first image acquisition sensor of the plurality of image acquisition sensors; analyzing the initial video data to determine an image quality parameter; based on the determined image quality parameter, instructing the PC device to acquire subsequent video data using a second image acquisition sensor of the plurality of(32736-2156)image acquisition sensors; identifying anatomical landmark data of the patient from the subsequent video data; and providing the identified anatomical landmark data to one or more trained computation models to facilitate classifying or quantifying one or more patient gait characteristics.

[0392] Clause 42. The method of clause 41 , wherein the second image acquisition sensor has a different aperture size than the first image acquisition sensor.

[0393] Clause 43. The method of clause 41 or 42, wherein the second image acquisition sensor has a different focal length than the first image acquisition sensor.

[0394] Clause 44. The method of any of clauses 41 through 43, further comprising: storing a list of the plurality of image acquisition sensors; and selecting the second image acquisition sensor based on the list and determined image quality parameter.

[0395] Clause 45. The method of any of clauses 41 through 44, wherein analyzing the initial video data comprises determining a direction of motion of the patient in the initial video data.

[0396] Clause 46 The method of any of clauses 41 through 44, wherein analyzing the initial video data comprises detecting a location of the patient in the initial video data.

[0397] Clause 47. The method of clause 46, wherein detecting a location of the patient in the initial video data comprises detecting a location of the patient relative to a bounding box.

[0398] Clause 48. The method of any of clauses 41 through 48, wherein the plurality of image acquisition sensors comprise a telephoto sensor, an ultra wide sensor, and a wide sensor.

[0399] Clause 49. The method of any of clauses 41 through 48, wherein providing the identified anatomical landmark data comprises providing the identified anatomical landmark data to one or more trained computation models to facilitate computing gait asymmetry for the patient.

[0400] Clause 50. The method of any of clauses 41 through 49, wherein providing th...

Claims

(32736-2156)WHAT IS CLAIMED IS:

1. A method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system, the method comprising:receiving video data from a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session;identifying, from the received video data, at least one fixed edge in an environment surrounding the patient;determining, based on the at least one identified fixed edge, an amount of camera tilt of the PC device relative to the patient;normalizing anatomical landmark data of the patient gait data by applying a transformation to the anatomical landmark data to compensate for the determined amount of camera tilt; andproviding the normalized anatomical landmark data to one or more trained computation models to facilitate classifying or quantifying one or more patient gait characteristics.

2. The method of claim 1 , wherein identifying at least one fixed edge comprises identifying at least one vertical edge in the environment.

3. The method of claim 1 , wherein identifying at least one fixed edge comprises identifying at least one horizontal edge in the environment.

4. The method of claim 1 , wherein identifying at least one fixed edge comprises identifying at least one fixed edge using an edge detection transform.

5. The method of any preceding claim, wherein the anatomical landmark data includes a distance between a left shoulder and a right shoulder of the patient.(32736-2156)6. The method of any of claims 1 through 5, wherein the anatomical landmark data includes a distance between i) a center of a hip of the patient and ii) a center of shoulders of the patient.

7. The method of any preceding claim, wherein providing the normalized anatomical landmark data comprises providing the normalized anatomical landmark data to one or more trained computation models to facilitate computing gait asymmetry for the patient.

8. The method of any of claims 1 through 7, wherein providing the normalized anatomical landmark data comprises providing the normalized anatomical landmark data to one or more trained computation models to facilitate assessing an effectiveness of stimulation on gait correction for the patient.

9. The method of any preceding claim, further comprising causing the PC device to display a notification that instructs the patient to adjust the camera tilt of the PC device based on the determined camera tilt.

10. The method of any preceding claim, further comprising causing the PC device to display a notification that instructs the patient to adjust illumination of the environment.

11. A method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system, the method comprising:receiving video data from a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session;identifying, from the received video data, a fixed frame of reference as the patient moves closer to a camera of the PC device;performing, using the identified fixed frame of reference, a depth invariance transformation on anatomical landmark data of the patient gait data to generate transformed anatomical landmark data; and(32736-2156)providing the transformed anatomical landmark data to one or more trained computation models to facilitate classifying or quantifying one or more patient gait characteristics.

12. The method of claim 11 , wherein the fixed frame of reference is associated with a center of a hip of the patient.

13. The method of claim 11 or 12, wherein performing a depth invariance transformation comprises normalizing at least one anatomic landmark point based on a reference distance.

14. The method of claim 13, wherein the reference distance is a hip width of a patient.

15. The method of claim 13, wherein the reference distance is a distance between i) a center of a hip of the patient and ii) a center of shoulders of the patient.

16. The method of any of claims 11 through 15, wherein providing the transformed anatomical landmark data comprises providing the transformed anatomical landmark data to one or more trained computation models to facilitate computing gait asymmetry for the patient.

17. The method of any of claims 11 through 15, wherein providing the transformed anatomical landmark data comprises providing the transformed anatomical landmark data to one or more trained computation models to facilitate assessing an effectiveness of stimulation on gait correction for the patient.

18. The method of any of claims 11 through 17, further comprising generating a scatter plot based on the transformed anatomical landmark data.

19. The method of any of claims 11 through 18, wherein the camera of the PC device is a two-dimensional camera.(32736-2156)20. The method of claim 19, wherein providing the transformed anatomical landmark data comprises providing the transformed anatomical landmark data to one or more trained computation models to facilitate computing lateral asymmetry of specific joints of the patient.

21. A method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system, the method comprising:receiving video data from a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session;analyzing the received video data to detect a depth of the patient relative to a camera of the PC device;performing one or more rescaling operations on one or more frames of the received video data;identifying anatomical landmark data of the patient from the one or more rescaled frames; andproviding the identified anatomical landmark data to one or more trained computation models to facilitate classifying or quantifying one or more patient gait characteristics.

22. The method of claim 21 , wherein performing one or more rescaling operations comprises performing one or more rescaling operations using a first deep neural network.

23. The method of claim 22, wherein detecting the depth of the patient in the video data comprises detecting the patient in the video data using the first deep neural network.

24. The method of claim 22, wherein identifying anatomical landmark data comprises identifying anatomical landmark data using a second deep neural network different from the first deep neural network.(32736-2156)25. The method of any of claims 21 through 24, wherein identifying anatomical landmark data comprises identifying anatomical landmark data based on a detected latency parameter.

26. The method of any of claims 21 through 24, wherein identifying anatomical landmark data comprises identifying anatomical landmark data based on a detected accuracy parameter.

27. The method of any of claims 21 through 26, wherein providing the identified anatomical landmark data comprises providing the identified anatomical landmark data to one or more trained computation models to facilitate computing gait asymmetry for the patient.

28. The method of any of claims 21 through 27, wherein providing the identified anatomical landmark data comprises providing the identified anatomical landmark data to one or more trained computation models to facilitate assessing an effectiveness of stimulation on gait correction for the patient.

29. The method of any of claims 21 through 28, wherein the video data is acquired while the patient walks towards the camera of the PC device.

30. The method of any of claims 21 through 29, wherein the identified anatomical landmark data includes at least one of a hip location and a shoulder location of the patient.

31. A method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system, the method comprising:receiving video data from a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session;processing anatomical landmark data in the patient gait data by identifying a fixed frame of reference of the patient's anatomy proximate to a hip of the patient as the patient moves closer to a camera of the PC device;(32736-2156)further processing the anatomical landmark data to generate a plurality of distance metrics relative to the identified fixed frame of reference; and providing the plurality of distance metrics to one or more trained computation models to facilitate classifying or quantifying one or more patient gait characteristics.

32. The method of claim 31 , wherein the fixed frame of reference is associated with a center of the hip of the patient.

33. The method of claim 31 , wherein the plurality of distance metrics include a tilt angle.

34. The method of claim 33, wherein the tilt angle is defined by i) a vertical line and ii) a line between a center of the patient’s hip and a center of the patient’s shoulders.

35. The method of claim 34, wherein the vertical line is determined based on vertical edges detected in the video data.

36. The method of any of claims 33 through 35, wherein providing the plurality of distance metrics to one or more trained computation models comprises providing the tile angle to the one or more trained computation models to identify temporal distortions in patient movement that are indicative of pain.

37. The method of any of claims 31 through 36, wherein the camera of the PC device is a two-dimensional camera.

38. The method of any of claims 31 through 37, wherein providing the plurality of distance metrics comprises providing the plurality of distance metrics to one or more trained computation models to facilitate computing gait asymmetry for the patient.

39. The method of any of claims 31 through 37, wherein providing the plurality of distance metrics comprises providing the plurality of distance metrics to one or more trained computation models to facilitate assessing an effectiveness of stimulation on gait correction for the patient.(32736-2156)40. The method of any of claims 31 through 39, wherein the video data is acquired while the patient walks towards the camera of the PC device.

41. A method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system, the method comprising:receiving initial video data from a patient controller (PC) device of the patient, the initial video data including initial patient gait data of the patient, the PC device comprising a camera comprising a plurality of image acquisition sensors, the initial video data acquired using a first image acquisition sensor of the plurality of image acquisition sensors;analyzing the initial video data to determine an image quality parameter; based on the determined image quality parameter, instructing the PC device to acquire subsequent video data using a second image acquisition sensor of the plurality of image acquisition sensors;identifying anatomical landmark data of the patient from the subsequent video data; andproviding the identified anatomical landmark data to one or more trained computation models to facilitate classifying or quantifying one or more patient gait characteristics.

42. The method of claim 41 , wherein the second image acquisition sensor has a different aperture size than the first image acquisition sensor.

43. The method of claim 41 or 42, wherein the second image acquisition sensor has a different focal length than the first image acquisition sensor.(32736-2156)44. The method of any of claims 41 through 43, further comprising: storing a list of the plurality of image acquisition sensors; andselecting the second image acquisition sensor based on the list and determined image quality parameter.

45. The method of any of claims 41 through 44, wherein analyzing the initial video data comprises determining a direction of motion of the patient in the initial video data.46 The method of any of claims 41 through 44, wherein analyzing the initial video data comprises detecting a location of the patient in the initial video data.

47. The method of claim 46, wherein detecting a location of the patient in the initial video data comprises detecting a location of the patient relative to a bounding box.

48. The method of any of claims 41 through 48, wherein the plurality of image acquisition sensors comprise a telephoto sensor, an ultra wide sensor, and a wide sensor.

49. The method of any of claims 41 through 48, wherein providing the identified anatomical landmark data comprises providing the identified anatomical landmark data to one or more trained computation models to facilitate computing gait asymmetry for the patient.

50. The method of any of claims 41 through 49, wherein providing the identified anatomical landmark data comprises providing the identified anatomical landmark data to one or more trained computation models to facilitate assessing an effectiveness of stimulation on gait correction for the patient.(32736-2156)51. A method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system, the method comprising:receiving video data from a patient controller (PC) device of the patient, the video data including patient gait data of the patient acquired by the PC device during the VC session;identifying anatomical landmark data from the patient gait data;generating, based on the identified anatomical landmark data, movement data indicative of movement of the patient;performing dynamic time warping operations on the movement data to generate one or more metrics indicating a degree of asymmetry of matching warped points within a time series; andclassifying or quantifying one or more patient gait characteristics of the patient using the one or more metrics of asymmetry.

52. The method of claim 51 , wherein the movement data is indicative of movement of a limb of the patient.

53. The method of claim 51 or 52, wherein the video data is acquired while the patient walks towards a camera of the PC device.

54. The method of any of claims 51 through 53, wherein performing dynamic time warping is performed using a full gait cycle approach.

55. The method of any of claims 51 through 53, wherein performing dynamic time warping is performed using a moving leg approach.

56. The method of any of claims 51 through 55, further comprising normalizing the identified anatomical landmark data.

57. The method of any of claims 51 through 56, further comprising automatically discarding at least one frame of the video data.(32736-2156)58. The method of any of claims 51 through 57, further comprising:detecting abnormal movement from the movement data; andgenerating an alert notifying a user of the detected abnormal movement.

59. The method of any of claims 51 through 58, wherein providing the generated metrics comprises providing the generated metrics to one or more trained computation models to facilitate computing gait asymmetry for the patient.

60. The method of any of claims 51 through 59, wherein providing the generated metrics comprises providing the generated metrics to one or more trained computation models to facilitate assessing an effectiveness of stimulation on gait correction for the patient.

61. A method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system, the method comprising:receiving video data from a patient controller (PC) device of the patient, the video data including patient gait data and patient facial data of the patient acquired by the PC device during the VC session;identifying first anatomical landmark data from the patient gait data;generating, based on the first anatomical landmark data, movement data indicative of movement of the patient;identifying second anatomical landmark data from the patient facial data;generating, based on the second anatomical landmark data, expression data indicative of one or more facial expressions of the patient;correlating repetition of one or more of the one or more facial expressions to relative time positions over multiple gait cycles of the patient using the movement data and the expression data; and(32736-2156)classifying or quantifying one or more patient gait characteristics using the correlated repetition.

62. The method of claim 61 wherein the classifying or quantifying calculates a degree of asymmetry of patient gait characteristics.

63. The method of any of claims 61 and 62 where the correlating comprises correlating one or more facial expression to limb movement.

64. The method of claim 63 wherein the correlating comprises: identifying movement of an anatomical feature of the patient correlated to pain using the facial expression data and the movement data.

65. The method of any of claims 61-64 comprising:comparing a first level of pain on a first side of a patient and second level of pain on a second side of the patient using the facial expression data and calculating a degree of gait asymmetry using the first level and the second level.

66. The method of any of claims 61-65 wherein the generating, based on the second anatomical landmark data, expression data indicative of one or more facial expressions of the patient comprises:comparing the second anatomical landmark data to one or more predefined templates defining muscle movement that are related to one or more respective facial expressions.

67. The method of claim 66 wherein the one or more predefined templates are defined to relate to composite facial expressions.

68. The method of any of claims 61-67 further comprising: performing dynamic time warping to at least one of the movement data and the expression data.-MO- PCT15854WOO1(32736-2156)69. The method of any of claims 61 through 68, wherein the classifying or quantifying one or more patient gait characteristics comprises providing the movement data and expression data to one or more computational models.

70. The method of any of claims 61 through 69, wherein providing the generated metrics comprises providing the generated metrics to one or more trained computational models to facilitate assessing an effectiveness of stimulation on gait correction for the patient.

71. A method of conducting a virtual clinic (VC) session between a patient and a clinician to facilitate medical services using a VC cloud computing system, the method comprising:receiving video data and audio data from a patient controller (PC) device of the patient, the video data including patient gait data;identifying anatomical landmark data from the patient gait data; generating, based on the first anatomical landmark data, movement data indicative of movement of the patient;processing the audio data to identify frequency content associated with the patient interaction with a floor;correlating patient interaction with the floor to the patient gait data over multiple gait cycles of the patient; andclassifying or quantifying one or more patient gait characteristics using the correlated patient interaction.

72. The method of claim 71 wherein the classifying or quantifying calculates a degree of asymmetry of patient gait characteristics.

73. The method of any of claims 71 and 72 where the correlating comprises correlating frequency content indicative of foot dragging within the multiple gait cycles.-MI- PCT15854WOO1(32736-2156)74. The method of any of claims 71-73 where the correlating comprises correlating frequency content indicative of shuffling within the multiple gait cycles.

75. The method of any of claims 71-74 where the correlating comprises correlating frequency content indicative of foot strikes with the ground within the multiple gait cycles.

76. The method of claim 75 wherein identifying frequency content indicative of foot strikes comprises identifying changes in pitch in the audio data.

77. The method of claim 76 further comprising: measuring differences in pitch between respective foot strikes to identify gait asymmetry.

78. The method of any of claims 71-77 wherein the processing the audio data to identify frequency content associated the patient interaction with the floor comprises:generating a spectrogram representation of the audio data and identifying a time series of spectral centroid across multiple gait cycles.

79. The method of any of claims 71-78 further comprising: generating an intensity curve representing levels of audio intensity across multiple gait cycles.

80. The method of any of claims 71-79 further comprising:generating a time series representative of spectral skewness or spectral kurtosis across multiple gait cycles.

81. A system comprising a patient controller device, a clinician programmer device, one or more cloud servers, and an implantable medical device for implementing any of the preceding methods.

82. A system comprising a patient controller device, a clinician programmer device, one or more cloud servers, and an implantable medical device, where the system is configured for remote programming of the implantable(32736-2156)medical device, wherein the implantable medica device is configured to facilitate therapy for a patient, wherein the system is configured to implement:establishing a first communication between a patient controller, PC, device and the implantable medical device, wherein the implantable medical device is configured to facilitate therapy to the patient according to one or more programmable parameters, the PC device communicates signals to the implantable medical device to set or modify the one or more programmable parameters, and the PC device comprises a video camera;establishing a video connection between the PC device and a clinician programmer, CP, device of a clinician for a remote programming session in a second communication that includes an audio / video, A / V session;communicating a value fora respective programmable parameter of the medical device from the CP device to the PC device during the remote programming session; andmodifying, by the PC device, the respective programming parameter of the implantable medical device according to the communicated value from the CP device during the remote programming session,wherein the patient controller is configured to process video data from the video camera to generate landmark data indicative of anatomical locations of the patient, communicate the landmark data to one or more of the PC device and one or more cloud servers in a separate data format from video data in the A / V session.

83. The method of claim 82 wherein the landmark data is communicated according to an asynchronous cloud communication protocol.

84. The method of any of claims 82 and 83 further comprising: downscaling video data from the video camera of the PC device before communicating the video data in the A / V session.(32736-2156)85. The method of claim 84 further comprising:conducting the downscaling by calculating one or more real-time session characteristic benchmarks to determine whether a respective video resolution may be supported by the A / V session.

Images