Methods and systems for closed-loop, artificial intelligence-enabled cognitive therapy

Abstract

Provided are methods comprising providing a sensory environment to an individual, identifying a cognitive and neural status of the individual, and providing an altered sensory environment to the individual to influence the cognitive and neural status. In certain embodiments, one or more machine learning models are used to identify the cognitive and neural status of the individual and / or to alter the sensory environment. According to some embodiments, providing a sensory environment to the individual comprises presenting to the individual a visual stimulus, an auditory stimulus, a tactile stimulus, an olfactory stimulus, a gustatory stimulus, or any combination thereof.

Description

[0001] METHODSAND SYSTEMS FOR CLOSED-LOOP, ARTIFICIAL INTELLIGENCE-ENABLED COGNITIVE THERAPY CROSS-REFERENCE To RELATED APPLICATION

[0002] This application claims benefit under 35 U.S.C. § 119(e) of provisional applications 63 / 748,332, filed January 22, 2025, which applications are hereby incorporated by reference in their entirety.

[0003] INT ODUCTION

[0004] Understanding the neurophysiological mechanism underlying emotion, mood, attention, and arousal sheds light on the mental well-being of individuals which could inform therapeutic solutions for the neuropsychiatric population. (See e.g., P. Lombardo, et al., BMC Public Health, vol. 18, no. 1, p. 342, Mar. 2018; the disclosure of which is incorporated by reference in its entirety.) Although studying human emotion, mood, attention, and arousal at a basic science level and across disciplines has a prolonged history, it has encountered significant challenges. One of such challenges is the subjective variations across individuals in response to identical paradigms that evoke cognitive responses. (See e.g., A. S. Cowen and D. Keltner, Proc. Natl. Acad. Sci., vol. 114, no. 38, pp. E7900-E7909, Sep. 2017; D. Keltner, et al. J. Nonverbal Behav., vol. 43, no.

[0005] 2, pp. 195-201, Jun. 2019; and A. Cowen, et al. Psychol. Sci. Public Interest, vol. 20, no. 1, pp.

[0006] 69-90, Jul. 2019; the disclosures of which are incorporated by reference in their entireties.) Psychological studies have surveyed various stimuli ranging from images (e.g., International affective picture system), videos, music, as well as autobiographical memories, imagery, and situational procedures. Recently, utilizing cloud-based technologies, many of these stimuli were surveyed across large populations to increase the generalizability of the induced emotion and cognition across individuals. However, these surveys are limited to self-reports, which are prone to uncertainties such as usage of language, background culture, societal definitions, and memory biases. (See e.g., A. Althubaiti, J. Multidiscip. Healthc., vol. 9, pp. 211-217, May 2016; the disclosure of which is incorporated by reference in its entirety.) Thus, there is a need in the field for methods and systems to study emotional and cognitive experience more efficiently and designing personalized paradigms that incorporate content tuning based on each individual’s experience in a systematic way.SUMMARY

[0007] Provided are methods comprising providing a sensory environment to an individual, identifying a cognitive and neural status, of the individual, and providing an altered sensory environment to the individual to influence the cognitive and neural status. In certain embodiments, one or more machine learning models are used to identify the cognitive and neural status of the individual and / or to alter the sensory environment. According to some embodiments, providing a sensory environment to the individual comprises presenting to the individual a visual stimulus, an auditory stimulus, a tactile stimulus, an olfactory stimulus, a gustatory stimulus, or any combination thereof. In certain embodiments, providing an altered sensory environment to the individual comprises providing a customized sensory environment to the individual in real-time based on the individual’s cognitive and neural status. According to some embodiments, the individual is suffering from a mental health condition selected from depression, anxiety, post-traumatic stress disorder (PTSD), addiction, and any combination thereof. Systems that find use in practicing the methods of the present disclosure are also provided.

[0008] BRIEF DESCRIPTION OF THE FIGURES

[0009] Figure 1 is a flow diagram illustrating steps of a method according to one embodiment of the present disclosure.

[0010] DETAILED DESCRIPTION

[0011] Before the methods and systems of the present disclosure are described in greater detail, it is to be understood that the methods and systems are not limited to particular embodiments described, as such may, of course, vary. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting, since the scope of the methods and systems will be limited only by the appended claims.

[0012] Where a range of values is provided, it is understood that each intervening value, to the tenth of the unit of the lower limit unless the context clearly dictates otherwise, between the upper and lower limit of that range and any other stated or intervening value in that stated range, is encompassed within the methods and systems. The upper and lower limits of these smaller ranges may independently be included in the smaller ranges and are also encompassed within the methods and systems, subject to any specifically excluded limit in the stated range. Where the stated range includes one or both of the limits, ranges excluding either or both of those included limits are also included in the methods and systems.Certain ranges are presented herein with numerical values being preceded by the term “about.” The term “about” is used herein to provide literal support for the exact number that it precedes, as well as a number that is near to or approximately the number that the term precedes. In determining whether a number is near to or approximately a specifically recited number, the near or approximating unrecited number may be a number which, in the context in which it is presented, provides the substantial equivalent of the specifically recited number.

[0013] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the methods and systems belong. Although any methods and systems similar or equivalent to those described herein can also be used in the practice or testing of the methods and systems, representative illustrative methods and systems are now described.

[0014] All publications and patents cited in this specification are herein incorporated by reference as if each individual publication or patent were specifically and individually indicated to be incorporated by reference and are incorporated herein by reference to disclose and describe the materials and / or methods in connection with which the publications are cited. The citation of any publication is for its disclosure prior to the filing date and should not be construed as an admission that the present methods and systems are not entitled to antedate such publication, as the date of publication provided may be different from the actual publication date which may need to be independently confirmed.

[0015] It is noted that, as used herein and in the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. It is further noted that the claims may be drafted to exclude any optional element. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as “solely,” “only” and the like in connection with the recitation of claim elements, or use of a “negative” limitation.

[0016] It is appreciated that certain features of the methods and systems, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the methods and systems, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable sub-combination. All combinations of the embodiments are specifically embraced by the present disclosure and are disclosed herein just as if each and every combination was individually and explicitly disclosed, to the extent that such combinations embrace operable processes and / or compositions. In addition, all sub-combinations listed in the embodiments describing such variables are also specifically embraced by the present methods and systemsand are disclosed herein just as if each and every such sub-combination was individually and explicitly disclosed herein.

[0017] As will be apparent to those of skill in the art upon reading this disclosure, each of the individual embodiments described and illustrated herein has discrete components and features which may be readily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present methods. Any recited method can be carried out in the order of events recited or in any other order that is logically possible.

[0018] Definitions

[0019] When describing the methods and compositions of the present disclosure, the following terms include the following meanings unless otherwise indicated within the present disclosure, but the terms are not to be understood to be limited to their accompanying meaning as rather it is to be understood to encompass any meaning in accordance with the teachings and present disclosure.

[0020] The term “cognition,” as used herein, can include, but is not limited to, domains such as perception, attention, memory, motor function, problem solving, language processing, decision making, emotional regulation, and intelligence.

[0021] The term “task” refers to a behavior to be accomplished by an individual who provides a response to a particular stimulus that may include a goal and / or objective. For example, the individual may be instructed to perform a specific behavior to achieve a particular goal. The “task” can serve as the baseline cognitive function that is being performed and, optionally, measured, which induces a particular neural activity, including e.g., activity in a particular brain region. Thus, a “task” often refers to the main behavior that an individual is instructed to perform, which will include a mental component and may or may not include a physical component. For example, in some instances, a task may include a mental component of identifying or recognizing a particular stimulus and may or may not require a physical component of responding to the mental component, e.g., indicating that the stimulus has been identified or recognized, e.g., by performing a physical action such as pressing a button or verbally identifying the stimulus.

[0022] The term “real-time” or “real time” refers to the immediate processing or action on data or events as they happen, with minimal delay. For example, it refers to systems or processes that can respond instantly or within milliseconds to new data inputs or changes, as opposed to delayed or batch processing.METHODS

[0023] The present disclosure provides methods, comprising providing a sensory environment (i.e., an environment presenting a stimulus to one or more sense) to an individual, identifying a cognitive and neural status of the individual, altering the sensory environment to influence a change in the cognitive and neural status of the individual, providing the altered sensory environment to the individual, and identifying an altered cognitive and neural status of the individual in response to altering the sensory environment. The methods of the present disclosure (as well as the related systems and computer-readable media) constitute an advancement in precision clinical tools. In certain embodiments, neural data is monitored and then the sensory environment (or “context”) of the individual is manipulated (e.g., via changing sensory stimulation) to optimize treatment effects. This advance is fundamental to advancing a new generation of highly effective, accessible, and personalized therapies. Such therapies can have uses, including (but not limited to) enhancing cognition, cognitive therapy, and treating a cognitive disorder. Cognitive disorders can include neurodegenerative disorders (e.g., Alzheimer’s Disease, Parkinson’s Disease Dementia, frontotemporal dementia, Lewy body dementia, Huntington’s disease), delirium Wernicke- Korsakoff Syndrome, traumatic brain injuries, (e.g., post-concussion syndrome, chronic traumatic encephalopathy), developmental disorders (e.g., intellectual disabilities, autism spectrum disorder, attention deficit disorder, attention deficit hyperactivity disorder), psychiatric disorders (e.g., schizophrenia), substance-induced cognitive impairment, vascular disorders (e.g., vascular dementia, stroke induced cognitive impairment), HIV-associated neurocognitive disorder, multiple sclerosis, Creutzfeldt- Jakob disease, amnestic disorders, mild cognitive impairment, chronic fatigue syndrome, among others.

[0024] According to some embodiments, the methods of the present disclosure are computer-implemented. By “computer-implemented” is meant at least one step of the method is implemented using one or more processors and one or more non-transitory computer-readable media. The computer-implemented methods of the present disclosure may further comprise one or more steps that are not computer-implemented.

[0025] In certain embodiments, the methods comprise a closed-loop experience in which the sensory environment presented to the individual (context) is customized in real time based upon their own cognitive and emotional state measured in the moment (recording). This approach allows for personalized and precisely-targeted experiences that are adaptively adjusted to create the ideal content, challenges, and rewards to harness each brain’s inherent plasticity to optimize function. Recent advances in digital technologies (sensory presentation devices, physiologicalrecording sensors, and machine learning algorithms) may be implemented to generate powerful, closed-loop experiences, unlocking entirely new treatment approaches.

[0026] In various embodiments, providing altering a multisensory environment comprises the use of a generative artificial intelligence (genAI) to create and / or manipulate (e.g., develop, create, change, etc.) the multisensory environment Various instances utilize a large language model (LLM) trained to generate a multisensory environment based on a provided prompt and / or in response to a physiological, psychological, and / or cognitive state of an individual. Some instances utilize an LLM trained to manipulate an extant multisensory environment in response to a physiological, psychological, and / or cognitive state of an individual.

[0027] Sensory Environment

[0028] The methods of the present disclosure comprise providing a sensory environment to an individual and altering the sensory environment based on the individual’s cognitive and neural status, physiological status, and / or emotional state. In the present context, “providing” includes presenting a static environment (e.g., image) to the individual and / or presenting a dynamic environment that allows interaction from the individual. Figure 1 provides an exemplary flow chart of an embodiment. Providing a sensory environment to the individual comprises presenting one or more sensory stimuli to the individual occurs at 102. In certain embodiments, providing a sensory environment to the individual comprises presenting to the individual one or more visual stimuli, one or more auditory stimuli, one or more tactile stimuli, one or more olfactory stimuli, one or more gustatory stimuli, or any combination thereof.

[0029] For simplicity, a number of embodiments are herein described with the presentation of a single stimulus. However, as will be readily understood and is further described below, in many instances neural activity may be monitored during the presentation of a plurality of stimuli including but not limited to e.g., 2 or more stimuli, 3 or more stimuli, 4 or more stimuli, 5 or more stimuli, 6 or more stimuli, 7 or more stimuli, 8 or more stimuli, 9 or more stimuli, 10 or more stimuli, 11 or more stimuli, 12 or more stimuli, 13 or more stimuli, 14 or more stimuli, 15 or more stimuli, 16 or more stimuli, 17 or more stimuli, 18 or more stimuli, 19 or more stimuli, 20 or more stimuli, etc.

[0030] Multiple stimuli may be presented individually, e.g., with an intermission between them, or may be presented as a defined series of stimuli, e.g., without an intermission between two or more of the stimuli. In some instances, series of stimuli may be presented in “blocks,” e.g., without an intermission between the stimuli of the series but with an intermission between blocks. Whenpresented as a series of stimuli, neural activity may be monitored before, during and / or after the series. As such, the monitoring may be time-locked to all or any portion of the series including but not limited to e.g., where the monitoring is time-locked to each stimulus of the series, to the first stimulus of the series, to the last stimulus of the series, to some interval of a portion of stimuli of the series (e.g., every other stimulus, every third stimulus, etc.), to a combination thereof, etc. It will be understood that, where appropriate, a stimulus described herein in the singular may be exchanged for a stimulus presented as a series of stimuli and vice versa.

[0031] Examples of visual stimuli include, but are not limited to, presenting to the individual one or more flashes of light, one or more shapes (e.g., colored or non-colored shapes), one or more colors, one or more scenes (e.g., a nature scene, a city scene, a live music scene, a scene from outer space, a scene that includes elements from the individual’s past and / or present living environment), one or more digital photos (e.g., one or more digital images from the individual’s past (e.g., the individual’s childhood), the individual’s family and / or friends, etc.), and / or the like.

[0032] When providing a sensory environment to the individual comprises presenting to the individual one or more visual stimuli, in certain embodiments, the one or more visual stimuli are presented to the individual via one or more displays. Non-limiting examples of displays that may be employed include a television (e.g., a high-definition television (HDTV)), a monitor, a projection screen, the display of a mobile device (e.g., tablet computer or smartphone), a head-mounted display, or any other suitable display for presentation of visual stimuli to the individual. According to some embodiments, one or more visual stimuli are presented to the individual via a headmounted display. Non-limiting examples of head-mounted displays include headsets or glasses, for example, virtual / augmented reality headsets or virtual / augmented reality glasses. A headmounted display that finds use in the methods and systems of the present disclosure include Microsoft’s HoloLens® 2 head-mounted display, or equivalent. In certain embodiments, when a head-mounted display is employed, the sensory environment presented to the individual comprises a virtual reality environment or augmented reality environment.

[0033] An “auditory stimulus” refers to a sound and may be characterized by, for example, frequency, loudness (i.e., intensity), timbre, or any parametric combination of these or any other sound features. Examples of auditory stimuli include, but are not limited to, presenting to the individual one or more tones, a rhythm, a beat, music (vocal, instrumental, or combinations thereof), nature sounds, one or more voices, and / or the like. When presenting a sensory environment to the individual comprises presenting to the individual one or more auditory stimuli, in certain embodiments, the one or more auditory stimuli are presented to the individual via one or more speakers, headphones (including wired or wireless headphones), and / or the like.Tactile stimuli are those related to the sense of touch and include, but are not limited to, objects that the individual touches with her / his hand(s), foot / feet, or other body part. Tactile stimuli also include stimuli that vibrate the individual or a portion thereof, including but not limited to, vibrations emitted from a chair, sofa, bed, and / or the like.

[0034] Olfactory stimuli are those relating to the sense of smell. When presenting a sensory environment to the individual comprises presenting to the individual one or more olfactory stimuli, in certain embodiments, the one or more olfactory stimuli comprise the smell of a plant (e.g., a tree, flower, fruit, and / or the like), aromatic essential oils, a food, and / or the like.

[0035] Gustatory stimuli are those relating to the sense of taste. Presenting a gustatory stimulus can include providing an edible item to the individual, such as a piece of food. Such items may be provided for consumption (e.g., masticating and / or swallowing) or solely as contact to a surface containing a taste bud (e.g., a tongue). Some instances provide an extract, a flavoring (natural and / or artificial), or other compound capable of stimulating a food.

[0036] A combination of stimuli from different sensory systems may be presented to the individual in the methods of the instant disclosure. For example, both auditory and visual stimuli may be presented concurrently or in sequence. The series can also present a sequence of auditory and visual stimuli that can be synchronized or unsynchronized. In a discrimination task, a target set can contain either a target auditory or a target visual stimulus or both. For example, a target stimulus may be a combination of the visual stimulus of a green circle as well as the spoken word “circle” as the auditory stimulus. Any of the above stimuli may, in certain instances, also serve as a distractor in a cognitive task, e.g., where the stimulus is a non-target stimulus meant to interfere with completion of the goal of the cognitive task as in, e.g., interfere with a subject’s sustained attention to or recognition of a target stimuli.

[0037] Identifying a Cognitive and Neural Status

[0038] The methods of the present disclosure comprise, during the providing of the sensory environment to an individual, identifying a cognitive and neural status of the individual at 104. Accordingly, in certain embodiments, the cognitive and neural status of the individual is monitored. The cognitive and neural status of the individual may be indicative of the cognitive and neural status of the individual comprises one or more of stress, mood, attention, arousal, concentration, awareness, mindfulness, flow, alertness, fatigue, confusion, memory, comprehension, decisionmaking, problem-solving, perception, distraction, insight, creativity, curiosity, daydreaming, overwhelmed, cognitive dissonance, and / or the like experienced by the individual during thepresentation of the sensory environment. In certain embodiments, the cognitive and neural status of the individual is identified using multi-modal physiological data using a machine learning model. In many instances, the cognitive and neural status includes one or more of a cognitive state and an emotional state. Additionally, the machine learning model is trained to interpret multi-modal physiological data to determine a cognitive and neural status selected from one or more of a cognitive state and an emotional state.

[0039] In many instances, the multi-modal physiological data is a noninvasive method, i.e. , where no device or portion thereof is implanted into the individual or under the skin of the individual. For example, in some instances, neural activity may be monitored using a EEG device that contacts, but does not penetrate, the scalp of the individual, or a noninvasive imaging device such as, e.g., a fMRI device. In certain embodiments, the cognitive and neural status of the individual is monitored by EEG using an EEG device (e.g., an EEG cap, such as a 64-channel EEG cap (or “headset”)) worn by the individual to enable recording of voltage fluctuations resulting from ionic current flows within the neurons of the brain at desired intervals, including in real-time, during presentation of the sensory environments.

[0040] According to some embodiments, the method employed for identifying a cognitive and neural status may be an invasive or minimally invasive method where a neural activity detection device, or a portion thereof, is implanted into the subject, including e.g., into the brain of the subject, onto the surface of the brain of the subject, under the skin of the scalp of the subject, etc. For example, in some instances, the electrodes of an ECoG device may be place or implanted onto the surface of the brain of the individual.

[0041] In certain embodiments, detected neural activity may be “co-registered” (i.e., “mapped”) onto a reference map or model of a brain. The reference map of the brain may be a general reference map or an individual-specific reference map. For example, in instances where an individual-specific reference map is used, the method may include mapping the individual’s brain with one or more brain imaging techniques and overlaying the detected neural activity onto the individual-specific map. A reference brain map of an individual may be obtained prior to monitoring cognitive and neural status during presentation of the sensory environments. Alternatively, a reference brain map may be obtained during the monitoring of the herein described methods, including, e.g., where the method of monitoring cognitive and neural status simultaneously produces a subject-specific brain map (e.g., as in fMRI) or where a neural activity monitoring technique is combined with a second technique for brain imaging (e.g., combined MRI and EEG recording).According to some embodiments, multi-modal physiological data high-density EEG is mapped in real-time or near real-time (e.g., computational lag of less than 1 second, including but not limited to e.g., less than 500 milliseconds, less than 400 milliseconds, less than 300 milliseconds, less than 200 milliseconds, between 200 to 100 milliseconds, etc.) onto a previously acquired Diffusion Tensor Imaging (DTI) 3D reconstruction of the individual’s brain. By “high-density EEG” is meant at least 64-channel EEG, however, EEG sensor density may vary and may include, in some instances, greater than 64-channel EEG (e.g., 128-channel EEG), less than 64-channel EEG (e.g., 32-channel EEG, 24-channel EEG, etc). In some instances, different physiologically relevant frequency bands may be differentiated including e.g., where only certain bands are displayed, where different bands are displayed at different intensities (including absolute intensities, relative intensities, threshold intensities, etc.), where different bands are displayed in different colors. Different physiologically relevant frequency bands include physiologically relevant alpha frequency bands (e.g., 8-12 Hz), physiologically relevant beta frequency bands (e.g., 12-20 Hz) and physiologically relevant theta frequency bands (e.g., 4-8 Hz). In some instances, before display, neural activity may be corrected for irrelevant interference including but not limited to e.g., ocular artifacts, muscular artifacts, etc. In some instances, effective connectivity may be and mapped and / or visualized calculated in real-time or near realtime onto a previously acquired reference brain model, including e.g., a DTI 3D reconstruction.

[0042] In some instances, an emotional state is selected from happiness, excitement, love, calmness, gratitude, relief, amusement, pride, contentment, euphoria, sadness, anger, fear, anxiety, frustration, jealousy, envy, shame, guilt, loneliness, disgust, hopelessness, regret, apathy, bitterness, confusion, surprise, anticipation, embarrassment, reluctance, and any combination thereof. In certain instances, emotional state or status may be determined using a verified survey, such as a Patient Health Questionnaire-9 (PHQ-9), Generalized Anxiety Disorder-7 (GAD-7), Depression Anxiety Stress Scales (DASS-21), Beck Depression Inventory (BDI), Profile of Mood States (POMS), Social Phobia Inventory (SPIN), Panic Disorder Severity Scale (PDSS), and any combination thereof.

[0043] In some instances, a cognitive state is selected from attention-related states (e.g., focused attention, divided attention, selective attention, sustained attention, distracted), memory-related states (e.g., short-term memory, long-term memory retrieval, working memory, forgetfulness, recollection, recognition, amnesia), learning and problem solving states (e.g., understanding, misunderstanding, insight, confusion, analysis, synthesis, logical reasoning, problem-solving), emotional influences on cognition (e.g., stress or anxiety that impacts focus, calmness that enhances clarity, motivation curiosity, overwhelm), perceptual states (e.g., sensory perception,illusion or hallucination, hyper-awareness, unawareness), metacognitive states (e.g., self-awareness, reflection, confidence in knowledge, doubt, overconfidence), decision-making and judgment states (e.g., deliberation intuition, hesitation, certainty, indecision), states of consciousness (e.g., fully awake, daydreaming hypnosis, meditation, sleep states, unconsciousness), creative and imaginative states (e.g., imagination, creativity, innovation, mental stimulation, brainstorming), other cognitive states (e.g., flow state, cognitive overload, cognitive dissonance, rumination, mind-wandering, zoning out), and any combination thereof. In some instances, the list of cognitive states may be reduced for convenience; as such, some embodiments select a cognitive state from attention, concentration, awareness, mindfulness, flow, alertness, fatigue, confusion, memory, comprehension, decision-making, problem-solving, perception, distraction, insight, creativity, curiosity, daydreaming, overwhelmed, cognitive dissonance, and any combination thereof.

[0044] In various instances, neural state may include a physiological state. In many instance, the machine learning model may further be trained to interpret the multi-modal physiological data to determine the cognitive and neural status based on the physiological state. Such physiological state may include homeostasis, resting, wakefulness, sleep, exercise, fatigue, relaxation, physical stress, recovery, tension, hunger, satiety, digestion, thirst, fight or flight, rest and digest, freezing, fever, hypothermia, hyperthermia, pain, inflammation, itch, tachycardia, bradycardia, hypertension, hypotension, hyperventilation, dyspnea, adrenaline rush, cortisol response, hypoglycemia, hyperglycemia, anabolism, catabolism, arousal, orgasm, menstruation, pregnancy, lactation, and / or the like experienced by the individual during the presentation of the sensory environment. Physiological state may be measured by systems or mechanisms to determine blood oxygen saturation, pulse rate, blood pressure, blood glucose level, blood lactate levels, tidal volume (of lungs), tidal carbon dioxide, respiration rate, expiratory flow rate, grip pressure, and / or any other applicable physiologic metric. Physiological metrics may be measured using applicable devices, such as one or more of a pulse oximetry, electromyography (EMG), respirometer, glucometer, implantable devices (e.g., continuous glucose monitoring), wearable devices (e.g., smart devices), other applicable devices or systems, and combinations thereof.

[0045] In certain embodiments, the methods of the present disclosure comprise, during the presenting of the sensory environment to an individual, monitoring the physiological state of the individual. The physiological state of the individual may be indicative of the stress, mood, attention, arousal, and / or the like experienced by the individual during the presentation of the sensory environment. According to some embodiments, monitoring the physiological state of the individual comprises monitoring one or more of the individual’s heart rate, heart rate variability,blood pressure, electrodermal activity (EDA) (e.g., galvanic skin response (GSR)), movement, eye movement, pupillary response, facial expression, and any combination thereof.

[0046] In certain embodiments, heart rate and / or heart rate variability are monitored using a mobile heart rate detection device, such as a device worn by the individual during presentation of the sensory environment. According to some embodiments, the device is worn on the individual’s wrist, an example of which includes the Peak fitness tracker by Basis, Inc. and the V800 sports watch by Polar Electro, Inc.

[0047] According to some embodiments, monitoring the physiological state of the individual comprises monitoring movement of the individual. Any convenient sensors or devices may be employed to detect the individual’s body movement, including wired or wireless motion sensors worn by the individual, a motion / position capture device, e.g., a motion / position capture device that includes a depth camera to capture depth images of feature points of interest on the individual's body, and the like.

[0048] In certain embodiments, the individual’s movement is monitored using sensors (e.g., wired or wireless sensors) worn on the body of the individual. Such sensors may be worn on one or both of the individual’s arms, one or both of the individual’s legs, and / or any other useful location on the individual’s body for monitoring the individual’s movement and / or position during the presentation of the sensory environment. In certain embodiments, the sensors are “wireless” sensors that communicate wirelessly with the system being implemented to the carry out the method. Such wireless sensors are known and include the PrioVR motion-tracking body suits available from YEI Corporation (Portsmouth, Ohio).

[0049] According to some embodiments, the individual’s movement is monitored using a motion / position capture device during the presentation of the sensory environment. A non-limiting example of a motion capture device includes a video camera, such as an RGB video camera and / or a depth camera. Movements made by the individual are captured in a sequence of images and then processed. The motion capture device, alone or in conjunction with a second component (e.g., a video game console) may perform functions including, but not limited to, detecting and performing gesture recognition on the individual's movements and monitoring direction and relative distance moved by the individual. Images of the individual's movements may be stored for analysis.

[0050] Further details regarding motion capture devices for detecting movement are found, e.g., in U.S. Patent No. 8,113,991, U.S. Patent Application Publication No. US 2010 / 0199228, U.S. Patent Application Publication No. US 2013 / 0342527, and U.S. Patent Application Publication No.US 2014 / 0244008, the disclosures of which are incorporated herein in their entireties for all purposes.

[0051] Motion / position capture devices that find use in monitoring movement are available and include Microsoft’s Kinect motion capture device (e.g., the Microsoft Kinect for Windows v2), Sony's EyeToy® motion capture device, and the like.

[0052] Electrodermal activity (e.g., electrodermal responses) of interest include, but are not limited to, the galvanic skin response (GSR). In certain embodiments, GSR is detected using a mobile GSR detection device, such as a device worn by the individual during presentation of the sensory environment. According to certain embodiments, the device is worn on the individual’s wrist, an example of which includes the Peak fitness tracker by Basis, Inc.

[0053] Suitable approaches for detecting the individual’s pupillary response include capturing images of one or both of the individual’s pupils during presentation of the sensory environment. In certain embodiments, a motion capture device employed to detect the subject’s body movement is also used to detect the individual’s pupillary response during presentation of the sensory environment. Pupillary responses of interest include constriction (miosis) and / or dilation (mydriasis).

[0054] Accordingly, one or more physiological parameters of the individual may be monitored using a wearable device, one or more cameras, and / or the like.

[0055] According to some embodiments, monitoring the cognitive and neural status, the physiological state, the emotional state, the cognitive state, or a combination thereof, of the individual is performed continuously for a period of time and is used to monitor the state of the individual in real-time, where the cognitive and neural status of the individual comprises one or more of stress, mood, attention, arousal, concentration, awareness, mindfulness, flow, alertness, fatigue, confusion, memory, comprehension, decision-making, problem-solving, perception, distraction, insight, creativity, curiosity, daydreaming, overwhelmed, cognitive dissonance, and any combination thereof. In some instances, an emotional state is selected from happiness, excitement, love, calmness, gratitude, relief, amusement, pride, contentment, euphoria, sadness, anger, fear, anxiety, frustration, jealousy, envy, shame, guilt, loneliness, disgust, hopelessness, regret, apathy, bitterness, confusion, surprise, anticipation, embarrassment, reluctance, and any combination thereof. In some instances, a physiological state being monitored includes homeostasis, resting, wakefulness, sleep, exercise, fatigue, relaxation, physical stress, recovery, tension, hunger, satiety, digestion, thirst, fight or flight, rest and digest, freezing, fever, hypothermia, hyperthermia, pain, inflammation, itch, tachycardia, bradycardia, hypertension,hypotension, hyperventilation, dyspnea, adrenaline rush, cortisol response, hypoglycemia, hyperglycemia, anabolism, catabolism, arousal, orgasm, menstruation, pregnancy, lactation, and any combination thereof.

[0056] In certain embodiments, the individual is suffering from a mental health condition selected from the group consisting of: depression, anxiety, post-traumatic stress disorder (PTSD), addiction, pain, and any combination thereof. When the individual is suffering from a mental health condition, according to some embodiments, the method is effective in treating the mental health condition. By “treat” or “treatment” is meant at least an amelioration of one or more symptoms associated with the mental health condition of the individual, where amelioration is used in a broad sense to refer to at least a reduction in the magnitude of a parameter, e.g., symptom, associated with the mental health condition being treated. As such, treatment also includes situations where the mental health condition, or at least one or more symptoms associated therewith, are completely inhibited, e.g., prevented from happening, or stopped, e.g., terminated, such that the individual no longer suffers from the mental health condition, or at least the symptoms that characterize the mental health condition.

[0057] According to many embodiments of the present disclosure, one or more steps may be computer-implemented. In certain embodiments, a computer system may perform the monitoring the multi-modal physiological data and present the monitoring data to a therapist, where the therapist visualizes the monitoring data (e.g., on a display) and presents one or more altered sensory environments to the individual based on the monitoring data. The therapist may present the one or more altered sensory environments solely based on her / his judgment based on the monitoring data. Alternatively, or additionally, in certain embodiments, a computer system implementing one or more steps of the method may suggest one or more alterations to the sensory environment based on the monitoring data, where the therapist presents a modified sensory environment to the individual based on one or more of the modifications suggested by the computer system.

[0058] Generative Al-Enabled Sensory Alteration

[0059] At 106, many embodiments provide an altered sensory environment to the individual. The altered sensory environment may be used to influence the cognitive and neural status of the individual — a non-limiting example of influencing a cognitive and neural status is to direct the cognitive and neural status to a healthier and / or improved state. In many embodiments, the altered sensory environment is generated using one or more machine learning models. In many embodiments, a machine learning model is a generative artificial intelligence (GenAI) model. TheGenAI model of some embodiments may generate and / or alter the sensory environment. In many instances, the GenAI model is a large language model (LLM). Such machine learning may be trained to alter the sensory environment based on the identified cognitive and neural status. Some embodiments further alter the sensory environment based on a desired cognitive and neural status. For example, if a desired cognitive and neural status is congruent with the identified cognitive and neural status, the GenAI model may alter the sensory environment to support or reinforce the individual’s identified cognitive and neural status. As another example, if a desired cognitive and neural status is incongruent with the identified cognitive and neural status, the GenAI model may alter the sensory environment to challenge or change the individual’s identified cognitive and neural status.

[0060] In some instances input for the GenAI model may be based on the output of the “first” machine learning model — i.e., the model used to identify a cognitive and neural status. In such instances, the first machine learning model can generate a composite score, which is used as a control signal by the GenAI model to modify the sensory environment.

[0061] Altering the sensory environment can use an existing model, including (but not limited to) one or more of OpenAI / Microsoft’s GPT-4, Google’s Gemini, OpenAI / Microsoft’s DALL-E, Anthropic’s Claude, Google’s Imagen, OpenAI / Microsoft’s JukeBox, Google’s MusicLM, OpenAI / Microsoft’s Point-E, Google’s DreamFusion, any other applicable model, and combinations thereof. Exemplary combinations may utilize a first model to develop a prompt or cognitive scenario, while a second model may be used to illustrate, animate, narrate, and / or generate any other stimulus. It should also be noted that the initial sensory environment — e g., as described in the section “Sensory Environment” may be generated using a GenAI model as described herein.

[0062] Providing the altered sensory environment to the individual at 106 may be followed by identifying an altered cognitive and neural status of the individual at 108. The providing of the altered sensory environment and identifying an altered cognitive and neural status may be performed in accordance with the sections entitled "Sensory Environment” and “Identifying a Cognitive and Neural Status” above.

[0063] Additionally, the features described in the present section (i.e., providing of the altered sensory environment and identifying an altered cognitive and neural status) may be iterated. Such features may be iterated until a desired cognitive and neural status is achieved — as described above, such cognitive and neural statuses can be congruent or incongruent. Such iteration can allow such methods to be Al-driven closed-loop system, such that it will continue to iterate withouthuman intervention until a desired end point (e.g., length time, number of iterations, achieved state, etc ). Additionally, the process of altering the sensory environment, providing of the altered sensory environment, and identifying an altered cognitive and neural status may occur in real time.

[0064] Alteration of sensory environments based on cognitive and neural status of individuals may be performed using any suitable approach. Adaptive methodologies / algorithms that may be employed include, but are not limited to, parameter estimation by sequential testing (PEST), maximum-likelihood procedures, and staircase procedures. According to certain embodiments, the modification / adaptation is achieved using PEST, which is characterized by an algorithm for threshold searching that changes both step sizes and direction (i.e., increasing and decreasing level) across a set of trials. Changes in step size are used to focus the adaptive track ever more finely, stopping the track when the estimate has been adequately defined. The final estimate is simply the final value determined by the trial placement procedure. The PEST algorithm is designed to place trials at the most efficient locations along the stimulus axis in order to increase measurement precision while minimizing the number of trials required to estimate a threshold.

[0065] According to certain embodiments, sensory environment alteration is achieved using an adaptive staircase algorithm / procedure. Staircase procedures generally use the previous one or more responses within an adaptive track to select the next trial placement, then provide a threshold estimate in a variety of ways, e.g., by averaging the levels at the direction reversals in the adaptive track (i.e., the turnaround points). Up-down staircases call for a reduction in stimulus level when the subject’s response (e.g., altered cognitive and neural status) is positive and an increase in stimulus level when the response is negative. Beginning at a level above threshold, positive responses lead to continued decreases in stimulus level until a negative response occurs. This triggers a reversal in the direction of the track, and levels on subsequent trials increase until the next change in response. The up-down staircase procedure targets the 50% performance level on a function that extends from 0% correct performance at chance to 100% correct performance. That is, the track targets the stimulus level for which the probability of a correct response equals the probability of a negative response or, equivalently, the level at which the track would move up or down on the stimulus axis with equal probability.

[0066] Therapeutic Methods

[0067] As summarized above, included in the present disclosure are methods of treating psychological and / or emotional conditions, including (but not limited to) depression, anxiety, PTSD, addition, etc. The treatment methods include presenting to a subject having apsychological and / or emotional disorder with a sensory environment alone or as part of a task. Based on a response to the sensory environment — e.g., cognitive and neural status, physiological state, cognitive state, emotional state, etc., the sensory environment may be altered then provided back to the individual. In various instances, the altered sensory environment is supportive of the individual’s emotional state (e.g., to reinforce a healthy state). In some instances, the altered sensory environment challenges the individual’s emotional state, such as to drive an individual away from an unhealthy state and / or toward a healthier state. The particular method steps may be performed using any of the approaches described herein. As with methods of cognitive enhancement, methods of treatment may be iterated until a desired cognitive and neural status is achieved — as described above, such cognitive and neural statuses can be congruent or incongruent. Such iteration can allow such methods to be Al-driven closed-loop system, such that it will continue to iterate without human intervention until a desired end point (e.g., length time, number of iterations, achieved state, etc.). As such methods of treatment may take any suitable approach, including adaptive methodologies / algorithms as described previously. In many instances, methods of treatment are effective in treating the mental condition.

[0068] Additional assessments of the treatment effects may use processes as described above, including surveys / assessments, including those described previously (e.g., GAD-7, PHQ-9, etc.), ADHD self-report scale, Positive and Negative Affect Schedule, PTSD Checklist, and any other types of surveys that can be conducted for a subject to report on their general feelings of symptoms of a condition or satisfaction with real-world functional status or improvement. In some instances, a third party (e.g., mental health professional may provide an assessment as to treatment success and effect.

[0069] Use of Methods in Conjunction with Other Therapeutics and Diagnostics

[0070] The methods described in the instant application can be used alone or with other interventions which are known to improve cognition and / or treat diseases and conditions. Other interventions include drugs as well as psychotherapeutic techniques. Sessions and training programs described herein may be used either consecutively or simultaneously with the other interventions. When used consecutively, the sessions and training programs can be used either prior to the other intervention or after the other intervention. The sessions and training programs may be used with one or multiple interventions.

[0071] Drug therapies that could be used in combination with the herein described methods or systems include, but are not limited to cholinesterase inhibitors, memantine, anti-depressants (e.g., selective serotonin-reuptake inhibitors, norepinephrine reuptake inhibitors, monoamineoxidase inhibitors, etc.), anxiolytics (e.g., benzodiazepines, buspirone, barbiturates, etc.) and antipsychotics.

[0072] Psychotherapeutic techniques that could be used in combination with the herein described methods or systems include, but are not limited to, behavior therapy, psychodynamic therapy, psychoanalytic therapy, group therapy, family counseling, art therapy, music therapy, vocational therapy, humanistic therapy, existential therapy, transpersonal therapy, client-centered therapy (also called person-centered therapy), Gestalt therapy, biofeedback therapy, rational emotive behavioral therapy, reality therapy, response based therapy, Sandplay therapy, status dynamics therapy, hypnosis and validation therapy.

[0073] SYSTEMSAND COMPUTER READABLE MEDIA

[0074] As summarized above, the present disclosure also provides systems. In certain embodiments, provided are systems for enhancing cognition. Such systems comprise one or more processors, an output device for providing the sensory environment and altered sensory environment to the individual, an input device for receiving multi-modal physiological data from the individual, and a non-transitory computer-readable medium. The non-transitory computer-readable medium comprises instructions stored thereon that cause the system to provide a sensory environment to an individual, identify a cognitive and neural status of the individual, alter the sensory environment, providing the altered sensory environment, and identifying an altered cognitive and neural status of the individual. Additionally, such systems can include an output device for providing the sensory environment and altered sensory environment to the individual and an input device for receiving multi-modal physiological data from the individual.

[0075] In certain embodiments, the output device is configured to provide one or more stimuli selected from: an auditory stimulus, a visual stimulus, a tactile stimulus, an olfactory stimulus, and a gustatory stimulus. When providing a visual stimulus, an output device can be selected from one or more of a video display, a virtual reality headset, and an augmented reality headset. When providing an auditory stimulus, an output device can include one or more of a speaker and a headphone. When providing a tactile stimulus, an output device can include one or more of haptic gloves, a vibramotor, a haptic body suit, a tactile electronic display, and a linear actuator. Gustatory and / or olfactory stimuli may be provided by any appropriate mechanism.

[0076] For input devices to obtain the multi-modal physiological data, the device can be any relevant device to the desired inputs, including one or more of electroencephalography (EEG) device, and electromyography (EMG) device, a functional magnetic resonance imaging (fMRI) device, a near-infrared spectroscopy (NIRS) device, an electrocortocography (ECoG) device, anelectrocardiography (EKG) device, a near-infrared spectroscopy (NIRS) device, an electrocortocography (ECoG) device, a blood glucometer, a spirometer, a sphygmomanometer, a camera, a LiDAR device, a blood lactate analyzer, a pulse oximeter, a movement tracker, an eye tracker, and a facial expression monitor.

[0077] Additional devices to obtain multi-modal physiological data can include one or more devices for measuring galvanic response, heart-rate, heart-rate variability, blood pressure, respiration, perspiration, oxygen saturation, expiratory flow rate, tidal volume, tidal carbon dioxide, blood glucose level, blood lactate levels, pupil dilation, eye movement, grip pressure, electrodermal activity, movement, and facial expression.

[0078] Additional systems may include a user interface configured to relay the multi-modal physiological data and / or the sensory environment to a medical practitioner. Such user interfaces can include one or more of an electronic display, a paper readout, a keyboard, a mouse, and a joystick.

[0079] According to some embodiments, the non-transitory computer-readable medium comprises instructions stored thereon that cause the system to providing a sensory environment to an individual. The sensory environment may be provided to an individual using one or more of the output devices described above depending on which stimuli (e.g., visual, auditory, tactile, gustatory, olfactory) and how many stimuli are desired.

[0080] In certain embodiments, the non-transitory computer-readable medium comprises instructions stored thereon that cause the system to identify a cognitive and neural status of the individual based on multi-modal physiological data from the individual using a first machine learning model, where the cognitive and neural status includes one or more of a cognitive state and an emotional state, where the first machine learning model is trained to interpret multi-modal physiological data to determine a cognitive and neural status selected from one or more of a cognitive state and an emotional state. Identifying the cognitive and neural status using multimodal physiological data can receive the multi-modal physiological data from one or more of the input devices described above. This multi-modal physiological data may be analyzed using a machine learning model.

[0081] According to some embodiments, the non-transitory computer-readable medium comprises instructions stored thereon that cause the system to alter the sensory environment to influence a change in the cognitive and neural status of the individual, where altering the sensory environment comprises utilizing a generative artificial intelligence (GenAI) model. In certain embodiments, the non-transitory computer-readable medium comprises instructions storedthereon that cause the system to provide the altered sensory environment to the individual using one or more of the output devices previously described. In some instances, the altering of the sensory environment and the providing of the sensory environment occur in real-time based on the identifying of the neural state using multi-modal physiological data.

[0082] According to some embodiments, the non-transitory computer-readable medium comprises instructions stored thereon that cause the system to identify an altered cognitive and neural status of the individual in response to altering the sensory environment. Identifying the altered cognitive and neural status using multi-modal physiological data can receive the multimodal physiological data from one or more of the input devices described above. This multi-modal physiological data may be analyzed using a machine learning model.

[0083] As noted, according to some embodiments, the non-transitory computer-readable medium comprises one or more machine learning models. As described herein, one machine learning model can be used to identify a cognitive and neural status. In various embodiments, a machine learning model may generate and / or alter the sensory environment. In certain embodiments, two machine learning models (e.g., one to identify a cognitive and neural status and one to generate and / or alter a sensory environment) may be trained jointly to form one unifying model. In some instances, a single machine learning model is capable of both interpreting a cognitive and neural status and generating or altering a sensory environment based on the cognitive and neural status.

[0084] Non-transitory physical computer readable media of the present disclosure include, but are not limited to, disks (e.g., magnetic or optical disks), solid-state storage drives, cards, tapes, drums, punched cards, barcodes, and magnetic ink characters and other physical medium that may be used for storing representations, instructions, and / or the like.

[0085] A system of the present disclosure may include one or more input or output devices (as described herein) electrically connected, either wired or wirelessly, to a computing device having a logic subsystem. The logic subsystem may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processors of the logic subsystem may be single-core or multi-core, and the programs executed thereon may be configured for sequential, parallel, or distributed processing. The logic subsystem may include individual components that are distributed among two or more devices, which can be remotely located and / or configured for coordinated processing. Aspects of the logic subsystem may be virtualized and executed by remotely accessible networked computing devices configured in a cloud-computing configuration.The system may further include a storage subsystem that includes one or more physical, non-transitory, devices configured to hold data and / or instructions executable by the logic subsystem to implement the methods of the present disclosure. When such methods and processes are implemented, the state of storage subsystem may be transformed, e.g., to hold different data.

[0086] The storage subsystem may include removable media and / or built-in devices. Storage subsystems may include optical memory devices (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory devices (e.g., RAM, EPROM, EEPROM, etc.) and / or magnetic memory devices (e.g., hard-diskdrive, floppy-disk drive, tape drive, MRAM, etc.), among others. Storage subsystems may include volatile, nonvolatile, dynamic, static, read / write, read-only, randomaccess, sequential-access, location-addressable, file-addressable, and / or content-addressable devices.

[0087] Storage subsystems may include one or more physical, non-transitory devices. However, in some embodiments, aspects of the instructions described herein may be propagated in a transitory fashion by a pure signal, e.g., an electromagnetic or optical signal, etc. that is not held by a physical device for a finite duration. Furthermore, data and / or other forms of information pertaining to the present disclosure may be propagated by a pure signal.

[0088] According to certain embodiments, aspects of the logic subsystem and of the storage subsystem may be integrated together into one or more hardware-logic components. Such hardware-logic components may include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC / ASICs), program- and application-specific standard products (PSSP / ASSPs), system-on-a-chip (SOC) systems, and complex programmable logic devices (CPLDs), for example.

[0089] Non-transitory computer-readable media such as those described above with respect to the systems of the present disclosure are also provided.

[0090] A variety of processor-based devices / systems may be employed to implement the embodiments of the present disclosure. Such systems may include system architecture wherein the components of the system are in electrical communication with each other using a bus. System architecture can include a processing unit (CPU, GPU, or other processor), as well as a cache, that are variously coupled to the system bus. The bus couples various system components including system memory, (e.g., read only memory (ROM) and random access memory (RAM), to the processor.System architecture can include a cache of high-speed memory connected directly with, in close proximity to, or integrated as part of the processor. System architecture can copy data from the memory and / or the storage device to the cache for quick access by the processor. In this way, the cache can provide a performance boost that avoids processor delays while waiting for data. These and other modules can control or be configured to control the processor to perform various actions. Other system memory may be available for use as well. Memory can include multiple different types of memory with different performance characteristics. Processor can include any general purpose processor and a hardware module or software module, such as first, second and third modules stored in the storage device, configured to control the processor as well as a special-purpose processor where software instructions are incorporated into the actual processor design. The processor may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multicore processor may be symmetric or asymmetric.

[0091] To enable user interaction with the computing system architecture, an input device can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech and so forth. An output device can also be one or more of a number of output mechanisms. In some instances, multimodal systems can enable a user to provide multiple types of input to communicate with the computing system architecture. A communications interface can generally govern and manage the user input and system output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

[0092] The storage device is typically a non-volatile memory and can be a hard disk or other types of computer-readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, random access memories (RAMs), read only memory (ROM), and hybrids thereof.

[0093] The storage device can include software modules for controlling the processor. Other hardware or software modules are contemplated. The storage device can be connected to the system bus. In one aspect, a hardware module that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as the processor, bus, output device, and so forth, to carry out various functions of the disclosed technology.Embodiments within the scope of the present disclosure may also include tangible and / or non-transitory computer-readable storage media or devices for carrying or having computerexecutable instructions or data structures stored thereon. Such tangible computer-readable storage devices can be any available device that can be accessed by a general purpose or special purpose computer, including the functional design of any special purpose processor as described above. By way of example, and not limitation, such tangible computer-readable devices can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other device which can be used to carry or store desired program code in the form of computer-executable instructions, data structures, or processor chip design. When information or instructions are provided via a network or another communications connection (either hardwired, wireless, or combination thereof) to a computer, the computer properly views the connection as a computer-readable medium. Thus, any such connection is properly termed a computer-readable medium. Combinations of the above should also be included within the scope of the computer-readable storage devices.

[0094] Computer-executable instructions include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Computer-executable instructions also include program modules that are executed by computers in stand-alone or network environments. Generally, program modules include routines, programs, components, data structures, objects, and the functions inherent in the design of special-purpose processors, etc. that perform tasks or implement abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of the program code means for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps.

[0095] Other embodiments of the disclosure may be practiced in network computing environments with many types of computer system configurations, including personal computers, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. Embodiments may also be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by a combination thereof) through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.For purposes of completeness, various aspects of the present disclosure are set out in the following numbered clauses.

[0096] Aspect 1. A method, comprising:

[0097] (a) providing a sensory environment to an individual;

[0098] (b) identifying a cognitive and neural status of the individual based on multi-modal physiological data from the individual using a first machine learning model, wherein the cognitive and neural status includes one or more of a cognitive state and an emotional state, wherein the first machine learning model is trained to interpret multi-modal physiological data to determine a cognitive and neural status selected from one or more of a cognitive state and an emotional state; and

[0099] (c) providing an altered sensory environment to the individual to influence the cognitive and neural status of the individual, wherein the altered sensory environment is generated using a generative artificial intelligence (GenAI) model.

[0100] Aspect 2. The method of Aspect 1 , wherein the first machine learning model and the GenAI model are trained jointly to form a unifying model.

[0101] Aspect 3. The method of Aspect 1 or 2, wherein the cognitive and neural status includes a physiological state of the individual based on the multi-modal physiological data, wherein the first machine learning model is further trained to interpret the multi-modal physiological data to determine a cognitive and neural status based on a physiological state.

[0102] Aspect 4. The method of any one of Aspects 1-3, wherein the multi-modal physiological data comprises one or more of the following: electroencephalography (EEG), electromyography (EMG), functional magnetic resonance imaging (fMRI), galvanic response, heart-rate, heart-rate variability, blood pressure, respiration, perspiration, oxygen saturation, electrocardiography (EKG), expiratory flow rate, tidal volume, tidal carbon dioxide, blood glucose level, blood lactate levels, pupil dilation, eye movement, grip pressure, near-infrared spectroscopy (NIRS), electrocortocography (ECoG), electrodermal activity, movement, and facial expression.

[0103] Aspect 5. The method of any one of Aspects 1-4, wherein a portion of the multi-modal physiological data is obtained from a wearable device.

[0104] Aspect 6. The method of any one of Aspects 1-5, wherein a portion of the multi-modal physiological data is obtained from a camera.

[0105] Aspect 7. The method of any one of Aspects 1-6, further comprising (d) identifying an altered cognitive and neural status of the individual in response to altering the sensory environment.

[0106] Aspect 8. The method of Aspect 7, further comprising iterating (c) and (d).Aspect 9. The method of Aspect 7 or 8, further comprising iterating (c) through (d) until a desired cognitive and neural status is achieved.

[0107] Aspect 10. The method of Aspect 9, wherein the desired neural status is incongruent with the cognitive and neural status identified in (b).

[0108] Aspect 11. The method of Aspect 9, wherein the desired cognitive and neural status is congruent with the cognitive and neural status identified in (b).

[0109] Aspect 12. The method of any one of Aspects 1-11, wherein the GenAI model alters the sensory environment based on the cognitive and neural status or altered cognitive and neural status of the individual

[0110] Aspect 13. The method of Aspect 12, wherein the generative Al model is a large language model.

[0111] Aspect 14. The method of Aspect 12 or 13, further comprising generating input for the GenAI model based on the output of the first machine learning model.

[0112] Aspect 15. The method of Aspect 14, wherein the first machine learning model generates a composite score, which is used as a control signal by the GenAI model to modify the sensory environment.

[0113] Aspect 16. The method of any one of Aspects 1-15, wherein the sensory environment comprises one or more of auditory, visual, tactile, olfactory, or gustatory stimuli.

[0114] Aspect 17. The method of any one of Aspects 1-16, wherein the sensory environment incorporates virtual reality (VR) or augmented reality (AR).

[0115] Aspect 18. The method according to any one of Aspects 1-17, wherein one or more steps of the method are computer-implemented.

[0116] Aspect 19. The method of any one of Aspects 1-18, wherein the cognitive state is selected from one or more of: attention, concentration, awareness, mindfulness, flow, alertness, fatigue, confusion, memory, comprehension, decision-making, problem-solving, perception, distraction, insight, creativity, curiosity, daydreaming, overwhelmed, and cognitive dissonance.

[0117] Aspect 20. The method of any one of Aspects 1-19, wherein the emotional state is selected from one or more of: happiness, excitement, love, calmness, gratitude, relief, amusement, pride, contentment, euphoria, sadness, anger, fear, anxiety, frustration, jealousy, envy, shame, guilt, loneliness, disgust, hopelessness, regret, apathy, bitterness, confusion, surprise, anticipation, embarrassment, and reluctance.

[0118] Aspect 21. The method of any one of Aspects 3-20, wherein the physiological state is selected from one or more of: homeostasis, resting, wakefulness, sleep, exercise, fatigue, relaxation, physical stress, recovery, tension, hunger, satiety, digestion, thirst, fight or flight, restand digest, freezing, fever, hypothermia, hyperthermia, pain, inflammation, itch, tachycardia, bradycardia, hypertension, hypotension, hyperventilation, dyspnea, adrenaline rush, cortisol response, hypoglycemia, hyperglycemia, anabolism, catabolism, arousal, orgasm, menstruation, pregnancy, and lactation.

[0119] Aspect 22. The method of any one of Aspects 1-21, wherein providing the sensory environment and altering the sensory environment occurs in real time.

[0120] Aspect 23. The method of any one of Aspects 1-22, wherein the individual is suffering from a mental health condition selected from the group consisting of: depression, anxiety, post-traumatic stress disorder (PTSD), addiction, pain, and any combination thereof.

[0121] Aspect 24. The method of Aspect 23, wherein the method is effective in treating the mental health condition.

[0122] Aspect 25. A system comprising:

[0123] an output device for providing a sensory environment to an individual;

[0124] an input device for receiving multi-modal physiological data from the individual;

[0125] a processor and a non-transitory computer memory, wherein the memory comprises instructions that when executed by the processor, direct the processor to perform the methods of any one of Aspects 1-24.

[0126] Aspect 26. The system of Aspect 25, wherein the output device is configured to provide one or more of an auditory stimulus, a visual stimulus, a tactile stimulus, an olfactory stimulus, and a gustatory stimulus.

[0127] Aspect 27. The system of Aspect 25 or 26, wherein the output device comprises a video display, a virtual reality headset, or an augmented reality headset.

[0128] Aspect 28. The system of any one of Aspects 25-27, wherein the output device comprises a speaker or a headphone.

[0129] Aspect 29. The system of any one of Aspects 25-28, wherein the output device comprises haptic gloves, a vibramotor, a haptic body suit, a tactile electronic display, and a linear actuator. Aspect 30. The system of any one of Aspects 25-29, wherein the output device comprises a scent delivery system or a system configured to provide a gustatory stimulus.

[0130] Aspect 31. The system of any one of Aspects 25-30, wherein the input device comprises one or more of an electroencephalography (EEG) device, and electromyography (EMG) device, a functional magnetic resonance imaging (fMRI) device, a near-infrared spectroscopy (NIRS) device, an electrocortocography (ECoG) device, an electrocardiography (EKG) device, a nearinfrared spectroscopy (NIRS) device, an electrocortocography (ECoG) device, a blood glucometer, a spirometer, a sphygmomanometer, a camera, a LiDAR device, a blood lactateanalyzer, a pulse oximeter, a movement tracker, an eye tracker, and a facial expression monitor.

[0131] Aspect 32. The system of any one of Aspects 25-30, wherein the system comprises one or more devices for measuring galvanic response, heart-rate, heart-rate variability, blood pressure, respiration, perspiration, oxygen saturation, expiratory flow rate, tidal volume, tidal carbon dioxide, blood glucose level, blood lactate levels, pupil dilation, eye movement, grip pressure, electrodermal activity, movement, and facial expression.

[0132] Aspect 33. The system of any one of Aspects 25-32, further comprising a user interface configured to relay the multi-modal physiological data and / or the sensory environment to a medical practitioner.

[0133] Aspect 34. The system of Aspect 33, wherein the user interface comprises one or more of: an electronic display, a paper readout, a keyboard, a mouse, and a joystick.

[0134] Aspect 35. A non-transitory, machine-readable media comprising instructions that when executed by a computer processor, direct the processor to perform the method of any one of Aspects 1-24.

[0135] The following examples are offered by way of illustration and not by way of limitation.

[0136] EXPERIMENTAL

[0137] Example 1 - Development of a multisensorv stimulus generation strategy using human-in-the-loop and generative Al

[0138] This example focuses on emotion, mood, attention, and arousal responses. It is hypothesized that generative machine learning is able to produce emotionally evocative stimuli and / or cognitive responses at a personalized level. To test this hypothesis, a multi-sensory environment is generated by generative Al technology (examples of existing architectures: Microsoft DALLE 3 and Google Gemini). The Al-generated multisensory environment is shown to a participant. Based on the participant’s ratings and their personal preference (e.g., experienced emotion, mood, attention, or arousal responses), the prompt for the next trial is another stimulus. This paradigm is repeated up to five times for each given target cognition.

[0139] For feasibility purposes, this experiment is limited number of basic cognitive states such as amusement, contentment, and adoration that have positive valence (but may differ in the arousal dimension) and are more likely to produce cognitive responses such as laughter and mirk. In this phase, self-reports are used as the only prompt to design the system in real-time.As an alternative, previously identified stimuli (e.g., video) are used to generate a response in the individual. Ratings regarding these videos (e.g., amusement, adoration, and contentment) and their intensities may be used as a control signal for future trials with the subject individual.

[0140] Example 2 - Integration of facial expression and multimodal peripheral physiology as objective readouts to the stimulus-generative Al strategy

[0141] To examine whether the personalized closed-loop stimulus generation strategy is effective, psychophysiological readouts (i.e., ECG, EEG, EDA) are combined with behavioral expressions such as facial cues and self-reports. This method measures the nuances of the cognitive experience in the generative stimulus paradigm and uses state-of-the-art technology to use non-contact sensors and to capture facial expressions.

[0142] These data define a composite score that is fed back into the system to inform the next trial. Based on this composite metric as a control signal for the closed-loop system, the presented multi-sensory stimuli in the next trial is tuned to generate the target image. In this phase, in addition to integrating multiple systems in the closed-loop generative model, an analytical framework combines facial expression outputs from the Al technology, with the peripheral physiology (heartbeat, skin conductance level, EEG gamma band) as the control signal. It is hypothesized that the composite metric is effective as an input to the stimulus generation system. For both steps, to ethically certify that the generated images are in line with the study purpose, a 30-second delay is introduced preceding image representation, such that experimenters validate the generated image that is shown in the subseguent trials. The efficacy of the composite score is compared based on the objective metrics vs. the self-reports as the control signal in the closed-loop generative strategy. It is predicted that the composite metric would be more effective compared to the self-reports.

[0143] Accordingly, the preceding merely illustrates the principles of the present disclosure. It will be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the invention and are included within its spirit and scope. Furthermore, all examples and conditional language recited herein are principally intended to aid the reader in understanding the principles of the invention and the concepts contributed by the inventors to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions. Moreover, all statements herein reciting principles, aspects, and embodiments of the invention as well asspecific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents and equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure. The scope of the present invention, therefore, is not intended to be limited to the exemplary embodiments shown and described herein.

Claims

1. WHAT is CLAIMED is:

1. A method, comprising:(a) providing a sensory environment to an individual;(b) identifying a cognitive and neural status of the individual based on multi-modal physiological data from the individual using a first machine learning model, wherein the cognitive and neural status includes one or more of a cognitive state and an emotional state, wherein the first machine learning model is trained to interpret multi-modal physiological data to determine a cognitive and neural status selected from one or more of a cognitive state and an emotional state; and(c) providing an altered sensory environment to the individual to influence the cognitive and neural status of the individual, wherein the altered sensory environment is generated using a generative artificial intelligence (GenAI) model.

2. The method of claim 1 , wherein the first machine learning model and the GenAI model are trained jointly to form a unifying model.

3. The method of claim 1 or 2, wherein the cognitive and neural status includes a physiological state of the individual based on the multi-modal physiological data, wherein the first machine learning model is further trained to interpret the multi-modal physiological data to determine a cognitive and neural status based on a physiological state.

4. The method of any one of claims 1-3, wherein the multi-modal physiological data comprises one or more of the following: electroencephalography (EEG), electromyography (EMG), functional magnetic resonance imaging (fMRI), galvanic response, heart-rate, heart-rate variability, blood pressure, respiration, perspiration, oxygen saturation, electrocardiography (EKG), expiratory flow rate, tidal volume, tidal carbon dioxide, blood glucose level, blood lactate levels, pupil dilation, eye movement, grip pressure, near-infrared spectroscopy (NIRS), electrocortocography (ECoG), electrodermal activity, movement, and facial expression.

5. The method of any one of claims 1-4, wherein a portion of the multi-modal physiological data is obtained from a wearable device.

6. The method of any one of claims 1-5, wherein a portion of the multi-modal physiological data is obtained from a camera.

7. The method of any one of claims 1-6, further comprising (d) identifying an altered cognitive and neural status of the individual in response to altering the sensory environment.

8. The method of claim 7, further comprising iterating (c) and (d).

9. The method of claim 7 or 8, further comprising iterating (c) through (d) until a desired cognitive and neural status is achieved.

10. The method of claim 9, wherein the desired neural status is incongruent with the cognitive and neural status identified in (b).

11. The method of claim 9, wherein the desired cognitive and neural status is congruent with the cognitive and neural status identified in (b).

12. The method of any one of claims 1-11, wherein the GenAI model alters the sensory environment based on the cognitive and neural status or altered cognitive and neural status of the individual13. The method of claim 12, wherein the generative Al model is a large language model.

14. The method of claim 12 or 13, further comprising generating input for the GenAI model based on the output of the first machine learning model.

15. The method of claim 14, wherein the first machine learning model generates a composite score, which is used as a control signal by the GenAI model to modify the sensory environment.

16. The method of any one of claims 1-15, wherein the sensory environment comprises one or more of auditory, visual, tactile, olfactory, or gustatory stimuli.

17. The method of any one of claims 1-16, wherein the sensory environment incorporates virtual reality (VR) or augmented reality (AR).

18. The method according to any one of claims 1-17, wherein one or more steps of the method are computer-implemented.

19. The method of any one of claims 1-18, wherein the cognitive state is selected from one or more of: attention, concentration, awareness, mindfulness, flow, alertness, fatigue, confusion, memory, comprehension, decision-making, problem-solving, perception, distraction, insight, creativity, curiosity, daydreaming, overwhelmed, and cognitive dissonance.

20. The method of any one of claims 1-19, wherein the emotional state is selected from one or more of: happiness, excitement, love, calmness, gratitude, relief, amusement, pride, contentment, euphoria, sadness, anger, fear, anxiety, frustration, jealousy, envy, shame, guilt, loneliness, disgust, hopelessness, regret, apathy, bitterness, confusion, surprise, anticipation, embarrassment, and reluctance.

21. The method of any one of claims 3-20, wherein the physiological state is selected from one or more of: homeostasis, resting, wakefulness, sleep, exercise, fatigue, relaxation, physical stress, recovery, tension, hunger, satiety, digestion, thirst, fight or flight, rest and digest, freezing, fever, hypothermia, hyperthermia, pain, inflammation, itch, tachycardia, bradycardia, hypertension, hypotension, hyperventilation, dyspnea, adrenaline rush, cortisol response, hypoglycemia, hyperglycemia, anabolism, catabolism, arousal, orgasm, menstruation, pregnancy, and lactation.

22. The method of any one of claims 1-21, wherein providing the sensory environment and altering the sensory environment occurs in real time.

23. The method of any one of claims 1-22, wherein the individual is suffering from a mental health condition selected from the group consisting of: depression, anxiety, post-traumatic stress disorder (PTSD), addiction, pain, and any combination thereof.

24. The method of claim 23, wherein the method is effective in treating the mental health condition.

25. A system comprising:an output device for providing a sensory environment to an individual;an input device for receiving multi-modal physiological data from the individual;a processor and a non-transitory computer memory, wherein the memory comprises instructions that when executed by the processor, direct the processor to perform the methods of any one of claims 1-24.

26. The system of claim 25, wherein the output device is configured to provide one or more of an auditory stimulus, a visual stimulus, a tactile stimulus, an olfactory stimulus, and a gustatory stimulus.

27. The system of claim 25 or 26, wherein the output device comprises a video display, a virtual reality headset, or an augmented reality headset.

28. The system of any one of claims 25-27, wherein the output device comprises a speaker or a headphone.

29. The system of any one of claims 25-28, wherein the output device comprises haptic gloves, a vibramotor, a haptic body suit, a tactile electronic display, and a linear actuator.

30. The system of any one of claims 25-29, wherein the output device comprises a scent delivery system or a system configured to provide a gustatory stimulus.

31. The system of any one of claims 25-30, wherein the input device comprises one or more of an electroencephalography (EEG) device, and electromyography (EMG) device, a functional magnetic resonance imaging (fMRI) device, a near-infrared spectroscopy (NIRS) device, an electrocortocography (ECoG) device, an electrocardiography (EKG) device, a near-infrared spectroscopy (NIRS) device, an electrocortocography (ECoG) device, a blood glucometer, a spirometer, a sphygmomanometer, a camera, a LiDAR device, a blood lactate analyzer, a pulse oximeter, a movement tracker, an eye tracker, and a facial expression monitor.

32. The system of any one of claims 25-30, wherein the system comprises one or more devices for measuring galvanic response, heart-rate, heart-rate variability, blood pressure,respiration, perspiration, oxygen saturation, expiratory flow rate, tidal volume, tidal carbon dioxide, blood glucose level, blood lactate levels, pupil dilation, eye movement, grip pressure, electrodermal activity, movement, and facial expression.

33. The system of any one of claims 25-32, further comprising a user interface configured to relay the multi-modal physiological data and / or the sensory environment to a medical practitioner.

34. The system of claim 33, wherein the user interface comprises one or more of: an electronic display, a paper readout, a keyboard, a mouse, and a joystick.

35. A non-transitory, machine-readable media comprising instructions that when executed by a computer processor, direct the processor to perform the method of any one of claims 1-24.

Images