Information processing device

Abstract

An information processing device according to the present disclosure comprises: a first input interface capable of acquiring a biological signal; and one or more processors capable of performing processing based on the biological signal. The one or more processors are capable of estimating an emotional state from the biological signal at a predetermined time interval. The one or more processors are capable of determining whether or not to perform intervention for changing the emotional state on the basis of first data indicating the estimated emotional state and second data obtained by accumulating the first data over a period longer than the predetermined time interval.

Description

Information processing apparatus

[0001] The present disclosure relates to an information processing apparatus.

[0002] Changes in human emotions are manifested on the body surface as physiological reactions such as brain waves, heartbeats, and sweating. For example, these physiological reactions can be read as biological signals by a sensor device, features such as physiological indices contributing to emotional reactions can be extracted by signal processing of the biological signals, and human emotions can be estimated from the features by a model formula obtained by machine learning.

[0003] International Publication No. 2021 / 199271 International Publication No. 2022 / 124085

[0004] Techniques for providing feedback of the result of emotion estimation to a user are disclosed in, for example, Patent Documents 1 and 2. Development of techniques for further expanding the scope of use of emotion estimation is required.

[0005] Therefore, it is desirable to provide an information processing apparatus capable of expanding the scope of use of emotion estimation.

[0006] An information processing apparatus according to an embodiment of the present disclosure includes a first input interface capable of acquiring a biological signal, and one or more processors capable of performing processing based on the biological signal. The one or more processors are capable of estimating an emotional state at a predetermined time interval from the biological signal. The one or more processors are capable of determining whether to perform an intervention to change the emotional state based on first data indicating the estimated emotional state and second data obtained by accumulating the first data over a period longer than the predetermined time interval.

[0007] In the information processing apparatus according to an embodiment of the present disclosure, it is determined whether to perform an intervention to change the emotional state based on first data indicating the emotional state estimated at a predetermined time interval and second data obtained by accumulating the first data over a period longer than the predetermined time interval.

[0008] Figure 1 is an explanatory diagram illustrating an overview of Kahneman's attention distribution model. Figure 2 is an explanatory diagram illustrating an overview of the vagus nerve tank theory. Figure 3 is an explanatory diagram illustrating an overview of the vagus nerve tank theory. Figure 4 is a block diagram schematically showing an example configuration of an information processing device according to one embodiment of this disclosure. Figure 5 is a flowchart illustrating an example of intervention timing operation by an information processing device according to one embodiment. Figure 6 is an explanatory diagram illustrating a specific example of dynamic change of integral weights by the integral weight selection unit of an information processing device according to one embodiment. Figure 7 is a flowchart illustrating an example of attenuation and resetting of the integral value of arousal level change by the integral weight selection unit of an information processing device according to one embodiment. Figure 8 is an explanatory diagram illustrating a specific example of a table for calculating the integral value adjustment gain g by the integral weight selection unit of an information processing device according to one embodiment. Figure 9 is an explanatory diagram illustrating a specific example of a UI image by the visualization unit in an information processing device according to one embodiment. Figure 10 is an explanatory diagram illustrating a specific example of a UI image by the visualization unit in an information processing device according to one embodiment. Figure 11 is an explanatory diagram illustrating a specific example of a UI image by the visualization unit in an information processing device according to one embodiment. Figure 12 is an explanatory diagram showing a specific example of a UI image created by the visualization unit in an information processing device according to one embodiment. Figure 13 is a flowchart showing a specific example of context state determination by the context state determination unit of an information processing device according to one embodiment. Figure 14 is a block diagram schematically showing an example of the configuration of an information processing device according to a modified example. Figure 15 is an explanatory diagram showing an image of the calculation of time integral associated with the accumulation of long-term data in fatigue and recovery states. Figure 16 is a flowchart showing an example of the operation of intervention timing by an information processing device according to a modified example. Figure 17 is a flowchart showing an example of the operation of attenuation and reset of the integral value of the arousal level change by the integral weight selection unit of an information processing device according to a modified example. Figure 18 is an explanatory diagram showing a specific example of a table for calculating the integral value adjustment gain g by the integral weight selection unit of an information processing device according to a modified example. Figure 19 is an explanatory diagram showing an image of a method for calculating the consumption and recovery amount of cognitive resources based on the ratio of fatigue and recovery states.Figure 20 is a conceptual diagram showing the estimated instantaneous arousal level used as an intervention trigger and the cognitive resources that constitute the accumulation of instantaneous arousal levels. Figure 21 is a conceptual diagram showing an example of calculating cognitive resources by time-series analysis that takes into account past instantaneous arousal level information. Figure 22 is a block diagram schematically showing one example configuration of the emotional state determination unit related to the modified version. Figure 23 is a block diagram schematically showing one example configuration of the long-term data storage unit related to the modified version. Figure 24 is a block diagram schematically showing one example configuration of the information processing device related to the modified version. Figure 25 is an explanatory diagram showing an example of an event signal and correction gain in the information processing device related to the modified version. Figure 26 is an explanatory diagram showing an example of an event signal and correction gain in the information processing device related to the modified version. Figure 27 is an explanatory diagram showing an example of an event signal and correction gain in the information processing device related to the modified version. Figure 28 is an explanatory diagram showing an example of an event signal and correction gain in the information processing device related to the modified version.

[0009] The embodiments of this disclosure will be described in detail below with reference to the drawings. The description will be in the following order: 0. Comparative Example 1. One Embodiment 1.1 Configuration 1.2 Operation 1.3 Effect 2. Modification A 2.0 Overview 2.1 Configuration 2.2 Operation 2.3 Effect 3. Modification B 3.0 Overview 3.1 Configuration 3.2 Effect 4. Modification C 4.1 Configuration 4.2 Effect 5. Modification D 5.1 Configuration 5.2 Effect 6. Other Embodiments

[0010] <0. Comparative Example> (Overview and Issues of the Information Processing Device Related to the Comparative Example) Figure 1 shows an overview of the information processing device related to the comparative example.

[0011] Changes in a person's emotions manifest on the body surface as physiological responses such as brain waves, heart rate, and sweating. For example, these physiological responses can be read as biosignals by sensor devices, and by processing these signals, features such as physiological indicators contributing to emotional responses can be extracted. It is then possible to estimate a person's emotions from these features using a model equation derived through machine learning. Emotions are classified along two axes: pleasure / displeasure and arousal level, as described in academic reference 1 below. [Academic Reference 1] Posner, Jonathan, James A. Russell, and Bradley S. Peterson. "The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology." Development and psychopathology 17.3 (2005): 715-734.

[0012] In application areas such as mental health, there is a high need to visualize cognitive resources, arousal resources, or fatigue levels by utilizing the balance of arousal levels determined by the activity of the autonomic and central nervous systems. Furthermore, there is a high need to use the obtained data on cognitive resources, arousal resources, or fatigue levels as intervention triggers to improve user performance and manage stress. However, existing methods mainly rely on approaches that estimate cognitive resources, arousal resources, or fatigue levels as the final output, and mechanism-based visualization methods and intervention detection methods have not yet been established.

[0013] In neurobiology and psychology, the relationship between arousal resources and performance has been studied for a long time. One example is Kahneman's attention distribution model, described in the following academic reference 2: [Academic Reference 2] Kahneman: Attention and effort. Prentice Hall (1973)

[0014] Figure 1 shows an overview of Kahneman's attention allocation model. This model assumes that attention has an overall capacity limit, and that a person's ability to perform tasks concurrently depends on how much "capacity" each task requires, thus moving researchers away from filter theory. Kahneman's attention allocation model suggests that arousal resources have a "capacity" and indicates a relationship between short-term and long-term changes in arousal.

[0015] As a second example, to better understand the complexity and mechanisms of autoregulatory interactions, there is the vagal tank theory described in the following academic reference 3: [Academic Reference 3] Laborde, Sylvain, et al., "Vagal tank theory: the three Rs of cardiac vagal control functioning-resting, reactivity, and recovery." Frontiers in neuroscience 12 (2018): 458.

[0016] Figures 2 and 3 illustrate the vagus nerve tank theory. Figure 2 shows the case when a stress event occurs as an event. Figure 3 shows the case when a relaxation event occurs as an event.

[0017] The vagus nerve tank theory is a theory that uses the vagus nerve as a tank to model cognitive resources and fatigue levels based on the type of stressor. In the vagus nerve tank theory, emotional changes originating from the physiological response of the vagus nerve in a person's heart are classified into instantaneous changes (Phasic) and gradual changes (Tonic). A person's resting state is the baseline, and the subsequent state is determined by external stimuli (Event). In the vagus nerve tank theory, the balance between stress and relaxation is metaphorically represented as a tank, depending on whether the external stimulus is a stressful event or a relaxing event.

[0018] The vagus nerve tank theory models how the state of the vagus nerve tank changes after an event (post-event). As shown in Figure 2, when a stress event occurs as an event from a resting state, the tank's resources decrease, and after the event (post-event), the tank's resources recover (increase). Also, as shown in Figure 3, when a relaxation event occurs as an event from a resting state, the tank's resources recover, and after the event (post-event), the tank's resources are maintained or increase / decrease.

[0019] These models are merely theoretical frameworks studied in the field of psychology, and their application to state estimation models such as arousal levels has not been fully established. When constructing physiologically valid emotion estimation models in a laboratory setting and applying them to real-world problems, the context dependence of human emotional and physiological responses becomes a challenge.

[0020] Furthermore, regarding the context dependence of emotional and physiological responses, there is the following academic reference 4. Academic reference 4 evaluates the exercise context dependence of stress reactivity effects based on the Vagal tank theory. Academic reference 4 compares heart rate variability (HRV) over four days in a laboratory resting state and in a real environment. The results suggest that resting and reactivity sensitivity in the real environment is lower than in the laboratory. It also suggests that the lower the exercise intensity, the higher the reactivity to stress events. In other words, it shows that human physiological responses are context-dependent in the real environment. [Academic reference 4] Melanie Bamert, et al., "Stress and recovery measured with heart rate variability: Do the findings of laboratory research translate to daily life?", Society for Ambulatory Assessment 2021

[0021] In particular, when applying this technology to everyday life applications, it is necessary to consider the signal quality of biosignals, differentiate between mental and motor arousal levels, and optimize processing according to the context. However, prior technologies have not taken these factors into account.

[0022] For example, Patent Document 1 (International Publication No. 2021 / 199271) proposes a technology that provides stress relief interventions based on changes in a user's short-term and long-term stress levels. Short-term and long-term stress levels are calculated separately, and stress relief interventions are flagged based on a combination of short-term and long-term stress information. However, the technology described in Patent Document 1 does not adequately consider the direct relationship between short-term and long-term stress levels, and is therefore limited in its ability to address the context-dependent nature of human physiological responses in real-world environments.

[0023] Patent Document 2 (International Publication No. 2022 / 124085) proposes a technology that divides the user's state of arousal into three stages, sets a certain time window to determine which of the three stages the user's state of arousal is in, and generates feedback information for the user based on the determination result. The technology described in Patent Document 2 does not propose using both instantaneous arousal levels and accumulated long-term changes in arousal levels, nor does it propose generating context-dependent feedback information. The technology described in Patent Document 2 has limitations in addressing the issue of context dependence of emotional and physiological responses in everyday and real-world environments.

[0024] If the accumulation of temporal changes in arousal and relaxation can be accurately visualized as an indicator of cognitive resources and fatigue levels, and intervention timing can be detected, it will lead to an expansion of the scope of use for emotion estimation (arousal level estimation), and an improvement in the user experience (UX) when using applications can be expected.

[0025] <1. One Embodiment> [1.1 Configuration] Figure 4 schematically shows one example of the configuration of an information processing device according to one embodiment of the present disclosure.

[0026] An information processing device according to one embodiment addresses the above-mentioned technical challenges by utilizing the instantaneous estimation results of the state of alertness to enable flagging for intervention when performance deteriorates and to visualize long-term stress balance for life log data.

[0027] An information processing device according to one embodiment comprises an emotion estimation unit 10 and an intervention determination and visualization unit 20.

[0028] An information processing device according to one embodiment may be composed of a computer equipped with, for example, one or more CPUs (Central Processing Units), one or more ROMs (Read Only Memory), and one or more RAMs (Random Access Memory). In this case, the processing of each block in the information processing device according to one embodiment can be realized by one or more CPUs executing processing based on a program stored in one or more ROMs or RAMs. Alternatively, the processing of each block in the information processing device according to one embodiment may be realized by one or more CPUs executing processing based on a program supplied from an external source, for example, via a wired or wireless network.

[0029] The emotion estimation unit 10 is an estimation unit capable of estimating the emotional state of a user at short time intervals (predetermined time intervals) from the vital signals (biological signals) of the user whose emotional state is to be estimated.

[0030] The emotion estimation unit 10 includes a sensor data acquisition unit 11, a filter preprocessing unit 12, a feature extraction unit 13, an emotion state determination unit 14, and a context state determination unit 15. The sensor data acquisition unit 11 corresponds to one specific example of the "first input interface" and the "second input interface" in one embodiment of the present disclosure. The filter preprocessing unit 12, the feature extraction unit 13, the emotion state determination unit 14, and the context state determination unit 15 are capable of processing the biosignals and physical information described later, and correspond to one specific example of the "one or more processors" in one embodiment of the present disclosure.

[0031] The sensor device 30 includes a vital sensor (biosensor) and a physical sensor such as a motion sensor. The sensor device 30 may be one or more devices. The biosensor and the physical sensor may be mounted on one device or on separate devices. The sensor device 30 may be a mobile device such as a smartphone or various wearable devices equipped with a biosensor or a physical sensor. The physical sensor may include a GPS (Global Positioning System), an IMU (Inertial Measurement Unit), and a microphone. The biosensor is capable of detecting biosignals for estimating the user's emotions, such as sweating, pulse waves, electromyography, blood pressure, blood flow, or body temperature. The emotion estimation unit 10 is capable of estimating the user's emotions based on the biosignals detected by the biosensor. This emotion allows for confirmation of the user's state of concentration, alertness, etc.

[0032] The sensor data acquisition unit 11 is an interface connected to the sensor device 30. The sensor data acquisition unit 11 functions as a communication device capable of receiving sensor information (biometric signals) from the biosensor of the sensor device 30 and outputting it as vital information (biometric information) to the filter preprocessing unit 12. The sensor data acquisition unit 11 also functions as a communication device capable of receiving sensor information from the physical sensor of the sensor device 30 and outputting it as physical information to the context state determination unit 15. If the sensor device 30 includes a smartphone, the sensor data acquisition unit 11 may also be capable of outputting information such as the history of smartphone application usage as physical information to the context state determination unit 15.

[0033] The filter preprocessing unit 12 is a preprocessing unit capable of preprocessing biological signals to remove unwanted components. The filter preprocessing unit 12 can perform preprocessing such as bandpass filtering and noise reduction on biological signals measured by the biosensor of the sensor device 30. The filter preprocessing unit 12 can output the preprocessed biological signal to the feature extraction unit 13.

[0034] For example, if the biological signal is an electroencephalogram (EEG), the EEG is measured by attaching electrodes to the scalp and detecting the brain's action potentials that leak through the scalp and skull. One characteristic of EEG measured in this way is that the signal-to-noise ratio (SNR) is known to be very low. Therefore, it is necessary to remove unwanted frequency components from the EEG, which is a time-series signal, and a bandpass filter is applied as the filter preprocessor 12 (for example, a passband of 0.1 Hz to 40 Hz).

[0035] Furthermore, body movement components caused by human body movement are superimposed on the biological signal as artifacts (noise other than the target signal). In response to this, signal processing techniques such as adaptive filtering and independent component analysis are applied to the filter preprocessing unit 12. Similarly, preprocessing is performed in the filter preprocessing unit 12 when the biological signal is psychogenic sweating (EDA), pulse wave (PPG), blood flow (LDF), etc.

[0036] The feature extraction unit 13 is an extraction unit capable of extracting physiological responses that contribute to emotion as features from preprocessed biological signals. The feature extraction unit 13 calculates feature quantities x = (x1, x2, ..., xN) as model input variables for estimating the emotional state. The feature extraction unit 13 is capable of outputting the extracted feature quantities x to the emotional state determination unit 14.

[0037] Specifically, the feature extraction unit 13 observes biological signals from vital sensors (biosensors) such as electroencephalograms (EEG), electropsychotic sweating (EDA), pulse waves (PPG), and blood flow (LDF) as time-series data, and is capable of extracting physiological indicators that contribute to emotional changes as feature quantities x. Generally, feature quantities x are extracted using an analysis window (sliding window) of about 5 minutes.

[0038] Furthermore, feature x is not limited to physiologically known features. The feature extraction unit 13 can also perform signal processing to extract feature x that contributes to emotion in a data-driven manner, for example, by using deep learning or an autoencoder.

[0039] The emotional state determination unit 14 can determine the emotional state using a machine learning model, taking the feature quantity x calculated by the feature quantity extraction unit 13 as input. The emotional state determination unit 14 can analyze the pattern of the time-series data of the feature quantity x calculated by the feature quantity extraction unit 13 and recognize what emotional state the user is in. The emotional state determination unit 14 can identify predicted labels for the time-series emotional state using a pre-built machine learning model, which is an identification model, and label them as Yk = (y1, y2, ..., yk). The emotional state determination unit 14 can output the emotional label Yk = (y1, y2, ..., yk) as the emotional state obtained by estimation (emotion estimation determination result). The emotion estimation determination result includes the instantaneous level of arousal.

[0040] The context state determination unit 15 is capable of determining the user's situation (context) for which emotional state is to be estimated, based on physical information acquired from the physical sensors of the sensor device 30. The process of determining the user's situation corresponds to one specific example of the "second determination process" in one embodiment of this disclosure. The context state determination unit 15 is capable of recognizing what kind of actions the user holding the sensor device 30 is taking, based on the physical information (motion information) acquired as sensor information from the physical sensors of the sensor device 30. The function of the context state determination unit 15 can be realized, for example, by pattern recognition of waveform data from a motion sensor, which is a physical sensor, and determining the type of movement or the type of vehicle, but the implementation method is not limited to this. If the sensor device 30 includes a smartphone, the context state determination unit 15 may also output information such as the history of smartphone application usage as context information.

[0041] The intervention determination / visualization unit 20 corresponds to the emotion control unit in the information processing apparatus according to one embodiment. The intervention determination / visualization unit 20 can determine whether to perform an intervention to change the emotional state based on the first data indicating the emotional state estimated by the emotion estimation unit 10 and the long-term data (second data) obtained by accumulating the first data over a period longer than a predetermined time interval. The process of determining whether to perform an intervention corresponds to a specific example of the "first determination process" in one embodiment of the present disclosure. Here, the first data includes instantaneous arousal. The intervention determination / visualization unit 20 can accumulate the long-term data of the determination result of emotion estimation and perform intervention determination and visualization.

[0042] The intervention determination / visualization unit 20 includes a long-term data accumulation unit 21, a processing block 22, an integration weight selection unit 23, and an integration parameter storage unit 24. The processing block 22 includes an intervention flag determination unit 25 and a visualization unit 26. The long-term data accumulation unit 21, the processing block 22, the integration weight selection unit 23, and the integration parameter storage unit 24 correspond to a specific example of the "one or more processors" in one embodiment of the present disclosure.

[0043] The long-term data accumulation unit 21 is an accumulation unit that can accumulate the long-term data of the determination result of emotion estimation and perform weighted time integration of the long-term data. The long-term data accumulation unit 21 can calculate the consumption amount or the remaining amount of cognitive resources by performing time integration of the change in instantaneous arousal. The long-term data accumulation unit 21 can buffer the samples of the emotional state determined by the emotional state determination unit 14 at each moment and analyze them as the arousal change of the long-term data. Specifically, the long-term data accumulation unit 21 can perform weighted time integration shown in, for example, the following formula (A) when a high arousal state is regarded as stress and a low arousal state is regarded as recovery. The following formula (A) formulates arousal resources based on the psychological model shown in FIGS. 1 to 3 described above.

[0044]

[0045] However, MF(t) indicates the consumption amount of cognitive resources. Wstress and Wrecovery indicate integral weights respectively. Pstress(t) takes a positive value in the case determined as stress at time t. Precovery(t) takes a negative value in the case determined as recovery at time t. The cases determined as recovery are specifically listening to relaxation music and stress-relieving actions (such as meditation and deep breathing). B(t = 0) indicates the initial value of the consumption amount of cognitive resources. The initial value of the consumption amount of cognitive resources is the consumption amount of cognitive resources at time t = 0.

[0046] Here, an example where MF(t) is taken as the consumption amount of cognitive resources has been given, but the remaining amount of cognitive resources may also be formulated with the above concepts. In that case, the signs of Pstress(t) and Precovery(t) will be the signs inverted in relation to the above-mentioned signs.

[0047] The integral parameter storage unit 24 can store a table or the like for calculating the integral value adjustment gain g shown in FIG. 8 described later.

[0048] The integral weight selection unit 23 is a weight selection unit capable of dynamically changing the weight of time integration by the long-term data accumulation unit 21 according to the determination result of the context state determination unit 15. Thereby, in the long-term data accumulation unit 21, the integral weight is dynamically changed according to the determination result of the context state determined by the context state determination unit 15. By dynamically changing the weight of time integration by the integral weight selection unit 23, it is possible to separate the motional arousal level and the mental arousal level. The integral weight selection unit 23 can dynamically change the weight of time integration, for example, by reducing the weight of time integration during exercise.

[0049] Also, the integral weight selection unit 23 can calculate an integral value adjustment gain g for dynamically attenuating or resetting the integral value of time integration by the long-term data accumulation unit 21 based on the table stored in the integral parameter storage unit 24 according to the determination result of the context state determination unit 15.

[0050] The intervention flag determination unit 25 is capable of detecting the timing of interventions for the user. Based on the instantaneous emotional state calculated by the emotional state determination unit 14 and the amount of arousal resource consumption that takes into account long-term changes in arousal level calculated by the long-term data storage unit 21, the intervention flag determination unit 25 can predict a decline in the user's performance and determine the stress-release intervention flag. Here, intervention refers to prompting the user to take stress-release actions. Stress-release actions include, for example, playing music suitable for stress relief.

[0051] As shown in the example of operation in Figure 5 described later, the intervention determination and visualization unit 20 can perform a stress-release intervention when the context state determination unit 15 determines that the situation is favorable for concentration and the instantaneous level of arousal is determined to be below a threshold. The intervention determination and visualization unit 20 can also perform a stress-release intervention when the context state determination unit 15 determines that the situation is favorable for concentration, the instantaneous level of arousal is above a threshold, and the amount of cognitive resources consumed is above a certain level.

[0052] As shown in Figures 9 to 12 described later, the visualization unit 26 is capable of visualizing instantaneous arousal levels as first data and long-term data as second data. The visualization unit 26 can provide feedback to the user based on the judgment results of the intervention flag determination unit 25, such as recommendations for stress-relief application experiences and visualized logs of emotional states and arousal resource states.

[0053] [1.2 Operation] Figure 5 is a flowchart showing an example of the operation of the intervention timing by an information processing device according to one embodiment.

[0054] The intervention determination and visualization unit 20 determines whether the situation is suitable for the user to release stress, based on the result of the context state determination. First, the intervention determination and visualization unit 20 determines, for example, whether the situation is conducive to concentration (step S101). For example, in the case of an office environment, a situation conducive to concentration would be a meeting. If it is determined that the situation is not conducive to concentration (step S101; N), the intervention determination and visualization unit 20 does not set a flag because it is not a situation for stress release (step S105).

[0055] On the other hand, if the system determines that the situation is favorable for concentration (step S101; Y), the intervention determination and visualization unit 20 then determines whether the instantaneous level of arousal is above a threshold (step S102). If the system determines that the instantaneous level of arousal calculated by the emotional state determination unit 14 is below the threshold (step S102; N), the system determines that the situation is favorable for concentration but the person is not aroused (unable to concentrate), and the intervention determination and visualization unit 20 sets an intervention flag as the timing for stress release (step S106).

[0056] Furthermore, if the instantaneous level of arousal is determined to be above a threshold (step S102; Y), the intervention determination and visualization unit 20 then determines whether the cognitive resources consumed are above a certain level (step S103). If the cognitive resources consumed are determined to be below a certain level (step S103; N), the intervention determination and visualization unit 20 determines that the state is one in which the level of arousal is high and sufficient cognitive resources are available, and does not set a flag (step S107). If the cognitive resources consumed are determined to be above a certain level (step S103; Y), the intervention determination and visualization unit 20 determines that the state is one in which the level of arousal is high but sufficient cognitive resources are not available (it is difficult to maintain concentration), and sets an intervention flag as the timing for stress release (step S104).

[0057] Figure 6 is an explanatory diagram showing a specific example of the dynamic modification of integral weights (Wstress and Wrecovery in equation (A)) by the integral weight selection unit 23 of an information processing device according to one embodiment.

[0058] The integral weight selection unit 23 may dynamically change the integral weights according to the context. By dynamically changing the weights of the time integral by the integral weight selection unit 23, a distinction may be made between motor arousal and mental arousal. By dynamically changing the weights of the time integral by the integral weight selection unit 23, a correction may also be made to the experiential value according to the situation and external stimuli even for mental arousal. Figure 6 shows an example where the context states are office work, exercise, rest, and sleep. In the example of distinguishing between motor arousal and mental arousal, the integral weight selection unit 23 takes measures such as reducing the integral weight during exercise. Also, in states such as rest and sleep, the user's state gradually recovers, but the instantaneous state may be judged as stressed. In that case, since the arousal resource recovers, the integral weight selection unit 23 controls the increase or decrease of the integral value of the arousal change by setting a negative value as the weight Wstress. Similarly, when the instantaneous arousal state is judged to be recovered, the positive or negative relationship of the integral value of the arousal change is determined.

[0059] Figure 7 is a flowchart showing an example of the operation of attenuation and resetting of the integral value of the arousal level change by the integral weight selection unit 23 of an information processing device according to one embodiment. Figure 8 is an explanatory diagram showing a specific example of a table for calculating the integral value adjustment gain g by the integral weight selection unit 23. Figure 8 shows an example where the context states include office work, exercise, rest, and sleep.

[0060] The integral weight selection unit 23 may dynamically reset the integral value of the arousal level change according to the context. This allows the intervention flag determination unit 25 to focus on the state of a specific time period or context and set an intervention flag. The integral weight selection unit 23 dynamically considers the influence of past arousal level resource values ​​when a person's contextual state changes significantly. Specifically, the integral weight selection unit 23 first determines whether or not there is a transition in the contextual state (step S201). If it is determined that there is no transition in the contextual state (step S201; N), the integral weight selection unit 23 continues the integration without attenuating or resetting the integral value of the arousal level change (step S202). On the other hand, if it is determined that there is a transition in the contextual state (step S201; Y), the integral weight selection unit 23 attenuates or resets the integral value of the arousal level change according to the state transition (step S203). When a context state transition occurs, the integral weight selection unit 23, referencing a table pre-held by the integral parameter storage unit 24, uses the previous specific context state and the current context state to determine the integral value adjustment gain g, and performs processing to attenuate or reset the integral value of the arousal level change. This enables the subsequent intervention flag determination unit 25 to set flags focused on a specific time period or context state. The MF(t) in equation (A) above is expressed as follows in equation (B) using the adjusted integral value adjustment gain g: MFupdated(t) = g * MF(t) ……(B)

[0061] Figures 9 to 12 show a specific example of a UI (User Interface) image generated by the visualization unit 26 in an information processing device according to one embodiment.

[0062] In the example in Figure 9, the emotional state for a day is visualized using a pie chart. In the example in Figure 9, the emotional state is divided into stress (high arousal state, fatigue state) and recovery state (low arousal state), each divided into three stages (low, mid, high), and the proportion of stress state and recovery state for a day is visualized using a pie chart. In addition, the example in Figure 9 also visualizes the total time spent in a stress state for a day (18 hours in the example in Figure 9). It is also possible to display the stress state and recovery state with different colors for easier understanding. For example, the stress state could be displayed with a red color and the recovery state with a blue color. Furthermore, the intensity of the color could be changed according to the stage of the state (low, mid, high) for both the stress state and the recovery state.

[0063] In the examples in Figures 10 and 11, the remaining cognitive resources at an arbitrary time (e.g., the present) are visualized using a meter. Figure 10 shows an example where cognitive resources are low, and Figure 11 shows an example where cognitive resources are high. The meter display color may also be changed depending on whether the resources are low or high. For example, low resources could be displayed in a reddish color, and high resources in a blued color.

[0064] In the example in Figure 12, emotional states are visualized over time. In the example in Figure 12, the time-series data for stress states (high arousal state) and recovery states (low arousal state) are visualized as bar graphs over time. In addition, the time-series data for remaining cognitive resources is visualized as a line graph over time. It is also possible to distinguish between stress states and recovery states by using different colors. For example, stress states could be displayed in red tones and recovery states in blue tones.

[0065] Figure 13 is a flowchart showing a specific example of context state determination by the context state determination unit 15 of an information processing device according to one embodiment.

[0066] The context state determination unit 15 determines the user's activity state from physical information (motion information) acquired by the sensor device 30. The sensor device 30 is a mobile device such as a smartphone, various wearable devices, or IoT (Internet of Things) devices. IoT devices are devices that include wireless communication devices that can connect to a network such as the internet, and can be installed in vehicles, smart home appliances, etc. The context state determination unit 15 recognizes what actions the user holding the sensor device 30 is taking from the sensor information of the physical sensors mounted on the sensor device 30. The context state determination unit 15 performs pattern recognition on the waveform data of the motion sensor, for example, when the sensor device 30 is a mobile device carried in a pocket or bag, and determines the type of movement and the type of vehicle. The context state determination unit 15 may also determine various means of transportation such as bicycles, trains, cars, and elevators, in addition to movements such as walking and running, without using positioning information.

[0067] First, the context state determination unit 15 acquires sensor data from the sensor device 30 (step S301). Next, the context state determination unit 15 extracts features from the sensor data of the sensor device 30 (step S302). Next, the context state determination unit 15 performs pattern recognition using machine learning (step S303). Next, the context state determination unit 15 determines movement and vehicle status as a result of the behavior recognition (step S304).

[0068] [1.3 Effects] As described above, according to one embodiment of the information processing device, a decision is made on whether or not to perform an intervention to change the emotional state based on first data indicating the emotional state estimated at predetermined time intervals and second data obtained by accumulating the first data over a period longer than the predetermined time intervals. This makes it possible to provide an information processing device that can expand the range of applications for emotional estimation.

[0069] Furthermore, according to one embodiment of the information processing device, the instantaneous state and weighted time integral of the arousal level estimation are separated, and an intervention flag is detected. This allows for the consideration of the context dependence of emotional and physiological responses, and enables improved user experience and user performance by making intervention decisions that take into account not only the user's instantaneous arousal level but also arousal resources.

[0070] Furthermore, according to one embodiment of the information processing device, by dynamically changing the weight of the integral value of the arousal level change according to the context, it is possible to differentiate between motor arousal and mental arousal. Even in the mental aspect, it becomes possible to correct the experiential value according to the situation and external stimuli, improving the user experience.

[0071] Furthermore, according to one embodiment of the information processing device, by dynamically attenuating or resetting the integral value of the arousal level change according to the context, it is possible to improve the user experience by focusing on the state of a specific time period or context and setting an intervention flag.

[0072] The effects described herein are merely illustrative and not limiting, and other effects may also exist. The same applies to the effects of subsequent modifications and other embodiments.

[0073] <2. Modified Example A> The information processing device according to Modified Example A will be described below. In the following, parts that are substantially the same as the components of the information processing device according to the above embodiment will be denoted by the same reference numerals, and their descriptions will be omitted as appropriate.

[0074] [2.0 Overview] The results of emotion estimation have various potential uses, one of which is to provide feedback to the user. In the technology according to the above embodiment, as an example of using emotion estimation results for feedback, a technology is proposed that visualizes fatigue states in past daily life based on physiological mechanisms and uses them as intervention triggers.

[0075] In the technology according to the above embodiment, the emotional state is estimated by dividing a person's level of arousal into an instantaneous state and a weighted time integral, and intervention flags are detected when the person is in a fatigued state. This approach enables visualization of fatigue during performance decline and triggers interventions. However, in the technology according to the above embodiment, the exact degree of recovery (recovery state) is not defined when fatigue recovery occurs during relaxation-related behavior (recovery behavior). Therefore, the cognitive resource (time integral) indicating the fatigue state is not estimated to reflect the recovery state. When there is a state transition accompanied by recovery behavior, only the time integral is attenuated or reset based on contextual information, and the accurate internal state during fatigue recovery is not reflected in the cognitive resource (time integral).

[0076] Therefore, if it is possible to define and visualize the degree of recovery based on a person's internal state when fatigue recovery occurs in conjunction with recovery behavior, and to estimate this by reflecting it in cognitive resources (time integral), then it will be possible to visualize the fatigue state during performance decline and detect the timing of intervention with greater accuracy even after fatigue recovery. This is expected to lead to further improvements in the user experience and expansion of applications. Furthermore, in the technology according to the above embodiment, the time integral of instantaneous arousal level is proposed as a method for calculating cognitive resources, but if various other calculation methods can be used, it is expected that the estimation of emotional states will become even more accurate.

[0077] This modified version includes the following technical differences from the technology of the above-described embodiment: • Difference Point 1: While the technology of the above-described embodiment only defined and visualized fatigue states due to prolonged arousal, this version defines and visualizes recovery states, which have a different time constant than fatigue states. A fatigue state due to prolonged arousal refers to a state in which a decline in user performance is expected, and where maintaining concentration becomes difficult. A recovery state with a different time constant refers to a state in which recovery of user performance is expected. For example, the level of arousal during a fatigue state due to prolonged arousal is used as a baseline, and the decrease in the level of arousal from that baseline is defined as a recovery index, and the recovery state is visualized.

[0078] - Difference Point 2: Regarding cognitive resources in fatigue and recovery states, it will be possible to use a calculation method that uses the ratio of the time integrals of fatigue and recovery states to a set standard, in addition to the calculation method using the time integral of instantaneous alertness. Furthermore, it will be possible to use a calculation method that incorporates past information in the time series of instantaneous alertness in fatigue and recovery states through time series analysis.

[0079] - Difference Point 3: By utilizing the newly defined and calculated amount of cognitive resource recovery in the recovery state, a highly accurate dynamic correction (attenuation and reset of fatigue state due to recovery) is achieved for the amount of cognitive resource consumed in the previous fatigue state.

[0080] • Difference Point 4: Based on the newly defined and calculated amount of cognitive resource recovery in the recovery state, a decision is made as to whether or not to implement an intervention that changes the state when the patient has fully recovered. This intervention refers to, for example, an intervention that encourages the timing of returning to work from rest.

[0081] [2.1 Structure]

[0082] Figure 14 schematically shows one example configuration of an information processing device according to Modification A.

[0083] In the information processing device according to the above embodiment (Figure 4), the instantaneous wakefulness state estimation results are utilized to enable flagging for intervention when performance declines and to visualize long-term stress balance for life log data. In this modified version, the instantaneous wakefulness state estimation results are utilized to enable flagging for intervention not only when performance declines, but also when performance recovers. Furthermore, in this modified version, the function of visualizing not only fatigue but also recovery is realized.

[0084] In this modified example, the configuration of the emotion estimation unit 10 may be substantially the same as that of the information processing device (Figure 4) according to the above embodiment. On the other hand, the components of the intervention determination and visualization unit 20 are substantially the same as those of the information processing device (Figure 4) according to the above embodiment, but the functions of some components are different.

[0085] In this modified example, the long-term data storage unit 21, similar to the information processing device according to the above embodiment, is capable of calculating the amount of cognitive resources consumed or the remaining amount of cognitive resources by performing a time integral of the instantaneous change in alertness. Furthermore, in this modified example, the long-term data storage unit 21 is capable of calculating the amount of cognitive resource recovery. For example, the long-term data storage unit 21 can define a recovery index as the decrease in alertness from a standard, using the alertness level during a fatigued state due to sustained alertness as a standard. For example, the long-term data storage unit 21 can calculate the amount of cognitive resource recovery by performing a time integral of the decrease in alertness from a standard, using the instantaneous alertness level (alertness level during sustained alertness) when fatigue accumulates and performance declines as a standard (alertness state standard). Specifically, the long-term data storage unit 21 can perform a time integral of the instantaneous change in alertness in stressful states and a time integral of the decrease in instantaneous alertness in relaxed states. The context state determination unit 15 is used to determine whether a state is stressful or relaxed.

[0086] Figure 15 shows an image illustrating the calculation of time integrals associated with the accumulation of long-term data in fatigued and recovered states.

[0087] In this modified version, the long-term data storage unit 21 can calculate the time integral of instantaneous arousal level as the amount of cognitive resources consumed, indicating a fatigued state, when performing stressful tasks. In this modified version, the long-term data storage unit 21 can further calculate the time integral of the decrease in instantaneous arousal level as the amount of cognitive resources recovered, indicating a recovery state, when performing relaxing activities. The intervention flag determination unit 25 can detect the timing of interventions for the user. Based on the instantaneous emotional state calculated by the emotional state determination unit 14 and the arousal resources (amount of cognitive resources consumed and recovered) calculated by the long-term data storage unit 21, the intervention flag determination unit 25 can predict the decline and recovery of the user's performance. Based on the predicted results of the decline and recovery of the user's performance, the intervention flag determination unit 25 can determine the intervention flag for stress release and the intervention flag for task restart. Here, the two interventions refer to encouraging the user to take stress-releasing actions and encouraging them to take actions that involve renewed concentration. Stress-relieving activities include, for example, playing music suitable for stress relief. Activities that involve concentration include, for example, doing work that requires concentration, such as desk work or meetings.

[0088] In this modified example, as shown in the operation example in Figure 16 described later, the intervention determination and visualization unit 20 can perform an intervention to encourage concentration-related behavior (task restart intervention) when the context state determination unit 15 determines that a change in behavior from concentration-related behavior to relaxation-related behavior is occurring, and when it is determined that the amount of cognitive resource recovery is above a certain level.

[0089] In this modified example, the visualization unit 26 is capable of visualizing instantaneous alertness as first data and long-term data as second data, similar to the information processing device according to the above embodiment. Furthermore, the visualization unit 26 is also capable of visualizing fatigue and recovery states determined based on the first and second data. The visualization unit 26 can provide feedback to the user not only on the judgment results of the intervention flag determination unit 25, but also on recommendations for stress-relieving application experiences, recommendations for returning to work when sufficient relaxation has been achieved, and visualized logs of emotional state and alertness resource status.

[0090] [2.2 Operation] Figure 16 is a flowchart showing an example of the operation of the intervention timing by the information processing device according to this modified example.

[0091] The intervention determination and visualization unit 20 determines, for example, whether the situation is suitable for the user to restart a task, based on the result of the context state determination. First, the intervention determination and visualization unit 20 determines, for example, whether the situation involves relaxation activities (step S301). For example, in the case of an office environment, a situation involving relaxation activities would be a situation where the user is sitting quietly and listening to music or relaxing sounds. If it is determined that the situation does not involve relaxation activities (step S301; N), the situation is not suitable for restarting a task, so the intervention determination and visualization unit 20 does not set a flag (step S304).

[0092] On the other hand, if it is determined that relaxation-related behaviors are being carried out (step S301; Y), the intervention determination and visualization unit 20 then determines whether the amount of cognitive resource recovery is above a certain level (step S302).

[0093] If the amount of cognitive resource recovery is determined to be less than a certain level (step S302; N), the intervention determination and visualization unit 20 determines that the recovery is insufficient and does not set a flag (step S305). On the other hand, if the amount of cognitive resource recovery is determined to be above a certain level (step S302; Y), the intervention determination and visualization unit 20 determines that the recovery is sufficient and performance recovery is expected, and sets an intervention flag as the timing to restart the task (step S303).

[0094] The integral parameter storage unit 24 stores tables and the like for calculating the integral value adjustment gain g. Here, the integral value refers to the amount of cognitive resources consumed (time integral) in a fatigued state, and this integral value can be adjusted by g.

[0095] The integral weight selection unit 23 calculates an integral value adjustment gain g to dynamically attenuate or reset the integral value of the cognitive resource consumption (time integral) by the long-term data storage unit 21, based on a table stored in the integral parameter storage unit 24, in response to feedback from the judgment result of the intervention flag judgment unit 25 as well as the judgment result of the context state determination unit 15.

[0096] Figure 17 is a flowchart illustrating an example of the operation of attenuation and resetting of the integral value of the arousal level change by the integral weight selection unit 23 of the information processing device according to this modified example. Figure 18 is an explanatory diagram showing a specific example of a table for calculating the integral value adjustment gain g by the integral weight selection unit 23 of the information processing device according to this modified example. Figure 18 shows an example where there are multiple states associated with recovery: recovery 1, recovery 2, recovery 3, and recovery 4. Recovery 4 represents a state in which the degree of recovery (amount of recovery) is greater than that of recovery 1.

[0097] The integral weight selection unit 23 can consider not only contextual information but also the influence according to the actual internal state when a person's contextual state changes, especially when they perform relaxation-related behaviors (recovery behaviors). Specifically, the integral weight selection unit 23 first determines whether or not a relaxation-related behavior has been performed (step S401). If it is determined that a relaxation-related behavior has not been performed (step S401; N), the long-term data storage unit 21 does not calculate the amount of cognitive resource recovery associated with recovery (step S405).

[0098] On the other hand, if it is determined that an action involving relaxation is being performed (step S401; Y), the long-term data storage unit 21 then calculates the amount of cognitive resource recovery associated with the recovery (step S402). Next, the integral weight selection unit 23 determines whether or not there is a state transition from relaxation to a task (step S403). If the integral weight selection unit 23 determines that relaxation has ended and there has been no transition to a task (step S403; N), the long-term data storage unit 21 continues to calculate the amount of cognitive resource recovery associated with the recovery (step S406).

[0099] On the other hand, if the integral weight selection unit 23 determines that relaxation has ended and the user has transitioned to a task (step S403; Y), the integral weight selection unit 23 reduces or resets the amount of cognitive resources consumed in the past (integral value of the change in arousal level in the fatigued state) according to the amount of cognitive resources recovered associated with the recovery calculated up to that point (step S404). When there is a state transition from a relaxed state to a task state, the integral weight selection unit 23 refers to the table held by the integral parameter storage unit 24 and selects an appropriate integral value adjustment gain g based on the amount of cognitive resources recovered associated with the recovery from the relaxed state. As a result, the integral weight selection unit 23 performs a process to reduce or reset the amount of cognitive resources consumed (integral value). This enables the subsequent intervention flag determination unit 25 to set a flag with higher accuracy, taking into account the internal state associated with the recovery.

[0100] (Variations in the calculation of cognitive resources) In the above explanation, we proposed a method for calculating cognitive resources in fatigue and recovery states using time integration. However, it is expected that state estimation will become more accurate by using various other calculation methods, not just this one. For example, a calculation method using the ratio of the time integration of fatigue and recovery states to a set standard can be used, or a calculation method using time series analysis that takes into account past information in the time series of fatigue and recovery states can be used.

[0101] Figure 19 shows an example of a method for calculating the consumption and recovery of cognitive resources based on the ratio of fatigued and recovered states.

[0102] The long-term data storage unit 21 may, for example, calculate the consumption of cognitive resources as a ratio to the consumption of cognitive resources when the instantaneous level of alertness is a first criterion indicating a fatigued state. For the fatigued state, for example, the instantaneous level of alertness at time t0 when fatigue accumulates and performance declines is taken as the alertness criterion yha (first criterion). In this case, if the alertness criterion yha is always maintained during the task, the maximum total consumption of cognitive resources will be given by equation (1). In contrast, the consumption of cognitive resources according to the actual change in instantaneous level of alertness f(t) will be given by equation (2). By using the ratio of equation (1) and equation (2), the consumption of cognitive resources can be calculated as a ratio as shown in equation (3). Similarly, the remaining amount of cognitive resources can be calculated as a ratio.

[0103]

[0104] Furthermore, the long-term data storage unit 21 may calculate the amount of cognitive resource recovery as a ratio to the amount of cognitive resource recovery when the instantaneous level of alertness is a second criterion indicating the recovery state. For the recovery state, for example, the instantaneous level of alertness at time t0 when fatigue accumulates and performance declines may be taken as the alertness state criterion yha, and the state in which the subject is most relaxed may be taken as the relaxation state criterion yla (second criterion). In this case, if the relaxation state criterion yla is always present during relaxation, the maximum total amount of cognitive resource recovery will be given by equation (4). In contrast, the amount of cognitive resource recovery according to the actual change in instantaneous level of alertness f(t) will be given by equation (5). By using the ratio of equation (4) and equation (5), the amount of cognitive resource recovery can be calculated as a ratio as shown in equation (6).

[0105] 1

[0106] Figure 20 is a conceptual diagram showing the estimated instantaneous level of arousal (Short-term arousal) used as an intervention trigger, and the cognitive resources (Long-term arousal) which represent the accumulation of instantaneous levels of arousal. Figure 21 is a conceptual diagram showing an example of calculating cognitive resources through time-series analysis that incorporates past information on instantaneous levels of arousal. Long-term arousal (Cognitive resource) is an estimated result obtained by accumulating the estimated instantaneous state of arousal over a longer period.

[0107] The long-term data storage unit 21 may store time-series data of instantaneous alertness levels and calculate the amount of cognitive resources consumed, or the remaining amount of cognitive resources, and the amount of cognitive resources recovered by performing time-series analysis that takes into account past information in the time-series of instantaneous alertness levels. Examples of time-series analysis that take into account past information from time series include DNN (Deep Neural Network), RNN (Recurrent Neural Network), LSTM (Long Short Term Memory), and Transformer. Specifically, for example, as shown in Figure 21, cognitive resources are calculated that reflect past information on instantaneous alertness levels.

[0108] [2.3 Effects] As described above, the information processing device according to this modified example makes it possible to visualize a recovery state having a different time constant than the fatigue state. Furthermore, it enables intervention of behavioral changes due to recovery and improves the accuracy of intervention of behavioral changes due to fatigue. In addition, it makes it possible to calculate the fatigue state and the recovery state not only by time integration but also by multifaceted methods, thereby improving the accuracy of state estimation.

[0109] <3. Modified Example B> The information processing device according to Modified Example B will be described below. In the following, parts that are substantially the same as the components of the information processing device according to the above embodiment will be denoted by the same reference numerals, and their descriptions will be omitted as appropriate.

[0110] [3.0 Overview] In the technology related to Modification A described above, the degree of recovery (recovery state) was accurately defined for fatigue recovery, and an estimate was made to reflect the recovery state in the cognitive resources indicating the fatigue state. Specifically, when there was a state transition accompanied by recovery, only the decay or reset of the time integral was performed from the context information, and the accurate update of the internal state during recovery was reflected in the recovery of cognitive resources. However, the estimation of cognitive resource consumption in Modification A described above did not take into account whether the cognitive resource consumption was due to task performance or due to mind wandering. Mind wandering refers to the phenomenon in which attention is diverted from the task at hand or the external environment and turns to internal thoughts and emotions. In particular, the mind wandering state is a state of low arousal, but it is known to consume cognitive resources separately from task performance (academic references 5, 6). Therefore, Modification A described above had the problem that errors could occur in the estimation of cognitive resource consumption.

[0111] [Academic Literature 5] Kiss, Luca, and Karina J. Linnell. "The effect of preferred background music on task-focus in sustained attention." Psychological research 85.6 (2021): 2313-2325. [Academic Literature 6] Thomson, David R., Derek Besner, and Daniel Smilek. "A resource-control account of sustained attention: Evidence from mind-wandering and vigilance paradigms." Perspectives on psychological science 10.1 (2015): 82-96.

[0112] Therefore, in the technology according to the above embodiment and the technology according to the above modified example A, if the emotion estimation unit has estimation units for both the attentional state and the mind-wandering state, and if the attentional state and the mind-wandering state can be separated and the temporal accumulation can be calculated, it is expected that the estimation of cognitive resource consumption will be made more accurate, and a technology for detecting the timing of intervention for performance decline will be provided.

[0113] This modified version includes the following technical differences from the technology of the above-described embodiment. In the technology of the above-described embodiment, cognitive resource consumption is estimated and intervention timing is determined by analyzing the instantaneous state of arousal and its temporal accumulation. However, in the technology of the above-described embodiment, the estimation of cognitive resource consumption does not take into account whether the cognitive resource consumption is due to task performance or due to mind wandering. In contrast, this modified version estimates cognitive resource consumption with high accuracy by the following means: - In the emotion estimation unit, the state of attention and the state of mind wandering are estimated using electroencephalogram and heart rate information. - In the intervention determination unit, the state of attention, the state of mind wandering, and the state of recovery are weighted and integrated during the calculation of temporal accumulation.

[0114] [3.1 Structure]

[0115] Figure 22 schematically shows one example configuration of the emotional state determination unit 14 in the information processing device according to Modification B. Figure 23 schematically shows one example configuration of the long-term data storage unit 21 in the information processing device according to Modification B.

[0116] The configuration of the information processing device according to Modification B may be substantially the same as that of the information processing device according to the above embodiment (Figure 4). In this modification, the emotional state determination unit 14 and the long-term data storage unit 21 have substantially the same components as those of the emotional state determination unit 14 and the long-term data storage unit 21 in the information processing device according to the above embodiment (Figure 4), but the functions of some components are different.

[0117] In this modified example, the emotional state determination unit 14 is capable of estimating the attentional state, mind-wandering state, and recovery state based on feature quantities extracted from biological signals. The emotional state determination unit 14 includes, for example, an attentional state determination unit 14A, a mind-wandering state determination unit 14B, and a recovery state determination unit 14C, as shown in Figure 22. The attentional state is a state of high arousal and is a stressful state. The mind-wandering state is a state of low arousal, but one in which cognitive resources are consumed separately from task performance. The recovery state is a state of low arousal in which recovery of the user's performance is expected.

[0118] The attention state determination unit 14A can estimate the attention state based, for example, on the feature quantity x calculated by the feature quantity extraction unit 13. The attention state determination unit 14A can use, for example, the techniques described in academic literature 5 and 6 to estimate the attention state.

[0119] [Academic Reference 5] Posner, MI and Petersen, SE, 1990. The attention system of the human brain. Annual review of neuroscience, 13(1), pp.25-42. [Academic Reference 6] Souza, Rhaira Helena Caetano E., and Eduardo Lazaro Martins Naves. "Attention detection in virtual environments using EEG signals: a scoping review." frontiers in physiology 12 (2021): 727840.

[0120] For example, suppose feature x is a theta wave, alpha wave, or beta wave signal from the frontal lobe, included in a measurement obtained from a sensor capable of measuring brain activity, such as an electroencephalogram (EEG) or magnetoencephalogram (MEG). In this case, the attention state determination unit 14A can determine the attention state by utilizing the difference in physiological responses between two classes, attentional and non-attentional states, based on the theta wave, alpha wave, or beta wave signal from the frontal lobe. The attention state determination unit 14A may also be capable of estimating the attention state using a technology different from the technology described in academic literature 5 and 6.

[0121] The mind-wandering state determination unit 14B can estimate the mind-wandering state based, for example, on the feature quantity x calculated by the feature quantity extraction unit 13. The mind-wandering state determination unit 14B can use, for example, the techniques described in academic references 7 and 8 to estimate the mind-wandering state.

[0122] [Academic Reference 7] Mittner, Matthias, et al. "A neural model of mind wandering." Trends in cognitive sciences 20.8 (2016): 570-578. [Academic Reference 8] Shinagawa, Kazushi, et al. "Brain-Body Interactions Influence the Transition from Mind Wandering to Awareness of Ongoing Thought." bioRxiv (2024): 2024-09.

[0123] For example, suppose feature x is theta, alpha, or beta wave signals from the frontal, central, and parietal regions, included in measurements obtained from a brain activity measurement sensor capable of measuring brain activity, such as electroencephalography (EEG) or magnetoencephalography (MEG). Hereinafter, "theta, alpha, or beta wave signals from the frontal, central, and parietal regions" will be referred to as the first signal. In this case, the mind-wandering state determination unit 14B can perform pattern analysis of the brain functional network on the first signal. Based on the results of the pattern analysis, the mind-wandering state determination unit 14B can determine the mind-wandering state by utilizing the difference in physiological responses between the two classes of mind-wandering state and non-mind-wandering state.

[0124] Furthermore, for example, suppose that feature quantity x includes the first signal and the signal of a heart-induced potential, which is included in the measurement value obtained by a sensor capable of measuring heart rate and heart rate variability using pulse waves, electrocardiograms, etc. A heart-induced potential is a potential that corresponds to a change in brain activity that appears in sync with the beating of the heart. Hereinafter, the "signal of a heart-induced potential" will be referred to as the second signal. In this case, the mind-wandering state determination unit 14B can perform an analysis of the activity decrease pattern for the first signal and an analysis of the amplitude increase pattern for the second signal. Based on the respective analysis results, the mind-wandering state determination unit 14B can determine the mind-wandering state by utilizing the difference in physiological responses between the two classes of mind-wandering state and non-mind-wandering state. The mind-wandering state determination unit 14B may be capable of estimating the mind-wandering state using a technology different from the technology described in academic literature 7 and 8.

[0125] The recovery state determination unit 14C can estimate the recovery state based, for example, on the feature quantity x calculated by the feature quantity extraction unit 13. The recovery state determination unit 14C can use, for example, the techniques described in academic literature 9 and 10 to estimate the recovery state.

[0126] [Academic Literature 9] Zhang, Yue, Lulu Zhang, Haoqiang Hua, Jianxiu Jin, Lingqing Zhu, Lin Shu, Xiangmin Xu, Feng Kuang, and Yunhe Liu. "Relaxation degree analysis using frontal electroencephalogram under virtual reality relaxation scenes." Frontiers in Neuroscience 15 (2021): 719869. [Academic Literature 10] Gaertner, Raphaela J., et al. "Relaxing effects of virtual environments on the autonomic nervous system indicated by heart rate variability: A systematic review." Journal of Environmental Psychology 88 (2023): 102035.

[0127] For example, suppose feature x is a signal of frontal delta waves, theta waves, low alpha waves, high alpha waves, low beta waves, high beta waves, or gamma waves included in a measurement obtained from a sensor capable of measuring brain activity, such as electroencephalography (EEG) or magnetoencephalography (MEG). Hereinafter, "frontal delta waves, theta waves, low alpha waves, high alpha waves, low beta waves, high beta waves, or gamma waves" will be referred to as the third signal. In this case, the recovery state determination unit 14C can determine the attention state based on the third signal by utilizing the difference in physiological responses between two classes: a recovered state and a non-recovered state.

[0128] Furthermore, for example, let's assume that feature x is a high-frequency (HF) component of heart rate variability, an RMSSD index that deals with the difference between adjacent heartbeat intervals, or a pNN50 index that deals with the difference between adjacent heartbeat intervals of 50 ms or more, all of which are included in the measurements obtained from a sensor that measures heart rate and heart rate variability using pulse waves, electrocardiograms, etc. Hereinafter, "high-frequency component of heart rate variability, RMSSD index, or pNN50 index" will be referred to as the fourth signal. In this case, the recovery state determination unit 14C can determine the attention state by utilizing the difference in physiological responses between two classes, the recovery state and the non-recovery state, based on the fourth signal. The recovery state determination unit 14C may be capable of estimating the recovery state using a technology different from the technology described in academic literature 9 and 10.

[0129] In this modified example, the long-term data storage unit 21 is a storage unit capable of storing long-term data of the judgment results of each emotion estimation and performing weighted time integration of the long-term data. By performing time integration of the changes in each emotion estimation, the long-term data storage unit 21 is capable of calculating the amount of cognitive resources consumed or the amount of cognitive resources remaining. The long-term data storage unit 21 includes, for example, a long-term data storage unit 21A capable of storing long-term data of the attentional state, a long-term data storage unit 21B capable of storing long-term data of the mind-wandering state, and a long-term data storage unit 21C capable of storing long-term data of the recovery state, as shown in Figure 23.

[0130] The long-term data storage unit 21A can calculate the amount of cognitive resources consumed or the remaining amount of cognitive resources by performing a time integral of the change in attentional state. The long-term data storage unit 21A can buffer samples of the attentional state at each time point determined by the attentional state determination unit 14A and analyze them as changes in long-term data. The long-term data storage unit 21A can perform a weighted time integral as shown in the first term on the right-hand side of equation (C) below.

[0131] The long-term data storage unit 21B can calculate the amount of cognitive resources consumed or the remaining amount of cognitive resources by performing a time integral of the changes in the mind-wandering state. The long-term data storage unit 21B can buffer samples of the mind-wandering state determined by the mind-wandering state determination unit 14B at each time point and analyze them as changes in long-term data. The long-term data storage unit 21B can perform a weighted time integral as shown in the second term on the right-hand side of equation (C) below.

[0132] The long-term data storage unit 21C can calculate the amount of cognitive resources consumed or the remaining amount of cognitive resources by performing a time integral of the change in the recovery state. The long-term data storage unit 21C can buffer samples of the recovery state at each time point determined by the recovery state determination unit 14C and analyze them as changes in long-term data. The long-term data storage unit 21C can perform a weighted time integral as shown in the third term on the right-hand side of equation (C) below.

[0133] The long-term data storage unit 21 further includes a synthesis unit 21D capable of synthesizing data obtained from the long-term data storage units 21A, 21B, and 1C, as shown in Figure 23, for example. The synthesis unit 21D is capable of calculating the cognitive resource consumption MF(t) by synthesizing the weighted time integral obtained from the long-term data storage unit 21A, the weighted time integral obtained from the long-term data storage unit 21B, and the weighted time integral obtained from the long-term data storage unit 21C, as shown in the following equation (C).

[0134]

[0135] Attention, Wmw, and Wrecovery represent the integral weights, respectively. Attention(t) is a positive value for cases where the user is determined to be in an attentional state at time t. Pmw(t) is a positive value for cases where the user is determined to be in a mind-wandering state at time t. Recovery(t) is a negative value for cases where the user is determined to be in a recovered state at time t. Cases determined to be in an attentional state specifically include a state in a driving environment where the driver is concentrating on driving while driving, a state in an office environment where the office worker is concentrating on a work task, or a state in a VR environment where the user is immersed in content while experiencing content. Cases determined to be in a mind-wandering state specifically include a state in a driving environment where the driver is distracted by things unrelated to driving while driving, a state in an office environment where the office worker is distracted by things unrelated to a work task, or a state in a VR environment where the user is distracted by things unrelated to the content while experiencing content. Cases judged to be in a recovered state are specifically those in which the individual is relaxed and physiologically recovered during activities such as listening to relaxing music or stress-relieving behaviors (meditation, deep breathing, etc.). B(t=0) represents the initial value of cognitive resource consumption. The initial value of cognitive resource consumption is the amount of cognitive resources consumed at time t=0.

[0136] Here, MF(t) is used as an example of the amount of cognitive resources consumed, but the remaining amount of cognitive resources can also be formulated using the above concept. In that case, the signs of Pattention(t), Pmw(t), and Precovery(t) will be inverted in relation to the signs mentioned above.

[0137] [3.2 Effects] As described above, the information processing device according to Modified Example B can calculate the amount of cognitive resources consumed or remaining for each state of attention, mind-wandering, and recovery. This makes it possible to estimate the amount of cognitive resources consumed or remaining with high accuracy.

[0138] <4. Modified Example C> The information processing device according to Modified Example C will be described below. In the following, parts that are substantially the same as the components of the information processing device according to the above embodiment will be denoted by the same reference numerals, and their descriptions will be omitted as appropriate.

[0139] [4.1 Structure]

[0140] Figure 24 schematically shows one example configuration of an information processing device according to modified example C.

[0141] The configuration of the information processing device according to Modification C may be substantially the same as that of the information processing device according to the above embodiment (Figure 4). In this modification, the context state determination unit 15 and the integral weight selection unit 23 have substantially the same components as the context state determination unit 15 and the integral weight selection unit 23 in the information processing device according to the above embodiment (Figure 4), but the functions of some components are different.

[0142] In this modified example, the context state determination unit 15 is capable of determining the user's situation (context) for which the emotional state is to be estimated, based on physical information. The context state determination unit 15 is capable of recognizing from the physical information what actions the user holding the sensor device 30 is taking. Furthermore, the context state determination unit 15 is capable of detecting from the physical information the timing of the event and the characteristics of the event that occurred.

[0143] The context state determination unit 15 can detect the timing of the occurrence of an external event related to concentration as the timing of the event occurrence, based on physical information obtained during the execution of a task requiring concentration, and can also detect the external event related to concentration as a characteristic of the event. The context state determination unit 15 can detect the timing of the occurrence of an external event related to relaxation as the timing of the event occurrence, based on physical information obtained during the execution of a task requiring relaxation, and can also detect the external event related to relaxation as a characteristic of the event.

[0144] The context state determination unit 15 is capable of generating an event signal δ(t) that indicates the time of the detected occurrence (event occurrence time). The event signal δ(t) corresponds to "1" for the event occurrence time, and to "0" for all other times. The context state determination unit 15 is capable of outputting the event signal δ(t) and the characteristics of the event to the integral weight selection unit 23 as external event trigger information. The event signal δ(t) generated when a concentration external event is detected as an event characteristic is, for example, the signal shown in Figure 25. The event signal δ(t) generated when a relaxation external event is detected as an event characteristic is, for example, the signal shown in Figure 26.

[0145] In this modified example, the integral weight selection unit 23 is a weight selection unit that can dynamically change the weights of the time integral by the long-time data storage unit 21 according to the determination result of the context state determination unit 15. The integral weight selection unit 23 can change the weight Wstress to a value greater than the initial value (e.g., 1) when it obtains an external event of the concentration as an event characteristic from the context state determination unit 15. For example, when a user is playing a game, if it obtains that an enemy character has appeared in the game as an external event of the concentration, the integral weight selection unit 23 can change the weight Wstress to a value greater than the initial value (e.g., 1).

[0146] The integral weight selection unit 23 can change the weight Wrecovery to a value greater than its initial value (e.g., 1) if it receives an external event of relaxation as a characteristic of the event from the context state determination unit 15. For example, if the integral weight selection unit 23 receives an external event of relaxation when a relaxing effect appears in the music while the user is playing music, it can change the weight Wrecovery to a value greater than its initial value (e.g., 1).

[0147] The integral weight selection unit 23 can vary the integral value adjustment gain g according to the characteristics of the acquired event when it acquires the characteristics of the event. When the integral weight selection unit 23 acquires an external event of a concentration as the characteristics of the event, it can vary the integral value adjustment gain gstress(Δt). The integral value adjustment gain gstress(Δt) is a correction gain of Pstress(t), and is a correction gain that changes from the event occurrence time as the starting point to the time thereafter. Let Δt be the time after the event occurrence time. When the integral weight selection unit 23 acquires an external event of a concentration as the characteristics of the event, it can change the integral value adjustment gain gstress(Δt) to a gradually larger value as time Δt progresses, for example as shown in Figure 25. Furthermore, when time Δt has passed a predetermined time x, the integral weight selection unit 23 can change the integral value adjustment gain gstress(Δt) to a gradually smaller value as time Δt progresses, for example as shown in Figure 25. The integral weight selection unit 23 can further stop the fluctuation of the integral value adjustment gain gstress(Δt) and fix the integral value adjustment gain gstress(Δt) to a constant value, as shown in Figure 25, when the integral value adjustment gain gstress(Δt) becomes the same value as the value at the time of event occurrence.

[0148] The integral weight selection unit 23 is configured such that, as an event characteristic, when an external event of a concentration is acquired, the integral value adjustment gain grecovery(Δt) can be fixed to a constant value without changing it. The integral value adjustment gain grecovery(Δt) is a correction gain of Precovery(t), and is a correction gain that changes from the event occurrence time onward.

[0149] The integral weight selection unit 23 is capable of varying the integral value adjustment gain greecovery(Δt) when it acquires an external event of relaxation as an event characteristic. When the integral weight selection unit 23 acquires an external event of relaxation as an event characteristic, it is possible to gradually change the integral value adjustment gain greecovery(Δt) to a larger value as time Δt progresses, for example as shown in Figure 26. Furthermore, when time Δt has passed a predetermined time x, the integral weight selection unit 23 is capable of gradually changing the integral value adjustment gain greecovery(Δt) to a smaller value as time Δt progresses, for example as shown in Figure 26. Furthermore, when the integral value adjustment gain greecovery(Δt) becomes the same value as the value at the time of event occurrence, the integral weight selection unit 23 is capable of stopping the variation of the integral value adjustment gain greecovery(Δt) and fixing the integral value adjustment gain greecovery(Δt) to a constant value, for example as shown in Figure 26.

[0150] The integral weight selection unit 23 is configured such that, as an event characteristic, when an external event of relaxation is acquired, the integral value adjustment gain gstress(Δt) can be fixed to a constant value without being varied.

[0151] The long-term data storage unit 21 is capable of performing weighted time integration as shown in equation (D) using integral weights Wstress, Wrecovery and integral value adjustment gains gstress(Δt), grecovery(Δt). As a result, the long-term data storage unit 21 adaptively weights the integral value of the emotional state (result of the emotional estimation judgment) in response to the occurrence of external events.

[0152] [4.2 Effects] As described above, the information processing device according to this modified example can adaptively weight the integral value of the emotional state (the result of the emotion estimation) according to the characteristics of the external event that occurs. This makes it possible to estimate the amount of cognitive resources consumed or remaining with high accuracy.

[0153] <5. Modified Example D> The information processing device according to Modified Example D will be described below. In the following, parts that are substantially the same as the components of the information processing device according to the above embodiment will be denoted by the same reference numerals, and their descriptions will be omitted as appropriate.

[0154] [5.1 Structure]

[0155] The configuration of the information processing device according to Modification D may be substantially the same as that of the information processing device according to Modification B. In this modification, the context state determination unit 15 and the integral weight selection unit 23 have substantially the same components as the context state determination unit 15 and the integral weight selection unit 23 in the information processing device according to Modification B, but the functions of some components are different.

[0156] In this modified example, the context state determination unit 15 is capable of determining the user's situation (context) for which the emotional state is to be estimated, based on physical information. The context state determination unit 15 is capable of recognizing from the physical information what actions the user holding the sensor device 30 is taking. Furthermore, the context state determination unit 15 is capable of detecting from the physical information the timing of the event and the characteristics of the event that occurred.

[0157] The context state determination unit 15 can detect the timing of the occurrence of an external event related to concentration as the timing of the event occurrence, based on physical information obtained during the execution of a task requiring concentration, and can further detect the external event related to concentration as a characteristic of the event that occurred. The context state determination unit 15 can detect the timing of the occurrence of an external event related to relaxation as the timing of the event occurrence, based on physical information obtained during the execution of a task requiring relaxation, and can further detect the external event related to relaxation as a characteristic of the event that occurred.

[0158] The context state determination unit 15 is capable of generating an event signal δ(t) that indicates the time of the detected occurrence (event occurrence time). The event signal δ(t) corresponds to "1" for the event occurrence time, and to "0" for all other times. The context state determination unit 15 is capable of outputting the event signal δ(t) and the characteristics of the occurred event to the integral weight selection unit 23 as external event trigger information. For example, the event signal δ(t) generated when a concentration external event is detected as an event characteristic will be the signal shown in Figure 27. For example, the event signal δ(t) generated when a relaxation external event is detected as an event characteristic will be the signal shown in Figure 28.

[0159] In this modified example, the integral weight selection unit 23 is a weight selection unit that can dynamically change the weights of the time integral by the long-time data storage unit 21 according to the determination result of the context state determination unit 15. When the integral weight selection unit 23 acquires an external event of the concentration as an event characteristic from the context state determination unit 15, it is possible to change the weight Wattention to a value greater than the initial value (e.g., 1) and change the weight Wmw to a value less than the initial value (e.g., 1). For example, when a user is playing a game, if the appearance of an enemy character in the game is acquired as an external event of the concentration, the integral weight selection unit 23 is possible to change the weight Wattention to a value greater than the initial value (e.g., 1) and change the weight Wmw to a value less than the initial value (e.g., 1).

[0160] The integral weight selection unit 23 can change the weight Wrecovery to a value greater than its initial value (e.g., 1) if it receives an external event of relaxation as a characteristic of the event from the context state determination unit 15. For example, if the integral weight selection unit 23 receives an external event of relaxation when a relaxing effect appears in the music while the user is playing music, it can change the weight Wrecovery to a value greater than its initial value (e.g., 1).

[0161] The integral weight selection unit 23 can vary the integral value adjustment gain g according to the characteristics of the acquired event when it acquires the characteristics of the event. When the integral weight selection unit 23 acquires an external event of a concentration as the characteristics of the event, it can vary the integral value adjustment gain gstress(Δt). The integral value adjustment gain gstress(Δt) is a correction gain of Pstress(t), and is a correction gain that changes from the event occurrence time as the starting point to the time thereafter. Let Δt be the time after the event occurrence time. When the integral weight selection unit 23 acquires an external event of a concentration as the characteristics of the event, it can change the integral value adjustment gain gstress(Δt) to a gradually larger value as time Δt progresses, for example as shown in Figure 27. Furthermore, when time Δt has passed a predetermined time x, the integral weight selection unit 23 can change the integral value adjustment gain gstress(Δt) to a gradually smaller value as time Δt progresses, for example as shown in Figure 27. Furthermore, when the integral value adjustment gain gstress(Δt) becomes the same value as the value at the time of the event occurrence, the integral weight selection unit 23 can stop the fluctuation of the integral value adjustment gain gstress(Δt) and fix the integral value adjustment gain gstress(Δt) to a constant value, as shown in Figure 27, for example.

[0162] The integral weight selection unit 23 is capable of varying the integral value adjustment gain gmw(Δt) when an external event of a concentration is acquired, as an event characteristic. The integral value adjustment gain gmw(Δt) is a correction gain of Pmw(t), and is a correction gain that changes from the event occurrence time as a starting point to the time thereafter. When an external event of a concentration is acquired, as an event characteristic, the integral weight selection unit 23 is capable of gradually changing the integral value adjustment gain gmw(Δt) to a smaller value as time Δt progresses, for example as shown in Figure 27. Furthermore, when time Δt has passed a predetermined time x, the integral weight selection unit 23 is capable of gradually changing the integral value adjustment gain gmw(Δt) to a larger value as time Δt progresses, for example as shown in Figure 27. Furthermore, when the integral value adjustment gain gstress(Δt) becomes the same value as the value at the time of the event occurrence, the integral weight selection unit 23 can stop the fluctuation of the integral value adjustment gain gmw(Δt) and fix the integral value adjustment gain gmw(Δt) to a constant value, as shown in Figure 27, for example.

[0163] The integral weight selection unit 23 is configured such that, as an event characteristic, when an external event of a concentration is acquired, the integral value adjustment gain grecovery(Δt) can be fixed to a constant value without changing it. The integral value adjustment gain grecovery(Δt) is a correction gain of Precovery(t), and is a correction gain that changes from the event occurrence time onward.

[0164] The integral weight selection unit 23 is capable of varying the integral value adjustment gain greecovery(Δt) when it acquires an external event of relaxation as an event characteristic. When the integral weight selection unit 23 acquires an external event of relaxation as an event characteristic, it is possible to gradually change the integral value adjustment gain greecovery(Δt) to a larger value as time Δt progresses, for example as shown in Figure 28. Furthermore, when time Δt has passed a predetermined time x, the integral weight selection unit 23 is capable of gradually changing the integral value adjustment gain greecovery(Δt) to a smaller value as time Δt progresses, for example as shown in Figure 28. Furthermore, when the integral value adjustment gain greecovery(Δt) becomes the same value as the value at the time of event occurrence, the integral weight selection unit 23 is capable of stopping the variation of the integral value adjustment gain greecovery(Δt) and fixing the integral value adjustment gain greecovery(Δt) to a constant value, for example as shown in Figure 28.

[0165] The integral weight selection unit 23 is configured such that, as an event characteristic, when an external event of relaxation is acquired, the integral value adjustment gains gstress(Δt) and gmw(Δt) can be fixed to a constant value without changing.

[0166] The long-term data storage unit 21 is capable of performing weighted time integration as shown in equation (E) using integral weights Wstress, Wmw, Wrecovery and integral value adjustment gains gstress(Δt), gmw(Δt), grecovery(Δt). As a result, the long-term data storage unit 21 adaptively weights the integral value of the emotional state (result of the emotional estimation judgment) in response to the occurrence of external events.

[0167] [5.2 Effects] As described above, the information processing device according to this modified example can adaptively provide a gain to the integral value of the emotional state (the result of the emotion estimation judgment) according to the characteristics and timing of the external event that occurs. This makes it possible to estimate the amount of cognitive resources consumed or remaining with high accuracy.

[0168] <6. Other Embodiments> The technology described herein is not limited to the above-described embodiment and can be implemented in various modified forms.

[0169] For example, this technology can also take the following configuration. According to this configuration of the technology, a decision is made on whether or not to perform an intervention to change the emotional state based on first data indicating the emotional state estimated at predetermined time intervals and second data obtained by accumulating the first data over a period longer than the predetermined time intervals. This makes it possible to provide an information processing device that can expand the range of applications for emotional estimation.

[0170] <1> An information processing device comprising one or more processors capable of performing processing based on biological signals, wherein the one or more processors are capable of estimating an emotional state from the biological signals at predetermined time intervals, and performing a first determination process to determine whether or not to perform an intervention to change the emotional state based on first data indicating the estimated emotional state and second data obtained by accumulating the first data over a period longer than the predetermined time interval. <2> The information processing device according to <1> above, wherein the one or more processors are capable of visualizing the first data and the second data. <3> The information processing device according to <1> or <2> above, wherein the one or more processors are capable of accumulating the second data and performing a weighted time integral of the second data. <4> The information processing device according to <3> above, wherein the one or more processors are capable of performing a second determination process to determine the situation of the user whose emotional state is to be estimated based on physical information, and dynamically changing the weights of the time integral according to the determination result of the second determination process. (5) The information processing apparatus according to <4> above, wherein one or more processors are capable of calculating an integral value adjustment gain for dynamically attenuating or resetting the integral value of the time integral according to the determination result of the second determination process. <6> The information processing apparatus according to <4> above, wherein one or more processors are capable of calculating an integral value adjustment gain for dynamically attenuating or resetting the integral value of the time integral according to the determination result of the first determination process and the determination result of the second determination process. <7> The information processing apparatus according to any one of <4> to <7> above, wherein the first data includes an instantaneous level of alertness, and one or more processors are capable of calculating the amount of cognitive resources consumed or the remaining amount of cognitive resources by performing a time integral of the change in the instantaneous level of alertness.<8> The information processing device according to <7> above, wherein the one or more processors are able to perform the intervention when the second determination process determines that it is a situation in which concentration is preferable, and the instantaneous level of arousal is determined to be below a threshold, or when the instantaneous level of arousal is above a threshold and the amount of cognitive resources consumed is above a certain level. <9> The information processing device according to <7> above, wherein the one or more processors are further able to calculate the amount of cognitive resources recovered. <10> The information processing device according to <9> above, wherein the one or more processors are able to dynamically correct the amount of cognitive resources consumed based on the amount of cognitive resources recovered. <11> The information processing device according to <9> or <10> above, wherein the one or more processors are able to perform the intervention to promote concentration when the second determination process determines that it is a situation in which a change in behavior from a concentration-related behavior to a relaxation-related behavior is being carried out, and the amount of cognitive resources recovered is above a certain level. <12> The information processing device according to any one of <9> to <11> above, wherein one or more processors are capable of calculating the amount of cognitive resource recovery by using the instantaneous level of alertness during sustained alertness as a reference and performing a time integral of the decrease from that reference. <13> The information processing device according to any one of <9> to <12> above, wherein one or more processors are capable of calculating the amount of cognitive resource consumption as a ratio to the amount of cognitive resource consumption when the instantaneous level of alertness is a first reference indicating a fatigued state, and calculating the amount of cognitive resource recovery as a ratio to the amount of cognitive resource recovery when the instantaneous level of alertness is a second reference indicating a recovered state.<14> The information processing device according to <1> or <2> above, wherein the first data includes instantaneous level of alertness, and the one or more processors accumulate time-series data of the instantaneous level of alertness, and by time-series analysis that takes into account past information in the time-series of the instantaneous level of alertness, it is possible to calculate the amount of cognitive resources consumed, or the remaining amount of cognitive resources, and the amount of cognitive resources recovered. <15> The information processing device according to any one of <2> to <14> above, wherein the one or more processors are capable of visualizing the fatigue state and recovery state determined based on the first data and the second data. <16> The information processing device according to any one of <1> to <15> above, wherein the one or more processors perform preprocessing on the biological signal to remove unnecessary components, extract physiological responses contributing to emotion as features from the biological signal after the preprocessing, and determine the emotional state based on the features. <17> An information processing device according to any one of <1> to <16> above, wherein one or more processors are capable of estimating an attention state, a mind-wandering state, and a recovery state based on features extracted from the biological signal. <18> An information processing device according to <17>, wherein one or more processors are capable of calculating the amount of cognitive resources consumed or the remaining amount of cognitive resources based on data obtained by accumulating the attention state over a period longer than the predetermined time interval, data obtained by accumulating the mind-wandering state over a period longer than the predetermined time interval, and data obtained by accumulating data indicating the recovery state over a period longer than the predetermined time interval. <19> An information processing device according to <18>, wherein one or more processors are capable of adaptively weighting the integral value of the attention state, the integral value of the mind-wandering state, and the integral value of the recovery state according to the characteristics of external events detected from the physical information.<20> The information processing apparatus according to <18>, wherein the one or more processors are capable of adaptively providing gains to the integral value of the attention state, the integral value of the mind-wandering state, and the integral value of the recovery state, depending on the characteristics and timing of occurrence of external events detected from the physical information.

[0171] This application claims priority based on Japanese Patent Application No. 2025-019544, filed with the Japan Patent Office on 7 February 2025, and all contents of that application are incorporated herein by reference.

[0172] Those skilled in the art will understand that various modifications, combinations, subcombinations, and changes can be conceived depending on design requirements and other factors, and that these fall within the scope of the attached claims and their equivalents.

Claims

1. An information processing device comprising one or more processors capable of performing processing based on biological signals, wherein the one or more processors are capable of estimating an emotional state from the biological signals at predetermined time intervals, and performing a first determination process to determine whether or not to perform an intervention to change the emotional state based on first data indicating the estimated emotional state and second data obtained by accumulating the first data over a period longer than the predetermined time interval.

2. The information processing apparatus according to claim 1, wherein the one or more processors are capable of visualizing the first data and the second data.

3. The information processing apparatus according to claim 1, wherein the one or more processors are capable of storing the second data and performing weighted time integration of the second data.

4. The information processing apparatus according to claim 3, wherein the one or more processors perform a second determination process to determine the status of the user whose emotional state is to be estimated based on physical information, and the weights of the time integral can be dynamically changed according to the determination result of the second determination process.

5. The information processing apparatus according to claim 4, wherein the one or more processors are capable of calculating an integral value adjustment gain for dynamically attenuating or resetting the integral value of the time integral according to the determination result of the second determination process.

6. The information processing apparatus according to claim 4, wherein one or more processors are capable of calculating an integral value adjustment gain for dynamically attenuating or resetting the integral value of the time integral according to the determination result of the first determination process and the determination result of the second determination process.

7. The information processing apparatus according to claim 4, wherein the first data includes an instantaneous level of alertness, and the one or more processors can calculate the amount of cognitive resources consumed or the remaining amount of cognitive resources by performing a time integral of the change in the instantaneous level of alertness.

8. The information processing apparatus according to claim 7, wherein the one or more processors are able to perform the intervention when the second determination process determines that it is preferable to concentrate, and the instantaneous level of alertness is determined to be below a threshold, or when the instantaneous level of alertness is above a threshold and the consumption of cognitive resources is above a certain level.

9. The information processing apparatus according to claim 7, wherein the one or more processors are further capable of calculating the amount of recovery of the cognitive resources.

10. The information processing apparatus according to claim 9, wherein the one or more processors are capable of dynamically correcting the consumption of the cognitive resources based on the amount of the cognitive resources recovered.

11. The information processing apparatus according to claim 9, wherein, when the one or more processors determine by the second determination process that a change in behavior from a behavior involving concentration to a behavior involving relaxation is being carried out, and when it is determined that the amount of recovery of the cognitive resources is above a certain level, the information processing apparatus is capable of performing an intervention to promote a behavior involving concentration as the intervention.

12. The information processing apparatus according to claim 9, wherein one or more processors are capable of calculating the amount of recovery of cognitive resources by using the instantaneous level of alertness during sustained alertness as a reference and performing a time integral of the decrease from that reference.

13. The information processing apparatus according to claim 9, wherein one or more processors are capable of calculating the consumption of cognitive resources as a ratio to the consumption of cognitive resources when the instantaneous level of alertness is a first criterion indicating a fatigued state, and calculating the recovery amount of cognitive resources as a ratio to the recovery amount of cognitive resources when the instantaneous level of alertness is a second criterion indicating a recovered state.

14. The information processing apparatus according to claim 1, wherein the first data includes instantaneous alertness, and the one or more processors accumulate time-series data of the instantaneous alertness, and by performing time-series analysis that takes into account past information in the time-series of the instantaneous alertness, it is possible to calculate the amount of cognitive resources consumed, or the remaining amount of cognitive resources, and the amount of cognitive resources recovered.

15. The information processing apparatus according to claim 2, wherein the one or more processors are capable of visualizing the fatigue state and recovery state determined based on the first data and the second data.

16. The information processing apparatus according to claim 1, wherein the one or more processors perform preprocessing on the biological signal to remove unwanted components, extract physiological responses contributing to emotion as features from the preprocessed biological signal, and determine the emotional state based on the features.

17. The information processing apparatus according to claim 1, wherein the one or more processors are capable of estimating an attention state, a mind-wandering state, and a recovery state based on feature quantities extracted from the biological signal.

18. The information processing apparatus according to claim 17, wherein one or more processors are capable of calculating the amount of cognitive resources consumed or the remaining amount of cognitive resources based on data obtained by accumulating the attention state over a period longer than the predetermined time interval, data obtained by accumulating the mind-wandering state over a period longer than the predetermined time interval, and data obtained by accumulating data indicating the recovery state over a period longer than the predetermined time interval.

19. The information processing apparatus according to claim 18, wherein the one or more processors are capable of adaptively weighting the integral value of the attention state, the integral value of the mind-wandering state, and the integral value of the recovery state according to the characteristics of the external events detected from the physical information.

20. The information processing apparatus according to claim 18, wherein the one or more processors are capable of adaptively providing gains to the integral value of the attention state, the integral value of the mind-wandering state, and the integral value of the recovery state, depending on the characteristics and timing of occurrence of external events detected from the physical information.

Images