Behavior control system, behavior control method, and program

Abstract

This behavior control system comprises: a communication device that communicates with a subject; a subject information acquisition unit that acquires subject information of the subject; an empathy level prediction unit that predicts an empathy level of the subject on the basis of the acquired subject information; an anxiety level prediction unit that predicts an anxiety level of the subject on the basis of the acquired subject information; and a behavior generation unit that generates a behavior for the communication device on the basis of the predicted empathy level and anxiety level.

Description

Behavior control system, behavior control method, and program

[0001] The present invention relates to a behavior control system, a behavior control method, and a program. This application claims priority under Japanese Patent Application No. 2024-206303, filed in Japan on November 27, 2024, the contents of which are incorporated herein by reference.

[0002] Generally, hospitals that treat diseases such as cancer focus on the clinical aspects of disease treatment, and much of the hospital's resources, such as personnel and funding, are allocated to these clinical aspects. On the other hand, for patients who require emotional support, such as children, in addition to the clinical aspects mentioned above, support is needed to maintain their mental, social, and emotional health.

[0003] Research is underway on methods using communication robots to address the emotional support needs of patients as described above. To facilitate more natural communication with patients, it is crucial to control the communication robot's behavior in accordance with the patient's responses (emotions). For example, systems have been proposed that determine the robot's actions after assessing a person's emotions (see, for example, Patent Documents 1 and 2).

[0004] Japanese Patent Publication No. 2020-17206, International Publication No. 2024 / 195570

[0005] Various types of information can be used to determine a patient's emotions, including their speech, facial expressions, and actions (gestures). There is a need for a method that can accurately determine a patient's emotions by comprehensively evaluating this diverse information, and that enables appropriate control of the communication robot's behavior in accordance with the determined emotions.

[0006] The present invention has been made in view of the above-mentioned problems, and aims to provide a behavior control system, a behavior control method, and a program that enable appropriate control of the behavior of a communication device in accordance with the emotions of a subject.

[0007] (1) To achieve the above objective, an action control system according to one aspect of the present invention comprises: a communication device for communicating with a subject; a subject information acquisition unit for acquiring subject information of the subject; an empathy level prediction unit for predicting the subject's empathy level based on the acquired subject information; an anxiety level prediction unit for predicting the subject's anxiety level based on the acquired subject information; and an action generation unit for generating the actions of the communication device based on the predicted empathy level and anxiety level.

[0008] (2) In the behavior control system described in (1) above, the subject information includes text information based on the subject's acoustic information and video information of the subject, and the empathy prediction unit extracts semantic cues from the text information, extracts video cues from the video information, combines the extracted semantic cues and video cues to generate combined information, and predicts the subject's empathy level based on the generated combined information.

[0009] (3) In the behavior control system of (1) or (2) above, the subject information includes video information of the subject, physiological information of the subject, and acoustic information of the subject, and the anxiety level prediction unit may extract the subject's facial expressions and behavioral characteristics from the video information, extract the subject's physiological characteristics from the physiological information, extract the subject's linguistic characteristics from the acoustic information, and predict the anxiety level based on the extracted facial expressions and behavioral characteristics, physiological characteristics, and linguistic characteristics.

[0010] (4) The behavior control system described in (1) or (2) above may further include a behavior control unit that controls the behavior of the communication device based on the generated behavior of the communication device.

[0011] (5) In the behavior control system described in (2) above, the empathy prediction unit may further extract acoustic cues from the acoustic information, combine the extracted semantic cues, visual cues, and acoustic cues to generate combined information, and predict the subject's empathy level based on the generated combined information.

[0012] (6) In the behavior control system described in (5) above, the empathy prediction unit may generate the combined information by weighting and adding the semantic arousal level and emotional valence included in the semantic cues, the visual arousal level and emotional valence included in the visual cues, and the acoustic arousal level and emotional valence included in the acoustic cues for each conversational exchange between the subject and the communication device.

[0013] (7) In the behavior control system of (1) or (2) above, the behavior generation unit may generate the behavior of the communication device using the output obtained by inputting the predicted degree of empathy and the degree of anxiety into a machine learning model which is a Generative Artificial Intelligence (AI).

[0014] (8) The behavior control system described in (7) above may further include a learning unit that performs learning processing on the machine learning model using a dataset that includes the speaker's utterance, the listener's utterance, and the listener's empathy level, which are obtained in a conversation between the speaker and the listener.

[0015] (9) A method for controlling the behavior of a communication device that communicates with a subject according to another aspect of the present invention involves a computer acquiring subject information of the subject, predicting the subject's level of empathy based on the acquired subject information, predicting the subject's level of anxiety based on the acquired subject information, generating the behavior of the communication device based on the predicted level of empathy and anxiety, and controlling the behavior of the communication device based on the generated behavior of the communication device.

[0016] (10) A program for controlling the behavior of a communication device that communicates with a subject according to another aspect of the present invention causes a computer to acquire subject information of the subject, predict the subject's level of empathy based on the acquired subject information, predict the subject's level of anxiety based on the acquired subject information, generate the behavior of the communication device based on the predicted level of empathy and anxiety, and control the behavior of the communication device based on the generated behavior of the communication device.

[0017] According to (1) to (10), it is possible to appropriately control the behavior of the communication device in accordance with the emotions of the subject. The behavior control system can be used, for example, to set up Human-Robot Interaction (HRI) between the communication device and a person. For example, when such a communication device is used in a hospital, the communication device can behave as a friend to subjects (patients) such as children, and through fun activities with the communication device, it can reduce the opportunities for children to feel bored or lonely, and even make children smile. The communication device can also behave as an (emotional) support for doctors, and can assist doctors when explaining the child's condition, treatment, etc. The communication device can also behave as a psychologist and can provide mental / emotional care to children.

[0018] This is a diagram illustrating the overview of the behavior control processing according to the embodiment. This is a diagram showing an example of the input and output data of the empathy prediction unit according to the embodiment. This is a diagram showing an example of the input and output data of the anxiety prediction unit according to the embodiment. This is a diagram showing an example of the input and output data of the behavior generation unit according to the embodiment. This is a diagram showing an example of the behavior control system S according to the embodiment. This is a diagram showing an example of the external appearance of the communication robot 1 according to the embodiment. This is a functional block diagram showing an example of the configuration of the communication robot 1 according to the embodiment. This is a flowchart showing an example of the processing flow of the behavior control device 10 according to the embodiment. This is a diagram illustrating the overview of the empathy prediction processing according to the embodiment. This is a flowchart showing an example of the flow of the empathy prediction processing by the behavior control device 10 according to the embodiment. This is a flowchart showing an example of the flow of the anxiety prediction processing by the behavior control device 10 according to the embodiment.

[0019] Hereinafter, an embodiment of the behavior control system, behavior control method, and program of the present invention will be described with reference to the drawings. The behavior control system according to this embodiment performs a process of understanding a person (Human Understanding) and a process of automatically controlling the behavior of a communication device (Autonomous Behavior Control). In the process of understanding a person, for example, emotional cues and internal states (anxiety, stress, etc.) are estimated using sensors, wearable devices, cameras, microphones, etc. Data obtained from these sensors, etc., is processed by machine learning algorithms, etc. On the other hand, in the process of automatically controlling the behavior of a communication device, the communication device behaves, for example, as a friend of the target person (hereinafter referred to as "subject" or "user"), making the subject smile and realizing empathetic conversation that reduces the subject's boredom or loneliness. Furthermore, when used in a medical institution such as a hospital, the communication device engages in empathetic conversation that supports doctors in explaining the subject's condition and treatment, and maintains the subject's motivation.

[0020] Such behavioral control systems facilitate communication with subjects by combining a communication device with a Generative Artificial Intelligence (Generative AI) module. The primary task of this communication device is to engage in empathetic conversation (facilitating communication with subjects) by carefully selecting the content, tone, and manner of speaking (e.g., empathetic words and tone). Furthermore, the actions of the communication robot, such as the movement of actuators on the communication device, screen changes on the display, and overall expression, are automatically generated without manual programming or scripting.

[0021] Figure 1 is a diagram illustrating the outline of the behavior control processing according to the embodiment. The process for understanding a person includes, for example, (1) multimodal data acquisition processing, (2) multimodal data processing, (3) empathy level prediction processing, and (4) anxiety level prediction processing. The process for automatically controlling the behavior of the communication device includes, for example, (5) behavior generation processing and (6) behavior control processing.

[0022] (1) In the multimodal data acquisition process, diverse data (multimodal data) is acquired from the subject. Multimodal data includes, for example, acoustic data of the subject acquired by a microphone (speech data, acoustic information, acoustic signals), video data of the subject acquired by a camera, and physiological data acquired by wearable devices worn by the subject.

[0023] (2) In multimodal data processing, processing is performed according to the type of data acquired. For example, audio processing is performed on acoustic data to generate acoustic information (waveform information) and semantic information (text information). For example, video processing is performed on video data to generate video information (for example, video information of faces).

[0024] (3) In the empathy level prediction process, the empathy level of the subject (or communication device) is predicted by the empathy level prediction unit, which will be described later. Empathy level indicates the degree (level) to which the subject empathizes with the actions of the communication device (or vice versa). Figure 2 shows an example of the input and output data of the empathy level prediction unit according to the embodiment. The empathy level prediction unit takes semantic information, acoustic information, and visual information as input data and the empathy level as output data. Details of the processing of the empathy level prediction unit will be described later.

[0025] (4) In the anxiety level prediction process, the anxiety level of the subject (or communication device) is predicted by the anxiety level prediction unit, which will be described later. The anxiety level indicates the degree (level) of anxiety that the subject feels in response to the behavior of the communication device (or vice versa). Figure 3 shows an example of the input and output data of the anxiety level prediction unit according to the embodiment. The anxiety level prediction unit takes acoustic information, visual information, and physiological information as input data and outputs the anxiety level as output data. Details of the processing of the anxiety level prediction unit will be described later.

[0026] (5) In the behavior generation process, the behavior generation unit (generation AI), described later, generates the behavior of the communication device (conversation content, actions, display content, etc.). Figure 4 shows an example of the input and output data of the behavior generation unit according to the embodiment. The behavior generation unit takes the degree of empathy, the degree of anxiety, the conversation that took place, and the task (prompt) as input data, and outputs information corresponding to the task (response behavior, e.g., conversation content, actions, display content, etc.). Details of the processing of the behavior generation unit will be described later.

[0027] (6) In the behavior control process, the behavior of the communication device is controlled according to the behavior (instruction information) generated by the behavior generation unit. The communication device performs, for example, set movements and actions, speech synthesis (TTS; Text to Speech), and conversation (improvised empathetic conversation as needed).

[0028] <System Configuration> Next, the configuration of the behavior control system of the embodiment will be described. Figure 5 is a diagram showing an example of the behavior control system S according to the embodiment. The behavior control system S includes, for example, a communication robot 1, a wearable device 3, and a behavior control device 10. These components are connected to each other so as to be able to communicate via a network. The network includes WAN (Wide Area Network), LAN (Local Area Network), the Internet, public lines, provider equipment, dedicated lines, wireless base stations, etc.

[0029] [Communication Robot] Figure 6 shows an example of the external appearance of the communication robot 1 according to the embodiment. The communication robot 1 is an example of a "communication device". The communication robot 1 according to the embodiment is a robot that communicates with one or more subjects U. Communication mainly consists of dialogue and gestures (movements). Movements are expressed not only by actual movements but also by images displayed on the display unit. Subject U responds to the actions presented by the communication robot 1 with facial expressions, gestures, voice, etc. For example, the communication robot 1 achieves communication by learning the content of dialogue and movements, which are forms of communication with humans, using machine learning (reinforcement learning) technology.

[0030] In Figure 6, the front view g101 and the side view g102 are diagrams showing an example of the external shape of the communication robot 1 according to the embodiment. The communication robot 1 is equipped with, for example, three display units 111 (eye display unit 111a, eye display unit 111b, and mouth display unit 111c). The imaging unit 102a is mounted on the upper part of the eye display unit 111a, and the imaging unit 102b is mounted on the upper part of the eye display unit 111b. The eye display units 111a and 111b correspond to human eyes and display images or image information corresponding to human eyes. The eye display units 111a and 111b are, for example, liquid crystal display devices or organic EL (Electro Luminescence) display devices. The screen size of the eye display units 111a and 111b is, for example, 3 inches. The speaker 112 is mounted near the mouth display unit 111c, which displays an image corresponding to a human mouth on the housing 120. The mouth display unit 111c is composed of, for example, multiple LEDs (light-emitting diodes), each of which is addressable and can be individually turned on or off. The sound pickup unit 103 is mounted on the housing 120.

[0031] Furthermore, the communication robot 1 is equipped with a boom 121. The boom 121 is movably attached to the housing 120 via a movable part 131. A horizontal bar 122 is rotatably attached to the boom 121 via a movable part 132. An eye display unit 111a is rotatably attached to the horizontal bar 122 via a movable part 133, and an eye display unit 111b is rotatably attached via a movable part 134. Note that the external shape of the communication robot 1 shown in Figure 1 is an example and is not limited thereto.

[0032] Figure 7 is a functional block diagram showing an example configuration of a communication robot 1 according to an embodiment. The communication robot 1 includes, for example, an imaging unit 102, a sound collection unit 103, an actuator 104, a display unit 111, a speaker 112, a control unit 200, and a storage unit 300.

[0033] The imaging unit 102 captures the environment where the communication robot 1 is installed. As shown in FIG. 6, the imaging unit 102 includes, for example, an imaging unit 102a and an imaging unit 102b. The imaging unit 102 may be installed in the environment where the communication robot 1 and the subject U are present. The imaging unit 102 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) imaging element, a CCD (Charge Coupled Device) imaging element (camera), or the like. The image information captured by the imaging unit 102 is, for example, an image including the face and body of the subject U. Note that the image may be any one of a still image, a series of still images, or a moving image.

[0034] The sound collection unit 103 collects (records) acoustic data (acoustic signals) in the environment where the communication robot 1 is installed. The sound collection unit 103 is, for example, a microphone array including a plurality of microphones. The sound collection unit 103 may be installed in the environment where the communication robot 1 and the subject U are present.

[0035] The actuator 104 drives each part of the communication robot 1 according to the control of the control unit 200.

[0036] As shown in FIG. 6, the display unit 111 includes, for example, an eye display unit 111a, an eye display unit 111b, and a mouth display unit 111c. The display unit 111 displays an image or an animation based on the control of the control unit 200.

[0037] The speaker 112 outputs an acoustic signal based on the control of the control unit 200.

[0038] The control unit 200 includes, for example, an acquisition unit 211, an image information processing unit 213, an audio information processing unit 215, a data providing unit 217, and an operation processing unit 219. Each of these functional units is realized, for example, by a hardware processor (computer) such as a CPU (Central Processing Unit) executing a program (software). Also, some or all of these components may be realized by hardware (including a circuit unit; circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), a SOC (System On Chip), etc., or may be realized by the cooperation of software and hardware. The program may be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as the HDD or flash memory of the communication robot 1, or may be stored in a removable storage medium such as a DVD or CD-ROM, and may be installed in the HDD or flash memory of the communication robot 1 when the storage medium (non-transitory storage medium) is mounted on the drive device.

[0039] The acquisition unit 211 acquires video data (image data) captured by the imaging unit 102 and acoustic data collected by the sound collection unit 103. Also, the acquisition unit 211 acquires the instruction information of the action transmitted from the action control device 10.

[0040] The image information processing unit 213 performs well-known image processing on the video data captured by the imaging unit 102. Well-known image processing is, for example, feature amount detection, binarization, edge detection, contour detection, clustering processing, etc. Further, the image information processing unit 213 uses the processed image information to detect, for example, a human face.

[0041] The audio information processing unit 215 performs well-known audio processing on the acoustic data collected by the sound collection unit 103. Well-known audio processing is, for example, speech recognition, speaker identification, etc.

[0042] The data provision unit 217 provides the action control device 10 with video data (video information) captured by the shooting unit 102 and sound data captured by the sound collection unit 103 at predetermined timings.

[0043] The motion processing unit 219 controls the operation of the communication robot 1. For example, the motion processing unit 219 generates images to be displayed on the eye display units 111a and 111b and the mouth display unit 111c, generates drive signals to drive the actuator 104, and generates sound to be output from the speaker 112. The motion processing unit 219 includes, for example, an image generation unit 221, a sound generation unit 223, and a drive unit 225.

[0044] The image generation unit 221 generates eye images (moving images) to be displayed on the eye display units 111a and 111b based on action instruction information transmitted from the action control device 10, and displays the generated eye images on the eye display units 111a and 111b. The displayed eye images are images corresponding to the movement of a person's eyes. The image generation unit 221 also generates images (still images, a series of still images, or moving images) to be displayed on the mouth display unit 111c, and displays the generated images on the mouth display unit 111c. The displayed images are images corresponding to the movement of a person's mouth. The image generation unit 221 may also generate images using a learning model that has learned the content of dialogue and actions, which are forms of communication with a person, using machine learning (reinforcement learning) technology.

[0045] The voice generation unit 223 generates an output audio signal to be output to the speaker 112 based on the action instruction information transmitted from the action control device 10, and outputs the generated output audio signal to the speaker 112. The voice generation unit 223 may also generate the output audio signal using the above-mentioned learning model, for example.

[0046] The drive unit 225 generates a drive signal to drive the neck actuator 104 based on action instruction information transmitted from the action control device 10, and drives the actuator 104 with the generated drive signal. The drive unit 225 may also generate the drive signal using the learning model described above, for example.

[0047] The memory unit 300 stores programs, algorithms, learning models, predetermined values, thresholds, etc., necessary for various control and processing of the communication robot 1. For example, the memory unit 300 stores a language model database, an acoustic model database, a dialogue corpus database, and acoustic features used in speech recognition, and a comparison image database and image features used in image recognition. The memory unit 300 may be implemented using the above-mentioned various storage devices, or an SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory). The memory unit 300 may be located on the cloud or connected via a network.

[0048] [Wearable Device] Returning to Figure 5, the wearable device 3 is worn by subject U and measures subject U's physiological data. The wearable device 3 measures, for example, subject U's body temperature, heart rate, blood pressure, blood oxygen saturation, and other physiological data. The wearable device 3 transmits the measured physiological data (physiological information) of subject U to the behavior control device 10.

[0049] [Action Control Device] The action control device 10 controls the actions of the communication robot 1. The action control device 10 is configured as a server device that can be accessed from the communication robot 1 or the wearable device 3 via a network. The action control device 10 may also be a personal computer or the like. The action control device 10 includes, for example, a control unit 20, a communication unit 30, and a storage unit 40. The communication unit 30 communicates with external devices such as the communication robot 1 via a network. The communication unit 30 is, for example, a network card for connecting to a network.

[0050] The storage unit 40 stores various types of information. The storage unit 40 is, for example, a semiconductor memory element such as RAM or flash memory, or a storage device such as a hard disk (a storage device equipped with a non-transient storage medium). The storage unit 40 may also be implemented by an external storage device connected via a network.

[0051] The control unit 20 includes, for example, an acquisition unit 21, an empathy level prediction unit 22, an anxiety level prediction unit 23, an action generation unit 24, an action control unit 25, and a learning unit 26. Each function of the control unit 20 is realized, for example, by a hardware processor such as a CPU executing a program (software).

[0052] The acquisition unit 21 acquires various types of information from external devices, etc. The acquisition unit 21 includes, for example, a subject information acquisition unit 21A. The subject information acquisition unit 21A acquires subject information such as video data and audio data transmitted from the communication robot 1, and physiological data transmitted from the wearable device 3.

[0053] The empathy level prediction unit 22 predicts the empathy level of subject U based on the acquired subject information. The empathy level prediction unit 22 includes, for example, a unit conversation extraction unit 22A, a semantic cue extraction unit 22B, a visual cue extraction unit 22C, an acoustic cue extraction unit 22D, a coupling unit 22E, and a prediction unit 22F.

[0054] The unit conversation extraction unit 22A acquires text information based on the acoustic data of subject U. For example, the acquisition unit 21 acquires text information by performing automatic speech recognition processing on the acoustic data of the conversation between subject U and communication robot 1 detected by the sound collection unit 103 of communication robot 1. The automatic speech recognition processing may be performed by the behavior control device 10 or by communication robot 1. Furthermore, the unit conversation extraction unit 22A separates the conversation between subject U and communication robot 1 contained in the text information into exchange units and extracts unit conversations (Exchanges). A unit conversation refers to a set of one statement from one party (subject U or communication robot 1) and one statement from the other party (subject U or communication robot 1) in response to that statement (a set of statements for one turn, a conversation exchange for one turn).

[0055] The semantic cue extraction unit 22B extracts semantic cues from text information. As semantic cues, the semantic cue extraction unit 22B extracts, for example, intention, arousal and valence (hereinafter referred to as "semantic AV value"), semantic imitation degree, emotion, and emotional response.

[0056] The visual cue extraction unit 22C extracts visual cues from the video data. The visual cue extraction unit 22C extracts emotional valence and arousal level (hereinafter referred to as "visual AV value") and visual imitation degree as visual cues. The visual imitation degree includes, for example, facial expression imitation degree, head posture imitation degree, and emotional imitation degree.

[0057] The acoustic cue extraction unit 22D extracts acoustic cues from acoustic data. These acoustic cues become important when clear video data of subject U (e.g., facial video data) cannot be obtained. The acoustic cue extraction unit 22D calculates emotional valence and arousal level (hereinafter referred to as "acoustic AV value"), emotion, etc., as acoustic cues using a deep learning-based model.

[0058] Acoustic cues include, for example, tone imitation and emotional imitation. Tone is defined as the average pitch and its variation. Voice tone is important because it influences how a message is received. Emotional imitation is calculated using the cosine similarity of the acoustic AV values ​​for each unit conversation. A value greater than 0.7 indicates similar emotions, while a value less than or equal to 0.7 indicates dissimilar emotions. Emotional imitation is confirmed by comparing the acoustic AV values ​​of the speaker and listener.

[0059] The degree of imitation indicates the extent to which one person imitates the facial expressions and actions (gestures) of the other person, for example, in a conversation between subject U and communication robot 1. For example, if one person has a positive facial expression (e.g., a "smile") and the other person also has a positive facial expression (e.g., a "smile"), the degree of imitation of the other person will be calculated to be high. Conversely, if one person has a positive facial expression (e.g., a "smile") while the other person has a negative facial expression (e.g., a "displeased face"), the degree of imitation of the other person will be calculated to be low. Alternatively, if one person's head is tilted (e.g., nodding) and the other person's head is tilted in the same way (e.g., nodding), the degree of imitation of the other person will be calculated to be high. Emotional valence is an indicator of how good (pleasant, enjoyable) or how bad (unpleasant, unpleasant) an emotion is. Arousal level is an indicator of the intensity of emotion.

[0060] Emotions are extracted from the voice using sound highlighting techniques such as MFCC (Mel Frequency Cepstral Coefficient). The emotions have a similar format to text-based emotions, using positive, neutral, and negative. The range is [0, 1], with the sum of the three being 1.

[0061] The coupling unit 22E generates combined information by combining semantic cues and visual cues. For example, for each unit conversation, the coupling unit 22E generates combined information by weighting and adding the semantic arousal level and emotional valence included in the semantic cues, the visual arousal level and emotional valence included in the visual cues, and the acoustic arousal level and emotional valence included in the acoustic cues. In addition, the coupling unit 22E generates combined information by quantifying the semantic imitation level included in the semantic cues, as well as the facial expression imitation level, emotional imitation level, and head position imitation level included in the visual cues, and adding up all the quantified values.

[0062] The prediction unit 22F predicts the empathy level of subject U based on the combined information. The prediction unit 22F also ranks the cues that have an influence on the empathy level prediction. For example, the prediction unit 22F uses PBC4emp as a classifier to predict the empathy level of subject U and rank the cues that have an influence on the empathy level prediction. For each conversational exchange between subject U and communication robot 1, the prediction unit 22F uses an explainable classifier that outputs a pattern-based prediction result and cues that have an influence on the prediction result to predict the empathy level of subject U (or communication robot 1). When unit conversation, semantic cues, visual cues, acoustic cues, and combined information are input, PBC4emp outputs the empathy level prediction result and explanatory information of cues that influence the empathy level. This explanatory information is the pattern of cues that is useful for predicting the empathy level and their values. The empathy level is expressed numerically, for example, as no empathy (1), moderate empathy (2), high empathy (3).

[0063] The anxiety level prediction unit 23 predicts the anxiety level of subject U based on the acquired subject information. The anxiety level prediction unit 23 includes, for example, a facial and behavioral feature extraction unit 23A, a physiological feature extraction unit 23B, a linguistic feature extraction unit 23C, and a prediction unit 23D.

[0064] The facial and behavioral feature extraction unit 23A extracts the facial and behavioral features of subject U by processing the video data using a machine learning model. The machine learning model is, for example, a convolutional neural network (CNN). This machine learning model is trained to output estimation results of subject U's facial and behavioral features when video data is input. The facial and behavioral features include, for example, AV values ​​based on facial expressions and AV values ​​based on behavior.

[0065] The physiological feature extraction unit 23B extracts the physiological features (biological signal features) of subject U by processing the physiological data using a machine learning model. The machine learning model is, for example, a deep neural network (DNN). This machine learning model is trained to output an estimated result of the physiological features of subject U when physiological data is input. The physiological features include, for example, AV values ​​based on the physiological data.

[0066] The linguistic feature extraction unit 23C extracts linguistic features of subject U's speech by processing the text information, which is generated by automatically processing the acoustic data of the conversation between subject U and the communication robot 1 using speech recognition, with a machine learning model. The machine learning model is, for example, a deep neural network (DNN). This machine learning model is trained to output an estimated result of subject U's linguistic features when text information is input. The linguistic features include, for example, AV values ​​based on linguistic information.

[0067] Alternatively, the linguistic feature extraction unit 23C extracts paralinguistic features of subject U's speech by processing the acoustic data (waveform data) of the conversation between subject U and communication robot 1 using a machine learning model. The machine learning model is, for example, a deep neural network (DNN). This machine learning model is trained to output estimation results of subject U's paralinguistic features when acoustic data is input. Paralinguistic features include, for example, AV values ​​based on paralinguistic information.

[0068] The prediction unit 23D predicts the anxiety level of subject U based on facial and behavioral characteristics, physiological characteristics, and linguistic characteristics (perilingual characteristics). For example, the prediction unit 23D predicts the anxiety level of subject U by processing facial and behavioral characteristics, physiological characteristics, and linguistic characteristics (and / or peripheral linguistic characteristics) using a machine learning model. The machine learning model is, for example, a deep neural network (DNN). This machine learning model is trained to output an estimated result of subject U's anxiety level when, for example, facial and behavioral characteristics, physiological characteristics, and linguistic characteristics (and / or peripheral linguistic characteristics) are input. The anxiety level is expressed numerically, such as no anxiety (1), moderate anxiety (2), and high anxiety (3).

[0069] The behavior generation unit 24 generates (determines) the behavior of the communication robot 1 based on the predicted empathy level and anxiety level. For example, the behavior generation unit 24 generates the behavior of the communication robot 1 (an behavior that gives the target level of empathy) by processing the empathy level and anxiety level using a machine learning model (using the output obtained by inputting the empathy level and anxiety level into the machine learning model). The machine learning model is, for example, a generative AI. This machine learning model outputs the behavior of the communication robot 1 when, for example, the empathy level and anxiety level, a conversation (for example, what subject U said), and a task (prompt) are input. The task (prompt) is, for example, "Generate empathetic behavior. If the anxiety level is high, generate behavior that reduces the seriousness and makes it fun."

[0070] The behavior control unit 25 controls the behavior of the communication robot 1 based on the behavior generated by the behavior generation unit 24. For example, the behavior control unit 25 controls the behavior of the communication robot 1 by outputting instruction information to the communication robot 1 that indicates the behavior generated by the behavior generation unit 24.

[0071] The learning unit 26 performs the learning process for the various machine learning models described above. For example, in the case of a machine learning model that is a generative AI used in the behavior generation unit 24, the learning unit 26 performs the learning process using an empathetic dataset based on a dataset of empathetic conversations (interactions) that have taken place in the past between a speaker and a listener (between two people, or between a person and a communication robot). In the empathetic conversation (interaction) dataset, each row represents the speaker's utterance, the listener's utterance, and the listener's empathy level. The goal of this learning process is to generate empathetic utterances from the listener that give a target empathy level. For this reason, a default (pre-learning) utterance, consisting of utterances (non-target) generated by the machine learning model before learning, is also prepared in advance. The learning unit 26 trains the machine learning model to minimize the possibility of generating non-target utterances while maximizing the possibility of generating target empathetic utterances. The trained machine learning model can adjust the empathy level of empathetic utterances to a certain target empathy level.

[0072] [Processing Flow] <Overall Processing> Next, the processing flow for controlling the behavior of the communication robot 1 will be explained. Figure 8 is a flowchart showing an example of the processing flow of the behavior control device 10 according to the embodiment. The processing shown in Figure 8 is executed repeatedly, for example, while communication takes place between the subject U and the communication robot 1.

[0073] First, the subject information acquisition unit 21A acquires multimodal data of subject U, including video data and audio data transmitted from the communication robot 1, and physiological data transmitted from the wearable device 3 (step S101).

[0074] Next, the empathy level prediction unit 22 predicts the empathy level of subject U based on the acquired multimodal data of subject U (step S103). Details of the processing of the empathy level prediction unit 22 will be described later.

[0075] Next, the anxiety level prediction unit 23 predicts the anxiety level of subject U based on the acquired multimodal data of subject U (step S105). Details of the processing of the anxiety level prediction unit 23 will be described later.

[0076] Next, the behavior generation unit 24 generates behavior for the communication robot 1 (instruction information indicating behavior that will give the target level of empathy) based on the predicted level of empathy and anxiety (step S107). For example, the behavior generation unit 24 generates behavior for the communication robot 1 by processing the level of empathy and anxiety using a machine learning model (generative AI).

[0077] Next, the action control unit 25 outputs instruction information indicating the generated action to the communication robot 1 and controls the action of the communication robot 1 (step S109).

[0078] <Empathy Prediction Process> Details of the empathy prediction process in step S103 above will be explained. Figure 9 is a diagram illustrating the overview of the empathy prediction process according to the embodiment. The empathy prediction process includes (1) an empathetic conversation implementation step, (2) a detection and recognition step, (3) a semantic cue extraction step, (4) an image cue extraction step, (5) an empathetic dataset collection step, (6) a merging step, and (7) an empathy prediction step.

[0079] (1) Empathic Conversation Implementation Process In the empathic conversation implementation process, for example, an empathic conversation is conducted between subject U and communication robot 1. Communication robot 1 speaks to subject U and plays the role of eliciting the desired response (acoustic data such as speech) from subject U.

[0080] (2) Detection and Recognition Process In the detection and recognition process, various detection and recognition processes are performed. Detection is performed using the imaging unit 102 and sound collection unit 103 provided on the communication robot 1. Recognition is performed by automatic speech recognition (ASR) and 3D model fitting. For 3D model fitting, for example, the EMOCA (Emotion-driven monocular face capture and animation) model, which is a 3DMM, is used (see Reference 1). A 3DMM is a statistical model that represents facial expressions and shapes. The EMOCA model provides, for example, facial expression coefficients (FE) represented by 50 coefficients and head pose coefficients (HP) represented by 6 coefficients. The facial expression coefficients FE represent a set of coefficients that show how the facial muscles change when a person makes an expression. By capturing the change in the facial expression coefficients FE over time, the facial expression sequence FESeq is obtained. Similarly, by capturing the change in the head pose coefficients HP over time, the head pose sequence HPSeq is obtained.

[0081] Reference 1: Danecek, Radek, Michael J. Black, Timo Bolkart, “EMOCA: Emotion driven monocular face capture and animation.”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022.

[0082] (3) Semantic Clue Extraction Process In the semantic cue extraction process, semantic cues are extracted from the text information of the speech acquired by automatic speech recognition. For example, the "intent," which is an empathetic response, is extracted using BERT classification (see References 2 and 3). This intent indicates the probability that the response of the listener, subject U, is confirmation, agreement, or suggestion. Furthermore, "sentiment" is extracted from the speaker and listener by fine-tuning the RoBERTa-based model, which has been trained on approximately 58 million tweets, using sentiment analysis technology (see Reference 4). In addition, "emotional reaction" is extracted using the EPITOME framework (see Reference 5). Furthermore, "emotional valence and arousal level (hereinafter referred to as "semantic AV value")" and "semantic mimicry" are extracted using NRC_VAD Lexicon (see Reference 6).

[0083] Reference 2: Welivita, Anuradha, and Pearl Pu, “A Taxonomy of Empathetic Response Intents in Human Social Conversations.”, Proceedings of the 28th International Conference on Computational Linguistics, International Committee on Computational Linguistics, 2020, pp. 4886-99. Reference 3: Chen, Mao Yan et al., “EmpHi: Generating Empathetic Responses with Human-like Intents.”, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, 2022, pp. 1063-74. Reference 4: Barbieri, Francesco et al., “TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification.”, Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, 2020, pp. 1644-50. Reference 5: Sharma, Ashish et al., “A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support.”, arXiv:2009.08441, arXiv, 17 Sept. 2020.Reference 6: Mohammad, Saif, “Obtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20,000 English Words.” Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, 2018.

[0084] (4) Visual cue extraction process In the visual cue extraction process, visual cues are extracted from the facial expression sequence FESeq and the head posture sequence HPSeq. For example, visual cues such as the presence or absence of outward expression of emotion or imitation are important in recognizing the presence or absence of empathy regarding behavior. These phenomena are the most basic criteria for judgment in predicting the degree of empathy. From the head posture sequence HPSeq, the visual AV values ​​for each frame (for each exchange of conversation) of the speaker and listener during the conversation are calculated. First, a step function is fitted to the AV signal in time series, and then the average of the obtained step function is calculated.

[0085] Furthermore, the degree of visual imitation is calculated by aligning the signals of the facial expression sequence FESeq and the head posture sequence HPSeq and capturing the changes in facial expression or head position over time. A deviation is applied to obtain the changes in features from one frame to the next. Next, the signals are aligned using, for example, a cross-correlation calculation in a 3-second window, and cosine similarity is calculated to determine how close the signals are to each other (see references 7 and 8). Finally, the degree of imitation is binarized using an empirically determined fixed threshold.

[0086] Reference 7: Tschacher, Wolfgang, Fabian Ramseyer, and Claudia Bergomi, "The subjective present and its modulation in clinical contexts.", Timing & Time Perception 1.2 (2013): 239-259. Reference 8: Dindar, Muhterem et al., "Leaders and followers identified by emotional mimicry during collaborative learning: A facial recognition study on emotional valence.", IEEE Transactions on Affective Computing 13.3 (2020): 1390-1400.

[0087] The degree of visual imitation includes, for example, the degree of facial imitation, the degree of head posture imitation, and the degree of emotional imitation. The degree of facial imitation is calculated by applying the facial expression coefficient FE (facial expression sequence FESeq) output by EMOCA to the degree of imitation calculation algorithm. The degree of head posture imitation is calculated by applying the degree of imitation calculation algorithm to the head posture coefficient HP (head posture sequence HPSeq) output by EMOCA. In particular, it is good to use only the coefficient for head movement in the pitch direction. The degree of emotional imitation is calculated based on the cosine similarity of the visual AV value for each interaction in the conversation. If the calculated cosine similarity value is greater than 0.7, it is determined that the emotions are similar, and if it is 0.7 or less, it is determined that the emotions are not similar. The degree of visual imitation is expressed, for example, as 0 (no imitation) or 1 (imitation present). To confirm the existence of video-based video imitation, three imitation degree outputs or combinations thereof are used. Combined information that is 0 is output only when all inputs are 0 (not imitation), and combined information that is 1 is output only when all inputs are 1 (imitation).

[0088] (5) Empathy Dataset Collection Process In the empathy dataset collection process, various types of information obtained in the processes (1) to (4) above are accumulated. This empathy dataset includes "Utterances" and "Length," which are information obtained in the detection and recognition process (2) above; "Intent," "Text AV," "Text mimicry," "Sentiment," and "Emotional Reaction," which are information obtained in the semantic cue extraction process (3) above; and "Video AV" and "Video mimicry," which are information obtained in the video cue extraction process (4) above. In addition, the empathy dataset includes "Empathy," which is manually labeled empathy level information.

[0089] (6) Fusion Process In the fusion process, the semantic cues obtained in the (3) semantic cue extraction process and the visual cues obtained in the (4) visual cue extraction process are combined to generate combined information.

[0090] (7) Empathy Prediction Process In the empathy prediction process, empathy prediction is performed using a pattern-based classifier (Explainable classifier). For example, PBC4emp is used as the classifier (see Reference 9). PBC4emp is a pattern-based classifier trained using only a dataset of empathy levels from text information. PBC4emp outputs prediction results in two levels ("1 (empathy)", "2 (no empathy)") or three levels ("1 (no empathy)", "2 (moderate empathy)", or "3 (high empathy)"). PBC4emp runs PBC4cip (Contrast Pattern-based Classifier for class imbalance problems). This algorithm extracts patterns using a random forest approach. PBC4emp uses a combination of empathetic cues extracted from text information, acoustic information, and video information as input. The classifier (model) of this embodiment can provide performance similar to that of deep learning methods (for example, BERT described in Reference 10) and large-scale language model methods.

[0091] Reference 9: Montiel-Vazquez, Edwin Carlos, Jorge Adolfo Ramirez Uresti, and Octavio Loyola-Gonzalez, "An explainable artificial intelligence approach for detecting empathy in textual communication." Applied Sciences 12.19 (2022): 9407. Reference 10: Hasan, Md Rakibul et al., "Empathy detection using machine learning on text, audiovisual, audio or physiological signals." arXiv preprint arXiv:2311.00721 ​​(2023).

[0092] Furthermore, in the empathy prediction process, PBC4emp is used to rank the cues that have an influence on predicting empathy. The current space is transformed into a future space, its empathy level is classified, patterns that cover the present are obtained, relevance scores for features are calculated, and the feature (cue) with the highest relevance score for empathy is selected.

[0093] Figure 10 is a flowchart showing an example of the empathy level prediction process by the behavior control device 10 according to the embodiment. First, the acquisition unit 21 acquires text information and video data based on acoustic data (step S201).

[0094] Next, the unit conversation extraction unit 22A separates the conversations between the subject U and the communication robot 1 contained in the text information into exchange units and extracts the unit conversations (Exchange) (step S203).

[0095] Next, the video cue extraction unit 22C performs recognition processing based on the video data (step S205). For example, the video cue extraction unit 22C inputs the video data into the EMOCA model, which is a 3DMM, to obtain the facial expression coefficient FE and the head posture coefficient HP. Furthermore, the video cue extraction unit 22C obtains the facial expression sequence FESeq by arranging the facial expression coefficient FE in chronological order. Also, the video cue extraction unit 22C obtains the head posture sequence HPSeq by arranging the head posture coefficient HP in chronological order.

[0096] Next, the acoustic cue extraction unit 22D performs recognition processing based on the acoustic characteristics of the acoustic data (step S207). For example, the acoustic cue extraction unit 22D calculates acoustic AV values, emotions, etc., based on the acoustic data.

[0097] Next, the semantic cue extraction unit 22B extracts semantic cues for each unit conversation (step S209). The semantic cue extraction unit 22B extracts, for example, intention, semantic AV value, semantic imitation degree, emotion, and emotional response as semantic cues.

[0098] Next, the video cue extraction unit 22C extracts video cues from the video data corresponding to the unit conversation in terms of time (step S211). The video cue extraction unit 22C extracts the video AV value and the video imitation degree as video cues. The video imitation degree includes, for example, the expression imitation degree, the head pose imitation degree, and the emotional imitation degree.

[0099] Next, the acoustic cue extraction unit 22D extracts acoustic cues from the acquired acoustic data (step S213). The acoustic cues include, for example, the tone imitation degree, the emotional imitation degree, and the like.

[0100] Next, the combining unit 22E combines the semantic cue, the video cue, and the acoustic cue to generate combined information (step S215). The combining unit 22E generates combined information, for example, by weighted addition of the semantic AV value included in the semantic cue, the video AV value included in the video cue, and the acoustic AV value included in the acoustic cue for each unit conversation. For example, the combining unit 22E generates combined information regarding the AV value based on the following formula (1).

[0101] Combined information (AV) = W audio *AV audio +W video *AV video +W text *AV text ... Formula (1)

[0102] In the above formula (1), W audio , W video , and W text are the weights for the acoustic cue, the video cue, and the semantic cue, respectively. AV audio , AV video , and AV text are the acoustic AV value, the video AV value, and the semantic AV value, respectively.

[0103] Also, for example, the combining unit 22E generates combined information by weighted addition of the text-based emotion and the acoustic-based emotion based on the following formula (2).

[0104] S = Waudio *S audio +W text *S text ...(Formula 2)

[0105] In the above equation (2), S audio and S text These are sound-based emotions and text-based emotions, respectively.

[0106] Next, the prediction unit 22F predicts the subject's empathy level based on the combined information (step S217). The prediction unit 22F also ranks the cues that have an influence on the prediction of empathy level. For example, the prediction unit 22F uses PBC4emp as a classifier to predict the subject's empathy level and rank the cues that have an influence on the prediction of empathy level. This completes the processing of this flowchart.

[0107] <Anxiety Level Prediction Processing> Details of the anxiety level prediction processing in step S105 above will be explained. Figure 11 is a flowchart showing an example of the flow of the anxiety level prediction processing by the behavior control device 10 according to the embodiment.

[0108] First, the facial and behavioral feature extraction unit 23A extracts the facial and behavioral features of subject U by processing the video data using a machine learning model (step S301). The facial and behavioral features include, for example, AV values ​​based on facial expressions and AV values ​​based on behavior.

[0109] Next, the physiological feature extraction unit 23B extracts the physiological features of subject U by processing the physiological data using a machine learning model (step S303). The physiological features include, for example, AV values ​​based on the physiological data.

[0110] Next, the linguistic feature extraction unit 23C extracts linguistic features of subject U's speech by processing the text information, which is converted from the acoustic data of the conversation between subject U and the communication robot 1 through automatic speech recognition processing, using a machine learning model (step S305). The linguistic features include, for example, AV values ​​based on linguistic information. The linguistic feature extraction unit 23C also extracts paralinguistic features of subject U's speech by processing the acoustic data of the conversation between subject U and the communication robot 1 using a machine learning model. The paralinguistic features include, for example, AV values ​​based on paralinguistic information.

[0111] Next, the prediction unit 23D predicts the anxiety level of subject U by processing facial and behavioral characteristics, physiological characteristics, and linguistic characteristics (and / or paralinguistic characteristics) using a machine learning model (step S307). This completes the processing of this flowchart.

[0112] As described above, the embodiments enable appropriate control of the communication robot 1's behavior according to the subject U. The behavior control system S can be used, for example, to set up Human-robot interaction (HRI) between the communication robot 1 and a person. For example, if such a communication robot 1 is used in a hospital, it can act as a friend to subjects (patients) such as children, reducing the opportunities for children to feel bored or lonely through fun activities with the communication robot 1, and even making children smile. The communication robot 1 can also act as an emotional support for doctors, assisting doctors when explaining the child's condition and treatment. Furthermore, the communication robot 1 can act as a psychologist, providing mental / emotional care to children.

[0113] Furthermore, some or all of the functions of the behavior control device 10 described above may be implemented by incorporating them into the communication robot 1.

[0114] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0115] 1...Communication robot, 3...Wearable device, 10...Behavior control device, 20...Control unit, 21...Acquisition unit, 21A...Subject information acquisition unit, 22...Empathy level prediction unit, 22A...Unit conversation extraction unit, 22B...Semantic cue extraction unit, 22C...Visual cue extraction unit, 22D...Acoustic cue extraction unit, 22E...Coupling unit, 22F...Prediction unit, 23...Anxiety level prediction unit, 23A...Facial expression / behavioral feature extraction unit, 23B...Physiological feature extraction unit, 23C...Linguistic feature extraction 23D…Prediction unit, 24…Action generation unit, 25…Action control unit, 26…Learning unit, 30…Communication unit, 40…Storage unit, 102…Shooting unit, 103…Sound collection unit, 104…Actuator, 111…Display unit, 112…Speaker, 200…Control unit, 211…Acquisition unit, 213…Image information processing unit, 215…Sound information processing unit, 217…Data provision unit, 219…Motion processing unit, 221…Image generation unit, 223…Sound generation unit, 225…Drive unit, 300…Storage unit, S…Action control system

Claims

1. A behavior control system comprising: a communication device for communicating with a subject; a subject information acquisition unit for acquiring subject information of the subject; an empathy level prediction unit for predicting the subject's empathy level based on the acquired subject information; an anxiety level prediction unit for predicting the subject's anxiety level based on the acquired subject information; and a behavior generation unit for generating the communication device's behavior based on the predicted empathy level and anxiety level.

2. The behavior control system according to claim 1, wherein the subject information includes text information based on the subject's acoustic information and video information of the subject, the empathy level prediction unit extracts semantic cues from the text information, extracts video cues from the video information, combines the extracted semantic cues and video cues to generate combined information, and predicts the subject's empathy level based on the generated combined information.

3. The behavior control system according to claim 1 or 2, wherein the subject information includes video information of the subject, physiological information of the subject, and acoustic information of the subject, and the anxiety level prediction unit extracts the subject's facial expressions and behavioral characteristics from the video information, extracts the subject's physiological characteristics from the physiological information, extracts the subject's linguistic characteristics from the acoustic information, and predicts the level of anxiety based on the extracted facial expressions and behavioral characteristics, physiological characteristics, and linguistic characteristics.

4. The behavior control system according to claim 1 or 2, further comprising a behavior control unit that controls the behavior of the communication device based on the generated behavior of the communication device.

5. The behavior control system according to claim 2, wherein the empathy prediction unit further extracts acoustic cues from the acoustic information, combines the extracted semantic cues, visual cues, and acoustic cues to generate combined information, and predicts the subject's empathy level based on the generated combined information.

6. The behavior control system according to claim 5, wherein the empathy prediction unit generates the combined information by weighting and adding the semantic arousal level and emotional valence included in the semantic cue, the visual arousal level and emotional valence included in the visual cue, and the acoustic arousal level and emotional valence included in the acoustic cue, for each conversational exchange between the subject and the communication device.

7. The behavior control system according to claim 1 or 2, wherein the behavior generation unit generates the behavior of the communication device using the output obtained by inputting the predicted degree of empathy and the degree of anxiety into a machine learning model which is a Generative Artificial Intelligence (AI).

8. The behavior control system according to claim 7, further comprising a learning unit that performs training processing on the machine learning model using a dataset including the speaker's utterance, the listener's utterance, and the listener's empathy level, obtained in a conversation between the speaker and the listener.

9. A method for controlling the behavior of a communication device that communicates with a subject, wherein a computer acquires subject information of the subject, predicts the subject's level of empathy based on the acquired subject information, predicts the subject's level of anxiety based on the acquired subject information, generates the behavior of the communication device based on the predicted level of empathy and anxiety, and controls the behavior of the communication device based on the generated behavior of the communication device.

10. A program for controlling the behavior of a communication device that communicates with a subject, the program causing a computer to acquire subject information of the subject, to predict the subject's level of empathy based on the acquired subject information, to predict the subject's level of anxiety based on the acquired subject information, to generate the behavior of the communication device based on the predicted level of empathy and anxiety, and to control the behavior of the communication device based on the generated behavior of the communication device.

Images