Information processing device

Abstract

A display control unit 52 of an information processing device 12 causes a display device 14 to display an agent that communicates with a user. A detection unit 56 of the information processing device 12 detects an operation of a predetermined imitation target. A frequency update unit 58 of the information processing device 12 updates frequencies of a plurality of predetermined operation patterns on the basis of the detected operation of the imitation target. An agent control unit 60 of the information processing device 12 controls the operation of the agent displayed on the display device 14 according to the frequencies of the plurality of operation patterns.

Description

Information processing apparatus

[0001] The present disclosure relates to data processing technology, and particularly to an information processing apparatus.

[0002] In an information processing apparatus, a character that communicates with a user may be displayed.

[0003] In order to effectively improve the quality of the user experience through interaction with the character, the inventor considered that it is important to enhance the sense of reality of the character. One object of the present disclosure is to provide a technique for enhancing the sense of reality of a character that communicates with a user.

[0004] To solve the above problems, an information processing apparatus according to an aspect of the present disclosure includes a display control unit that displays an agent that communicates with a user, a detection unit that detects an operation of a predetermined imitation target, and a frequency update unit that updates the frequencies of a plurality of predetermined operation patterns based on the detected operation of the imitation target, and an agent control unit that controls the operation of the agent according to the frequencies of the plurality of operation patterns.

[0005] Another aspect of the present disclosure is also an information processing apparatus. This apparatus includes a display control unit that displays an agent that communicates with a user, a detection unit that detects the expression of the user, and an agent control unit that controls the expression of the agent to imitate the detected expression of the user after a predetermined delay time has elapsed since the expression of the user was detected.

[0006] Still another aspect of the present disclosure is also an information processing apparatus. This apparatus includes a display control unit that displays an agent that communicates with a user, a detection unit that detects the tone of the user's voice, and an agent control unit that controls the tone of the agent's voice to imitate the detected tone of the user's voice.

[0007] Another aspect of this disclosure is also an information processing device. This device comprises a display control unit that displays an agent communicating with a user, a detection unit that detects the rhythm of an output sound, and an agent control unit that controls the agent to hum the detected rhythm.

[0008] Another aspect of this disclosure is also an information processing device. This device comprises a display control unit that displays a scene of an application played by a user and an agent that communicates with the user; a video acquisition unit that acquires a video from a video sharing service showing a person playing the same scene as the displayed application scene; a detection unit that detects the movements of the person in the acquired video; and an agent control unit that controls the actions of the agent to mimic the movements of the detected person.

[0009] Furthermore, any combination of the above components, as well as conversions of the expressions of this disclosure between systems, methods, computer programs, recording media storing computer programs, etc., are also valid forms of this disclosure.

[0010] According to this disclosure, it is possible to enhance the sense of realism of characters that communicate with users.

[0011] This is a diagram showing the configuration of the information processing system of the first embodiment. This is a block diagram showing the functional blocks of the information processing device of the first embodiment. This is a diagram showing an example of agent display. This is a block diagram showing the functional blocks of the information processing device of the second embodiment. This is a diagram showing the configuration of the information processing system of the third embodiment. This is a diagram showing the configuration of the information processing system of the fifth embodiment. This is a block diagram showing the functional blocks of the information processing device of the fifth embodiment.

[0012] The embodiments will now be outlined. In these embodiments, the user is a player of a game application. The information processing system of these embodiments provides an agent that communicates continuously with the user, in other words, an agent that interacts with the user over a relatively long period of time. Short-term interactions, such as advertisements, may also be included. The agent is a character that acts as the user's friend or companion, for example, by playing games with the user or providing the user with game hints.

[0013] The inventors of this invention believe that enhancing the sense of presence of the agent is important in order to effectively improve the user experience (in an embodiment, the game experience) through communication with the agent. The sense of presence of the agent can be described as making the user feel as if the agent exists in the same world as the user, or as making the user feel as if they are sharing the same world as the agent.

[0014] To enhance the sense of presence of the agent, the information processing system in each embodiment operates the agent to mimic predetermined objects (e.g., user actions, facial expressions, background music, etc.). This enhances the sense of presence of the character communicating with the user and improves the user experience.

[0015] <First Embodiment> Figure 1 shows the configuration of the information processing system 10 of the first embodiment. In the information processing system 10 of the first embodiment, the agent is made to imitate the actions of a user, which is a predetermined target for imitation. The information processing system 10 comprises an information processing device 12, a display device 14, a controller 16, and a camera 18.

[0016] The functions of the components in each computer (e.g., information processing device 12) constituting the information processing system 10 may be realized in circuits or processing circuits, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (a Central Processing Unit), conventional circuits, and / or combinations thereof, which are configured or programmed to realize the functions described herein. A processor is considered to be a circuit or processing circuit that includes transistors and other circuits. A processor may also be a programmed processor that executes a program stored in memory.

[0017] In this specification, circuits, units, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0018] If the hardware is a processor that is considered to be a type of circuit, then the circuit, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0019] The information processing device 12 includes a computer capable of running system software and various application software. In the first embodiment, the information processing device 12 is a stationary game console, but as a modification, the information processing device 12 may be a portable game console, a PC, a tablet terminal, a smartphone, digital signage, etc.

[0020] The controller 16 is an operating device that receives user inputs to the system software and application software executed by the information processing device 12. The controller 16 is connected to the information processing device 12 by wire or wireless connection and sequentially transmits operation information indicating the user inputs to the information processing device 12.

[0021] The controller 16 is equipped with a speaker and can output sound based on an audio signal transmitted from the information processing device 12. The controller 16 is also equipped with a vibrator and can vibrate the vibrator in a manner based on a haptic signal transmitted from the information processing device 12.

[0022] The display device 14 has a display that outputs images and a speaker that outputs sound. The display device 14 may be a television or a computer display. The display device 14 is connected to the information processing device 12 by wire or wireless. When the information processing device 12 receives operation information provided by the controller 16, it reflects that operation information in the processing of system software or application software, displays an image related to the processing result on the display device 14, and outputs sound related to the processing result from the speaker of the display device 14.

[0023] The display device 14 may be a head-mounted display worn on the user's head. In this case, the information processing device 12 and the display device 14 (head-mounted display) may be configured as a system in which a panoramic image is displayed on the head-mounted display, and when the user wearing the head-mounted display rotates their head, a panoramic image corresponding to the direction of their gaze is displayed. The display device 14 may be a head-mounted display configured to block out external light, and may display a virtual reality image generated by the information processing device 12. Alternatively, the display device 14 may be an optically transparent head-mounted display, and may display augmented reality images generated by the information processing device 12.

[0024] Camera 18 is a device that captures images of a user playing a game. Camera 18 is connected to the information processing device 12 by wire or wireless connection and transmits data of images of the user (hereinafter also referred to as "captured images") to the information processing device 12. The captured images in the first embodiment include at least still images or videos showing the user's face.

[0025] Figure 2 is a block diagram showing the functional blocks of the information processing device 12 according to the first embodiment. Each block shown in the block diagrams of this specification can be realized in hardware terms by a computer processor (CPU, etc.), memory and other elements, electronic circuits, and mechanical devices, and in software terms by a computer program, etc., but here we are depicting functional blocks that are realized through the cooperation of these. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways by combinations of hardware and software.

[0026] The information processing device 12 comprises a processing unit 30, a storage unit 32, and a communication unit 34. The processing unit 30 performs various data processing. The storage unit 32 stores data that is referenced or updated by the processing unit 30. The communication unit 34 communicates with external devices via wired or wireless communication according to a predetermined communication protocol. The processing unit 30 sends and receives data to and from the display device 14, controller 16, and camera 18 via the communication unit 34.

[0027] The storage unit 32 includes a program storage unit 40 and an operation pattern storage unit 42. The program storage unit 40 stores system software and application software. The application software in the first embodiment includes a game application (e.g., a video game).

[0028] The motion pattern storage unit 42 stores information indicating the frequency of each of a predetermined set of motion patterns relating to the user. The set of motion patterns may include movements and states of various parts of the user's body, or movements and states of various items worn by the user. For example, the set of motion patterns may include blinking or clearing the throat. The frequency may be expressed as the number of times per minute.

[0029] The processing unit 30 includes a game execution unit 50, a display control unit 52, an audio control unit 54, a detection unit 56, a frequency update unit 58, and an agent control unit 60. The functionality of at least one of these multiple functional blocks may be implemented in a computer program (for example, system software and / or application software). This computer program may be installed in the storage of the information processing device 12. The processor of the information processing device 12 may perform the functions of these multiple functional blocks by reading this computer program into main memory and executing it.

[0030] The game execution unit 50 executes system software and application software stored in the program storage unit 40. For example, the game execution unit 50 executes a game application in response to user input to the controller 16. Based on the execution results of the game application, the game execution unit 50 generates image data and sound data for the game.

[0031] The display control unit 52 controls the content displayed on the display device 14. For example, the display control unit 52 causes the display device 14 to display images of the game generated by the game execution unit 50. The display control unit 52 also causes the display device 14 to display images of agents that communicate with the user. For example, the display control unit 52 causes the display device 14 to display images of agents operating in a manner determined by the agent control unit 60.

[0032] The audio control unit 54 controls the audio output from the speaker of the display device 14. For example, the audio control unit 54 causes the audio of the game generated by the game execution unit 50 to be output from the speaker of the display device 14. The audio control unit 54 also causes the audio that the agent, as determined by the agent control unit 60, should emit, to be output from the speaker of the display device 14.

[0033] The detection unit 56 detects the user's movements by analyzing the captured image of the user output from the camera 18 using known techniques. When a head-mounted display is used as the display device 14, various user behaviors can be detected by attaching various sensors to the head. For example, blinking can be detected using a gaze sensor or an electroencephalogram (EEG) sensor.

[0034] The frequency update unit 58 updates the frequencies of multiple action patterns stored in the action pattern storage unit 42 based on the user actions detected by the detection unit 56. For example, if the frequency update unit 58 detects the user blinking as a user action, it updates the blinking frequency stored in the action pattern storage unit 42. Also, if the user clears their throat as a user action, it updates the throat clearing frequency stored in the action pattern storage unit 42.

[0035] The agent control unit 60 controls the actions of the agent displayed on the display device 14 according to the frequencies of multiple action patterns stored in the action pattern storage unit 42 and updated by the frequency update unit 58. For example, the agent control unit 60 makes the agent blink at a blinking frequency (e.g., 20 times / minute) stored in the action pattern storage unit 42. The agent control unit 60 also makes the agent clear its throat at a throat-clearing frequency (e.g., 4 times / minute) stored in the action pattern storage unit 42.

[0036] Figure 3 shows an example of the agent display. The operation of the information processing system 10 of the first embodiment will be described below with reference to Figure 3.

[0037] The game execution unit 50 of the information processing device 12 generates image data and sound data for the game according to the user's gameplay status. The display control unit 52 of the information processing device 12 displays the game images generated by the game execution unit 50 on the display of the display device 14. The game images include, for example, (1) images of various scenes in the game, (2) images of various characters appearing in the game (player characters, ally characters, enemy characters, etc.), (3) images of items (weapons, accessories, etc.), and (4) images of other objects (backgrounds, etc.). The sound control unit 54 of the information processing device 12 outputs the game sounds generated by the game execution unit 50 from the speaker of the display device 14.

[0038] In parallel with this, the display control unit 52 of the information processing device 12 displays the agent 100 on the display device 14. The agent control unit 60 of the information processing device 12 determines the content of the voice that the agent 100 will speak to the user. The content of the voice that the agent 100 will speak to the user may include, for example, advice to the user, hints for the game the user is playing, or recommendation information tailored to the user's attributes. The voice control unit 54 of the information processing device 12 outputs the voice of the agent 100 determined by the agent control unit 60 from the speaker.

[0039] Camera 18 captures images of the user playing the game and the user responding to interactions with Agent 100 and inputting actions. Camera 18 inputs the video data of the user to the information processing device 12. The detection unit 56 of the information processing device 12 analyzes the video input from Camera 18 and detects the user's actions. Based on the detected user actions, the detection unit 56 of the information processing device 12 updates the frequency of multiple action patterns stored in the program storage unit 40.

[0040] The agent control unit 60 of the information processing device 12 controls the operation of the agent 100 displayed on the display device 14 according to the frequency of a plurality of operation patterns stored in the program storage unit 40. For example, the agent control unit 60 may generate an image of the agent 100 blinking at the blinking frequency stored in the operation pattern storage unit 42, and display that image on the display device 14 via the display control unit 52. Alternatively, the agent control unit 60 may generate audio data of the agent 100 clearing its throat at the throat clearing frequency stored in the operation pattern storage unit 42, and output the throat clearing audio from the speaker via the audio control unit 54.

[0041] In the information processing device 12 of the first embodiment, the agent 100 is made to perform actions that abstract (patternize) the user's actions at the frequency of the user's actions. This allows the user to feel a sense of presence from the agent 100 while suppressing the feeling of annoyance from the user, unlike simple action imitation. It also allows the user to feel a sense of familiarity with the agent 100.

[0042] A modified example of the first embodiment will now be described. If the motion of a running character is directly imitated by agent 100, agent 100 will appear to be floating in the air. Therefore, when the agent control unit 60 of the information processing device 12 is to have agent 100 imitate the motion of a running character, it may convert the motion into stepping in place and control agent 100 to perform stepping in place.

[0043] Another modification relating to the first embodiment will now be described. The agent control unit 60 of the information processing device 12 may change the behavior of agent 100 regarding imitation according to the shape of agent 100, its characteristics (e.g., dexterity), and its state (e.g., physical strength). For example, the motion of agent 100 may be exaggerated or reduced (in other words, degraded).

[0044] Another modified example related to the first embodiment will be described. In the first embodiment, the object imitated by the agent was the user who was the communication partner, but the object imitated by the agent is not limited to the user. For example, the object imitated by the agent may include the character displayed on the screen of the display device 14 (the screen that displays the execution result of the application, for example, the game screen). Further, the object imitated by the agent may include both the user and the character displayed on the screen.

[0045] When the object imitated by the agent is the character displayed on the screen, the detection unit 56 may acquire information regarding the movement of the character from the game execution unit 50. The information regarding the movement of the character is information of a predetermined item that can identify the movement mode of the character, and may include, for example, the values of one or more parameters indicating the movement mode of the character. The detection unit 56 may detect the movement of the character based on the information regarding the movement of the character acquired from the game execution unit 50.

[0046] As another aspect, the information processing system 10 may further include a camera that captures the screen of the display device 14. The detection unit 56 may detect the movement of the character displayed on the screen by analyzing the captured image in which the screen of the display device 14 is captured using a known technique. The processing after detecting the movement of the character is the same as in the case where the object imitated by the agent is the user. According to the information processing apparatus 12 of this modified example, by causing the agent 100 to perform an operation obtained by abstracting the movement of the character at the frequency of the movement of the character, it is possible to suppress the user from feeling annoyed and to make the user feel the presence of the agent 100.

[0047] <Second Embodiment> The second embodiment of the present disclosure will be described focusing on the differences from the first embodiment. Among the components of the second embodiment, the components that are the same as or equivalent to the components of the first embodiment are assigned the same reference numerals as the components of the first embodiment, and the descriptions that overlap with the description of the first embodiment are omitted as appropriate.

[0048] The configuration of the information processing system 10 according to the second embodiment is the same as that of the information processing system 10 according to the first embodiment shown in FIG. 1. FIG. 4 is a block diagram showing the functional blocks of the information processing apparatus 12 according to the second embodiment. The information processing apparatus 12 according to the second embodiment is different from the information processing apparatus 12 according to the first embodiment in that it does not include the operation pattern storage unit 42 and the frequency update unit 58.

[0049] The operation of the information processing system 10 according to the second embodiment will be described. Here, as an operation different from that of the information processing system 10 according to the first embodiment, the subsequent processing after the display control unit 52 of the information processing apparatus 12 displays the game image and the agent 100 on the display device 14 will be described.

[0050] The camera 18 inputs a captured image in which the user appears to the information processing apparatus 12. The detection unit 56 of the information processing apparatus 12 analyzes the captured image in which the user appears output from the camera 18 using a known technique, and detects the expression of the user. For example, the detection unit 56 may detect (estimate) the expression of the user by comparing the captured image in which the user appears with a plurality of template images showing a plurality of patterns of expressions. Further, the detection unit 56 may detect (estimate) the expression of the user appearing in the captured image using an AI model that performs expression recognition.

[0051] The agent control unit 60 of the information processing apparatus 12 waits until a predetermined delay time elapses after the expression of the user is detected by the detection unit 56. After a predetermined delay time elapses after the expression of the user is detected, the agent control unit 60 controls the expression of the agent 100 so as to imitate the detected expression of the user. For example, the agent control unit 60 may generate an image of an agent having the same expression (or a relatively similar expression) as the detected expression of the user, and display the image on the display device 14 via the display control unit 52.

[0052] The user's facial expressions that can be detected or estimated by the detection unit 56 may include at least joy, surprise, fear, disgust, anger, sadness, and a neutral expression. The facial expression patterns of the pre-prepared agent 100 may also include at least joy, surprise, fear, disgust, anger, sadness, and a neutral expression. The detection unit 56 may also detect or estimate at least one of the user's emotions, namely the level of arousal and the emotional valence. The emotional valence can also be called the degree of positive / negative sentiment. The above delay time may be a value between 500 milliseconds and 2 seconds. Furthermore, the delay time may be determined to an appropriate value based on the knowledge of the system developer or the results of experiments using the information processing system 10.

[0053] According to the information processing device 12 of the second embodiment, by having the agent 100 imitate the user's facial expressions while providing a delay time, the user can feel the presence of the agent and develop a sense of familiarity with the agent.

[0054] Modifications of the user imitation of the first and second embodiments will be described. User imitation performed by the agent may include at least one of the following: (1) imitation of the user's blinking, (2) imitation of the user's coughing or yawning, (3) imitation of the user's habits (body movements or hand gestures), or (4) imitation of the user's reactions.

[0055] <Third Embodiment> The third embodiment of this disclosure will be described focusing on the differences from the previously described embodiments. Components of the third embodiment that are the same as or equivalent to the components of the previously described embodiments will be denoted by the same reference numerals as those of the previously described embodiments, and explanations that overlap with the descriptions of the previously described embodiments will be omitted as appropriate.

[0056] Figure 5 shows the configuration of the information processing system 10 of the third embodiment. The information processing system 10 of the third embodiment further includes a microphone 20 in addition to the components of the information processing system 10 of the first embodiment. The microphone 20 receives the voice emitted by the user (air vibrations accompanying the user's voice) and converts it into an electrical signal, and inputs the electrical signal based on the voice emitted by the user (hereinafter also referred to as "user voice data") to the information processing device 12.

[0057] The functional blocks provided by the information processing device 12 of the third embodiment are the same as the functional blocks provided by the information processing device 12 of the second embodiment shown in Figure 4.

[0058] The operation of the information processing system 10 of the third embodiment will be described. Here, the operation will differ from that of the information processing system 10 of the first embodiment, and will describe the processing after the display control unit 52 of the information processing device 12 displays the game image and agent 100 on the display device 14.

[0059] The microphone 20 inputs the user's voice data to the information processing device 12. The detection unit 56 of the information processing device 12 analyzes the user's voice data input from the microphone 20 using known techniques and detects the tone of the user's voice. The tone of the user's voice can also be called intonation, and includes patterns of pitch.

[0060] The agent control unit 60 of the information processing device 12 controls the tone of the agent 100's voice to mimic the tone of the user's voice detected by the detection unit 56. For example, multiple patterns of the agent 100's voice tone may be prepared in advance. The agent control unit 60 may generate voice data of the agent 100 (for example, a voice speaking to the user) set to the same or similar tone as the user's voice tone, and output that voice from the speaker via the voice control unit 54.

[0061] As a variation, the agent control unit 60 may estimate the user's emotions based on video or audio related to the content being processed by the information processing device 12, and may control the agent 100 to speak in a voice based on the estimated user's emotions. The agent control unit 60 may also estimate the user's emotions based on game scenes or events (for example, being disappointed if their character loses), and cause the agent 100 to speak in a voice based on the user's emotions.

[0062] According to the information processing device 12 of the third embodiment, by having the agent 100 imitate the tone of the user's voice, the user can feel the presence of the agent and develop a sense of familiarity with the agent.

[0063] <Fourth Embodiment> The fourth embodiment of this disclosure will be described focusing on the differences from the previously described embodiments. Components of the fourth embodiment that are the same as or equivalent to the components of the previously described embodiments will be denoted by the same reference numerals as those of the previously described embodiments, and explanations that overlap with the descriptions of the previously described embodiments will be omitted as appropriate.

[0064] The configuration of the information processing system 10 in the fourth embodiment is the same as the configuration of the information processing system 10 in the third embodiment shown in Figure 5. Furthermore, the functional blocks provided by the information processing device 12 in the fourth embodiment are the same as the functional blocks provided by the information processing device 12 in the second embodiment shown in Figure 4.

[0065] The operation of the information processing system 10 of the fourth embodiment will now be described. Here, the operation will differ from that of the information processing system 10 of the first embodiment, and will describe the processing after the display control unit 52 of the information processing device 12 displays the game image and agent 100 on the display device 14.

[0066] The detection unit 56 of the information processing device 12 detects the rhythm of the output sound. The sound whose rhythm is to be detected may be sound output in conjunction with the execution of an application in the information processing device 12. For example, the sound whose rhythm is to be detected may be the theme music or background music of a game (such as music played during battles with enemy characters or music played during dungeon exploration), or it may be the voices or music spoken by characters appearing in the game. The rhythm of sound can also be called prosody, and it defines the structure of the temporal changes in sound, for example, it defines when notes and rests are played.

[0067] The detection unit 56 of the information processing device 12 may acquire music data being output from the speaker of the display device 14 from the game execution unit 50 or the sound control unit 54. Alternatively, the microphone 20 may input music data obtained by converting the music being output from the speaker of the display device 14 into an electrical signal to the information processing device 12, and the detection unit 56 may acquire that music data. The detection unit 56 may use known techniques to extract the rhythm of the music from the acquired music data.

[0068] The agent control unit 60 of the information processing device 12 controls the agent 100 to hum the rhythm detected by the detection unit 56. For example, the agent control unit 60 may generate audio data of the agent 100 humming the detected rhythm and output that audio from the speaker via the audio control unit 54. The audio of the agent 100 humming the detected rhythm may be the agent 100 whispering the rhythm or the agent 100 humming the rhythm.

[0069] According to the information processing device 12 of the fourth embodiment, by controlling the agent 100 to hum along to the rhythm of the outputted voice, the user can feel the presence of the agent and develop a sense of familiarity with the agent.

[0070] A modification relating to the fourth embodiment will now be described. The agent control unit 60 of the information processing device 12 may control the agent 100 to emit a response to the sound detected by the detection unit 56. For example, the agent control unit 60 may cause the agent 100 to emit a response at the timing of a rest in the music detected by the detection unit 56.

[0071] Another modification relating to the fourth embodiment will now be described. The agent control unit 60 of the information processing device 12 may cause the agent 100 to utter (imitate) the same sound if the detection unit 56 detects the same sound a predetermined number of times or more. For example, the agent control unit 60 may measure the number of times the same sound has been output using the cross-correlation of the acoustic signals and / or the time-frequency histogram of the acoustic signals, and recognize that the same sound has been output a predetermined number of times or more than a threshold. In this modification, the sound that the agent is to imitate may be a catchphrase, for example, "You did it," or "What?".

[0072] Further variations relating to the fourth embodiment will now be described. The agent control unit 60 of the information processing device 12 may cause the agent 100 to mimic a game environment. Mimicking a game environment may include at least one of the following: (1) humming along to (or dancing to, or moving to the rhythm with) the background music or fanfare; (2) interjecting with comments to the background music or fanfare; (3) singing parodies of songs; or (4) behaving as if in the game world (for example, acting cold in the case of an ice country).

[0073] Further modifications relating to the fourth embodiment will now be described. The agent control unit 60 of the information processing device 12 may control the agent 100 to harmonize with the sound output in conjunction with the execution of the application in the information processing device 12. For example, the agent control unit 60 may cause the agent 100 to speak a sound of a different pitch (for example, a sound five degrees higher or a sound five degrees lower) than the sound output in conjunction with the execution of the application in the information processing device 12.

[0074] Further variations relating to the fourth embodiment will now be described. The agent control unit 60 of the information processing device 12 may control agent 100 to sing the rhythm and / or bassline of music output in conjunction with the execution of an application in the information processing device 12. For example, a bassline (e.g., "boom boom") may be extracted and agent 100 may be made to imitate it. Alternatively, agent 100 may be made to imitate a part of a specific instrument (e.g., drums). Alternatively, agent 100 may be made to imitate a sound resulting from a combination of pitch and a specific vocalization (e.g., "ta" and "ra") (e.g., "tarararattatta!"). Alternatively, agent 100 may be made to imitate a sound resulting from a combination of pitch and a specific line (e.g., "level up!").

[0075] Further modifications relating to the fourth embodiment will now be described. The agent control unit 60 of the information processing device 12 may, in parallel with controlling the agent 100 to hum along to the rhythm of the output sound, also control the agent 100 to move its body (for example, move its head or shoulders) in time with the output sound (rhythm, etc.).

[0076] <Fifth Embodiment> The fifth embodiment of this disclosure will be described focusing on the differences from the previously described embodiments. Components of the fifth embodiment that are the same as or equivalent to the components of the previously described embodiments will be denoted by the same reference numerals as those of the previously described embodiments, and explanations that overlap with the descriptions of the previously described embodiments will be omitted as appropriate.

[0077] Figure 6 shows the configuration of the information processing system 10 of the fifth embodiment. The information processing system 10 of the fifth embodiment further includes a video sharing site 24 in addition to the components of the information processing system 10 of the first embodiment. The video sharing site 24 includes an information processing device (such as a web server) that provides a known video sharing service. The information processing device 12 is connected to the video sharing site 24 via a communication network 22 including a LAN, WAN, the Internet, etc.

[0078] Figure 7 is a block diagram showing the functional blocks of the information processing device 12 of the fifth embodiment. The information processing device 12 of the fifth embodiment differs from the information processing device 12 of the first embodiment in that it does not have an operation pattern storage unit 42 and a frequency update unit 58, but instead has a video acquisition unit 62.

[0079] The operation of the information processing system 10 of the fifth embodiment will now be described. Here, as an operation different from that of the information processing system 10 of the first embodiment, the process after the display control unit 52 of the information processing device 12 displays the image of the application scene played by the user (hereinafter referred to as the game scene image) and the agent 100 on the display device 14 will be described.

[0080] The video acquisition unit 62 of the information processing device 12 identifies the game scene displayed on the display device 14. For example, the video acquisition unit 62 may identify the scene by analyzing the image of the game scene displayed on the display device 14 and based on the characters, etc., contained in the image. Alternatively, the video acquisition unit 62 may identify the scene by acquiring identification information of the currently displayed scene (such as the scene name or mission name) from the game execution unit 50.

[0081] The video acquisition unit 62 accesses the video sharing site 24 and acquires videos from the video sharing service provided by the video sharing site 24 that show a person playing the same scene as the game scene displayed on the display device 14. For example, the video acquisition unit 62 may search the video sharing site 24 for gameplay videos using scene identification information (such as the scene name or mission name) as a key, thereby acquiring videos that are gameplay videos related to the game played by the user and that show a person playing the same scene.

[0082] The detection unit 56 of the information processing device 12 detects the movements of a person in a video by analyzing the video acquired by the video acquisition unit 62 using known technology. The movements of a person detected by the detection unit 56 include not only physical movements such as shaking the body, raising hands, and jumping, but also the facial expressions of the person. The expressions that can be detected may include joy, surprise, fear, disgust, anger, sadness, and a blank expression.

[0083] The agent control unit 60 of the information processing device 12 controls the operation of the agent 100 to mimic the actions of the person detected by the detection unit 56. For example, the agent control unit 60 may generate an image of the agent 100 performing the same actions (including facial expressions) as the person detected by the detection unit 56, and display that image on the display device 14 via the display control unit 52.

[0084] According to the information processing device 12 of the fifth embodiment, the agent is made to mimic the actions (including facial expressions) of another person playing the same scene as the application (game, etc.) that the user is playing. This makes the user feel that the agent is real and fosters a sense of familiarity with the agent.

[0085] A modification of the fifth embodiment will now be described. The agent control unit 60 of the information processing device 12 may control the agent 100 to perform a specific reaction when a scene is displayed in which a specific reaction from the user is expected. For example, the information processing device 12 may further include a learning unit that analyzes videos acquired from a video sharing site 24 to learn the relationship between scenes (in-game events indicated by the scenes) and reactions. The agent control unit 60 may refer to the learning results and cause the agent 100 to perform a reaction corresponding to the displayed scene (in-game events indicated by the scenes). This allows, for example, the agent 100 to perform a surprised action when a scene that is likely to surprise the user is displayed.

[0086] In the learning process described above, the learning scope may be limited to videos from a specific video streamer, or to videos from video streamers with specific attributes (for example, male streamers or streamers who behave like friends). This makes it easier to maintain consistency in the agent's responses, or in other words, to stabilize the agent's personality. It also makes it easier for users to perceive the agent's personality, thereby enhancing the agent's sense of presence.

[0087] Another modification of the fifth embodiment will be described. The video acquisition unit 62 may recognize fine-grained game events. For example, the video acquisition unit 62 may recognize that it is a scene in which the player's character has lost and is looking up at the sky. The video acquisition unit 62 may analyze the video acquired from the video sharing site 24 and associate the events (fine-grained game events) shown in the video with the player's reactions and save them. The agent control unit 60 may control the agent 100 to perform a reaction associated with a fine-grained game event when that event occurs.

[0088] The present disclosure has been described above based on the first to fifth embodiments. Each embodiment is illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of components and processing processes, and that such modifications are also within the scope of the present disclosure.

[0089] Modifications applicable to each of the above embodiments will now be described. The scope of application of the agent control shown in each of the above embodiments is not limited to games, but can also be applied to characters in advertisements (e.g., advertisements using digital signage), movies, dramas, etc. For example, the areas in which the agent can be provided may include at least one of the following: digital signage, robots (real robots installed in the real world), androids, in-car navigation systems, in-home smart speakers (including display units), home agents, virtual humans, virtual characters (including humanoid and animal-type characters), VR (virtual reality), or AR (augmented reality) characters. When applied to advertisements, the information processing system 10 does not need to include the controller 16. Also, the controller 16 does not need to include a speaker.

[0090] Another variation applicable to each of the above embodiments is described. The actions of the person that the agent imitates may include at least one of yawning, tilting the head, or adjusting glasses. The information processing device 12 may also have a reaction database that associates situations with the agent's reactions. The agent control unit 60 of the information processing device 12 may cause the agent to take a reaction that is associated with the current situation in the reaction database. For example, in a situation where the character controlled by the user has been defeated by an enemy, if multiple reactions such as looking up at the sky, saying "Oh no," or saying "Seriously?" are associated with that situation, the agent control unit 60 may randomly select a reaction for the agent from among these multiple reactions.

[0091] Furthermore, if a user takes an action unrelated to the game, the agent control unit 60 of the information processing device 12 may ignore that user's action and exclude that user's action from being imitated. Also, if a user takes an action unrelated to the game and there is no game event, the agent control unit 60 may ignore that user's action and exclude that user's action from being imitated. An action unrelated to the game may be the user reacting to an event outside the house, or the user turning to look off-screen (for example, behind them).

[0092] Further variations applicable to each of the above embodiments are described. The agent control unit 60 of the information processing device 12 may slow down the agent's movements, such that the agent's internal state changes (for example, its stamina decreases) as the agent continues to imitate. Slowing down the agent's movements may include at least one of the following: slowing down the operating speed, reducing the imitation modal, or reducing the beat of the movements (slowing down the operating tempo).

[0093] Further modifications applicable to each of the above embodiments will be described. When the agent control unit 60 of the information processing device 12 has the agent imitate the body movements or facial features of the object to be imitated, it may reflect the rate of change of the body movements or facial features of the object to be imitated in the rate of change of the agent's body movements or facial features. For example, if the agent control unit 60 detects that the size of the eyes of the object to be imitated (e.g., the user) has been reduced to 0.4 times, it may reduce the size of the agent's eyes to 0.4 times, or for example, it may generate an image in which the size of the agent's eyes has been reduced to 0.4 times.

[0094] Further variations applicable to each of the above embodiments will be described. The agent control unit 60 of the information processing device 12 may manage the agent's emotions. If the agent is experiencing joy, the agent control unit 60 may control the agent to not only mimic the user but also mimic a character (taking the same pose as the in-game character) while making spontaneous utterances (for example, utterances expressing joy, such as "Yay!").

[0095] Further variations applicable to each of the above embodiments will be described. The agent control unit 60 of the information processing device 12 may include expressions using manga symbols (in other words, manga symbol animations) in the emotion imitation expressions by the agent.

[0096] The effects achieved by the information processing system 10 in each of the above embodiments will be explained in more detail below. In the information processing system 10 of each embodiment, the imitation control of the agent makes it easier for the user to recognize that the agent understands the game. By perceiving the agent as an entity that understands the game, the user's social skills can be improved. For example, it can suppress behaviors such as unfair play or abusive language.

[0097] Furthermore, the information processing system 10 in each embodiment produces the following effects: (1) The game experience (enjoyment) is enhanced by playing the game while maintaining a sense of unity with the agent. For example, the user can understand that the agent understands the game space. It can also make the user feel like they are playing the game together with the agent. Even if the user gets bored with the game, a metagame can be provided to the user by making the agent perform interesting actions. (2) The agent's sense of presence (in other words, sense of life) can be enhanced. For example, by having the agent say words such as "I'm tired," the user can feel the agent's presence. (3) The agent can create opportunities to intervene in the user's actions. For example, by having the agent say words such as "I'm tired now," the agent can suggest to the user that they take a break from the game. (4) The agent can perform actions that enhance the user's sense of closeness (attachment) to the agent. For example, when the character controlled by the user is frustrated, the agent is also frustrated, and when the character controlled by the user is happy, the agent is also happy, thereby increasing the user's sense of closeness to the agent. (5) By highlighting the creator's attention to detail, it is possible to encourage users to become more immersed in the game world and to increase the creator's motivation to create a detailed game. For example, by having the agent mimic actions that are small and difficult to see on the screen, users can feel the creator's dedication to creating the world.

[0098] As yet another modification applicable to each of the above embodiments, a highlight estimation process will be described. The information processing device 12 may further include an estimation unit that estimates highlights in games, etc. The estimation unit may prioritize estimating scenes in which camera processing has been performed as highlights. The estimation unit may also estimate scenes in which the pose of a character in the game screen is fixed as highlights. The estimation unit may detect if the same scene is repeated by analyzing autocorrelation, changes in the screen's color histogram, movement of feature points, etc. If the same scene is repeated a predetermined number of times or more, the estimation unit may estimate that scene as a highlight.

[0099] Furthermore, the estimation unit may estimate scenes in which significant visual effects are played as highlights. Scenes in which significant visual effects are played may include at least one of the following: scenes with rapid changes in brightness, scenes in which the protagonist's face is displayed prominently, scenes with many high-brightness colors in the screen histogram (or scenes with many whites), or scenes in which various effects are applied to the background.

[0100] Furthermore, the estimation unit may estimate highlights based on sound effects. For example, the estimation unit may estimate a highlight as a scene in which a loud sound effect is played, or at least one of two previously observed scenes in which a loud sound effect is played. The estimation unit may also estimate highlights based on the recognition results of audio acquired by a predetermined microphone. For example, the estimation unit may estimate a highlight if it acquires audio that signifies a level up (e.g., "Level up! Yay!") or audio that signifies a finishing move.

[0101] The agent control unit 60 of the information processing device 12 may determine the agent's behavior based on the internal information of the content being executed (game, movie, advertisement, etc.). For example, if the content to be executed is loading, the agent control unit 60 may determine the agent's behavior to be one of excitement (for example, anticipation and excitement). The content may output information (content internal information) to the agent control unit 60 indicating how the agent should behave. The console (game machine, etc.) may also output information to the agent control unit 60 indicating how the agent should behave, and the agent control unit 60 may determine the agent's behavior based on that information. The agent control unit 60 may operate the agent while a scene that the estimation unit has estimated to be a highlight is being displayed.

[0102] Further modifications applicable to each of the above embodiments will be described. The agent control unit 60 of the information processing device 12 may recognize the environment or situation of the content being played (game, movie, advertisement, etc.) and determine the agent's behavior. For example, in a scene showing lava, the agent may be determined to react by saying "ouch, it's hot." In a scene of the open sky, depending on the agent's personality, the agent may be determined to sway back and forth comfortably, or to react by showing fear.

[0103] Furthermore, in scenes with swaying motions, the agent's actions may be to maintain balance or to say "Whoa!". Also, depending on the ambient temperature, the agent's actions may be to feel hot or cold. In scenes with an icy world, the agent's actions may be to feel cold or to exhale visible breath.

[0104] In this modified example, the agent control unit 60 may determine the agent's reaction to the in-game situation based on the agent's personality. For example, in a roller coaster scene, the agent's actions based on their personality may be to feel exhilarated and say "Yay!" or to feel scared and act timidly.

[0105] Further variations applicable to each of the above embodiments will be described. The agent control unit 60 of the information processing device 12 may cause the agent to react positively (e.g., by showing joy) if a positive event (e.g., an event favorable to the user) occurs, even if neither the user nor the game character is responding. On the other hand, in the same situation, if a negative event (e.g., an event unfavorable to the user) occurs, the agent control unit 60 may cause the agent to react negatively (e.g., by showing disappointment).

[0106] Further modifications applicable to each of the above embodiments will be described. The agent control unit 60 of the information processing device 12 may prevent the agent from becoming stationary by having the agent repeat the previous operation if there are no estimation results regarding the environment or situation of the content being executed (game, movie, advertisement, etc.). Alternatively, the agent control unit 60 may prevent the agent from becoming stationary by having the agent perform a pre-prepared idling operation.

[0107] Any combination of the embodiments and modifications described above is also useful as an embodiment of the present invention. The new embodiments resulting from these combinations possess the combined effects of each of the combined embodiments and modifications. Furthermore, it will be understood by those skilled in the art that the functions to be performed by each component described in the claims can be achieved by each component shown in the embodiments and modifications individually or in combination thereof.

[0108] The technical concepts described in this disclosure can be expressed as follows: [Technology 1] An information processing device comprising a circuit configured to the following, wherein the circuit displays an agent that communicates with a user, detects the operation of a predetermined imitation target, updates the frequency of a predetermined set of operation patterns based on the detected operation of the imitation target, and controls the operation of the agent according to the frequency of the set of operation patterns. [Technology 2] The information processing device according to Technology 1, wherein the imitation target includes at least one of the user and a character displayed on a screen. [Technology 3] An information processing device comprising a circuit configured to the following, wherein the circuit displays an agent that communicates with a user, detects the user's facial expression, and controls the agent's facial expression to imitate the detected user's facial expression after a predetermined delay time has elapsed since the detection of the user's facial expression. [Technical 4] An information processing device comprising a circuit configured to the following, wherein the circuit displays an agent that communicates with a user, detects the tone of the user's voice, and controls the tone of the agent's voice to mimic the detected tone of the user's voice. [Technical 5] An information processing device comprising a circuit configured to the following, wherein the circuit displays an agent that communicates with a user, detects the rhythm of the output voice, and controls the agent to hum the detected rhythm.[Technical 6] Information processing device comprising a circuit configured to the following: the circuit displays a scene of an application played by a user and an agent that communicates with the user; it acquires a video from a video sharing service showing a person playing the same scene as the displayed application scene; it detects the actions of the person in the acquired video; and it controls the actions of the agent to mimic the detected actions of the person.

[0109] This invention can be applied to information processing devices, information processing systems, and the like.

[0110] 10 Information processing system, 12 Information processing device, 14 Display device, 18 Camera, 20 Microphone, 52 Display control unit, 54 Audio control unit, 56 Detection unit, 58 Frequency update unit, 60 Agent control unit, 62 Video acquisition unit.

Claims

1. An information processing device comprising: a display control unit that displays an agent that communicates with a user; a detection unit that detects the actions of a predetermined target to be imitated; a frequency update unit that updates the frequency of a predetermined set of action patterns based on the detected actions of the target to be imitated; and an agent control unit that controls the actions of the agent according to the frequency of the set of action patterns.

2. The information processing apparatus according to claim 1, wherein the object to be imitated includes at least one of the user and a character displayed on the screen.

3. An information processing device comprising: a display control unit that displays an agent that communicates with a user; a detection unit that detects the user's facial expression; and an agent control unit that controls the agent's facial expression to mimic the detected user's facial expression after a predetermined delay time has elapsed since the detection of the user's facial expression.

4. An information processing device comprising: a display control unit that displays an agent that communicates with a user; a detection unit that detects the tone of the user's voice; and an agent control unit that controls the tone of the agent's voice to mimic the detected tone of the user's voice.

5. An information processing device comprising: a display control unit that displays an agent that communicates with the user; a detection unit that detects the rhythm of the voice being output; and an agent control unit that controls the agent to hum the detected rhythm.

6. An information processing device comprising: a display control unit that displays a scene of an application played by a user and an agent that communicates with the user; a video acquisition unit that acquires a video from a video sharing service showing a person playing the same scene as the displayed application scene; a detection unit that detects the movements of a person in the acquired video; and an agent control unit that controls the actions of the agent to mimic the detected movements of the person.

Images