Information processing device, information processing method, and computer program

Abstract

An information processing device 12 controls the operation of a robot 20. The information processing device 12 detects at least one user eye and at least one user mouth on the basis of an image of user(s) captured by a camera 18. When a character communicates or attempts to communicate with the user(s), the information processing device 12 operates a part corresponding to the eyes of the robot 20 such that the line of sight moves along sides of a polygon connecting the at least one user eye and the at least one user mouth.

Description

Information Processing Apparatus, Information Processing Method, and Computer Program

[0001] The present disclosure relates to data processing technology, and more particularly to an information processing apparatus, an information processing method, and a computer program.

[0002] In an information processing apparatus, a character that communicates with a user may be displayed.

[0003] In order to effectively improve the quality of the user's experience through interaction with the character, the inventor considered that it is important to enhance the sense of reality of the character. One object of the present disclosure is to provide a technique for enhancing the sense of reality of a character that communicates with a user.

[0004] To solve the above problems, an information processing apparatus according to an aspect of the present disclosure includes a robot control unit that controls the operation of a robot, and a detection unit that detects one or more eyes and one or more mouths of a user based on an image of the user. When the robot is communicating with the user or attempting to communicate with the user, the robot control unit operates a portion corresponding to the eyes of the robot so that the line of sight moves along the sides of a polygon connecting one or more eyes and one or more mouths of the user.

[0005] Another aspect of the present disclosure is an information processing apparatus. The apparatus includes a display control unit that causes a predetermined character to be displayed on a display, and a detection unit that detects one or more eyes and one or more mouths of a user based on an image of the user. When the character is communicating with the user or attempting to communicate with the user, the display control unit operates a portion corresponding to the eyes of the character so that the line of sight moves along the sides of a polygon connecting one or more eyes and one or more mouths of the user.

[0006] Another aspect of this disclosure is an information processing method. This method involves an information processing device detecting one or more eyes and one or more mouths of a user based on an image of the user, controlling the movement of a robot, and when the robot communicates with or attempts to communicate with the user, moving the parts of the robot corresponding to the eyes so that the line of sight moves along the sides of a polygon connecting the one or more eyes and one or more mouths of the user.

[0007] Another aspect of this disclosure is also an information processing method. This method involves an information processing device detecting one or more eyes and one or more mouths of a user based on an image of the user, displaying a predetermined character on a display, and when the character communicates with or attempts to communicate with the user, moving the parts of the character corresponding to the eyes so that the line of sight moves along the sides of a polygon connecting the user's one or more eyes and one or more mouths.

[0008] Furthermore, any combination of the above components, as well as any conversion of the expressions of this disclosure between systems, computer programs, recording media storing computer programs, etc., are also valid forms of this disclosure.

[0009] According to this disclosure, it is possible to enhance the sense of realism of characters that communicate with users.

[0010] This is a diagram showing the configuration of the information processing system of the first embodiment. This is a block diagram showing the functional blocks of the robot of the first embodiment. This is a block diagram showing the functional blocks of the information processing device of the first embodiment. This is a flowchart showing the operation of the information processing device of the first embodiment. This is a diagram schematically showing an example of robot control. This is a diagram schematically showing an example of triangular gaze control for an agent. This is a diagram showing the configuration of the information processing system of the second embodiment. This is a block diagram showing the functional blocks of the information processing device of the second embodiment. This is a flowchart showing the operation of the information processing device of the second embodiment. This is a diagram schematically showing an example of agent display. This is a diagram showing the configuration of the information processing system of the third embodiment. This is a diagram schematically showing an example of agent display.

[0011] The embodiments are outlined below. In each embodiment, the user is a player of a game application. The information processing system in each embodiment controls an agent that communicates continuously with the user. The agent acts as a friend or companion to the user, enjoying the game together and providing hints for the game. The agent includes a robot placed in real space (first embodiment) and a character placed in a virtual space (game space, etc.) (second embodiment). The communication between the user and the agent can also be described as interaction between the user and the agent.

[0012] The inventors of this invention believe that enhancing the sense of presence of the agent is important in order to effectively improve the user experience (in an embodiment, the game experience) through communication with the agent. The sense of presence of the agent can be described as making the user feel as if the agent exists in the same world as the user, or as making the user feel as if they are sharing the same world as the agent.

[0013] To enhance the agent's sense of presence, the information processing system in each embodiment makes the character move so that its gaze moves between the user's eyes and mouth. This improves the user experience through communication with the agent.

[0014] <First Embodiment> Figure 1 shows the configuration of the information processing system 10 according to the first embodiment. The information processing system 10 includes an information processing device 12, a display device 14, a controller 16, a camera 18, and a robot 20.

[0015] The functions of the components in each computer (e.g., information processing device 12, robot 20, etc.) constituting the information processing system 10 may be realized in circuits or processing circuits, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (a Central Processing Unit), conventional circuits, and / or combinations thereof, which are configured or programmed to realize the functions described herein. A processor is considered to be a circuit or processing circuit that includes transistors and other circuits. A processor may also be a programmed processor that executes a program stored in memory.

[0016] In this specification, circuits, units, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0017] If the hardware is a processor that is considered to be a type of circuit, then the circuit, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0018] The information processing device 12 includes a computer capable of running system software and various application software. In the first embodiment, the information processing device 12 is a stationary game console, but as a modification, the information processing device 12 may be a portable game console, a PC, a tablet terminal, a smartphone, etc.

[0019] The controller 16 is an operating device that receives user inputs to the system software and application software executed by the information processing device 12. The controller 16 is connected to the information processing device 12 by wire or wireless connection and sequentially transmits operation information indicating the user inputs to the information processing device 12.

[0020] The controller 16 is equipped with a speaker and can output sound based on an audio signal transmitted from the information processing device 12. The controller 16 is also equipped with a vibrator and can vibrate the vibrator in a manner based on a haptic signal transmitted from the information processing device 12.

[0021] The display device 14 has a display that outputs images and a speaker that outputs sound. The display device 14 may be a television or a computer display. The display device 14 is connected to the information processing device 12 by wire or wireless. When the information processing device 12 receives operation information provided by the controller 16, it reflects that operation information in the processing of system software or application software, displays an image related to the processing result on the display device 14, and outputs sound related to the processing result from the speaker of the display device 14.

[0022] The display device 14 may be a head-mounted display worn on the user's head. In this case, the information processing device 12 and the display device 14 (head-mounted display) may be configured to display a panoramic image on the head-mounted display, and when the user wearing the head-mounted display rotates their head, a panoramic image corresponding to the direction of their gaze is displayed. The display device 14 (head-mounted display) may be configured to block out external light and may display a virtual reality image generated by the information processing device 12. Alternatively, the display device 14 (head-mounted display) may be optically transparent and may display augmented reality images generated by the information processing device 12.

[0023] Camera 18 is a device that captures images of a user playing a game. Camera 18 is connected to the information processing device 12 by wire or wireless connection and transmits data of images of the user (hereinafter also referred to as "captured images") to the information processing device 12. The captured images in the first embodiment include at least still images or videos showing the user's face.

[0024] Robot 20 is a robot that is installed in real space and operates as an agent. Robot 20 is equipped with parts that correspond to human eyes, or in other words, parts that look like eyes externally. Robot 20 may be a humanoid robot (a so-called android) or a pet-type robot.

[0025] Figure 2 is a block diagram showing the functional blocks of the robot 20 according to the first embodiment. Each block shown in the block diagrams of this specification can be realized in hardware terms by a computer processor (CPU, etc.), memory and other elements, electronic circuits, and mechanical devices, and in software terms by a computer program, etc., but here we are depicting functional blocks that are realized through the cooperation of these. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways by combinations of hardware and software.

[0026] The robot 20 includes actuators 22, a speaker 23, a control unit 24, and a communication unit 26. The actuators 22 are mechanical elements (in other words, drive devices) that move each part of the robot 20. The speaker 23 is an acoustic device that outputs instructed sounds as sounds emitted by the robot. The communication unit 26 communicates with external devices according to a predetermined communication protocol. The control unit 24 sends and receives data with the information processing device 12 via the communication unit 26.

[0027] The control unit 24 controls the operation of the actuator 22 according to the control signals transmitted from the information processing device 12, thereby causing the robot 20 to perform the actions and postures instructed by the information processing device 12. The control unit 24 also controls the robot 20's vocalization by outputting the sound specified by the control signals transmitted from the information processing device 12 through the speaker 23.

[0028] Figure 3 is a block diagram showing the functional blocks of the information processing device 12 according to the first embodiment. The information processing device 12 comprises a processing unit 30, a storage unit 32, and a communication unit 34. The processing unit 30 performs various data processing. The storage unit 32 stores data that is referenced or updated by the processing unit 30. The communication unit 34 communicates with external devices via wired or wireless communication according to a predetermined communication protocol. The processing unit 30 sends and receives data to and from the display device 14, controller 16, camera 18, and robot 20 via the communication unit 34.

[0029] The storage unit 32 includes a program storage unit 40 and a closeness storage unit 42. The program storage unit 40 stores system software and application software. The application software in the first embodiment includes a game application (e.g., a video game). The closeness storage unit 42 stores information indicating the closeness between the user and the agent.

[0030] The processing unit 30 includes a game execution unit 50, a display control unit 52, an image acquisition unit 54, a detection unit 56, an action determination unit 58, a closeness update unit 60, and a robot control unit 62. The functionality of at least one of these multiple functional blocks may be implemented in a computer program (for example, system software and / or application software). This computer program may be installed in the storage of the information processing device 12. The processor of the information processing device 12 may perform the functions of these multiple functional blocks by reading this computer program into main memory and executing it.

[0031] The game execution unit 50 executes system software and application software stored in the program storage unit 40. For example, the game execution unit 50 executes a game application in response to user input to the controller 16. It generates game images based on the execution results of the game application.

[0032] The display control unit 52 controls the content displayed on the display device 14. For example, the display control unit 52 causes the game image generated by the game execution unit 50 to be displayed on the display device 14.

[0033] The image acquisition unit 54 acquires the captured image of the user output from the camera 18. The detection unit 56 analyzes the captured image to detect the position of one or more eyes and one or more mouths of the user in the captured image. Typically, the detection unit 56 detects the positions of two eyes and one mouth of the user in the captured image. The detection unit 56 also analyzes the captured image to further detect the user's posture.

[0034] The operation determination unit 58 determines the agent's operation mode. The operation determination unit 58 determines whether predetermined conditions for the agent to communicate with the user (hereinafter also referred to as "communication conditions") have been met. If the communication conditions have been met, the operation determination unit 58 determines the agent's operation mode for communicating with the user. The agent's operation mode for communicating with the user may include, for example, speaking to the user (and what to say), smiling at the user, waving to the user, etc.

[0035] The communication conditions may be met if (1) the game the user is playing reaches a predetermined state, (2) a predetermined amount of real time has been reached, or (3) the user's state, as detected by a predetermined sensor, reaches a predetermined state. The predetermined state in (1) may be that the game's objective has been achieved, that the game's objective has not been achieved, or that the game has lasted for a predetermined amount of time or longer. The predetermined state in (3) may be that the user's gaze is directed towards the agent.

[0036] The communication conditions in the first embodiment include the user assuming a predetermined posture. That is, the operation determination unit 58 determines whether or not to allow the agent to communicate with the user based on the user's posture detected by the detection unit 56.

[0037] Furthermore, the operation determination unit 58 determines that, as an operation mode for the agent accompanying the action for communicating with the user, the part of the agent corresponding to the eyes will be operated so that the line of sight moves along the sides (which can also be called the outer edge) of a polygon connecting one or more eyes and one or more mouths of the user detected by the detection unit 56 (hereinafter also referred to as "triangular gaze control"). The part of the agent corresponding to the eyes can also be said to be the part of the agent corresponding to the human eye among the multiple parts that the agent has, and also includes the agent's eye itself. In the first embodiment, since the robot 20 operates as the agent, the part of the agent corresponding to the eyes will also be referred to as "the eyes of the robot 20" below.

[0038] The robot control unit 62 controls the operation of the robot 20. The robot control unit 62 operates the robot 20 in the manner determined by the operation determination unit 58 by transmitting a control signal to the robot 20 instructing it to operate in the manner determined by the operation determination unit 58. For example, the robot control unit 62 may operate the robot 20 to speak to the user by transmitting a control signal to the robot 20 instructing it to output a voice (for example, a conversation voice with the user) determined by the operation determination unit 58, and a control signal to the robot 20 instructing it to move its mouth in accordance with that voice output. Alternatively, the robot control unit 62 may cause the robot 20 to perform a gesture action determined by the operation determination unit 58 by transmitting a control signal to the robot 20 instructing it to perform that gesture action.

[0039] When the communication conditions are met, this includes when the agent communicates with the user (for example, while communicating) and when the agent attempts to communicate with the user (for example, immediately before communicating). When the communication conditions are met, the robot control unit 62 operates the robot's eyes as triangular gaze control for the agent, so that the line of sight moves along the sides of a polygon connecting one or more eyes and one or more mouths of the user detected by the detection unit 56. Controlling the robot's eyes so that the line of sight moves to a specific position in three-dimensional space can be achieved using known techniques.

[0040] The intimacy update unit 60 updates the intimacy level between the user and the agent stored in the intimacy storage unit 42 based on the status or performance of communication between the user and the agent. The communication conditions in the first embodiment include the intimacy level between the user and the agent being above a predetermined threshold. Unless otherwise specified, the threshold described in this disclosure may be set to an appropriate value based on the knowledge of the system developer or through experiments using the information processing system 10.

[0041] The operation determination unit 58 decides whether or not to allow the agent to communicate with the user based on the level of intimacy between the user and the agent. The robot control unit 62 operates the eyes of the robot 20 so that the line of sight moves along the sides of a polygon connecting one or more of the user's eyes and one or more of their mouths, when the agent communicates with the user or attempts to communicate with the user and the level of intimacy between the user and the agent is above a predetermined threshold.

[0042] Figure 4 is a flowchart showing the operation of the information processing device 12 of the first embodiment. The operation of the information processing system 10 of the first embodiment will be described below with reference to Figure 4.

[0043] The game execution unit 50 of the information processing apparatus 12 executes a game application according to the user's operation input to the controller 16, and generates a game image indicating the execution result of the game. The display control unit 52 of the information processing apparatus 12 causes the display device 14 to display the game image.

[0044] The camera 18 transmits data of a captured image in which the user appears to the information processing apparatus 12. When the image acquisition unit 54 of the information processing apparatus 12 receives the captured image in which the user appears, which is transmitted from the camera 18 (Y in S10), the detection unit 56 of the information processing apparatus 12 analyzes the captured image and detects the positions of the user's eyes and mouth (S11). The detection unit 56 may detect the three-dimensional coordinate positions of the user's eyes and mouth in the world coordinate system (in other words, the global coordinate system).

[0045] Further, the detection unit 56 of the information processing apparatus 12 analyzes the captured image in which the user appears, and detects the user's posture with respect to the display device 14 on which the agent is displayed (in other words, the user's posture with respect to the agent) (S12). For example, the detection unit 56 detects that the user's posture is a posture facing the display device 14, in other words, a posture facing the agent displayed on the display device 14. If the captured image has not been acquired (N in S10), the processes of S11 and S12 are skipped.

[0046] The intimacy update unit 60 determines whether or not a predetermined intimacy update condition is satisfied based on the communication situation or results between the user and the agent. For example, the intimacy update condition may be satisfied when the number of times the agent has spoken to the user is equal to or greater than a predetermined threshold, or when the number of times the user has input an operation to respond to the agent is equal to or greater than a predetermined threshold.

[0047] When the intimacy update condition is satisfied (Y in S13), the intimacy update unit 60 derives a new intimacy between the user and the agent, and stores the new intimacy in the intimacy storage unit 42 (S14). For example, the intimacy update unit 60 may determine a higher intimacy as the number of times the agent talks to the user increases and / or as the number of times the user inputs an operation to respond to the agent increases. If the intimacy update condition is not satisfied (N in S13), the process of S14 is skipped.

[0048] When the communication condition is satisfied based on the state (progress) of the game in which the user is playing and the user's posture (Y in S15), and the intimacy between the user and the agent is equal to or greater than a predetermined threshold (Y in S16), the operation determination unit 58 of the information processing apparatus 12 determines triangular line-of-sight control for the agent. The operation determination unit 58 may determine the operation mode of the eyes (particularly the pupils) of the agent (the robot 20 in the first embodiment) so that the line of sight moves along the line connecting the three-dimensional coordinate positions of the user's eyes and mouth in the world coordinate system detected in S11.

[0049] The robot control unit 62 of the information processing apparatus 12 operates the eyes of the robot 20 in the mode determined by the operation determination unit 58 as triangular line-of-sight control for the agent. Specifically, the robot control unit 62 operates the eyes of the robot 20 (particularly the part corresponding to the pupils) so that the tip of the line of sight moves clockwise or counterclockwise along the side of the polygon connecting the user's eyes and mouth (S17).

[0050] For example, the robot control unit 62 may send a control signal to the robot 20 specifying the three-dimensional coordinate position of one vertex of a polygon connecting the user's eyes and mouth as the initial position of the line of sight, and specifying the three-dimensional coordinate position of another vertex of that polygon as the destination position of the line of sight. The control unit 24 of the robot 20 may operate the actuator 22 for the robot 20's eyes in accordance with the control signal sent from the information processing device 12 so that the line of sight of the robot 20 moves along the side of the polygon connecting the user's eyes and mouth. Operating the actuator 22 for the robot 20's eyes so that the line of sight moves in a specific direction in three-dimensional space may be achieved by known technology. If the communication conditions are not met (N in S15), or if the intimacy between the user and the agent is below a predetermined threshold (N in S16), the process in S17 is skipped.

[0051] Figure 5 schematically shows an example of robot 20 control. The display device 14 in Figure 5 displays a game image including multiple objects A to D related to the game. Objects A to D are various characters and objects in the game world. In Figure 5, the robot 20's line of sight is indicated by a dashed line.

[0052] Figure 6 schematically shows an example of triangular gaze control for an agent. In the figure, the right eye, left eye, and mouth represent the user's right eye, left eye, and mouth as seen from the agent's perspective. The robot control unit 62 of the information processing device 12 may move the robot 20's line of sight clockwise (for example, right eye → left eye → mouth → right eye) as shown by the solid line. Alternatively, the robot control unit 62 may move the robot 20's line of sight counterclockwise (for example, right eye → mouth → left eye → right eye) as shown by the dashed line.

[0053] According to the information processing device 12 of the first embodiment, the robot 20 is made to feel a sense of presence to the user (for example, an impression that the robot 20 is human), making it easier for the user to feel a sense of familiarity with the robot 20. Furthermore, according to the information processing device 12, the robot 20 can communicate with the user at an appropriate timing based on the user's posture. In addition, according to the information processing device 12, by having the robot 20 communicate with the user when the level of familiarity with the user is high, the robot 20 can be made to behave in a human-like manner.

[0054] Modifications relating to the first embodiment will now be described. Instead of having eyes, the robot 20 may have a part that is assumed to be perceived by the user as the robot 20's eyes due to the pareidolia phenomenon (in other words, a part corresponding to the eyes). And / or, instead of having a mouth, the robot 20 may have a part that is assumed to be perceived by the user as the robot 20's mouth due to the pareidolia phenomenon (in other words, a part corresponding to the mouth). In this case, the robot control unit 62 of the information processing device 12 may perform triangular gaze control on the relevant part of the robot 20.

[0055] <Second Embodiment> The second embodiment of this disclosure will be described focusing on the differences from the first embodiment. Components of the second embodiment that are the same as or equivalent to components of the first embodiment will be denoted by the same reference numerals as those of the first embodiment, and explanations that overlap with the description of the first embodiment will be omitted as appropriate.

[0056] Figure 7 shows the configuration of the information processing system 10 of the second embodiment. The information processing system 10 of the second embodiment differs from the information processing system 10 of the first embodiment in that, instead of a robot 20, it displays an image of an agent as a character in a virtual space (hereinafter also simply referred to as "agent") on the display device 14.

[0057] Figure 8 is a block diagram showing the functional blocks of the information processing device 12 of the second embodiment. The information processing device 12 of the second embodiment differs from the information processing device 12 of the first embodiment in that it does not include a robot control unit 62.

[0058] In the second embodiment, the display control unit 52 displays a predetermined image of an agent, which is a character in the virtual space, on the display device 14 and controls the agent display mode. Specifically, the display control unit 52 operates the agent to communicate with the user in a manner determined by the operation determination unit 58. For example, the display control unit 52 may operate the agent to speak to the user, or to perform a predetermined gesture to the user. As previously described, the display device 14 may be a head-mounted display, and the display control unit 52 may display the agent, which is a character placed in the VR space, on the display device 14 (head-mounted display).

[0059] Furthermore, when the communication conditions described in the first embodiment are met, the display control unit 52 operates the part corresponding to the agent's eyes (hereinafter simply referred to as "agent's eyes") so that the point of gaze moves along the sides of a polygon connecting one or more eyes and one or more mouths of the user detected by the detection unit 56, as triangular gaze control for the agent. Controlling the agent's eyes so that the gaze of the agent displayed on the display device 14 moves to a specific position in three-dimensional space can be achieved using known technology.

[0060] Furthermore, when the agent communicates with the user, or when the agent attempts to communicate with the user, and the level of intimacy between the user and the agent is above a predetermined threshold, the display control unit 52 operates the agent's eyes so that the line of sight moves along the sides of a polygon connecting one or more of the user's eyes and one or more of their mouths.

[0061] Figure 9 is a flowchart showing the operation of the information processing device 12 in the second embodiment. The display control unit 52 of the information processing device 12 causes the game image to be displayed on the display device 14. The display control unit 52 also causes the image of the agent assisting the user to be displayed on the display device 14 (S20). The display control unit 52 may display the agent image together with the game image, or it may suppress the display of the agent image when the game image is displayed.

[0062] The processes S21 to S25 in Figure 9 are the same as the processes S10 to S14 in Figure 4, so their explanation is omitted. Based on the state (progress) of the game being played by the user and the user's posture, if the communication conditions are met (Y in S26) and the level of intimacy between the user and the agent is above a predetermined threshold (Y in S27), the operation determination unit 58 of the information processing device 12 determines triangular gaze control for the agent. The operation determination unit 58 may also determine the operation mode of the agent's eyes (especially the pupils) so that the gaze moves along a line connecting the three-dimensional coordinate positions of the user's eyes and mouth in the world coordinate system detected in S12.

[0063] The display control unit 52 of the information processing device 12 displays an image of the agent on the display device 14, in which the agent's eyes move in a manner determined by the operation determination unit 58, as triangular gaze control for the agent. Specifically, the display control unit 52 moves the agent's eyes (especially the pupils) so that the point of gaze moves clockwise or counterclockwise along the sides of a polygon connecting the user's eyes and mouth (S28). If the communication conditions are not met (N in S26), or if the intimacy between the user and the agent is below a predetermined threshold (N in S27), the process in S28 is skipped.

[0064] Figure 10 schematically shows an example of agent display. The display device 14 in Figure 10 displays a game image including multiple game-related objects A to D and agent 100. Objects A to D are various characters and objects in the game world. In Figure 10, the agent's line of sight is indicated by a dashed line. In the second embodiment, the display control unit 52 of the information processing device 12 performs triangular line of sight control, as explained in relation to Figure 6, on the agent 100 displayed on the display device 14.

[0065] According to the information processing device 12 of the second embodiment, the agent is made to give the user a sense of presence (for example, the impression that the agent is human), making it easier for the user to feel a sense of familiarity with the agent. Furthermore, according to the information processing device 12, the agent can communicate with the user at an appropriate timing based on the user's posture. In addition, according to the information processing device 12, by having the agent communicate with the user when the level of familiarity with the user is high, the agent can be made to behave in a human-like manner.

[0066] A modification relating to the second embodiment will now be described. The display control unit 52 of the information processing device 12 may, instead of having eyes, display on the display device 14 an image of an agent having a part that the user is expected to recognize as the agent's eyes due to the pareidolia phenomenon (in other words, a part corresponding to the eyes). And / or, the display control unit 52 may, instead of having a mouth, display on the display device 14 an image of an agent having a part that the user is expected to recognize as the agent's mouth due to the pareidolia phenomenon (in other words, a part corresponding to the mouth). In this case, the display control unit 52 of the information processing device 12 may perform triangular gaze control on that part of the agent.

[0067] <Third Embodiment> The third embodiment of this disclosure will be described focusing on the differences from the second embodiment. Components of the third embodiment that are the same as or equivalent to components of the second embodiment will be denoted by the same reference numerals as those of the second embodiment, and explanations that overlap with the explanation of the second embodiment will be omitted as appropriate.

[0068] Figure 11 shows the configuration of the information processing system 10 of the third embodiment. The information processing system 10 of the third embodiment differs from the information processing system 10 of the second embodiment in that (1) it includes a plurality of display devices 14 (specifically, display devices 14a and display devices 14b) connected to the information processing device 12, and (2) it selectively displays an agent on one of the plurality of display devices 14.

[0069] Display devices 14a and 14b may both be computer displays. Alternatively, one of the display devices 14a and 14b may be a television and the other a computer display. Alternatively, one of the display devices 14a and 14b may be a television and the other a tablet terminal. The information processing system 10 may be equipped with three or more display devices 14.

[0070] The functional blocks of the information processing device 12 in the third embodiment are the same as those of the information processing device 12 in the second embodiment shown in Figure 8. The display control unit 52 of the information processing device 12 causes the agent image to be displayed on a plurality of display devices 14. In the third embodiment, the display control unit 52 causes the agent to be displayed on either the display device 14a or the display device 14b depending on the game situation or user operation. The display control unit 52 switches between displaying the agent on the display device 14a and displaying the agent on the display device 14b depending on the game situation or user operation.

[0071] For example, the display device 14a may be a television. The display device 14b may be a tablet device or a spatial display. A spatial display is a display capable of displaying three-dimensional images and is also called a three-dimensional display. The display control unit 52 may display images of the game being played on the information processing device 12 on the display device 14a. The display control unit 52 may display an agent on the display device 14a when the game images are not being displayed on the display device 14a. On the other hand, during the period when the game images are being displayed on the display device 14a, the display control unit 52 may terminate the display of the agent on the display device 14a and display the agent on the display device 14b. The display control unit 52 applies the technology described in the first embodiment to perform triangular gaze control on the agent displayed on the display device 14a and the agent displayed on the display device 14b, respectively.

[0072] Figure 12 schematically shows an example of agent display. In Figure 12, a game image including multiple game-related objects A to D is displayed on the display device 14a, and agent 100 is displayed on the display device 14b. In Figure 12, the agent's line of sight is indicated by a dashed line. The display control unit 52 performs triangular line of sight control for the agent displayed on the display device 14b.

[0073] Each of the display devices 14a and 14b is equipped with a camera 18. The detection unit 56 of the information processing device 12 detects the distance between the user and each display device 14 based on the image captured by the camera 18 of each display device 14. The camera 18 of each display device 14 may include a stereo camera. The detection unit 56 may derive the direction of the user and the distance to the user based on the parallax shown in the image captured by the stereo camera. In addition, the camera 18 of each display device 14 may include a Time of Flight (ToF) camera. The detection unit 56 may derive the distance between the user and each display device 14 based on the output information of the ToF camera of each display device 14.

[0074] The display control unit 52 of the information processing device 12 sets the size of the agent to be displayed on each display device 14 based on the distance between the user and each display device 14. In the second embodiment, the display control unit 52 reduces the size of the agent to be displayed on the display device 14 as the distance between the user and the display device 14 increases.

[0075] For example, the display control unit 52 may calculate the physical size of the agent displayed on the screen of the display device 14a (referred to as the "first size") based on the screen resolution (ppi (Pixels per inch)) of the display device 14a and the display size of the agent (number of pixels, etc.). The display control unit 52 may also calculate the physical size of the agent to be displayed on the screen of the display device 14b (referred to as the "second size") based on the ratio of the distance between the user and the display device 14a to the distance between the user and the display device 14b. That is, the display control unit 52 may decide how much to scale the first size to obtain the second size. The display control unit 52 may also calculate the number of pixels required to display the second-size agent based on the screen resolution of the display device 14b.

[0076] The information processing device 12 of the third embodiment also provides the same effects as the information processing device 12 of the second embodiment. According to the information processing device 12 of the third embodiment, when displaying agents on multiple displays, it is possible to maintain consistency in the size of the characters displayed on each display as seen by the user, and it further provides the effect of suppressing the user from feeling uncomfortable with the size of the characters.

[0077] The present disclosure has been described above based on the first to third embodiments. Each embodiment is illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of each component and each processing process, and that such modifications are also within the scope of the present disclosure.

[0078] Modifications relating to the first to third embodiments will now be described. When predetermined termination conditions are met, the operation determination unit 58 of the information processing device 12 may decide to terminate the agent's triangular gaze control. In the first embodiment, the robot control unit 62 of the information processing device 12 may terminate the robot 20's triangular gaze control according to the decision of the operation determination unit 58. The display control unit 52 of the information processing device 12 may terminate the agent's triangular gaze control according to the decision of the operation determination unit 58.

[0079] The termination conditions may be met if at least one of the following states (1) to (4) occurs: (1) The user is no longer visible in the image captured by the camera 18. (2) A predetermined amount of time has elapsed since the start of triangular gaze control. (3) Based on the image captured by the camera 18, it is detected that the time the user has been gazing at the agent has exceeded a predetermined amount of time. (4) A high-priority event has occurred. The high-priority event may be a predetermined event related to the game (for example, the game has started).

[0080] Another modification relating to the first to third embodiments will be described. If, due to false detection or the like, two or more mouths are detected on the target (the user in the first embodiment) that is the focus of the agent's gaze, even though it actually has only one mouth, it may be impossible to create an appropriate gaze movement for the agent. Therefore, if it is known that the target has only one mouth, such as when the target is a human, the operation determination unit 58 of the information processing device 12 may determine the gaze control of the agent by targeting only one of the multiple mouths detected by the detection unit 56.

[0081] The operation determination unit 58 may determine the target mouth from among the multiple mouths that has a relatively high Confidence value. Alternatively, the operation determination unit 58 may determine the target mouth from among the multiple detected mouths that is located between two eyes.

[0082] Further modifications relating to the first to third embodiments will now be described. In each of the above embodiments, the target of triangular gaze control was an agent that behaves as a friend or companion of the user, but the target of triangular gaze control is not limited to agents. It is also useful to perform triangular gaze control on various robots or characters that communicate with the user and for which it is preferable to enhance the sense of realism (for example, characters that appear in VR (virtual reality) games).

[0083] Further modifications relating to the first to third embodiments will be described. The scope of application of the triangular gaze control shown in each of the above embodiments is not limited to games, but can also be applied to characters in advertisements (e.g., advertisements using digital signage), movies, dramas, etc. For example, the field of application of an agent that performs triangular gaze control may include at least one of the following: digital signage, robots (real robots installed in the real world), in-car navigation systems, in-home smart speakers (including display units), home agents, virtual humans, virtual characters (including humanoid and animal-type characters), VR (virtual reality), or AR (augmented reality) characters. When applied to advertisements, the information processing system 10 does not need to include the controller 16. Also, the controller 16 does not need to include a speaker.

[0084] Further modifications relating to the first to third embodiments will now be described. The information processing device 12 may start controlling the agent's triangular gaze upon detecting a user. For example, the information processing device 12 may start controlling the agent's triangular gaze regardless of the level of intimacy with the user, or without deriving the level of intimacy with the user.

[0085] Further modifications relating to the first to third embodiments will be described. The agent's gaze control may involve randomly moving between elements detected as the user's eyes or mouth. For example, if the detection unit 56 detects three eyes and one mouth, the operation determination unit 58 of the information processing device 12 may decide to move the gaze from the middle of the three eyes to the mouth.

[0086] Further modifications relating to the first to third embodiments will be described. If the user is assumed to have two eyes and one mouth, and the detection unit 56 detects only one eye from the captured image, the detection unit 56 may estimate the position of the user's other eye based on the position of the detected eye. If the position of the mouth can also be detected, the detection unit 56 may estimate the position of the user's other eye based on the position of the detected eye and the position of the mouth. The operation determination unit 58 may determine the gaze movement of the agent to follow the detected eye and mouth positions and the estimated eye positions.

[0087] Similarly, if the position of the user's mouth is not detected, the detection unit 56 may estimate the position of the user's mouth based on the position of one or more eyes that have been detected. The operation determination unit 58 may determine the agent's gaze movement to follow the position of one or more eyes that have been detected and the estimated position of the mouth.

[0088] Further modifications relating to the first to third embodiments will be described. The operation determination unit 58 of the information processing device 12 may randomly switch between the agent's line of sight moving clockwise (in other words, to the right) and moving counterclockwise (in other words, to the left). The robot control unit 62 (display control unit 52) ​​of the information processing device 12 may randomly switch between moving the robot 20 (agent)'s line of sight clockwise and moving counterclockwise, in accordance with the determination of the operation determination unit 58.

[0089] For example, the action determination unit 58 may use a state machine or the like to transition between three states—clockwise movement, counterclockwise movement, and hold (waiting without moving)—each time the user's part (i.e., eyes or mouth) towards which the agent's gaze is directed changes. The action determination unit 58 may also set the agent's dominant eye and set the dwell time on the dominant eye side to be longer than the dwell time on the non-dominant eye side. For example, if the agent's dominant eye is the right eye, the time the agent spends looking at the user's left eye, which is directly facing the agent's right eye, may be set to be longer than the time the agent spends looking at the user's right eye. Furthermore, random numbers may be used to determine the order of transitions and dwell times so that the user does not perceive any patterns or regularities in the agent's gaze transitions.

[0090] Further modifications relating to the first to third embodiments will be described. The detection unit 56 of the information processing device 12 may detect multiple users appearing in the captured image. If multiple users are detected, the operation determination unit 58 of the information processing device 12 may determine which user the agent should direct its gaze towards, according to its relationship with each user.

[0091] For example, the operation determination unit 58 may determine an agent's fan as a user that the agent directs its gaze towards. An agent's fan may be a user whose level of intimacy with the agent is above a predetermined threshold, or it may be the user with the highest level of intimacy with the agent among multiple users. Furthermore, an agent's fan may include at least one of the following: a user who reacted to the agent's behavior, a user who purchased content related to the agent, a user who made a payment, a user who spent a long time watching the agent, or a user who spent a long time interacting with the agent.

[0092] Furthermore, the operation determination unit 58 may measure the social distance between the agent and each user, and determine which users the agent will direct its gaze towards when the social distance between them has decreased to below a predetermined threshold. The social distance between the agent and the user may be measured in a way that reduces it based on actions such as giving a gift, shaking hands, or cheering.

[0093] Any combination of the embodiments and modifications described above is also useful as an embodiment of the present invention. The new embodiments resulting from these combinations possess the combined effects of the respective embodiments and modifications. Furthermore, it will be understood by those skilled in the art that the functions to be performed by each component described in the claims can be achieved by each component shown in the embodiments and modifications individually or in combination thereof.

[0094] The technical concepts described in the above embodiments and modifications can be expressed as follows: [Technology 1] An information processing device comprising a circuit configured to the following, wherein the circuit controls the movement of a robot, detects one or more eyes and one or more mouths of a user based on an image of the user, and operates a part of the robot corresponding to the eyes so that the point of view moves along the sides of a polygon connecting the one or more eyes and one or more mouths of the user when the robot communicates with or attempts to communicate with the user. [Technology 2] The information processing device according to Technology 1, wherein the circuit operates a part of the robot corresponding to the eyes so that the point of view moves clockwise or counterclockwise along the sides of the polygon. [Technology 3] The information processing device according to Technology 1, wherein the circuit further detects the posture of the user, and the circuit determines whether or not to communicate with the user based on the user's posture. [Technology 4] The information processing device according to Technology 1, wherein the circuit stores the degree of intimacy with the user, and when the robot communicates with or attempts to communicate with the user, and the degree of intimacy with the user is above a predetermined threshold, the circuit operates a part corresponding to the robot's eyes so that the point of gaze moves along the sides of the polygon. [Technology 5] An information processing device comprising a circuit configured to as follows, wherein the circuit detects one or more eyes and one or more mouths of the user based on an image of the user, displays a predetermined character on a display, and when the character communicates with or attempts to communicate with the user, the circuit operates a part corresponding to the eyes of the character so that the point of gaze moves along the sides of the polygon connecting one or more eyes and one or more mouths of the user.[Technical 6] The information processing apparatus according to Technical 5, wherein the circuit operates a part corresponding to the eyes of the character so that the line of sight moves clockwise or counterclockwise along the sides of the polygon. [Technical 7] The information processing apparatus according to Technical 5, wherein the circuit further detects the posture of the user, and the circuit decides whether or not to communicate with the user based on the posture of the user. [Technical 8] The information processing apparatus according to Technical 5, wherein the circuit stores the level of intimacy with the user, and the circuit operates a part corresponding to the eyes of the character so that the line of sight moves along the sides of the polygon when the character communicates with or attempts to communicate with the user and the level of intimacy with the user is above a predetermined threshold. [Technical 9] The information processing apparatus according to Technical 5, wherein the circuit displays the character on a plurality of displays, and the circuit sets the size of the character to be displayed on each display based on the distance between the user and each display. [Technical 10] An information processing method comprising: an information processing device detecting one or more eyes and one or more mouths of a user based on an image of the user; controlling the movement of a robot; and when the robot communicates with or attempts to communicate with the user, operating the parts of the robot corresponding to the eyes so that the line of sight moves along the sides of a polygon connecting one or more eyes and one or more mouths of the user. [Technical 11] An information processing method comprising: an information processing device detecting one or more eyes and one or more mouths of a user based on an image of the user; displaying a predetermined character on a display; and when the character communicates with or attempts to communicate with the user, operating the parts of the character corresponding to the eyes so that the line of sight moves along the sides of a polygon connecting one or more eyes and one or more mouths of the user.[Technology 12] A computer program for causing an information processing device to detect one or more eyes and one or more mouths of a user based on an image of the user, control the movement of a robot, and, when the robot communicates with or attempts to communicate with the user, to operate the parts of the robot corresponding to the eyes so that the point of view moves along the sides of a polygon connecting one or more eyes and one or more mouths of the user. [Technology 13] A computer-readable non-temporary recording medium storing a computer program for causing an information processing device to detect one or more eyes and one or more mouths of a user based on an image of the user, display a predetermined character on a display, and, when the character communicates with or attempts to communicate with the user, to operate the parts of the character corresponding to the eyes so that the point of view moves along the sides of a polygon connecting one or more eyes and one or more mouths of the user.

[0095] This invention can be applied to information processing devices, information processing systems, and the like.

[0096] 10 Information processing system, 12 Information processing device, 14 Display device, 18 Camera, 20 Robot, 32 Memory unit, 42 Intimacy memory unit, 52 Display control unit, 54 Image acquisition unit, 56 Detection unit, 58 Action determination unit, 60 Intimacy update unit, 62 Robot control unit.

Claims

1. An information processing device comprising: a robot control unit for controlling the movement of a robot; and a detection unit for detecting one or more eyes and one or more mouths of a user based on an image of the user, wherein the robot control unit operates a part corresponding to the eyes of the robot so that when the robot communicates with or attempts to communicate with the user, the point of view moves along the sides of a polygon connecting one or more eyes and one or more mouths of the user.

2. The information processing device according to claim 1, wherein the robot control unit operates a part corresponding to the robot's eye so that the point of view moves clockwise or counterclockwise along the side of the polygon.

3. The information processing apparatus according to claim 1, further comprising a determination unit, wherein the detection unit further detects the user's posture, and the determination unit determines whether or not to communicate with the user based on the user's posture.

4. The information processing apparatus according to claim 1, further comprising a storage unit for storing the degree of intimacy with the user, wherein the robot control unit operates a part corresponding to the robot's eye so that the point of gaze moves along the side of the polygon when the robot communicates with or attempts to communicate with the user and the degree of intimacy with the user is above a predetermined threshold.

5. An information processing device comprising: a display control unit that displays a predetermined character on a display; and a detection unit that detects one or more eyes and one or more mouths of a user based on an image of the user, wherein the display control unit operates the part of the character corresponding to the eyes so that the line of sight moves along the sides of a polygon connecting one or more eyes and one or more mouths of the user when the character communicates with or attempts to communicate with the user.

6. The information processing apparatus according to claim 5, wherein the display control unit operates a part corresponding to the character's eyes so that the line of sight moves clockwise or counterclockwise along the side of the polygon.

7. The information processing apparatus according to claim 5, further comprising a determination unit, wherein the detection unit further detects the user's posture, and the determination unit determines whether or not to communicate with the user based on the user's posture.

8. The information processing apparatus according to claim 5, further comprising a storage unit for storing the degree of intimacy with the user, wherein the display control unit operates a part corresponding to the eyes of the character so that the line of sight moves along the side of the polygon when the character communicates with or attempts to communicate with the user and the degree of intimacy with the user is above a predetermined threshold.

9. The information processing apparatus according to claim 5, wherein the display control unit causes the character to be displayed on a plurality of displays, and the display control unit sets the size of the character to be displayed on each display based on the distance between the user and each display.

10. An information processing method comprising: an information processing device detecting one or more eyes and one or more mouths of a user based on an image of the user; controlling the movement of a robot; and, when the robot communicates with or attempts to communicate with the user, operating the parts of the robot corresponding to the eyes so that the robot's line of sight moves along the sides of a polygon connecting the one or more eyes and one or more mouths of the user.

11. An information processing method comprising: an information processing device detecting one or more eyes and one or more mouths of a user based on an image of the user; displaying a predetermined character on a display; and, when the character communicates with or attempts to communicate with the user, operating the parts of the character corresponding to the eyes so that the line of sight moves along the sides of a polygon connecting the one or more eyes and one or more mouths of the user.

12. A computer program for an information processing device that, based on an image of the user, detects one or more eyes and one or more mouths of the user, controls the movement of a robot, and, when the robot communicates with or attempts to communicate with the user, causes the part of the robot corresponding to the eyes to move along the sides of a polygon connecting the one or more eyes and one or more mouths of the user.

13. A computer program for an information processing device that, based on an image of the user, detects one or more eyes and one or more mouths of the user, displays a predetermined character on a display, and, when the character communicates with or attempts to communicate with the user, operates the parts of the character corresponding to the eyes so that the line of sight moves along the sides of a polygon connecting the one or more eyes and one or more mouths of the user.

Images