Information processing device, information processing method, and computer program

Abstract

An information processing device 12 causes a display device 14 to display a character. The information processing device 12 detects the line-of-sight direction of a user. The information processing device 12 estimates a gazing position of the user on content displayed on the display device 14 on the basis of the line-of-sight direction of the user. When the character communicates with the user or attempts to communicate with the user, the information processing device 12 controls the character so that the character sees a position in the content based on the gazing position of the user.

Description

Information Processing Apparatus, Information Processing Method, and Computer Program

[0001] The present disclosure relates to data processing technology, and more particularly to an information processing apparatus, an information processing method, and a computer program.

[0002] In an information processing apparatus, a character that communicates with a user may be displayed.

[0003] In order to effectively improve the quality of the user's experience through interaction with the character, the inventor considered that it is important to enhance the sense of reality of the character. One object of the present disclosure is to provide a technique for enhancing the sense of reality of a character that communicates with a user.

[0004] To solve the above problems, an information processing apparatus according to an aspect of the present disclosure includes a display control unit that causes a character to be displayed on a display, a detection unit that detects a user's line-of-sight direction, and an estimation unit that estimates a user's fixation position in the content displayed on the display based on the user's line-of-sight direction. The display control unit controls the character so as to look at a position in the content based on the user's fixation position when the character communicates or attempts to communicate with the user.

[0005] Another aspect of the present disclosure is an information processing method. This method includes an information processing apparatus detecting a user's line-of-sight direction, estimating a user's fixation position in the content displayed on the display based on the user's line-of-sight direction, causing a character to be displayed on the display, and controlling the character so as to look at a position in the content based on the user's fixation position when the character communicates or attempts to communicate with the user.

[0006] Note that any combination of the above components, as well as those obtained by converting the expressions of the present disclosure among a system, a computer program, a recording medium storing the computer program, etc., are also effective as aspects of the present disclosure.

[0007] According to the present disclosure, the sense of reality of a character that communicates with a user can be enhanced.

[0008] This is a diagram showing the configuration of the information processing system of the first embodiment. This is a block diagram showing the functional blocks of the information processing device of the first embodiment. This is a flowchart showing the operation of the information processing device of the first embodiment. This is a diagram schematically showing an example of agent control. This is a diagram schematically showing an example of agent control. This is a diagram showing the configuration of the information processing system of the second embodiment. This is a diagram schematically showing an example of agent control.

[0009] The embodiments are outlined below. In the information processing system of each embodiment, a character that communicates continuously with the user is displayed. The user in each embodiment is a player of a game application. The character behaves as the user's friend or companion, enjoying the game together and providing hints for the game, and is hereinafter referred to as the "agent." The communication between the user and the agent can also be called the interaction between the user and the agent.

[0010] The inventors of this invention believe that enhancing the sense of presence of the agent is important in order to effectively improve the user experience (in an embodiment, the game experience) through communication with the agent. The sense of presence of the agent can be described as making the user feel as if the agent exists in the same world as the user, or as making the user feel as if they are sharing the same world as the agent.

[0011] To enhance the agent's sense of presence, the information processing system in each embodiment controls the agent to view its position within the content displayed on the screen based on the user's gaze position when the agent should communicate with the user. This enhances the agent's sense of presence and supports smooth communication between the user and the agent.

[0012] <First Embodiment> Figure 1 shows the configuration of the information processing system 10 according to the first embodiment. The information processing system 10 includes an information processing device 12, a display device 14, a controller 16, and a camera 18.

[0013] The functions of the components in each computer (e.g., information processing device 12) constituting the information processing system 10 may be realized in circuits or processing circuits, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (a Central Processing Unit), conventional circuits, and / or combinations thereof, which are configured or programmed to realize the functions described herein. A processor is considered to be a circuit or processing circuit that includes transistors and other circuits. A processor may also be a programmed processor that executes a program stored in memory.

[0014] In this specification, circuits, units, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0015] If the hardware is a processor that is considered to be a type of circuit, then the circuit, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0016] The information processing device 12 includes a computer capable of running system software and various application software. In the first embodiment, the information processing device 12 is a stationary game console, but as a modification, the information processing device 12 may be a portable game console, a PC, a tablet terminal, a smartphone, etc.

[0017] The controller 16 is an operating device that receives user inputs to the system software and application software executed by the information processing device 12. The controller 16 is connected to the information processing device 12 by wire or wireless connection and sequentially transmits operation information indicating the user inputs to the information processing device 12.

[0018] The controller 16 is equipped with a speaker and can output sound based on an audio signal transmitted from the information processing device 12. The controller 16 is also equipped with a vibrator and can vibrate the vibrator in a manner based on a haptic signal transmitted from the information processing device 12.

[0019] The display device 14 has a display that outputs images and a speaker that outputs sound. The display device 14 may be a television or a computer display. The display device 14 is connected to the information processing device 12 by wire or wireless. When the information processing device 12 receives operation information provided by the controller 16, it reflects that operation information in the processing of system software or application software, displays an image related to the processing result on the display device 14, and outputs sound related to the processing result from the speaker of the display device 14.

[0020] The display device 14 may be a head-mounted display worn on the user's head. In this case, the information processing device 12 and the display device 14 (head-mounted display) may be configured to display a panoramic image on the head-mounted display, and when the user wearing the head-mounted display rotates their head, a panoramic image corresponding to the direction of their gaze is displayed. The display device 14 (head-mounted display) may be configured to block out external light and may display a virtual reality image generated by the information processing device 12. The display device 14 (head-mounted display) may also be optically transparent and may display augmented reality images generated by the information processing device 12.

[0021] Camera 18 is a device that captures images of the user playing the game. Camera 18 is connected to the information processing device 12 by wire or wireless connection and transmits data of images of the user (hereinafter also referred to as "captured images") to the information processing device 12. The captured images in the first embodiment include content necessary for measuring and tracking the user's gaze direction (in other words, viewpoint) (i.e., eye tracking). For example, the subject of the captured images in the first embodiment may include the user's pupil. If the display device 14 is a head-mounted display, camera 18 may be an eye-tracking camera mounted on the head-mounted display that captures the user's pupil.

[0022] Figure 2 is a block diagram showing the functional blocks of the information processing device 12 according to the first embodiment. Each block shown in the block diagrams of this specification can be realized in hardware terms by a computer processor (CPU, etc.), memory and other elements, electronic circuits, and mechanical devices, and in software terms by a computer program, etc., but here we are depicting functional blocks that are realized through the cooperation of these. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways by combinations of hardware and software.

[0023] The information processing device 12 comprises a processing unit 30, a storage unit 32, and a communication unit 34. The processing unit 30 performs various data processing. The storage unit 32 stores data that is referenced or updated by the processing unit 30. The communication unit 34 communicates with external devices via wired or wireless communication according to a predetermined communication protocol. The processing unit 30 sends and receives data to and from the display device 14, controller 16, and camera 18 via the communication unit 34.

[0024] The storage unit 32 includes a program storage unit 40. The program storage unit 40 stores system software and application software. The application software in the first embodiment includes a game application (e.g., a video game).

[0025] The processing unit 30 includes a game execution unit 50, a display control unit 52, an audio output unit 54, a detection unit 56, an estimation unit 58, and an action determination unit 60. The functionality of at least one of these multiple functional blocks may be implemented in a computer program (for example, system software and / or application software). This computer program may be installed in the storage of the information processing device 12. The processor of the information processing device 12 may perform the functions of these multiple functional blocks by reading this computer program into main memory and executing it.

[0026] The game execution unit 50 executes system software and application software stored in the program storage unit 40. For example, the game execution unit 50 executes a game application in response to user input to the controller 16. Based on the execution results of the game application, it generates game images and sound.

[0027] The display control unit 52 controls the content displayed on the display of the display device 14. For example, the display control unit 52 causes the game image generated by the game execution unit 50 to be displayed on the display of the display device 14. The display control unit 52 also causes a predetermined agent image to be displayed on the display of the display device 14.

[0028] The audio output unit 54 controls the audio output from the speaker of the display device 14. For example, the audio output unit 54 causes the audio of the game generated by the game execution unit 50 to be output from the speaker of the display device 14. The audio output unit 54 also causes the audio that the agent should emit, as determined by the operation determination unit 60, to be output from the speaker of the display device 14.

[0029] The detection unit 56 detects the user's gaze direction based on the image captured by the camera 18 and known eye-tracking technology. The estimation unit 58 estimates the user's gaze position in the displayed content based on the user's gaze direction detected by the detection unit 56 and the content displayed on the display of the display device 14 (hereinafter also referred to as "displayed content").

[0030] The operation determination unit 60 determines the operation mode of the agent. The operation determination unit 60 determines whether the predetermined conditions for the agent to communicate with the user (hereinafter also referred to as "communication conditions") have been met.

[0031] The communication conditions may be met if (1) the game being played by the user reaches a predetermined state, (2) a predetermined time is reached in real time, or (3) the user's state detected by a predetermined sensor reaches a predetermined state. The predetermined state in (1) may be that the game's objective has been achieved, that the game's objective has not been achieved, or that the game duration has exceeded a predetermined time. The predetermined state in (3) may be that the user has gazed at a specific position in the displayed game image for a predetermined time or longer, or that the user's posture is facing the agent displayed on the display device 14.

[0032] If the communication conditions are met, the operation determination unit 60 determines the operation mode of the agent for communicating with the user. The operation mode of the agent for communicating with the user may include, for example, speaking to the user, smiling at the user, waving to the user, etc. If the communication conditions are met, the operation determination unit 60 further determines that the operation mode of the agent is to look at a position within the displayed content based on the user's gaze position estimated by the estimation unit 58.

[0033] The display control unit 52 operates the agent to communicate with the user in a manner determined by the operation determination unit 60. For example, the display control unit 52 may operate the agent to speak to the user, or it may operate the agent to perform a predetermined gesture to the user.

[0034] When the communication conditions are met, this includes when the agent communicates with the user (for example, while communicating) and when the agent attempts to communicate with the user (for example, immediately before communicating). When the communication conditions are met, the display control unit 52 controls the agent to view the position within the display content based on the user's gaze position, according to the decision made by the operation decision unit 60. In other words, the display control unit 52 displays the agent which is operating to view the position within the display content based on the user's gaze position.

[0035] Figure 3 is a flowchart illustrating the operation of the information processing device 12 of the first embodiment. The operation of the information processing system 10 of the first embodiment will be described below with reference to Figure 3.

[0036] The display control unit 52 of the information processing device 12 displays an image of an agent assisting the user on the display device 14 (S10). The game execution unit 50 of the information processing device 12 executes a game application in response to user input to the controller 16 and generates a game image showing the game execution result. The display control unit 52 displays the game image on the display device 14 (S11).

[0037] Camera 18 transmits captured images for eye tracking to information processing device 12. The detection unit 56 of the information processing device 12 detects the user's gaze direction based on the captured images from camera 18 (S12). The estimation unit 58 of the information processing device 12 estimates the user's gaze position within the displayed content based on the user's gaze direction detected in S12 (S13).

[0038] In S13, the estimation unit 58 may estimate the coordinate values ​​of the user's gaze position within the screen of the display device 14. The estimation unit 58 may also store the placement positions (coordinate values, coordinate ranges, etc.) of multiple objects included in the game image (for example, various characters and objects in the game world), and may estimate the object placed at the user's gaze position as the user's gaze object.

[0039] The detection unit 56 analyzes the image captured by the camera 18 that shows the user and detects the user's posture relative to the display device 14 on which the agent is displayed (in other words, the user's posture relative to the agent). For example, the detection unit 56 detects if the user's posture is facing directly towards the display device 14, or in other words, if the user's posture is facing directly towards the agent displayed on the display device 14.

[0040] The operation determination unit 60 of the information processing device 12 determines whether the communication conditions have been met based on the state and progress of the game being played by the user, and the user's posture. If the communication conditions are met (Y in S14), the operation determination unit 60 determines the direction of the agent's gaze and also determines the agent's posture and actions that are consistent with that gaze direction. The operation determination unit 60 also determines the voice that the agent will emit (for example, what to say to the user).

[0041] The display control unit 52 of the information processing device 12 controls the gaze direction of the agent displayed on the display device 14 so that it is the gaze direction determined by the action determination unit 60, and also controls the agent to perform the posture and actions determined by the action determination unit 60 (S15). The audio output unit 54 of the information processing device 12 outputs the voice of the agent determined by the action determination unit 60 from the speaker of the display device 14 (S16). If the communication conditions are not met (N in S14), processes S15 and S16 are skipped.

[0042] Figure 4 schematically shows an example of agent control. The display device 14 in Figure 4 displays a game image containing multiple game-related objects A to D, and agent 100. Objects A to D are various characters and objects in the game world. In Figure 4, the user's line of sight is shown with a solid line, and the agent's line of sight is shown with a dashed line. The following explanation will describe an example of agent control related to S15 and S16 in Figure 3, with reference to Figure 4.

[0043] Example 1 of Agent Control: The operation determination unit 60 of the information processing apparatus 12 determines, as the operation mode of the agent 100, to alternately look at the position in the display content based on the user's gaze position and the user's face. The display control unit 52 of the information processing apparatus 12 controls the display of the agent 100 to alternately look at the position in the display content based on the user's gaze position and the user's face.

[0044] For example, the operation determination unit 60 may determine the line-of-sight direction, posture, or operation of the agent 100 so that the agent 100 looks at the same position as the user's gaze position or the same object as the user's gazed object (object A in FIG. 4) as the first operation mode of the agent 100. Further, the detection unit 56 may detect the position of the user's face in the world coordinate system based on the captured image of the camera 18 in which the user is captured. The operation determination unit 60 may determine the line-of-sight direction, posture, or operation of the agent 100 so that the line of sight of the agent 100 faces the position of the user's face as the second operation mode of the agent 100. The display control unit 52 may switch the display of the agent 100 in the first operation mode and the display of the agent 100 in the second operation mode each time a predetermined time (for example, 3 seconds) elapses. Note that the display control unit 52 may control the posture and operation of the agent 100 according to the line-of-sight direction of the agent 100 using a known technique.

[0045] Example 2 of Agent Control: The operation determination unit 60 of the information processing apparatus 12 determines, as the voice emitted by the agent 100, a voice including information about an object arranged near the user's gaze position. The voice output unit 54 of the information processing apparatus 12 causes a voice including information about an object arranged near the user's gaze position to be output from the speaker of the display device 14 as the voice emitted by the agent 100.

[0046] For example, the operation determination unit 60 may express the object at the user's gaze position (object A in FIG. 4) as "this object", and determine the voice of the agent 100 that indicates the information about the object at the user's gaze position as the information of "this object". Further, the operation determination unit 60 may express the object near the user's gaze position (object D in FIG. 4) as "the adjacent object" or "the object right nearby", and determine the voice of the agent 100 that indicates the information about the object near the user's gaze position as the information of "the adjacent object" or "the object right nearby".

[0047] Example 3 of agent control: The operation determination unit 60 of the information processing apparatus 12 determines, as the voice emitted by the agent 100, a voice including information for designating a position based on the user's gaze position (hereinafter also referred to as "position designation information"). The voice output unit 54 of the information processing apparatus 12 causes the speaker of the display apparatus 14 to output, as the voice emitted by the agent 100, a voice including position designation information based on the user's gaze position.

[0048] For example, the operation determination unit 60 may express the position above the user's gaze position as "above", and determine the voice of the agent 100 that indicates the information about the position above the user's gaze position as the information of "above". Further, the operation determination unit 60 may express the position to the right of the user's gaze position as "right", and determine the voice of the agent 100 that indicates the information about the position to the right of the user's gaze position as the information of "right".

[0049] The positional information included in the voice output by agent 100, based on the user's gaze position, may include at least one of the following: "further back," "in front," "front," "back," "side," "here," "below here," or "below there." The positional information may also include information other than direction. For example, the positional information may include a background color specification; if the right side of the background is red, it may include the specification "the red side." The positional information may also include an object specification; if there is a rocky area on the left side of the screen, it may include the specification "please focus on the rocky side." In this case, the operation determination unit 60 may determine the positional information to be spoken by agent 100 based on the results of object recognition and color scheme recognition displayed on the screen.

[0050] The position information included in the voice output by agent 100, which is based on the user's gaze position, may include not only the relative position of the object (e.g., "above") but also instructions to limit the target object when the target object is not clearly identified. For example, if many red enemies are displayed on the screen, and both red and blue enemies are positioned near the gaze point, the position information may include the instruction "the red one" in the sense of the red one at the gaze point. Furthermore, since the word "edge" is more likely to be used to mean an absolute position rather than a relative position from the gaze point, "edge" may be used to specify objects positioned at the left or right edge of the screen.

[0051] Agent control example 4: Here, we assume a situation where the user should be instructed to look at a specific location within the display content that is different from the user's gaze position (hereinafter also referred to as the "instruction target location"). In this case, the operation determination unit 60 of the information processing device 12 decides that the agent 100 should look at the instruction target location, and also decides that the agent 100 should emit audio that includes information indicating the relationship between the user's gaze position and the instruction target location. The display control unit 52 of the information processing device 12 controls the display of the agent 100 to look at the instruction target location. The audio output unit 54 of the information processing device 12 causes the agent 100 to output audio that includes information indicating the relationship between the user's gaze position and the instruction target location from the speaker of the display device 14.

[0052] Figure 5 also schematically illustrates an example of agent control. In Figure 4, the user's gaze is directed to object A, while the target location is object C. In this case, the operation determination unit 60 decides that the agent 100 should look at object C, and determines that the agent 100 should emit a voice message that includes information indicating the relationship between object A and object C (e.g., "below it") (e.g., "look below it"). The display control unit 52 controls the display of agent 100 to look at object C. The voice output unit 54 causes the voice message "look below it" emitted by agent 100 to be output from the speaker of the display device 14.

[0053] According to the information processing device 12 of the first embodiment, the user can be made to feel as if they and the agent are sharing the same space, making it easier for the user to feel a sense of familiarity with the agent and enhancing the sense of the agent's presence. Furthermore, the information processing device 12 can clearly present to the user viewing content displayed on the screen a position that encourages their attention, for example, it can clearly present a position that encourages attention that is different from the position the user is currently looking at.

[0054] <Second Embodiment> The second embodiment of this disclosure will be described focusing on the differences from the first embodiment. Components of the second embodiment that are the same as or equivalent to components of the first embodiment will be denoted by the same reference numerals as those of the first embodiment, and explanations that overlap with the description of the first embodiment will be omitted as appropriate.

[0055] Figure 6 shows the configuration of the information processing system 10 of the second embodiment. The information processing system 10 of the second embodiment differs from the information processing system 10 of the first embodiment in that (1) it includes a plurality of display devices 14 (specifically, display devices 14a and display devices 14b) connected to the information processing device 12, and (2) it selectively displays an agent on one of the plurality of display devices 14.

[0056] Display devices 14a and 14b may both be computer displays. Alternatively, one of the display devices 14a and 14b may be a television and the other a computer display. Alternatively, one of the display devices 14a and 14b may be a television and the other a tablet terminal. The information processing system 10 may be equipped with three or more display devices 14.

[0057] The functional blocks of the information processing device 12 in the second embodiment are the same as those of the information processing device 12 in the first embodiment shown in Figure 2. The display control unit 52 of the information processing device 12 causes the agent image to be displayed on a plurality of display devices 14. In the second embodiment, the display control unit 52 causes the agent to be displayed on either the display device 14a or the display device 14b depending on the game situation or user operation. The display control unit 52 switches between displaying the agent on the display device 14a and displaying the agent on the display device 14b depending on the game situation or user operation.

[0058] For example, the display device 14a may be a television. The display device 14b may be a tablet device or a spatial display. A spatial display is a display capable of displaying three-dimensional images and is also called a three-dimensional display. The display control unit 52 may display images of the game being played on the information processing device 12 on the display device 14a. The display control unit 52 may display an agent on the display device 14a when the game images are not being displayed on the display device 14a. On the other hand, during the period when the game images are being displayed on the display device 14a, the display control unit 52 may terminate the display of the agent on the display device 14a and display the agent on the display device 14b.

[0059] The display control unit 52 applies the technology described in the first embodiment to control the gaze of the agent displayed on the display device 14a and the agent displayed on the display device 14b. The audio output unit 54 applies the technology described in the first embodiment to control the audio output related to the agent displayed on the display device 14a and the agent displayed on the display device 14b.

[0060] Figure 7 schematically shows an example of agent control. In Figure 7, a game image including multiple game-related objects A to D is displayed on the display device 14a, and agent 100 is displayed on the display device 14b. In Figure 7, the user's line of sight is shown with a solid line, and the agent's line of sight is shown with a dashed line. The display control unit 52 controls the line of sight of agent 100 displayed on the display device 14b. The audio output unit 54 controls the audio output related to agent 100 displayed on the display device 14b.

[0061] Each of the display devices 14a and 14b is equipped with a camera 18. The detection unit 56 of the information processing device 12 detects the distance between the user and each display device 14 based on the image captured by the camera 18 of each display device 14. The camera 18 of each display device 14 may include a stereo camera. The detection unit 56 may derive the direction of the user and the distance to the user based on the parallax shown in the image captured by the stereo camera. In addition, the camera 18 of each display device 14 may include a Time of Flight (ToF) camera. The detection unit 56 may derive the distance between the user and each display device 14 based on the output information of the ToF camera of each display device 14.

[0062] The display control unit 52 of the information processing device 12 sets the size of the agent to be displayed on each display device 14 based on the distance between the user and each display device 14. In the second embodiment, the display control unit 52 reduces the size of the agent to be displayed on the display device 14 as the distance between the user and the display device 14 increases.

[0063] For example, the display control unit 52 may calculate the physical size of the agent displayed on the screen of the display device 14a (referred to as the "first size") based on the screen resolution (ppi (Pixels per inch)) of the display device 14a and the display size of the agent (number of pixels, etc.). The display control unit 52 may also calculate the physical size of the agent to be displayed on the screen of the display device 14b (referred to as the "second size") based on the ratio of the distance between the user and the display device 14a to the distance between the user and the display device 14b. That is, the display control unit 52 may decide how much to scale the first size to obtain the second size. The display control unit 52 may also calculate the number of pixels required to display the second-size agent based on the screen resolution of the display device 14b.

[0064] The information processing device 12 of the second embodiment also provides the same effects as the information processing device 12 of the first embodiment. The information processing device 12 of the second embodiment further provides the effect of maintaining consistency in the size of the characters displayed on each display as seen by the user when displaying agents on multiple displays, thereby suppressing the user from feeling any discomfort with the size of the characters.

[0065] In the first and second embodiments described above, the character performing gaze control and voice control was an agent acting as a friend or companion to the user. However, the target of gaze control and voice control is not limited to agents. It is also useful to apply the gaze control and voice control of each embodiment to various characters that communicate with the user and for which it is desirable to enhance the sense of realism (for example, characters appearing in VR (virtual reality) games).

[0066] The present disclosure has been described above based on the first and second embodiments. Each embodiment is illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of each component and each processing process, and that such modifications are also within the scope of the present disclosure.

[0067] Modifications relating to the first and second embodiments will now be described. The scope of application of the agent control shown in each of the above embodiments is not limited to games, but can also be applied to characters in advertisements (e.g., advertisements using digital signage), movies, dramas, etc. For example, the areas in which the agent can be provided may include at least one of the following: digital signage, robots (real robots installed in the real world), in-car navigation systems, in-home smart speakers (including display units), home agents, virtual humans, virtual characters (including humanoid and animal-type characters), VR (virtual reality), or AR (augmented reality) characters. When applied to advertisements, the information processing system 10 does not need to include the controller 16. Also, the controller 16 does not need to include a speaker.

[0068] Another modification relating to the first and second embodiments will be described. The operation determination unit 60 of the information processing device 12 may acquire the user's voice (user's utterance) input to a predetermined microphone. The microphone may be provided in at least one of the information processing device 12, the display device 14 (head-mounted display, etc.), and the controller 16. The operation determination unit 60 may determine the agent's operation mode according to the user's instructions indicated by the user's voice (for example, "down here," "over there," etc.). Furthermore, when the operation determination unit 60 instructs the agent to focus on a location indicated by the user's voice, it may determine the agent's operation mode so that not only the agent's eyes but its entire body is directed to that location.

[0069] If the user's instructions are unclear or not clear, the action determination unit 60 may determine the agent's behavior to ask the user questions about the instructions. For example, if multiple enemy characters are displayed and it is not possible to determine which one to fight, the action determination unit 60 may decide to output voice messages such as "Right? Left?", "The bigger one?", or "The enemy in the shadow?" as questions to the user. The action determination unit 60 may also input the image displayed on the display device 14 into a predetermined VQA (Visual Question Answering) to recognize the type and position of each object in the image, and then decide what questions to ask the user based on the recognition results.

[0070] Further modifications relating to the first and second embodiments will be described. When the agent is facing the user, the agent's use of "left" can mean different directions depending on whether it refers to the agent's left or the user's left. Therefore, the information processing device 12 may position the agent with its back to the user and facing the same direction as the user, so that the agent's line of sight and the user's line of sight coincide. Alternatively, the information processing device 12 may position the agent facing the same direction as the user, or it may position the agent behind, above, or below the user.

[0071] The detection unit 56 of the information processing device 12 may detect the agent's gaze position based on the agent's posture, eye position, etc. The display control unit 52 of the information processing device 12 may display the agent's gaze position on the display of the display device 14 in a predetermined manner, for example, by placing a predetermined object or mark at the agent's gaze position. This indicates that the reference point is shared between agent users when giving instructions such as "up," providing the user with a sense of security. The agent's gaze point may also be indicated to the user using sound or vibration.

[0072] Any combination of the above-described embodiments and modifications is also useful as an embodiment of the present invention. The new embodiments resulting from these combinations possess the combined effects of each of the embodiments and modifications. Furthermore, it will be understood by those skilled in the art that the functions to be performed by each component described in the claims can be achieved by each component shown in the embodiments and modifications individually or in combination thereof.

[0073] The technical concepts described in the above embodiments and modifications can be expressed as the following technologies: [Technology 1] An information processing device comprising a circuit configured to the following, wherein the circuit displays a character on a display, detects the direction of the user's gaze, estimates the user's gaze position in the content displayed on the display based on the direction of the user's gaze, and controls the character to look at a position in the content based on the user's gaze position when the character communicates with or attempts to communicate with the user. [Technology 2] The information processing device according to Technology 1, wherein the circuit controls the character to alternately look at a position in the content based on the user's gaze position and the user's face. [Technology 3] The information processing device according to Technology 1, wherein the circuit outputs sound containing information about an object placed near the user's gaze position as sound emitted by the character. [Technology 4] The information processing device according to Technology 1, wherein the circuit outputs sound containing information specifying a position based on the user's gaze position as sound emitted by the character. [Technical 5] When it is necessary to instruct the user to look at a specific location within the content that is different from the user's gaze position, the circuit controls the character to look at the specific location, and the circuit outputs sound as sound emitted by the character that includes information indicating the relationship between the user's gaze position and the specific location, the information processing apparatus according to Technical 1. [Technical 6] The circuit causes the character to be displayed on a plurality of displays, and the circuit sets the size of the character to be displayed on each display based on the distance between the user and each display, the information processing apparatus according to Technical 1.[Technical 7] An information processing method comprising: an information processing device detecting the direction of a user's gaze; estimating the user's gaze position in content displayed on a display based on the direction of the user's gaze; displaying a character on the display; and controlling the character to view a position within the content based on the user's gaze position when the character communicates with or attempts to communicate with the user. [Technical 8] A computer-readable non-temporary recording medium storing a computer program for causing an information processing device to perform the following: detecting the direction of a user's gaze; estimating the user's gaze position in content displayed on a display based on the direction of the user's gaze; displaying a character on the display; and controlling the character to view a position within the content based on the user's gaze position when the character communicates with or attempts to communicate with the user.

[0074] This invention can be applied to information processing devices, information processing systems, and the like.

[0075] 10 Information processing system, 12 Information processing device, 14 Display device, 52 Display control unit, 54 Audio output unit, 56 Detection unit, 58 Estimation unit, 60 Operation determination unit.

Claims

1. An information processing device comprising: a display control unit for displaying a character on a display; a detection unit for detecting the direction of the user's gaze; and an estimation unit for estimating the position of the user's gaze in the content displayed on the display based on the direction of the user's gaze, wherein the display control unit controls the character to look at the position in the content based on the position of the user's gaze when the character communicates with or attempts to communicate with the user.

2. The information processing apparatus according to claim 1, wherein the display control unit controls the character to alternately view the position within the content based on the user's gaze position and the user's face.

3. The information processing apparatus according to claim 1, further comprising an audio output control unit that outputs audio containing information about an object located near the user's gaze position as audio emitted by the character.

4. The information processing apparatus according to claim 1, further comprising an audio output control unit that outputs audio, as audio emitted by the character, which includes information specifying a position based on the user's gaze position.

5. The information processing apparatus according to claim 1, further comprising an audio output control unit, wherein when it is necessary to instruct the user to look at a specific location within the content that is different from the user's gaze position, the display control unit controls the character to look at the specific location, and the audio output control unit outputs audio as audio emitted by the character, which includes information indicating the relationship between the user's gaze position and the specific location.

6. The information processing apparatus according to claim 1, wherein the display control unit causes the character to be displayed on a plurality of displays, and the display control unit sets the size of the character to be displayed on each display based on the distance between the user and each display.

7. An information processing method comprising: an information processing device detecting the direction of a user's gaze; estimating the position of the user's gaze in content displayed on a display based on the direction of the user's gaze; displaying a character on the display; and, when the character communicates with or attempts to communicate with the user, controlling the character to look at the position in the content based on the position of the user's gaze.

8. A computer program that causes an information processing device to perform the following actions: detect the direction of the user's gaze; estimate the position of the user's gaze on content displayed on a display based on the direction of the user's gaze; display a character on the display; and, when the character communicates with or attempts to communicate with the user, control the character to look at a position within the content based on the position of the user's gaze.

Images