Display control device, display control method, and display control program

Abstract

This agent system acquires a video of a user interacting with an agent. Next, the agent system calculates, from the video of the user, the orientation of the user when the user views the agent. The agent system estimates the display direction of the agent viewed from the position of the user on the basis of the calculated orientation of the user. The agent system then calculates, on the basis of the estimated display direction of the agent, the positional relationship between the imaging device that has captured the video of the user, the position of the user, and the display position of the agent. The agent system uses the positional relationship between the imaging device, the position of the user, and the display position of the agent to perform display control of the agent system such that the agent will face in the direction of the user.

Description

Display control device, display control method, and display control program

[0001] The present invention relates to a display control device, a display control method, and a display control program for controlling the display of an agent.

[0002] There are services that use humanoid computer graphics (CG) or robots with physical representations (hereinafter referred to as "agents") to provide functions such as assistants, presenters, and counselors. In such services, a method has been proposed to estimate the user's gaze and standing position and adjust the orientation of the agent's body and face displayed on the screen (see Non-Patent Document 1).

[0003] Naoto Yoshida et al., Verification of gaze communication in real space using a drawing agent that responds to the user's viewpoint position, IEICE Transactions on Electronics, Information and Communication Engineers, Vol.J99-D, No.9, pp.915-925, 2016.

[0004] However, changes in the position of the camera capturing the user, changes in the position of the screen displaying the agent, additions of screens, changes in the user's position, etc., can alter the relative positions of the agent and the user. In such cases, the system cannot easily grasp the relative position of the user from the agent's perspective. As a result, the system cannot adjust the agent's body or face to face the user, making appropriate interaction between the agent and the user difficult.

[0005] For example, if you want to adjust the camera that captures the user and the screen that displays the agent independently, or if the user receives the service at any position where they can view the screen (see reference numeral 101 in Figure 1), or if the camera position is easily changed during system operation (see reference numeral 102 in Figure 1), or if the screen is added or expanded (see reference numeral 103 in Figure 1), the system cannot easily determine the positional relationship between the agent and the user. As a result, the system has the problem that it cannot adjust the orientation of the agent's body and face on the screen, taking the user's position into consideration.

[0006] This problem will be explained in detail using Figure 2. For example, if the agent's position changes from the state shown by reference numeral 201 to the state shown by reference numeral 202 in Figure 2, a discrepancy will occur between the user's position assumed by the agent and the actual user's position. As a result, the system will not be able to make the agent look in the user's direction and speak.

[0007] Therefore, the present invention aims to solve the aforementioned problems and enable the agent to adjust the orientation of its body and face so that it can properly interact with the user even in environments where the camera that captures the user, the screen (display) that displays the agent, and the user's position are changed.

[0008] To solve the aforementioned problems, the present invention is characterized by comprising: an input unit that acquires video of a user interacting with an agent on a screen; a direction estimation unit that estimates the orientation of the user when the user is looking at the agent from the video of the user; a display direction estimation unit that estimates the display direction of the agent as seen from the user's position based on the estimated orientation of the user; a position relationship calculation unit that calculates the position relationship between the shooting device that captured the video, the user's position, and the display position of the agent based on the estimated display direction of the agent; and an agent control unit that controls the display of the agent so that the agent faces the direction of the user, using the position relationship between the shooting device, the user's position, and the display position of the agent.

[0009] According to the present invention, even in environments where the camera that captures the user, the display that shows the agent, and the user's position are changed, the orientation of the agent's body and face can be adjusted so that the agent can interact with the user appropriately.

[0010] Figure 1 is a diagram illustrating an example of the positional relationship between the camera, screen, and user. Figure 2 is a diagram illustrating an example of changing the positional relationship between the camera, screen, and user. Figure 3 is a diagram illustrating an example of a method for estimating the display direction of the agent. Figure 4 is a diagram illustrating an example of a change in the positional relationship between the position of the measuring device, the display position of the agent, and the position of the user. Figure 5 is a diagram illustrating an example of the configuration of the agent system. Figure 6 is a diagram illustrating an example of the configuration of the agent system. Figure 7 is a diagram illustrating an example of the information held in the storage unit of Figure 5. Figure 8 is a flowchart illustrating an example of a processing procedure executed by the agent system. Figure 9 is a diagram illustrating an example of the processing procedure of S4 in Figure 8. Figure 10 is a diagram illustrating an example of calculation example 1 for estimating the display position of the agent. Figure 11 is a diagram illustrating an example of calculation example 2 for estimating the display position of the agent. Figure 12 is a diagram illustrating an example of calculation for estimating the vertical display position of the agent. Figure 13 is a diagram illustrating a method 1 for calculating the display position of the agent when there are multiple users. Figure 14 is a diagram illustrating a method 2 for calculating the display position of the agent when there are multiple users. Figure 15 is a diagram illustrating an example of calculating the position of the user. Figure 16 shows an example of a computer that executes a display control program.

[0011] The following describes embodiments for carrying out the present invention with reference to the drawings. The present invention is not limited to these embodiments.

[0012] [Overview] This section describes the overview of the display control device (agent system) of this embodiment. The agent system controls the display of an agent on the display screen based on the user's video captured by a camera, for example, and realizes interactions such as dialogue between the user and the agent.

[0013] Here, given the user's position or the agent's display position in a coordinate system with the camera as the origin, the agent system estimates the direction of the agent's display from the user's perspective based on the orientation of the user's face estimated from the video of the user's face while interacting with the agent. Then, the agent system calculates the agent's display position (or the user's position) from the estimated direction of the agent's display.

[0014] For example, given the user's location, the agent system calculates the agent's display position from the estimated agent's display direction. Similarly, given the agent's display position, the agent system calculates the user's location from the estimated agent's display direction.

[0015] The agent system then uses the agent's display position and the user's position to estimate the user's direction relative to the agent's display position.

[0016] Thus, the agent system can estimate the user's direction from the agent's display position, even in environments where the camera, display, and user's positions are changed. As a result, the agent system can have the agent look in the user's direction when speaking or wave to the user. In other words, the agent system can adjust the orientation of the agent's body and face so that the agent can interact with the user appropriately, even in environments where the camera, display, and user's positions are changed.

[0017] In the following explanation, the agent system will be described using an example where it calculates the agent's display position in a coordinate system with the camera as the origin, based on the user's position and the orientation of the user's face. However, the origin is not limited to the camera's position; it may also be a coordinate system obtained by correcting the coordinates obtained from the camera so that the origin is the agent's position or the user's position. Furthermore, the orientation of the user's face can be replaced with information about the direction the user is looking, such as the user's gaze.

[0018] [Overview of Estimating the Agent's Display Direction] Next, with reference to Figure 3, we will explain the overview of the method for estimating the agent's display direction from the user's position.

[0019] Method 1: For example, the agent system gives the user an instruction to look at the agent (e.g., "Look at me"), and estimates the direction the user turns their face after this instruction as the direction the agent is displayed from the user's perspective.

[0020] Method 2: For example, the agent system estimates the direction the user is most likely to be facing during interaction with the agent as the display direction of the agent from the user's position.

[0021] In Method 2, for example, as shown in Figure 3, the agent system has the agent and the user interact multiple times and obtains the direction the user's face is facing during those interactions. The agent system then calculates statistical values ​​of the user's face orientation during a predetermined time (or predetermined number of interactions) and estimates these values ​​as the display direction of the agent as seen from the user's position. The above-mentioned statistical values ​​of face orientation may be, for example, the average value of the face orientation or the direction the face was facing for the longest time.

[0022] [Overview of the Method for Estimating the Agent's Display Position] Next, an overview of the method for estimating the agent's display position will be explained using Figure 4. In the following, it is assumed that the user is in a position where the agent is easily visible, and that there is no change in the depth distance between the user and the agent. Based on this assumption, the agent system will determine that there has been a change in the positional relationship between the agent and the measuring device if the distance between the measuring device (e.g., camera) and the user changes, or if the orientation of the user's face changes from the initial orientation.

[0023] Furthermore, an example of estimating the display position of an agent on a two-dimensional plane in the horizontal direction and the depth direction is shown here. Regarding the display position of the agent in the height direction, it is assumed that the agent is positioned on a straight line extending vertically upward and downward from the display position on the plane, and is determined using the vertical angle of the user's neck (details will be described later).

[0024] Note that, regarding the initial positional relationship among the agent, the measurement device and the user, for example, as indicated by reference numeral 401 in FIG. 4, it is assumed that the position of the measurement device on the two-dimensional plane matches the display position of the agent, and the horizontal position matches among the user, the agent, and the measurement device.

[0025] When the positional relationship among the measurement device, the user, and the agent changes from the initial positional relationship indicated by reference numeral 401 to the positional relationships indicated by reference numerals 402 to 405, the agent system estimates the display position of the agent in a coordinate system with the measurement device as the origin by using the distance between the measurement device and the user (l´ or l) and the user's direction (angle θ) in each positional relationship.

[0026] For example, as indicated by reference numerals 402 and 403, consider a case where the distance in the depth direction between the measurement device and the user does not change, but the horizontal distance between the measurement device and the display position of the agent changes. In this case, if it is assumed that no change occurs in the distance in the depth direction between the user and the agent, the agent system can estimate the display position of the agent in the coordinate system with the measurement device as the origin by using the distance l (or distance l´) and the angle θ shown in FIG. 4.

[0027] Furthermore, for example, as indicated by reference numerals 404 and 405, consider a case where the distance in the depth direction between the measurement device and the user changes, and the horizontal distance between the measurement device and the display position of the agent also changes. Also in this case, if it is assumed that no change occurs in the distance in the depth direction between the user and the agent, the agent system can estimate the display position of the agent in the coordinate system with the measurement device as the origin by using the distance l´ and the angle θ shown in FIG. 4.

[0028] As described above, even when the position of the measurement device or the display position of the agent is changed, the agent system can estimate the display position of the agent in a coordinate system with the measurement device as the origin. As a result, the agent system can grasp the positional relationship between the user's position and the agent's display position in said coordinate system.

[0029] Accordingly, even when, for example, the position of the measurement device or the display for displaying the agent is changed, the agent system can adjust the orientation of the agent's body and face so that the agent can appropriately interact with the user. As a result, even when the position of the measurement device or the display is changed, the agent system can provide interactive services between the user and the agent.

[0030] In addition, for example, when it is necessary to independently adjust the position of the camera and the display position of the agent due to environmental constraints such as a projection-type display system, the camera can be installed without affecting the external shape representation of the agent, and the aforementioned interactive services can be provided.

[0031] In addition, even when the agent system moves the display position of the agent within the screen of the display, the aforementioned interactive services can be provided without being affected by the display position of the agent. Further, even when a display is added or expanded, the agent system can easily recalculate the positional relationship among the added or expanded display, the measurement device, and the user, so the aforementioned interactive services can be provided.

[0032] [Configuration Example] Next, a configuration example of the agent system 10 will be described with reference to FIG. 5. The agent system 10 includes, for example, an input unit 11, a face recognition unit (direction estimation unit) 12, a positional relationship correction unit 13, an agent control unit 14, and an output unit 15.

[0033] [Input Unit] The input unit 11 acquires the user's position and the user's video. For example, the input unit 11 includes a video input device 111 and a position measuring device 112. The video input device 111 acquires the user's video using a camera. The position measuring device 112 acquires the user's position (the user's position information in a coordinate system with the position measuring device 112 as the origin) using a distance measuring sensor or the like.

[0034] The input unit 11 acquires the user's video and position using the video input device 111 and position measurement device 112, and outputs them to the face recognition unit 12. This input unit 11 can be implemented, for example, by an RGB camera or a combination of an RGB camera and a distance measuring sensor. Hereinafter, the video input device 111 and the position measurement device 112 will be collectively referred to as the measurement device. The measurement device may be a single device or a combination of multiple devices.

[0035] [Face Recognition Unit] The face recognition unit 12 measures the user's orientation from the user's video. For example, the face recognition unit 12 measures the orientation of the user's face (angle relative to the normal direction of the measuring device) from the user's video. The face recognition unit 12 then outputs the measured orientation of the user's face and the user's position (coordinates with the measuring device as the origin) to the position relationship correction unit 13. In the following explanation, the face recognition unit 12 will be described using the example of measuring and outputting the orientation of the user's face as the user's orientation, but it may also measure and output the direction the user is looking, such as the user's gaze.

[0036] [Position Relationship Correction Unit] The position relationship correction unit 13 estimates the display direction of the agent as seen from the user's position, based on the orientation of the user's face output from the face recognition unit 12. Then, based on the estimated display direction of the agent as seen from the user's position, the position relationship correction unit 13 calculates the display position of the agent in a coordinate system with the measuring device as the origin, and outputs the display position of the agent and the position of the user. The position relationship correction unit 13 comprises a display direction estimation unit 131, a position relationship calculation unit 133, and a storage unit 134.

[0037] [Display Direction Estimation Unit] The display direction estimation unit 131 takes the measurement result of the user's face orientation as input, estimates the display direction of the agent as seen from the user (angle based on the direction from the user to the measuring device), and outputs it.

[0038] The display direction estimation unit 131 includes an estimation time management unit 132. When the display direction estimation unit 131 estimates the display direction of the agent as seen from the user using the measurement results of the user's face orientation, the estimation time management unit 132 increases the measurement time of the user's face orientation used for estimation as the distance between the measuring device and the user's position increases.

[0039] In this way, for example, the decrease in the accuracy of measuring the direction of the user's face due to an increase in the distance between the measuring device and the user's position can be mitigated. As a result, the accuracy of the display direction estimation unit 131's estimation of the display direction of the agent as seen from the user can be improved. Details of the display direction estimation unit 131 and the estimated time management unit 132 will be described later.

[0040] [Position Relationship Calculation Unit] The position relationship calculation unit 133 calculates and outputs the position relationship between the position of the measuring device, the position of the user, and the display position of the agent, based on the estimated display direction of the agent as seen from the user's position.

[0041] For example, the position relationship calculation unit 133 calculates the display position of the agent in a coordinate system with the measuring device as the origin, based on the display direction of the agent estimated by the display direction estimation unit 131 and the user's position. The position relationship calculation unit 133 then outputs the user's position and the agent's display position in the above coordinate system to the agent control unit 14.

[0042] Furthermore, when the positional relationship calculation unit 133 calculates the display position of the agent in a coordinate system with the measuring device as the origin, it refers, for example, to the display position of the agent and the position of the user, as well as the orientation of the user's face (see Figure 7), during the initial setup of the agent system 10.

[0043] In the following explanation, the positional relationship between the position of the measuring device, the user's position, and the agent's display position will be described using, as an example, a coordinate system in which the position of the measuring device is the origin, but it is not limited to this. For example, the above positional relationship may also represent the position of the measuring device and the agent's display position with the user's position as the origin, or it may represent the position of the measuring device and the user's position with the agent's display position as the origin.

[0044] [Storage Unit] The storage unit 134 stores information indicating the display position of the agent and the user in a coordinate system with the measuring device as the origin, as well as the orientation of the user's face, in the initial setup of the agent system 10 (see Figure 7).

[0045] The information shown by reference numeral 701 in Figure 7 is an example of information when, in the initial settings, the display position of the agent and the position of the measuring device coincide, and the user's position is directly in front of the measuring device. The information shown by reference numeral 702 is an example of information when, in the initial settings, the display position of the agent is shifted to the right relative to the measuring device, and the user's position is directly in front of the agent. The information shown by reference numeral 703 is an example of information when, in the initial settings, the display position of the agent is shifted to the right and rear relative to the measuring device, and the user's position is directly in front of the measuring device.

[0046] The above information may be actual measured values, or values ​​manually set by the user or the administrator of agent system 10.

[0047] [Agent Control Unit] Returning to the explanation of Figure 5, the agent control unit 14 takes the user's voice and video acquired from the input unit 11 as input, controls the agent's speech, actions, and display, and outputs the expressions performed by the agent to the output unit 15.

[0048] Here, for example, the agent control unit 14 outputs to the output unit 15 an image that has been controlled to face the user, using the positional relationship between the display position of the measuring device and the agent and the user's position, which is output from the positional relationship correction unit 13.

[0049] Furthermore, when the user initiates interaction (e.g., a conversation) with the agent, the agent control unit 14 may output instructions via the output unit 15 to cause the user to face the agent. For example, the agent control unit 14 may output at least one of the video and audio of the agent speaking to the user via the output unit 15. This allows the agent system 10 to acquire video when the user is facing the agent.

[0050] The agent control unit 14 includes a speech control unit 141 that controls the agent's speech, an action control unit 142 that controls the agent's movements, and a display control unit 143 that controls the agent's display.

[0051] [Output Unit] The output unit 15 outputs the agent's video and audio output from the agent control unit 14. For example, the output unit 15 outputs the agent's video and audio output from the agent control unit 14 to a display, speaker, etc.

[0052] Each of the above components is realized, for example, by the execution of programs by the CPU of the agent system 10 or by dedicated hardware.

[0053] Furthermore, while the agent system 10 measures the user's position using the position measuring device 112, it is not limited to this. For example, the agent system 10 may measure (estimate) the user's position from the user's video acquired by the video input device 111. In that case, the agent system 10 may have a configuration that does not include the position measuring device 112, as shown in Figure 6, for example.

[0054] [Example of Processing Procedure] Next, an example of a processing procedure performed by the agent system 10 will be explained using Figures 8 and 9.

[0055] First, the agent control unit 14 of the agent system 10 has the agent converse with the user (S1 in Figure 8). Then, the face recognition unit 12 acquires video of the user during the conversation with the agent via the input unit 11, and recognizes the position and orientation of the user's face from the video of the conversation (S2).

[0056] After S2, the display direction estimation unit 131 estimates the display direction of the agent as seen from the user's position, based on the orientation of the user's face recognized in S2 (S3).

[0057] After S3, the position relationship calculation unit 133 calculates the display position of the agent from the user's position with the measuring device as the origin and the agent's display direction (the agent's display direction estimated in S3) (S4).

[0058] After S4, the agent control unit 14 corrects the display of the agent based on the user's position and the agent's display position (the agent's display position calculated in S4) (S5). For example, the agent control unit 14 corrects the display of the agent so that the agent faces the direction of the user's position. After that, the agent control unit 14 outputs the corrected display of the agent to the output unit 15.

[0059] Next, the process of S4 in Figure 8 will be explained in detail using Figure 9. For example, the position relationship calculation unit 133 obtains the initial position relationship (the display position of the agent and the user in a coordinate system with the measuring device as the origin, and the orientation of the user's face) from the storage unit 134 (S41 in Figure 9).

[0060] After S41, the position relationship calculation unit 133 calculates the horizontal coordinates of the agent's display position using the position relationship between the user and the agent at the time of initial setup, which was obtained in S41, and the display direction of the agent (the display direction of the agent estimated in S3 in Figure 8) (S42). The position relationship calculation unit 133 also calculates the depth coordinates of the agent's display position using the current user's position and the position relationship between the user and the agent at the time of initial setup (S43).

[0061] The positional relationship calculation unit 133 performs the above processing to calculate the horizontal and depth coordinates of the agent's display position.

[0062] [Calculation Example for Estimating the Agent's Display Direction] Next, an example of calculation for estimating the agent's display direction will be explained. The display direction estimation unit 131 measures the user's face orientation for a predetermined period of time or longer, as in method 2 described above (see Figure 3), and uses the statistical values ​​of the measured face orientation to estimate the agent's display direction as seen from the user. At this time, the display direction estimation unit 131 measures the user's face orientation (angle) for a longer period of time the longer the distance between the measuring device and the user. This makes it possible to mitigate the effect of reduced accuracy in estimating the agent's display direction due to the distance between the measuring device and the user.

[0063] The measurement time τ of the user's face orientation used to estimate the agent's display direction is calculated by the estimation time management unit 132. The estimation time management unit 132 calculates τ by, for example, one of the following equations (1) to (3).

[0064]

[0065]

[0066]

[0067] t0: Start time for estimating the agent's display direction τ: Length of time for measuring the angle used to estimate the agent's display direction τ0: Initial setting value for the length of time for measuring the angle used to estimate the agent's display direction l: Distance between the measuring device and the user measured by the measuring device τ max : The maximum value to add to the initial value τ0 when calculating τ θ(t): An angle indicating the direction the user was facing at time t θ: An angle indicating the display direction of the agent α: A constant that adjusts the influence of distance l on τ

[0068] For example, l = 2 meters, τ0 = 1 second, τ max When τ = 5 seconds and α = 10, the estimated time management unit 132 calculates τ using the above equation (3), and the result is τ = approximately 4.6 seconds.

[0069] The display direction estimation unit 131 uses the τ calculated by the estimated time management unit 132 to estimate the agent's display direction θ, for example, by the following equation (4). In other words, the display direction estimation unit 131 estimates the average angle of the angles observed between time t0 and time t0+τ as the agent's display direction θ.

[0070]

[0071] Furthermore, the display direction estimation unit 131 may divide the range of angles measurable by the measuring device into intervals of any width, and use as θ the representative value (mean or median) of the interval to which the most angles among those measured between time t0 and time t0+τ belong. Alternatively, the display direction estimation unit 131 may use as θ the mode of the observed angles.

[0072] [Variations of Estimation of Agent's Display Direction] The display direction estimation unit 131 treats the angle between the normal direction of the measuring device and the normal direction of the user's face as the orientation of the user's face, but this is not limited to this. Also, the above angle does not have to be an angle that was measured directly, but may be an angle calculated using multiple measured values.

[0073] For example, if the orientation of the measuring device and the orientation of the user's face are measured or given in advance in the form of vectors, the display direction estimation unit 131 may calculate the angle between those vectors. The orientation of the user's face may be the rotation angle of the user's face relative to the direction when the user is facing forward, or it may be an angle relative to the normal direction of the measuring device.

[0074] [Example of an instruction to the user to face the agent] In addition, in the method 1 described above (see Figure 3), the agent system 10 instructed the user to face a specific direction (the direction of the agent) through the agent's utterances, etc., but the means of instruction are not limited to utterances.

[0075] For example, the agent system 10 may attract the user's attention in a specific direction in the following manner and use this as an alternative to the above instructions.

[0076] The agent control unit 14, which attracts attention through the agent's physical movements, may, for example, guide the user to face the agent by performing display control such as the agent waving, beckoning, jumping around on the screen, or moving around on the screen.

[0077] - The agent system 10 may play a beep or music of its choice at the agent's display location to guide the user towards the agent. For example, if the display showing the agent has a built-in speaker, or if sound can be emitted from the agent's display location, the agent system 10 may guide the user towards the agent by playing a beep or music of its choice at the agent's display location.

[0078] - Presenting a specific light stimulus at the agent's display location: For example, taking advantage of the fact that humans are sensitive to changes in brightness, the agent system 10 may present a flashing light stimulus or the like at the agent's display location on the display to attract the user's attention.

[0079] [Example of user-agent interaction] Furthermore, in method 2 described above (see Figure 3), the interaction between the user and the agent that the agent control unit 14 has the agent perform is not limited to voice interaction. For example, the interaction with the user may be done using sign language or gestures, or it may be done through text-based communication such as writing or chat.

[0080] [Calculation Example 1 for Estimating the Display Position of an Agent] Next, with reference to Figure 10, an example of the calculation for estimating the display position of an agent by the position relationship calculation unit 133 will be explained.

[0081] a: Agent's coordinates at initial setup o: Measurement device coordinates (origin) u: User's coordinates at initial setup a': Estimated agent coordinates u': User's coordinates when estimating agent coordinates θ: User's face orientation (direction of agent display as seen from the user). This is the angle between the line parallel to the measurement device's normal direction and the face's normal direction, with positive values ​​to the left of the user and negative values ​​to the right.

[0082] For example, consider a case as shown in FIG. 10 where the agent moves in the horizontal direction while maintaining the depth-direction distance from the user, starting from an initial positional relationship (see reference numeral 401 in FIG. 4), and the measuring device moves in the depth direction. Let o(0,0), a(a x ,a y ), u(u x ,u y ), a'(a' x ,a' y ), u'(u' x ,u' y ), and 0 < |θ| < 90. In this case, the positional relationship calculation unit 133 can obtain the coordinates a'=(a' x ,a' y ) of the agent in the coordinate system with the measuring device as the origin by the following equation (5).

[0083]

[0084] [Calculation Example 2 for Estimating Agent Display Position] Further, for example, as shown in FIG. 11, it is assumed that the distance between a' and u' does not change from the initial state (the distance remains l). In this case, the positional relationship calculation unit 133 can obtain the coordinates a'=(a' x ,a' y ) of the agent in the coordinate system with the measuring device as the origin by the following equation (6).

[0085]

[0086] [Vertical Calculation Example 1] Next, a calculation example for estimating the vertical display position of the agent will be described. For example, as shown in FIG. 12, it is assumed that the position of the measuring device in the z-direction is the same as the position of the user's face, and the measuring device faces toward the user. Further, as in the previous calculation examples, it is assumed that the depth-direction distance between the agent and the user does not change from the initial state.

[0087] In this case, the positional relationship calculation unit 133 can obtain the vertical (z-direction) position a' z of the agent by the following equation (7).

[0088]

[0089] In equation (7), φ represents the orientation of the user's face (the direction in which the agent is displayed as seen from the user's perspective). φ is the angle of the perpendicular between the line representing the normal direction of the measuring device and the normal direction of the user's face.

[0090] Furthermore, assuming that the straight-line distance l between the agent and the user does not change from the initial state, the position relationship calculation unit 133 calculates the agent's vertical (z-direction) position a'. z This can be calculated using the following formula (8).

[0091]

[0092] [Vertical Calculation Example 2] The position relationship calculation unit 133 also calculates the vertical position (z-direction) of the agent a'. z This can also be calculated as follows. Here, the positional relationship calculation unit 133 calculates the coordinates of the agent on the xy-plane a'=(a' x ,a' ｙ It is assumed that ) has already been calculated. Also, the direction of the user's face obtained from the measuring device is given in vector form, and the direction on the plane in the depth direction (y direction) and the vertical direction (z direction) is d u (d y d ｙ )

[0093] Furthermore, the user's coordinates on the plane in the depth and vertical directions are obtained by the measuring device, u'(u' y ,u´ ｚ ) then u'(u' y ,u´ z ) passing through d u (d y ,d' y A line parallel to ) is represented by the following equation (9).

[0094]

[0095] Here, a' ｙ Since is known, y=a' y Substitute this into equation (9) above and get a' z When calculated, it is as shown in equation (10).

[0096]

[0097] In this way, the positional relationship calculation unit 133 calculates the vertical (z-direction) coordinate a' of the agent. z It is possible to calculate this.

[0098] [Calculation method when there are multiple users] Note that there may be multiple users interacting with the agent. When there are multiple users, the position relationship calculation unit 133 calculates the agent's display position a' using one of the following calculation methods, for example.

[0099] Calculation method 1: The position relationship calculation unit 133 selects one user from among multiple users that meets specific conditions, and uses the coordinates u' of the selected user and the orientation of the user's face to calculate the display position a' of the agent using the aforementioned equation (5) or equation (6) (see Figure 13).

[0100] Calculation Method 2: The positional relationship calculation unit 133 obtains the orientation of any two users from among the multiple users, and sets the intersection of the lines indicating the orientation of the two users' faces as the display position a' of the agent. Note that the user's coordinates u' are the positions of the centroids of the multiple users (see Figure 14).

[0101] [Calculation Method 1 when there are multiple users] The above calculation method 1 will be explained in detail with reference to Figure 13. The position relationship calculation unit 133 selects one user from among multiple users (u1', u2', u3', u4', u5') to be used to estimate the agent's display position a' by one of the following methods (1) to (5), and sets the coordinates of the selected user as u'.

[0102] (1) For example, the positional relationship calculation unit 133 individually records the voice of each user, associates each user's voice with the individual in the video, and selects the user who spoke for the longest time. Alternatively, the positional relationship calculation unit 133 may select the user whose mouth was moving for the longest time from the video without using voice.

[0103] (2) Similar to (1) above, the positional relationship calculation unit 133 individually records the voice of each user and selects the user with the loudest voice.

[0104] (3) The positional relationship calculation unit 133 records the content of each user's speech and selects the user who has made a statement of high importance (for example, a question to the agent or a decision regarding the previous topic).

[0105] (4) The positional relationship calculation unit 133 estimates the skeleton of the user in the video using an existing method and selects the user whose hand and upper body movements are greatest or whose movements last the longest.

[0106] (5) The position relationship calculation unit 133 selects the user who is closest to the measuring device.

[0107] [Calculation Method 2 when there are multiple users] Next, calculation method 2 described above will be explained in detail with reference to Figure 14. For example, the position relationship calculation unit 133 obtains the orientation of the faces of any two users among the multiple users. The position relationship calculation unit 133 then sets the intersection of the lines (for example, d1 and d4) that indicate the orientation of the faces of the two users as the display position a' of the agent. The position relationship calculation unit 133 also sets u' as the average of the coordinates (u1', u2', u3', u4', u5') of the multiple users.

[0108] For example, the positional relationship calculation unit 133 selects the two users who are furthest apart from among multiple users (the user at coordinates u1' and the user at coordinates u4' shown in Figure 14). Let d1(m1,n1) and d4(m4,n4) be vectors indicating the orientation of the faces of the selected users. Here, the equations of the line passing through coordinate u1' with direction d1(m1,n1) and the line passing through coordinate u4' with direction d4(m4,n4) are given by equation (11) below.

[0109]

[0110] Therefore, the position relationship calculation unit 133 can calculate the intersection point of the two lines by solving the equation shown in equation (11). Let the coordinates of this intersection point be a'. The position relationship calculation unit 133 also sets u' to be the average of the coordinates of multiple users (u1', u2', u3', u4', u5').

[0111] In this way, the positional relationship calculation unit 133 can estimate the display position of the agent even when there are multiple users.

[0112] In previous calculation examples, the agent system 10 was assumed to be able to measure the user's position, and the distance between the agent and the user in the depth direction did not change from the initial state. Under these assumptions, the agent's coordinates a' were estimated with the measuring device as the origin. However, if the agent system 10 cannot measure the user's position, the user's coordinates u' can be estimated (calculated) if the agent's coordinates a' are known.

[0113] In the above case, since the user's location is not provided, the agent system 10 does not need to be equipped with a location measuring device 112 (see Figure 6).

[0114] [Example 1 of User Position Calculation] Referring to Figure 15, an example of calculating the user's coordinates u' when the agent's coordinates a' are known (Calculation Example 1) will be explained. For example, if it can be assumed that the distance in the depth direction between the agent and the user does not change from the initial state, the position relationship calculation unit 133 calculates the user's coordinates u' = (u') using the following equation (12). ｘ ,u´ y ) can be calculated. Note that in equation (12), a' = (a' ｘ ,a' y ) is the coordinates of the agent, which are given in advance.

[0115]

[0116] [Example 2 of user position calculation] In the above calculation example 1, the position relationship calculation unit 133 calculated the user's coordinates u' under the assumption that the distance in the depth direction between the agent and the user does not change from the initial state. However, if we assume that the distance between the agent and the user does not change from the initial state, the user's coordinates u' can be calculated as shown in the following calculation example 2.

[0117] Next, with reference to Figure 15, we will explain the second example of calculating the user's coordinates u'. For example, if the distance between the agent's coordinates a' and the user's coordinates u' is L, the position relationship calculation unit 133 calculates the user's coordinates u' = (u') using the following equation (13). ｘ ,u´ y ) can be calculated.

[0118]

[0119] Thus, even if the agent system 10 cannot measure the user's coordinates u', if the agent's coordinates a' are known, the position relationship calculation unit 133 can estimate (calculate) the user's coordinates u'. As a result, the position relationship calculation unit 133 can output the agent's coordinates a' and the user's coordinates u' to the agent control unit 14.

[0120] [Example of agent system operation] The agent system 10 estimates the display direction of the agent and calculates the display position of the agent by any of the following processing procedures (1) to (3).

[0121] (1) First, the agent system 10 has the agent continue the conversation with the user facing a random or fixed direction, and calculates the agent's display position during this time. Then, when the calculation of the agent's display position is complete, the agent system 10 has the agent face the user and continue the conversation.

[0122] (2) First, when the agent system 10 begins calculating the display position of the agent, it has the agent make an utterance that draws attention to itself, and calculates the display position of the agent. Then, during or after the conversation with the user that was triggered by that utterance, the agent system 10 controls the display of the agent, such as having the agent face the direction of the user or moving according to the user's position.

[0123] (3) When the agent system 10 starts calculating (estimating) the display position of the agent, it has the agent make a statement indicating the start of estimation, and when the estimation of the display position of the agent is finished, it has the agent make a statement indicating the end.

[0124] According to the agent system 10 described above, even in environments where the position of the measuring device, the position of the display (the agent's display position), and the user's position change, the agent's display position and the user's position in a coordinate system with the measuring device as the origin can be calculated. As a result, the agent system 10 can adjust the orientation of the agent's body and face so that the agent can interact with the user appropriately.

[0125] [System Configuration, etc.] Furthermore, the components of each part shown in the diagram are functional concepts and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown in the diagram, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. In addition, all or any part of the processing functions performed by each device can be realized by a CPU and the program executed on that CPU, or by hardware using wired logic.

[0126] Furthermore, among the processes described in the embodiments described above, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.

[0127] [Program] The agent system 10 described above can be implemented by installing a program (display control program) as packaged software or online software on a desired computer. For example, by having the computer run the above program, the computer can function as the agent system 10. The term "computer" here includes mobile communication terminals such as smartphones, mobile phones and PHS (Personal Handyphone System), as well as terminals such as PDA (Personal Digital Assistant).

[0128] Figure 16 shows an example of a computer that executes a display control program. Computer 1000 has, for example, memory 1010 and CPU 1020. Computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0129] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM (Random Access Memory) 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0130] The hard disk drive 1090 stores, for example, the OS 1091, application programs 1092, program modules 1093, and program data 1094. That is, the programs that define each process executed by the agent system 10 are implemented as program modules 1093 in which executable code is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to the functional configuration in the agent system 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0131] Furthermore, the data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes them.

[0132] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070.

[0133] 10 Agent system (display control device) 11 Input unit 12 Face recognition unit (method estimation unit) 13 Position relationship correction unit 14 Agent control unit 15 Output unit 111 Video input device 112 Position measurement device 131 Display direction estimation unit 132 Estimated time management unit 133 Position relationship calculation unit 134 Storage unit 141 Speech control unit 142 Motion control unit 143 Display control unit

Claims

1. A display control device comprising: an input unit that acquires video of a user interacting with an agent on a screen; a direction estimation unit that estimates the user's orientation when the user is looking at the agent from the user's video; a display direction estimation unit that estimates the display direction of the agent as seen from the user's position based on the estimated user orientation; a position relationship calculation unit that calculates the positional relationship between the shooting device that captured the video, the user's position, and the agent's display position based on the estimated display direction of the agent; and an agent control unit that controls the display of the agent so that the agent faces the user, using the positional relationship between the shooting device, the user's position, and the agent's display position.

2. The display control device according to claim 1, wherein the position relationship calculation unit acquires the position of the user in a coordinate system with the imaging device as the origin, calculates the display position of the agent in the coordinate system based on the acquired position of the user and the estimated display direction of the agent, and outputs the position of the user and the display position of the agent.

3. The display control device according to claim 1, wherein the position relationship calculation unit acquires the display position of the agent in a coordinate system with the imaging device as the origin, calculates the position of the user in the coordinate system based on the acquired display position of the agent and the estimated display direction of the agent, and outputs the position of the user and the display position of the agent.

4. The display control device according to claim 1, characterized in that the agent control unit outputs an instruction to the user to face the agent by controlling at least one of the agent's video and audio before estimating the agent's display direction, and the display direction estimation unit estimates the agent's display direction based on the user's orientation when the user looks at the agent in response to the instruction.

5. The display control device according to claim 1, characterized in that the agent control unit outputs an instruction to cause interaction between the agent and the user to continue for a predetermined time by controlling at least one of the agent's video and audio before estimating the agent's display direction, and the display direction estimation unit estimates the agent's display direction based on statistical values ​​of the user's orientation measured during the predetermined time.

6. The display control device according to claim 5, characterized in that the display direction estimation unit increases the measurement time of the user's orientation as the distance between the shooting device and the user increases.

7. A display control method performed by a display control device, comprising the steps of: acquiring video footage of a user interacting with an agent on a screen; calculating the user's orientation when the user is looking at the agent from the video footage of the user; estimating the display direction of the agent as seen from the user's position based on the calculated user orientation; calculating the positional relationship between the imaging device that captured the video footage, the user's position, and the agent's display position based on the estimated display direction of the agent; and performing display control of the agent so that the agent faces the user, using the positional relationship between the imaging device, the user's position, and the agent's display position.

8. A display control program that causes a computer to perform the following steps: acquiring video of a user interacting with an agent on a screen; calculating the user's orientation when the user is looking at the agent from the video of the user; estimating the display direction of the agent as seen from the user's position based on the calculated user orientation; calculating the positional relationship between the recording device that captured the video, the user's position, and the agent's display position based on the estimated display direction of the agent; and controlling the display of the agent so that the agent faces the user, using the positional relationship between the recording device, the user's position, and the agent's display position.

Images