Interaction generation device, interaction generation method, and interaction generation program

Abstract

This interaction generation device comprises a tracking unit and an event occurrence assessment unit. The tracking unit captures movements of a user and a counterpart, who interact using XR technology, on the basis of captured videos of the user and the counterpart, and outputs movement information pertaining to the user and the counterpart. The event occurrence assessment unit assesses whether an event has occurred on the basis of the motion information, and, when it is assessed that an event has occurred, generates an action generation instruction for a device that generates an action for motivating the user to interact.

Description

Interaction generation device, interaction generation method, and interaction generation program

[0001] The present invention relates to an interaction generation device, an interaction generation method, and an interaction generation program.

[0002] Development of XR (Extended Reality) technologies related to the integration of the real world and the virtual world is in progress. As XR technologies, VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality) and the like are known.

[0003] For example, apparatuses and systems that provide an environment for performing interaction using XR technology between a user in the real world and a counterpart in the virtual world have been proposed. As an example, a naked-eye XR shared-seat dialogue system has been proposed, which provides an environment for interacting with a counterpart visible through a half mirror, by displaying the counterpart on a display arranged behind the half mirror for a user positioned in front of the half mirror without requiring wearing of goggles or the like, while the user views their own reflection in the half mirror.

[0004] "Latest XR Technology × Hatsune Miku, Providing a Futuristic Experience That Allows Enjoying Shared Seating With Virtual Characters ~ Integrating the Real and the Virtual at 'Super Kabuki powered by NTT'", "[online]", [retrieved on April 2, 2025], Internet <https: / / group.ntt / jp / newsrelease / 2023 / 04 / 29 / 230429a.html>

[0005] In the aforementioned naked-eye XR shared-seat dialogue system, the interaction between the user and the counterpart is limited to dialogue. For this reason, there is a view that the interactivity is low, and there is also a demand for improvement of the interactivity.

[0006] This invention has been made in view of the above circumstances, and its purpose is to provide an interaction generation device, an interaction generation method, and an interaction generation program that enhance interactivity in an environment where XR technology is used for interaction.

[0007] One aspect of the present invention is an interaction generation device. The interaction generation device includes a tracking unit and an event occurrence determination unit. The tracking unit captures the movements of the user and the other party based on the captured video of the user and the other party performing an interaction using XR technology, and calculates movement information of the user and the other party. The event occurrence determination unit determines whether an event has occurred based on the movement information, and when it determines that an event has occurred, it generates an action generation instruction for a device that generates an action to prompt the user to interact.

[0008] One aspect of the present invention is an interaction generation method. The interaction generation method includes the steps of: capturing images of a user and another person performing an interaction using XR technology; generating movement information of the user and the other person based on the captured images of the user and the other person; determining whether an event has occurred based on the movement information; and, when it is determined that an event has occurred, generating an action to prompt the user to interact.

[0009] One aspect of the present invention is an interaction generation program. The interaction generation program causes a computer to perform at least some of the functions of the components of the above-described interaction generation device.

[0010] According to the present invention, an interaction generation device, an interaction generation method, and an interaction generation program are provided that enhance interactivity in an environment where interaction using XR technology is performed.

[0011] Figure 1 is a diagram showing an interaction generation system according to an embodiment. Figure 2 is a flowchart showing the operation flow in the interaction generation system according to an embodiment. Figure 3 is a diagram showing the hardware configuration of the interaction generation device according to an embodiment.

[0012] The interaction generation system according to the embodiment will be described below with reference to the drawings. The interaction generation system is a system that enhances interactivity in an environment where interaction using XR technology takes place between a user in the real world and a partner in the virtual world. In other words, the interaction generation system is a system that enhances interactivity in XR devices and XR systems that provide an environment for interaction using XR technology.

[0013] Figure 1 shows an interaction generation system 100 according to an embodiment. Figure 1 also shows a naked-eye XR seated conversation system 10 as an XR system to which the interaction generation system 100 is applied.

[0014] Below, we will first describe the naked-eye XR seated dialogue system 10, to which the interaction generation system is applied, and then describe the interaction generation system 100.

[0015] (Analogless XR seated conversation system) The Analogless XR seated conversation system 10 is a system that provides an environment in which the user and the other party can converse while seated, without wearing special goggles or the like.

[0016] The naked-eye XR seated conversation system 10 includes a half-mirror 11, a darkroom 12, and a display 13.

[0017] The half-mirror 11 is installed upright. The darkroom 12 is located behind the half-mirror 11. The display 13 is installed upright in the corner of the darkroom 12, at an angle to the half-mirror 11. For example, the display 13 is large enough to display the other person at life size.

[0018] The naked-eye XR seated conversation system 10 also has a sofa 21 and tables 22A and 22B.

[0019] The sofa 21 is positioned in front of the half-mirror 11. The sofa 21 has, for example, three seats, and the seats are arranged parallel to the half-mirror 11. The rightmost seat is where user U sits when in use. The leftmost seat is positioned symmetrically with respect to the display 13 with respect to the half-mirror 11.

[0020] The two tables 22A and 22B are identical in structure and are arranged symmetrically with respect to the half mirror 11. Table 22A is positioned between the half mirror 11 and the sofa 21, and table 22B is positioned between the half mirror 11 and the display 13.

[0021] Additionally, wine glasses 23A and 23B are placed on tables 22A and 22B, respectively. The wine glasses 23A and 23B are arranged symmetrically with respect to the half-mirror 11.

[0022] The naked-eye XR seated conversation system 10 also includes a camera 15 and a video controller 16.

[0023] Camera 15 is positioned in front of the half-mirror 11, at an angle to the half-mirror 11. Camera 15 captures the user U reflected in the half-mirror 11.

[0024] The video controller 16 receives the video signal S1 and displays the other party's video on the display 13. The video controller 16 also acquires the video captured by the camera 15 and transmits the corresponding video signal S2.

[0025] The video controller 16 performs image processing to reverse the left-right image reflected on the half-mirror 11, which is captured by the camera 15.

[0026] The video signal includes both a video signal and an audio signal. In other words, the video captured by camera 15 includes both video and audio. Furthermore, display 13 has the function of playing back both the video signal and the audio signal.

[0027] When using the naked-eye XR seated conversation system 10, user U sits in the seat at the far right of the sofa 21 and faces the display 13. The other person is displayed on the display 13 at life size.

[0028] The other party may be a real-world person who is not present at the scene, or a virtual world character such as an anime character. In this context, real-world people are considered virtual world partners because they are not present at the scene. If the other party is a virtual world character and does not need to act in accordance with user U's actions, camera 15 is necessary, but may be omitted.

[0029] The other person is displayed in a seated position. Because user U sees both their own mirrored image and the image of the other person displayed on the display 13 simultaneously via the half-mirror 11, user U perceives the other person as sitting in the leftmost seat of the sofa 21 on which they are sitting. As a result, user U feels as if they are conversing with someone sitting next to them on the same sofa 21.

[0030] In the naked-eye XR seated conversation system 10, tables 22A and 22B conceal the knees and surrounding areas of the other person from the user U, thereby reducing discomfort caused by unfamiliarity.

[0031] Furthermore, the wine glasses 23A and 23B placed on tables 22A and 22B contribute to creating a strong sense of three-dimensionality.

[0032] (Interaction generation system) The interaction generation system 100 includes an interaction generation device 110, a camera 140, and a device 160.

[0033] Camera 140 is positioned behind the sofa 21 and captures images of user U and the other person engaging in conversation using the naked-eye XR seated conversation system 10. Specifically, camera 140 captures the mirrored image of user U reflected in the half-mirror 11 and the image of the other person displayed on the display 13 through the half-mirror 11. Camera 140 outputs the captured images to the interaction generation device 110.

[0034] Device 160 receives an action generation instruction from the interaction generation device 110 and generates an action that prompts user U to interact. For efficient transmission of the action, in one example, device 160 is carried by user U. In another example, device 160 is placed near user U.

[0035] For example, interaction includes actions such as user U and the other person putting their palms together, or user U and the other person toasting with wine glasses. Action is a stimulus that acts on a sensory organ. For example, action is a stimulus that acts on a sensory organ other than sight, and includes vibrations that act on touch, sounds that act on hearing, and so on. Action may also be a stimulus that acts on balance or proprioception.

[0036] Device 160 may be capable of generating one type of action, or it may be capable of generating multiple types of actions. Device 160 represents a functional entity that generates actions, and may consist of a single device, or it may consist of a combination of multiple devices.

[0037] The interaction generation device 110 receives video footage from the camera 140, detects the occurrence of an event corresponding to an interaction from the video footage, and when it detects the occurrence of an event, generates an action generation instruction for the device 160 and outputs the action generation instruction to the device 160.

[0038] Therefore, the interaction generation device 110 includes an input unit 111, a tracking unit 112, an event occurrence determination unit 113, an output unit 114, and an event database (DB) 121.

[0039] The input unit 111 receives the captured video from the camera 140 and passes the captured video to the tracking unit 112.

[0040] The tracking unit 112 captures the movements of the user U and the other party based on the captured video, calculates movement information of the user U and the other party, and outputs the movement information to the event occurrence determination unit 113. For example, the tracking unit 112 captures the movements of the hand of the user U and the hand of the other party, and calculates movement information for each hand. The hand movement information is time-series information of the hand position. That is, the hand movement information includes hand position information in each frame of the captured video.

[0041] For example, the tracking unit 112 uses a hand tracking technique to recognize a hand as a 21-joint model, recognize the 21 joints of the hand, and output three-dimensional position information of each joint as the hand position information.

[0042] The event occurrence determination unit 113 refers to the event DB 121 based on the movement information to determine whether an event has occurred. The event DB 121 holds event information relating to events to be determined. The event information is registered in advance.

[0043] The number of events to be determined may be one or a plurality. That is, the event information held by the event DB 121 may be one or a plurality.

[0044] The event corresponds to an interaction performed by the user U and the other party. The event is a state that occurs when the interaction is performed. For example, the event is represented by the relative positional relationship between the hand of the user U and the hand of the other party.

[0045] For example, when the interaction is an action in which the user U and the other party put their palms together, the event is a state in which the palms of the user U and the other party face each other and are close to each other.

[0046] Further, when there is an action in which the user U and the other party toast with wine glasses, the event is a state in which the hand of the user U and the hand of the other party are adjacent to each other while holding the wine glasses.

[0047] The event information includes information on the relative positional relationship between the hand of the user U and the hand of the counterpart under the event occurrence situation, and information on an action generated for the interaction corresponding to the event. The information on the relative positional relationship between the hand of the user U and the hand of the counterpart is reference information (threshold information) used in determining the occurrence of an event.

[0048] For example, when the interaction is an action in which the user U and the counterpart press their palms together, the information on the relative positional relationship of the hands includes a predetermined value (threshold value) of the distance between the two hands that serves as a criterion for determining whether the palm of the user U and the palm of the counterpart are close to each other, and a predetermined value (threshold value) of the relative direction between the two palms that serves as a criterion for determining whether the orientation of the palm of the user U and the orientation of the palm of the counterpart are close to anti-parallel.

[0049] Further, when the interaction is an action in which the user U and the counterpart toast with wine glasses, the information on the relative positional relationship of the hands includes a predetermined value (threshold value) of the posture of the hand and fingers that serves as a criterion for determining whether the shape of the hand is close to the shape of a hand holding a wine glass, and a predetermined value (threshold value) of the distance between the two hands that serves as a criterion for determining whether the hand of the user U and the hand of the counterpart are adjacent to each other.

[0050] The event occurrence determination unit 113 calculates the relative positional relationship between the hand of the user U and the hand of the counterpart for each frame based on the three-dimensional position information of each joint of the hand of the user U and the hand of the counterpart. Next, the event occurrence determination unit 113 searches the calculated relative positional relationship against the event information held in the event DB 121.

[0051] As a result of the search, when there is no event corresponding to the relative positional relationship in the event information, the event occurrence determination unit 113 determines that no event has occurred.

[0052] As a result of the search, when there is an event corresponding to the relative positional relationship in the event information, the event occurrence determination unit 113 determines that the event has occurred. In this case, the event occurrence determination unit 113 acquires the information of the action corresponding to the event from the event information, and generates an action generation instruction for the device 160.

[0053] If device 160 is capable of generating multiple types of actions, the action generation instruction includes information specifying the action to be generated. Furthermore, if device 160 is composed of multiple devices, each for a specific action, the instruction includes information specifying the device.

[0054] The output unit 114 receives an action generation instruction from the event occurrence determination unit 113 and outputs the action generation instruction to the device 160.

[0055] When device 160 receives an action generation instruction from output unit 114, it generates an action according to the action generation instruction.

[0056] For example, if the interaction is an action in which user U and the other party bring their palms together, device 160 continuously generates vibrations for a predetermined period of time. For example, if device 160 receives another action generation instruction before stopping the generation of vibrations, it will continue generating vibrations.

[0057] Furthermore, if user U and the other party perform an action of toasting with wine glasses, device 160 generates the sound that occurs when the wine glasses touch. For example, to avoid the unnatural continuous generation of glass contact sounds, device 160 will not generate sound again even if it receives another action generation instruction, until a predetermined period has elapsed after generating a sound.

[0058] User U, by receiving actions from device 160, perceives the interaction through other sensory organs in addition to vision, for example. This enhances the interactivity between User U, who is using the naked-eye XR co-op dialogue system 100, and the other party.

[0059] (Example of Operation) Next, an example of the operation of the interaction generation system 100 will be described with reference to Figure 2. Figure 2 is a flowchart showing the flow of operation in the interaction generation system 100 according to the embodiment.

[0060] In the following operation description, it is assumed that the naked-eye XR seated conversation system 10 is in operation, and that user U is seated on the sofa 21 and begins using the naked-eye XR seated conversation system 10.

[0061] Furthermore, the interaction generation system 100 is in a standby state, starts operating when user U sits down, and ends operating when user U leaves the seat. For example, detection of user U sitting down and leaving the seat may be performed by a pressure sensor provided on the sofa 21.

[0062] Furthermore, device 160 is a smartphone, and user U will carry the smartphone and use the naked-eye XR co-op dialogue system 10.

[0063] Furthermore, the interaction between user U and the other party is to be performed by the actions of user U and the other party. For this reason, the tracking unit 112 of the interaction generation device 110 calculates information on the hand movements of user U and the other party, and the event occurrence determination unit 113 determines the occurrence of an event corresponding to the interaction based on the hand movement information of user U and the other party.

[0064] The calculation of hand movement information by the tracking unit 112 and the determination of event occurrence by the event occurrence determination unit 113 are performed at high speed for each frame of the video captured by the camera 140.

[0065] When user U takes device 160 in their hand, sits on sofa 21, and begins using the naked-eye XR seated conversation system 10, in step S11, camera 140 starts capturing images of user U and the other person, and starts supplying the captured images of user U and the other person to the interaction generation device 110.

[0066] The input unit 111 receives the captured video from the camera 140 and passes the captured video to the tracking unit 112.

[0067] In step S12, the tracking unit 112 uses hand tracking technology to capture the movements of user U's hand and the opponent's hand based on the captured video, and calculates movement information for user U's hand and the opponent's hand. In other words, the tracking unit 112 calculates three-dimensional position information for each part of user U's hand and the opponent's hand for each frame of the captured video. The tracking unit 112 then passes the three-dimensional position information for each part of the hand to the event occurrence determination unit 113.

[0068] In step S13, the event occurrence determination unit 113 calculates the relative positional relationship between user U's hand and the opponent's hand based on the three-dimensional positional information of each part of user U's hand and the opponent's hand.

[0069] In step S14, the event occurrence determination unit 113 determines whether an event has occurred based on the calculated relative positional relationship between user U's hand and the opponent's hand. To make this determination, the event occurrence determination unit 113 searches for the relative positional relationship between user U's hand and the opponent's hand against the event information held in the event DB 121.

[0070] If the search results show that there is no event in the event information that matches the relative positional relationship, the event occurrence determination unit 113 determines that no event has occurred (step S14: No). In this case, the operation of the interaction generation device 110 skips steps S15 and S16 and proceeds to step S17.

[0071] If the search results show that there is an event in the event information that matches the relative positional relationship, the event occurrence determination unit 113 determines that an event has occurred (step S14: Yes). In this case, the event occurrence determination unit 113 obtains information on the action corresponding to the event from the event information and generates an action generation instruction for the device 160 in step S15.

[0072] The output unit 114 receives an action generation instruction from the event occurrence determination unit 113 and outputs the action generation instruction to the device 160.

[0073] When device 160 receives an action generation instruction from output unit 114, it generates an action in step S16.

[0074] In step S17, the interaction generation device 110 determines whether user U has left their seat. If user U has not left their seat (step S17: No), the interaction generation device 110 returns to the process in step S12 and performs the processes in steps S12 to S16 for the next frame of the captured video. If user U has left their seat (step S17: Yes), the interaction generation device 110 terminates its operation.

[0075] (Specific Example 1) A specific example is described below. In this specific example, the interaction is an action in which user U and the other person bring their palms together. In this case, the event is the state in which user U and the other person's palms are facing each other and close together. The event information includes information on the relative positional relationship between user U's hand and the other person's hand under the circumstances in which the event occurred, including a predetermined value (threshold) for the distance between the two hands which serves as a criterion for determining whether user U's palm and the other person's palm are close together, and a predetermined value (threshold) for the relative direction between the two palms which serves as a criterion for determining whether the orientation of user U's palm and the orientation of the other person's palm are nearly antiparallel.

[0076] In this specific example, in step S13, the event occurrence determination unit 113 calculates the distance between user U's palm and the opponent's hand, and the relative direction between user U's palm and the opponent's palm, as the relative positional relationship between user U's hand and the opponent's hand.

[0077] In step S14, the event occurrence determination unit 113 searches the event information to see if there is an event in which the distance between user U's palm and the other person's hand is less than or equal to a predetermined value (threshold) for distance in the event information, and the relative direction between user U's palm and the other person's palm is less than or equal to a predetermined value (threshold) for relative direction in the event information.

[0078] If the search finds an event that satisfies these conditions, the event occurrence determination unit 113 generates an action generation instruction for the device 160 in step S15. The action generation instruction instructs the device 160 to continuously generate vibrations for a predetermined period of time.

[0079] In step S16, device 160 continuously generates vibrations for a predetermined period of time. For example, if device 160 receives an action generation instruction again before stopping the generation of vibrations, it continues to generate vibrations. If the predetermined period of time has elapsed before receiving an action generation instruction again, device 160 stops generating vibrations.

[0080] (Specific Example 2) Let's consider another specific example. In this example, the interaction is the action of user U and the other person toasting with wine glasses. In this case, the event is the state in which user U's hand and the other person's hand are adjacent to each other, both holding wine glasses. The event information includes information about the relative positional relationship between user U's hand and the other person's hand under the circumstances in which the event occurred, including a predetermined value (threshold) for the posture of the hand and fingers, which is the criterion for determining whether the shape of the hand is close to the shape of a hand holding a wine glass, and a predetermined value (threshold) for the distance between the two hands, which is the criterion for determining whether user U's hand and the other person's hand are adjacent to each other.

[0081] In this specific example, in step S13, the event occurrence determination unit 113 calculates the relative positional relationship between user U's hand and the opponent's hand, including the posture of the hands and fingers of user U and the opponent's hand, and the distance between user U's hand and the opponent's hand.

[0082] In step S14, the event occurrence determination unit 113 searches the event information to see if there is an event in which the posture of the hands and fingers of user U and the other party falls below a predetermined value (threshold) for the posture of the hands and fingers in the event information, and the distance between user U's hand and the other party's hand falls below a predetermined value (threshold) for the distance between user U's hand and the other party's hand in the event information.

[0083] If the search finds an event that satisfies these conditions, the event occurrence determination unit 113 generates an action generation instruction for the device 160 in step S15. The action generation instruction instructs the device 160 to generate the sound that occurs when wine glasses touch.

[0084] In step S16, device 160 generates a sound that occurs when wine glasses touch each other. For example, to avoid the unnatural continuous generation of glass contact sounds, device 160 does not generate sound again even if it receives an action generation instruction for a predetermined period of time after generating a sound. For this reason, for example, event occurrence determination unit 113 does not generate an action generation instruction for a predetermined period of time after generating an action generation instruction.

[0085] (Hardware Configuration of Interaction Generation Device) Next, the hardware configuration of the interaction generation device 110 will be described with reference to Figure 3. For example, the interaction generation device 110 is composed of a computer. Figure 3 is a block diagram showing an example of the hardware configuration of the computer 200 that constitutes the interaction generation device 110 according to this embodiment.

[0086] As shown in Figure 3, the computer 200 includes a processor 211, a ROM (Read Only Memory) 212, a RAM (Random Access Memory) 213, an auxiliary storage device 214, an input / output interface 215, and a communication interface 216.

[0087] The processor 211, ROM 212, RAM 213, auxiliary storage device 214, input / output interface 215, and communication interface 216 are electrically connected to each other via a bus 217, and data is exchanged via the bus 217.

[0088] The processor 211 is composed of a general-purpose hardware processor, such as a CPU (Central Processing Unit) and a GPU (Graphical Processing Unit). The processor 211 controls the entire system, including the ROM 212, RAM 213, auxiliary storage device 214, input / output interface 215, and communication interface 216.

[0089] ROM 212 is a non-volatile memory that constitutes part of the main memory. ROM 212 non-temporarily stores the startup program necessary for starting the computer 200. The processor 211 starts the computer 200 by executing the program in ROM 212. ROM 212 is, for example, composed of EPROM (Erasable Programmable Read Only Memory) and stores various startup settings in addition to the startup program.

[0090] RAM 213 is a volatile memory that constitutes part of the main memory. RAM 213 temporarily stores the program necessary for processing by the processor 211 and the data necessary for executing the program. By executing the program in RAM 213, the processor 211 performs calculations on the data in RAM 213 and stores the calculation results in RAM 213.

[0091] The auxiliary storage device 214 consists of non-volatile memory such as an HDD (Hard Disk Drive) or SSD (Solid State Drive). The auxiliary storage device 214 non-temporarily stores programs executed by the processor 211 and data necessary for program execution. The processor 211 reads the programs and data from the auxiliary storage device 214 into the RAM 213 and executes various functions by running the programs.

[0092] The input / output interface 215 is connected to external input devices 231 and output devices 232, etc., enabling the input of information from the input device 231 and the output of information to the output device 232. For example, the input / output interface 215 may be a wired interface or a wireless interface. A wired interface includes ports to which the device is connected. A wireless interface includes Bluetooth®, WiFi®, etc.

[0093] The input device 231 may include a keyboard, mouse, touch panel, receiver, disk drive, etc. However, the input device 231 is not limited to these and may include any other input device. The output device 232 may include a display, transmitter, disk drive, etc. However, the output device 232 is not limited to these and may include any other output device. The input device 231 and the output device 232 may be configured as an input / output device having the functions of both.

[0094] The communication interface 216 enables communication with the outside world. This allows the interaction generation device 110 to exchange information with the outside world.

[0095] Furthermore, the input device 231 includes the camera 140 of the interaction generation system 100, and the output device 232 includes the device 160 of the interaction generation system 100. The input / output interface 215 constitutes the input section 111 and the output section 114 of the interaction generation device 110. The main memory (RAM 213) and the auxiliary memory 214 constitute the action DB 121 of the interaction generation device 110.

[0096] Programs stored non-temporarily in the auxiliary storage device 214 are provided to the computer 200, for example, via a recording medium 250 that is readable by the computer 200 on which the program was stored non-temporarily. Such a recording medium 250 is called a non-temporarily computer-readable recording medium. Non-temporarily computer-readable recording media include disks such as flexible disks, optical disks (CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), magneto-optical disks (MO, etc.), and semiconductor memory.

[0097] The programs stored non-temporarily in the auxiliary storage device 214 include an interaction generation program. The interaction generation program is a program that causes the computer 200 to execute the functions of a part of the interaction generation device 110, for example, the tracking unit 112 and the event occurrence determination unit 113.

[0098] Programs stored non-temporarily in the auxiliary storage device 214 are read into the auxiliary storage device 214 and stored non-temporarily via the input device 231 (disk drive) and input / output interface 215 if the recording medium 250 is a disk, or via the input / output interface 215 (port) if the recording medium 250 is semiconductor memory. Alternatively, the program may be stored on a server on a network, downloaded from the server via the communication interface 216, and stored non-temporarily in the auxiliary storage device 214.

[0099] When the computer 200 starts up, the processor 211 executes a program in the ROM 212 and loads the OS into the RAM 213 to start up. Under the control of the OS, the processor 211 monitors instruction inputs and connections of external devices. Also, under the control of the OS, the processor 211 sets up a program area and a data area in the RAM 213. In response to an instruction input to start the interaction generation device 110, the processor 211 loads the interaction generation program from the auxiliary storage device 214 into the program area of ​​the RAM 213, and loads the data necessary for program execution from the auxiliary storage device 214 into the data area of ​​the RAM 213. The processor 211 performs calculations on the data in the data area according to the program and writes the calculation results to the data area. Through these operations, the processor 211, RAM 213, auxiliary storage device 214, input / output interface 215, and communication interface 216 work together to execute the functions of each component of the interaction generation device 110.

[0100] (Effect) The interaction generation system 100 according to the embodiment detects the occurrence of an event corresponding to an interaction based on the captured video of user U and the other party using the naked-eye XR co-seat dialogue system 100, and when it detects the occurrence of an event, it gives user U an action that prompts interaction. By receiving the action, user U perceives the interaction not only through vision but also through other sensory organs.

[0101] This enhances the interactivity between user U using the naked-eye XR co-seat dialogue system 100 and the other party. In other words, the interaction generation system 100 enhances the interactivity of the naked-eye XR co-seat dialogue system 100.

[0102] Embodiments of the present invention have been described above with reference to the drawings. However, the above embodiments are merely examples of configurations that embody the present invention. In other words, it is clear that the present invention is not limited to the above embodiments. Therefore, additions, omissions, substitutions, and other modifications of components are permitted without departing from the technical spirit of the present invention.

[0103] For example, in the embodiment, an example of a glasses-free XR seated dialogue system 100 was shown as a means of providing an environment for interaction using XR technology. However, the application of the present invention is not limited to the glasses-free XR seated dialogue system 100, but is broadly applicable to any environment for interaction using XR technology.

[0104] The present invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate, and in that case, the combined effects can be obtained. Moreover, the above embodiments include various inventions, and various inventions can be extracted by selecting combinations from the multiple constituent elements disclosed. For example, if the problem can be solved and effects obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiment, then the configuration with these deleted constituent elements can be extracted as an invention.

[0105] 10...Analogless XR seated conversation system 11...Half mirror 12...Darkroom 13...Display 15...Camera 16...Video controller 21...Sofa 22A, 22B...Table 100...Interaction generation system 110...Interaction generation device 111...Input unit 112...Tracking unit 113...Event occurrence determination unit 114...Output unit 121...Event database 140...Camera 160...Device 200...Computer 211...Processor 212...ROM 213...RAM 214...Auxiliary storage device 215...Input / output interface 216...Communication interface 217...Bus 231...Input device 232...Output device 250...Recording medium

Claims

1. An interaction generation device comprising: a tracking unit that captures the movements of a user and another party based on captured video footage of the user and the other party performing an interaction using XR technology, and outputs movement information of the user and the other party; and an event occurrence determination unit that determines, based on the movement information, whether an event occurs when the interaction takes place, and when it is determined that an event has occurred, generates an action generation instruction for a device that generates an action to prompt the user to interact.

2. The interaction generation device according to claim 1, wherein the tracking unit captures the movement of the user's hand and the movement of the opponent's hand, the movement information includes positional information of the user's hand and the opponent's hand, and the event occurrence determination unit calculates the distance between the user's hand and the opponent's hand, determines that the event has occurred when the distance falls below a threshold, and generates the action generation instruction.

3. An interaction generation method comprising the steps of: photographing a user and another person who are performing an interaction using XR technology; generating information about the movements of the user and the other person based on the captured images of the user and the other person; determining, based on the movement information, whether an event occurs when the interaction takes place; and, when it is determined that an event has occurred, generating an action to prompt the user to perform an interaction.

4. An interaction generation program that causes a computer to execute at least some of the functions of the components of the interaction generation device described in claim 1.

Images