Simulation system, simulation method, and simulation program

Abstract

Provided is a simulation system (10) having a first virtual video generation device (100) and a second virtual video generation device (200), wherein: the first virtual video generation device comprises a first virtual space information acquisition unit (101) that acquires virtual object information indicating a virtual object in a first virtual space and virtual moving body information indicating the position of a virtual moving body corresponding to a simulated moving body in the first virtual space, a first video generation unit (103) that generates, on the basis of the virtual object information and the virtual moving body information, a three-dimensional video of the first virtual space in which the virtual moving body is moving and that generates a first virtual video by converting the three-dimensional video into a virtual two-dimensional video seen from a user in the virtual moving body, and a first output unit (106) that outputs the first virtual video; and the second virtual video generation device comprises a virtual video acquisition unit (201) that acquires the first virtual video, a second video generation unit (205) that acquires virtual position information indicating a prescribed position in a second virtual space having the same coordinate system as the coordinate system of a real space and that generates a second virtual video which is a virtual three-dimensional video obtained by including the virtual position information in the first virtual video, and a second output unit (206) that outputs the second virtual video to a virtual space display device.

Description

Simulation System, Simulation Method, and Simulation Program Cross - reference to Related Applications 【0001】 This application is based on Japanese Patent Application No. 2024 - 219355 filed on December 13, 2024, the content of which is incorporated herein by reference. 【0002】 The present disclosure relates to a simulation system that, by using a virtual space, can provide a user with a moving body experience by a moving body without actually moving the moving body in the real space, and further can provide the user with information regarding the functions of the moving body while experiencing the moving body. 【0003】 In recent years, technologies for providing virtual reality (VR: Virtual Reality), augmented reality (AR: Augmented Reality), or mixed reality (MR: Mixed Reality) to users have been proposed. In particular, a technology for providing virtual reality or the like using a head - mounted display (hereinafter referred to as HMD) that can be worn on a user's head is known to be able to provide a more immersive experience to the user. 【0004】 Patent Document 1 discloses a presentation system that adds explanations of a vehicle while driving the vehicle in a virtual space. According to Patent Document 1, in addition to being able to experience operations that cannot be experienced in an actual test drive while virtually experiencing the driving of a vehicle in the virtual space, a user can receive real - time explanations of the vehicle according to various situations encountered during the driving of the vehicle. However, in Patent Document 1, since the virtual space is displayed on a flat display screen, the sense of reality is lower compared to the actual driving of a vehicle. 【0005】Therefore, a vehicle simulation system is being considered in which the user rides in an actual vehicle and uses an HMD (Head-Mounted Display) to provide a virtual space within the vehicle. In such a vehicle simulation system, the user can virtually drive a vehicle in a virtual space while touching the interior structure of the actual vehicle, such as the steering wheel and seats, and can also visually perceive the interior structure of the vehicle through an HMD using MR (Mixed Reality) technology, thereby providing the user with a more realistic virtual experience. 【0006】 Furthermore, Patent Document 2 discloses a vehicle HMD device. According to Patent Document 2, the vehicle HMD device is worn on the driver's head while driving and can display various information necessary for safe driving (for example, speed information) regardless of the direction the driver's head is facing. 【0007】 JP-A No. 2002-8064 JP-A No. 10-206789 【0008】 Here, the inventors have identified the following problems after detailed investigation. Conventional MR technology employs a method of depicting mixed reality on the HMD according to the depth of real-world objects and the depth of virtual objects in the virtual space. For example, if there are real-world objects at a depth lower than the depth of the virtual object, the realism of the virtual space can be enhanced by depicting the virtual image so that the real object is positioned in front of the virtual object. However, in vehicle simulation systems that use an HMD inside an actual vehicle, real-world objects outside the vehicle (e.g., ceiling, floor, furniture, etc.) may be located at a lower depth than the virtual objects in the virtual space. In such cases, real-world objects will be depicted in the virtual space while the vehicle is in motion, which impairs the realism of the virtual space. 【0009】 Therefore, this disclosure aims to realize a vehicle simulation system, etc., that can provide a virtual driving experience in a virtual space without displaying real-world objects existing outside the vehicle. Furthermore, this disclosure aims to realize a vehicle simulation system, etc., that can provide an appropriate virtual space according to the user's facial orientation. 【0010】 One aspect of the present disclosure is a simulation system that outputs images to a virtual space display device attached to the head of a user inside a simulated mobile body, the simulation system comprising a first virtual image generation device and a second virtual image generation device, the first virtual image generation device comprising: a first virtual space information acquisition unit that acquires virtual object information indicating virtual objects in a first virtual space which is a virtual three-dimensional space, and virtual mobile body information indicating the position of a virtual mobile body corresponding to the simulated mobile body in the first virtual space; a first image generation unit that generates a three-dimensional image of the first virtual space on which the virtual mobile body moves based on the virtual object information and the virtual mobile body information, and converts the three-dimensional image into a virtual two-dimensional image visible to the user inside the virtual mobile body to generate a first virtual image; and a first output unit that outputs the first virtual image, the second virtual image generation device comprising: a virtual image acquisition unit that acquires the first virtual image, The system includes a second image generation unit that acquires virtual position information indicating a predetermined position in a second virtual space, which is a virtual three-dimensional space having the same coordinate system as the real space, and generates a second virtual image, which is a virtual three-dimensional image that includes the virtual position information in the first virtual image, and a second output unit that outputs the second virtual image to the virtual space display device. 【0011】 With this configuration, the simulation system and the like of this disclosure can provide users with a highly realistic virtual experience without displaying real objects that are unnecessary for the virtual driving experience, even when real objects exist outside the vehicle. 【0012】 The numbers in parentheses in the claims indicate the correspondence between the present invention and the embodiments described below, and are not intended to limit the present invention. 【0013】The above-mentioned and other purposes, features and advantages of this disclosure will become clearer from the following detailed description with reference to the accompanying drawings. The drawings are as follows: Figure 1 is a diagram illustrating the apparatus related to the simulation system of each embodiment; Figure 2 is a diagram illustrating the arrangement of the simulation system and related apparatus of each embodiment; Figure 3 is a diagram illustrating the real space around the user using the simulation system of each embodiment; Figure 4 is a diagram illustrating the real space visible to the user in each embodiment; Figure 5 is a diagram illustrating an example configuration of the virtual space display device of each embodiment; Figure 6 is a block diagram illustrating an example configuration of the simulation system of Embodiment 1; Figure 7 is a diagram illustrating the generation process of the first virtual image in Embodiment 1; Figure 8 is a diagram illustrating the deformation process of the first virtual image in Embodiment 1; Figure 9 is a diagram illustrating the first virtual images for the right eye and left eye in Embodiment 1; Figure 10 is a diagram illustrating the virtual object in Embodiment 1. Figure 11 is a diagram illustrating the shape of a virtual object in Embodiment 1, Figure 12 is a diagram illustrating the generation process of a second virtual image in Embodiment 1, Figure 13 is a diagram illustrating a virtual image using the prior art, Figure 14 is a diagram illustrating the operation of the simulation system in Embodiment 1, Figure 15 is a block diagram illustrating an example configuration of the second virtual image generation device in Embodiment 2, Figure 16 is a diagram illustrating the image displayed on the virtual space display device in Embodiment 2, Figure 17 is a diagram illustrating the operation of the second virtual image generation device in Embodiment 2, Figure 18 is a diagram illustrating the operation of the second virtual image generation device in Embodiment 2, Figure 19 is a diagram illustrating the operation of the second virtual image generation device in Embodiment 2, and Figure 20 is a block diagram illustrating a modified simulation device of Embodiment 2. 【0014】 Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings. 【0015】The present invention as described below means the invention described in the claims and is not limited to the embodiments described below. Furthermore, at least the words enclosed in double quotation marks mean the words described in the claims and are not limited to the embodiments described below. 【0016】 The configurations and methods described in the dependent claims are optional configurations and methods in the invention described in the independent claims. The configurations and methods in embodiments corresponding to the configurations and methods described in the dependent claims, as well as configurations and methods described only in embodiments and not in the claims, are optional configurations and methods in the present invention. The configurations and methods described in embodiments when the claims are broader than the descriptions in embodiments are also optional configurations and methods in the present invention, in the sense that they are illustrative examples of the configurations and methods of the present invention. In any case, by being described in the independent claims, they become essential configurations and methods of the present invention. 【0017】 The effects described in the embodiments are those that occur when the configuration is that of an exemplary embodiment of the present invention, and are not necessarily effects that the present invention possesses. 【0018】 When there are multiple embodiments (including examples and variations; the same applies hereinafter), the disclosed configuration in each embodiment is not confined to that embodiment alone, but can be combined across embodiments. For example, the disclosed configuration may be combined in one embodiment with another. Alternatively, the disclosed configuration may be collected and combined in each of multiple embodiments. 【0019】 The problems described in this disclosure are not publicly known problems, but rather things that the inventors have discovered independently, and these facts, along with the structure and method of this disclosure, affirm the inventive step of the invention. 【0020】 1. Prerequisites for Each Embodiment (1) Relationship between the Simulation System and Related Equipment of Each Embodiment The overall configuration of the simulation system 10 and the equipment related to the simulation system 10 of each embodiment will be described with reference to Figures 1 and 2. 【0021】Figure 1 schematically shows the simulation system 10, the virtual space display device 20 connected to the simulation system 10, and the in-vehicle device 30 for each embodiment. 【0022】 The simulation system 10 generates virtual images. For example, it generates images of the road and scenery visible from inside a car as it is driven. During this process, events such as traffic lights changing, the scenery changing from day to night as time passes, or obstacles appearing in the direction of travel may be included. In this disclosure, the simulation system 10 is a simulation system for vehicles, and the virtual space display device 20 is configured to generate virtual images related to the vehicle's driving events. However, the simulation system 10 is not limited to vehicles and can be applied to simulation systems for any moving object, such as airplanes or trains. 【0023】 The simulation system 10 shown in Figure 1 has a first virtual image generation "device" 100 and a second virtual image generation "device" 200, and is shown as an example where the first virtual image generation device 100 and the second virtual image generation device 200 are physically independent devices. However, the first virtual image generation device 100 and the second virtual image generation device 200 do not have to be physically independent devices, and may be configured to be included in a single physical device. In this case, the first virtual image generation device 100 and the second virtual image generation device 200 are collectively referred to as a simulation device. Here, "device" does not necessarily have to be independent, and multiple devices may be provided as functions within a single physical device. 【0024】 The virtual space display device 20 displays virtual images generated and output by the simulation system 10. The virtual space display device 20 is a head-mounted display that can be worn on the user's head and is connected to the simulation system 10 wirelessly or via a wired connection. 【0025】The in-vehicle device 30 is installed in a simulated vehicle, which is a "simulated mobile object" in which a user equipped with a virtual space display device 20 rides. The in-vehicle device 30 generates operation information indicating the user's operating status of the simulated vehicle and transmits it to the second virtual image generation device 200. For example, when the user operates the steering wheel of the simulated vehicle, the in-vehicle device 30 detects the steering wheel operation and transmits operation information indicating that the steering wheel has been operated to the second virtual image generation device 200. Here, a "simulated mobile object" is a device that simulates a mobile object, and may be a device that cannot actually move, or an actual mobile object (for example, a car, airplane, train, etc.) may be used as a simulated mobile object. 【0026】 The first virtual image generation device 100 and the second virtual image generation device 200, the first virtual image generation device 100 and the second virtual image generation device 200 and the virtual space display device 20, and the second virtual image generation device 200 and the in-vehicle device 30 are connected using either a wireless communication method or a wired communication method. Examples of wireless communication methods include IEEE 802.11 (Wi-Fi®), IEEE 802.16 (WiMAX®), W-CDMA (Wideband Code Division Multiple Access), HSPA (High Speed ​​Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution Advanced), 4G, 5G, etc. Examples of wired communication methods include LAN (Local Area Network), the Internet, and fixed telephone lines. The connection methods between each device may be the same or different. The choice of communication method should be appropriately selected based on the amount of information to be transmitted and received, and the installation locations of each device constituting the simulation system 10. 【0027】 Figure 2 shows the arrangement of the simulation system 10 and the equipment related to the simulation system 10. 【0028】As shown in Figure 2(a), both the first virtual image generation device 100 and the second virtual image generation device 200, which constitute the simulation system 10, may be provided outside the simulated vehicle. Alternatively, as shown in Figure 2(b), both the first virtual image generation device 100 and the second virtual image generation device 200 may be mounted on the simulated vehicle. As shown in Figure 2(c), the first virtual image generation device 100 may be provided outside the simulated vehicle, and the second virtual image generation device 200 may be mounted on the simulated vehicle. Alternatively, instead of the arrangement in Figure 2(c), the first virtual image generation device 100 may be mounted on the simulated vehicle, and the second virtual image generation device 200 may be provided outside the simulated vehicle. Although not shown in Figure 2, at least one of the first virtual image generation device 100 and the second virtual image generation device 200 may be built into the virtual space display device 20. 【0029】 Figure 3 illustrates the real-world space surrounding the user. Figure 3(a) schematically shows the simulated vehicle and its exterior viewed from the side, while Figure 3(b) schematically shows the simulated vehicle and its exterior viewed from above. As shown in Figure 3, the exterior of the simulated vehicle contains furniture such as desks and chairs, as well as the ceiling and walls of the room in which the simulated vehicle is installed. 【0030】 Figure 4 shows the view from the user inside the simulated vehicle, given that the real-world space surrounding the user is in the state shown in Figure 3. As shown in Figure 4, the simulated vehicle is equipped with interior features such as a steering wheel, car navigation system, and instrument panel, just like a real vehicle. In addition, the user can see furniture positioned in front of the outside of the simulated vehicle. 【0031】 In each embodiment described below, "real object" refers to an object that exists in real space. In the case of Figures 3 and 4, not only the equipment inside the simulated vehicle (e.g., steering wheel, car navigation system, instrument panel, etc.) but also objects outside the vehicle (e.g., ceiling, walls, floor, people, furniture, etc.) are considered real objects. "Virtual object" refers to a virtual object located in the virtual space on which the vehicle virtually travels. For example, traffic lights, trees, obstacles, buildings, etc., as well as roads on which the vehicle virtually travels, are considered virtual objects. 【0032】 (2) The configuration of the virtual space display device 20 will be described with reference to the configuration diagram 5 of the virtual space display device 20. The virtual space display device 20 includes an external camera 21, a distance sensor 22, a head tracking unit 23, an eye tracking unit 24, and a display screen 25. 【0033】 Figure 5 illustrates a case where the virtual space display device 20 is a so-called video see-through type device that displays real objects captured by an external camera 21 on a display screen 25 by superimposing them with virtual images. However, the device of this disclosure may also be applied to a so-called optical see-through type device that displays virtual images on a display screen 25 by superimposing them onto real objects viewed with the naked eye through the lens of the virtual space display device 20. If the virtual space display device 20 is an optical see-through type device, the external camera 21 in Figure 5 is omitted. 【0034】 The external camera 21 is an imaging device that captures real objects in the real space surrounding the user. The external camera 21 outputs video information showing the captured real images to the second virtual image generation device 200. In the examples in Figures 3 and 4, the camera captures the steering wheel, car navigation system, instrument panel, etc. inside the simulated vehicle, as well as the desk, chair, floor, and walls outside the simulated vehicle. 【0035】 The distance sensor 22 is a sensor that measures the actual distance from the virtual space display device 20 to a real object in the real space. Distance sensors are also called depth sensors. The distance sensor 22 outputs the actual distance information, which indicates the measured actual distance, to the second virtual image generation device 200. 【0036】 The head tracking unit 23 acquires the position of the user's head and the orientation of their face while the virtual space display device 20 is attached. For example, the head tracking unit 23 has sensors (not shown) such as an accelerometer and a gyroscope, and these sensors measure the position of the head and the orientation of the face. However, the values ​​measured using the sensors are acceleration and angular velocity, and cannot directly measure the position of the head or the orientation of the face. Therefore, the head tracking unit 23 calculates the position of the user's head and the orientation of their face using the acceleration and angular velocity measured using the sensors. 【0037】 For example, the initialization process of the head tracking unit 23 sets the position of the user's head and the orientation of their face when the user wears the virtual space display device 20 as the initial position and initial orientation in the virtual space. Then, using a coordinate system transformation matrix, the acceleration and angular velocity measured by sensors such as the accelerometer and gyroscope are converted into amounts of movement from the initial position and initial orientation in the virtual space, and calculated as the position of the user's head and the orientation of their face in the virtual space. 【0038】 In each embodiment, the head tracking unit 23 calculates the position of the user's head and the orientation of their face in the virtual space from the sensor measurements and outputs the calculated information to the first virtual image generation device 100 and the second virtual image generation device 200. However, the head tracking unit 23 may output the sensor measurements to the first virtual image generation device 100 and the second virtual image generation device 200, and the first virtual image generation device 100 and the second virtual image generation device 200 may calculate the position of the user's head and the orientation of their face in the virtual space from the sensor measurements. 【0039】 In the following embodiment, the coordinate system of the virtual space used by the first virtual image generation device 100 is different from the coordinate system of the virtual space used by the second virtual image generation device 200. Therefore, the head tracking unit 23 calculates the user's head position and face orientation in each virtual space from the sensor measurements. 【0040】 The eye-tracking unit 24 detects the pupils of the user wearing the virtual space display device 20 and determines the direction of the user's gaze. The eye-tracking unit 24 outputs gaze information indicating the direction of the user's gaze to the second virtual image generation device 200. 【0041】The display screen 25 is a display that shows the virtual video obtained from the simulation system 10. By displaying the virtual video obtained from the simulation system 10, the display screen 25 can provide the virtual video to the user wearing the virtual space display device 20. Note that the display screen 25 has a display screen for the right eye (corresponding to the "right-eye display unit") and a display screen for the left eye (corresponding to the "left-eye display unit"), and different virtual videos may be displayed on the display screen for the right eye and the display screen for the left eye respectively. By displaying different virtual videos on the display screen for the right eye and the display screen for the left eye respectively, the video can be shown more stereoscopically. 【0042】 Note that in FIG. 5, a configuration in which the virtual space display device 20 has an external camera 21, a distance sensor 22, and a head tracking unit 23 is illustrated and described. However, some of these configurations may be provided outside the virtual space display device 20. 【0043】 As an example, the distance sensor may be provided inside the simulated vehicle instead of the virtual space display device 20. Using the distance sensor or infrared camera provided inside the simulated vehicle, the positions of the virtual space display device 20 or the user inside the simulated vehicle and the positions of the real objects inside the simulated vehicle are measured respectively, and the relative distance between the virtual space display device 20 or the user and the real object is calculated using the measured position information. Since the positions of the real objects inside the simulated vehicle do not change, they may be measured and saved in advance. 【0044】 As another example, the head tracking unit 23 may be provided inside the simulated vehicle instead of the virtual space display device 20. By photographing or sensing the user's head using a camera or sensor, the position of the user's head and the direction of the face may be detected. 【0045】 Although omitted in FIG. 5, the virtual space display device 20 naturally has a configuration that serves as an interface with the first virtual video generation device 100 and the second virtual video generation device 200 described later. 【0046】2. Embodiment 1 (1) Referring to FIG. 6 which is a configuration diagram of the first virtual video generation device 100, the configuration of the first virtual video generation device 100 will be described. The first virtual video generation device 100 includes a first virtual space information acquisition unit 101, a first virtual direction information acquisition unit 102, a first video generation unit 103, a second virtual space information acquisition unit 104, a virtual video deformation unit 105, and a first output unit 106. 【0047】 The first virtual space information acquisition unit 101 (corresponding to the "virtual space information acquisition unit") acquires virtual object information indicating virtual objects in the first virtual space, which is a virtual three-dimensional space. The virtual object information includes the coordinates, shape, and image of the virtual object. The first virtual space information acquisition unit 101 further acquires virtual vehicle information (corresponding to the "virtual moving object information") indicating the position of a virtual vehicle (corresponding to the "virtual moving object") in the first virtual space. The position of the virtual vehicle may be an absolute position in the first virtual space or a relative position with respect to a reference position in the first virtual space. Here, the virtual vehicle is a virtual vehicle corresponding to a simulated vehicle in the first virtual space and travels in the first virtual space. Therefore, the position (i.e., coordinates) of the virtual vehicle changes as the virtual vehicle moves. Similarly, the coordinates, shape, and image of the virtual object in the first virtual space also change as the virtual vehicle moves. 【0048】 The virtual object information and the virtual vehicle information are created in advance according to the vehicle driving simulation event and stored in a storage device (not shown) provided outside or inside the simulation system 10. Therefore, the first virtual space information acquisition unit 101 acquires the virtual object information and the virtual vehicle information from such a storage device. 【0049】The first virtual direction information acquisition unit 102 acquires first virtual direction information from the head tracking unit 23, which indicates the "face orientation" of the user in the first virtual space. Here, the face orientation of the user in the first virtual space corresponds to the face orientation of the user in the simulated vehicle. In other words, if the user in the simulated vehicle turns their face to the right, the face orientation of the user in the first virtual space also becomes to the right. The face orientation indicated by the first virtual direction information may also be expressed as a relative direction to the virtual vehicle. Here, "face orientation" includes not only the direction the face is facing forward, but also the tilt of the face. 【0050】 The first video generation unit 103 generates a three-dimensional image of the first virtual space in which the virtual vehicle moves, based on the virtual object information and virtual vehicle information acquired by the first virtual space information acquisition unit 101. Next, the first video generation unit 103 "converts" the three-dimensional image of the first virtual space into a virtual two-dimensional image to generate a first virtual image, based on the first virtual direction information acquired by the first virtual direction information acquisition unit 102. Here, the first virtual image is the image seen by a user inside a virtual vehicle traveling within the virtual space. "Conversion" includes not only the process of generating the first virtual image using all the information before conversion, but also the process of generating the first virtual image using only a portion of the information before conversion. 【0051】 Referring to Figure 7, the image generated by the first image generation unit 103 will be described. Figure 7(a) schematically shows the three-dimensional image of the first virtual space generated by the first image generation unit 103. As shown in Figure 7(a), the first virtual space contains virtual objects such as a tree, a preceding virtual vehicle, and a signal. Since the virtual object information representing these virtual objects includes three-dimensional coordinates, it is generated as a three-dimensional image. 【0052】Figure 7(b) schematically shows the first virtual space shown in Figure 7(a) as viewed from above. Figure 7(c) shows the orientation of the user's face within the first virtual space in the schematic diagram of Figure 7(b). Here, it is assumed that the user's face in the simulated vehicle is turned to the right relative to the front of the vehicle. Therefore, in the example in Figure 7(c), the user in the first virtual space also turns their face to the right, corresponding to the orientation of the user's face in the simulated vehicle. The image seen by the user in the virtual vehicle driving through the virtual space is a three-dimensional image within the area indicated by Area A in the first virtual space. 【0053】 Figure 7(d) is a two-dimensional image obtained by converting the three-dimensional image of Area A in Figure 7(c). In Figure 7(d), virtual objects included in Area A, such as a tree, a preceding vehicle, and a traffic light, are depicted. Note that the virtual objects shown in Figure 7(d) do not include coordinates in the depth direction. Therefore, the image shown in Figure 7(d) is a two-dimensional image. 【0054】 Returning to Figure 6, the second virtual space information acquisition unit 104 (corresponding to the "virtual position information acquisition unit") acquires virtual position information indicating a predetermined "position" in the second virtual space, which is a virtual three-dimensional space different from the first virtual space. The second virtual space is a virtual space having the same coordinate system as the coordinate system of the real space. The second virtual space will be described later in the explanation of the second virtual image generation device 200. Here, "position" may be an absolute position represented by coordinates, etc., or a relative position such as the distance from a predetermined reference. 【0055】 The second virtual space information acquisition unit 104 (corresponding to the “virtual shape information acquisition unit”) further acquires virtual shape information indicating the shape of virtual objects in the second virtual space. The shape of virtual objects is also called a mesh. The shape of virtual objects in the second virtual space indicated by the virtual shape information will be described later in the explanation of the second virtual image generation device 200. 【0056】 The virtual image deformation unit 105 deforms the first virtual image generated by the first image generation unit 103 based on the virtual position information and virtual shape information. 【0057】Figure 8 illustrates an example of the transformation process of the first virtual image by the virtual image transformation unit 105. Figure 8(a) is the first virtual image before transformation, which is the same as Figure 7(d). In contrast, Figure 8(b) is the first virtual image after transformation. In this example, the size of the virtual object indicated by the virtual shape information (i.e., the size of the rectangle shown by the solid lines in Figure 8(b)) is smaller than the size of the first virtual image shown in Figure 8(a) (i.e., the size of the rectangle shown by the dotted line in Figure 8(b)). Therefore, the virtual image transformation unit 105 transforms the first virtual image to match the shape (i.e., size) of the virtual object indicated by the virtual shape information. By transforming the first virtual image, the virtual image transformation unit 105 can make the virtual image displayed on the display screen of the virtual space display device 20 look more natural and realistic. 【0058】 The first output unit 106 outputs the first virtual image that has been transformed by the virtual image transformation unit 105. In this embodiment, the first virtual image is transformed by the virtual image transformation unit 105 in order to generate a highly realistic virtual image, but the transformation process in the virtual image transformation unit 105 may be omitted in order to reduce the processing load on the first virtual image generation device 100. In this case, the first output unit 106 outputs the first virtual image before transformation instead of the first virtual image after transformation. 【0059】 Furthermore, in order to provide a three-dimensional image with a sense of depth by utilizing the difference in the user's left and right field of view, the first image generation unit 103 may generate a first virtual image for the right eye (corresponding to the "first virtual image for the right eye") and a first virtual image for the left eye (corresponding to the "first virtual image for the left eye") as the first virtual image. 【0060】Figure 9 illustrates the first virtual images for the right and left eyes. Figure 9(a), like Figure 7(c), shows a schematic representation of the first virtual space viewed from above. However, unlike Figure 7(c), Area B represents the image seen by the user's left eye within the virtual vehicle, and Area C represents the image seen by the user's right eye. Furthermore, Figure 9(b) is the result of converting the three-dimensional image of Area B into a two-dimensional image, and Figure 9(c) is the result of converting the three-dimensional image of Area C into a two-dimensional image. By viewing the image in Figure 9(b) with the left eye and the image in Figure 9(c) with the right eye, it is possible to generate an image with a sense of depth due to the difference in the left and right fields of view. 【0061】 (2) Configuration of the second virtual image generation device 200 Next, the configuration of the second virtual image generation device 200 will be described with reference to Figure 6. The second virtual image generation device 200 includes a virtual image acquisition unit 201, a second virtual direction information acquisition unit 202, a virtual space information setting unit 203, a real object information acquisition unit 204, a second image generation unit 205, and a second output unit 206. 【0062】 The virtual image acquisition unit 201 acquires the first virtual image. Figure 6 illustrates the case where the virtual image acquisition unit 201 directly acquires the first virtual image output from the first virtual image generation device 100. However, the virtual image acquisition unit 201 may also acquire the first virtual image output from the first virtual image generation device 100 and stored in memory (not shown) located outside the simulation system 10 by reading it. 【0063】The second virtual direction information acquisition unit 202 acquires second virtual direction information indicating the "face orientation" of the user in the second virtual space. Here, the face orientation of the user in the second virtual space corresponds to the face orientation of the user in the simulated vehicle. In other words, if the user in the simulated vehicle turns their face to the right, the face orientation of the user in the second virtual space will also be to the right. The face orientation indicated by the second virtual direction information may also be expressed as a relative direction to the virtual vehicle. Note that while the first virtual direction information acquired by the first virtual direction information acquisition unit 102 of the first virtual image generation device 100 indicates the face orientation of the user in the first virtual space, the second virtual direction information acquired by the second virtual direction information acquisition unit 202 of the second virtual image generation device 200 indicates the face orientation of the user in the second virtual space. 【0064】 The virtual space information setting unit 203 sets a predetermined position of a virtual object in the second virtual space, which is a virtual three-dimensional space, based on the second virtual direction information. Here, the second virtual space is a space having the same coordinate system as the coordinate system of the real space. Therefore, the predetermined position set by the virtual space information setting unit 203 can be said to correspond to a position in the real space. In addition to the second virtual direction information, the virtual space information setting unit 203 may further set the predetermined position using real distance information acquired by the real object information acquisition unit 204, which will be described later. When using real distance information, it is desirable to set the predetermined position to be a position between a real object, which is equipment inside the vehicle, and a real object located outside the simulated vehicle. 【0065】 The virtual space information setting unit 203 further sets the "shape" of the virtual object in the second virtual space. Here, "shape" may include not only the form of the virtual object but also the size of the virtual object. 【0066】Figure 10 is a diagram illustrating the shape indicated by the virtual shape information, that is, the shape of the virtual object. Figure 10(a) shows the case where the virtual object is a planar rectangle, Figure 10(b) shows the case where it is spherical, and Figure 10(c) shows the case where it is cylindrical. The virtual objects shown in Figures 10(b) and 10(c) have spherical and cylindrical shapes that "contain" the virtual vehicle inside. Note that the shapes shown in Figure 10 are merely examples and are not limited to these shapes. Here, "contained inside" means that it is sufficient if at least a part of the virtual moving body is inside the spherical or cylindrical shape, and the entire virtual moving body does not necessarily have to be inside the spherical or cylindrical shape. 【0067】 Next, we will explain the shape indicated by the virtual shape information, that is, the size of the virtual object. As shown in Figure 10(a), if the virtual object is a planar rectangle, it may be necessary to change the size of the virtual object depending on the orientation of the user's face indicated by the second virtual direction information. In this case, the virtual space information setting unit 203 may set the shape of the virtual object based on the second virtual direction information, similar to the predetermined position. 【0068】 Figure 11 illustrates a configuration for setting the size of a virtual object according to the orientation of the user's face. Similar to Figure 3(b), Figure 11 shows a simulated vehicle and its exterior viewed from above. In Figure 11, the user's field of vision is indicated by a shaded area, and the thick lines inside the field of vision indicate the shape of the virtual object set by the virtual space information setting unit 203. As shown in Figure 11(a), when the user's face is facing forward relative to the virtual vehicle, setting the position of the virtual object between the simulated vehicle and furniture located in front of the simulated vehicle requires the virtual object to be relatively small. In contrast, as shown in Figure 11(b), when the user's face is facing to the right relative to the virtual vehicle, the size of the virtual object can be made larger compared to Figure 11(a). Thus, the virtual space information setting unit 203 may set the shape, which is the size of the virtual object, according to the orientation of the user's face. 【0069】The virtual space information setting unit 203 then outputs virtual position information indicating a predetermined set position and virtual shape information indicating a set shape to the second video generation unit 205 and the first virtual video generation device 100. 【0070】 In this embodiment, the virtual space information setting unit 203 is described as being provided in the second virtual image generation device 200. However, a configuration equivalent to the virtual space information setting unit 203 may be provided outside the second virtual image generation device 200. In this case, the second virtual image generation device 200 acquires virtual position information and virtual shape information from an external source. 【0071】 The real object information acquisition unit 204 acquires real distance information, which indicates the actual distance to a real object in real space as measured by the distance sensor 22. The real object information acquisition unit 204 also acquires video information, which indicates the actual video footage captured by the external camera 21 of a real object in real space. 【0072】 The second video generation unit 205 generates a second virtual video, which is a virtual three-dimensional video that includes virtual position information and virtual shape information in addition to the first virtual video acquired by the virtual video acquisition unit 201. 【0073】 In this case, if the virtual space display device 20 is a video see-through type device, the second image generation unit 205 further uses the real distance information and image information acquired by the real object information acquisition unit 204 to generate a second virtual image. Specifically, the second image generation unit 205 generates a second virtual image by combining a real image showing a real object whose real distance indicated by the real distance information is shorter than the distance to a predetermined position indicated by the virtual position information with the first virtual image. 【0074】If the virtual space display device 20 is an optical see-through type device, the second image generation unit 205 further uses the real distance information acquired by the real object information acquisition unit 204 to generate a second virtual image. Specifically, if the real distance indicated by the real distance information indicates that there is a real object whose distance is shorter than the distance to a predetermined position indicated by the virtual position information, the second image generation unit 205 generates the second virtual image such that the virtual image is not displayed in the part that overlaps with the real object. 【0075】 Referring to Figure 12, the image generated by the second image generation unit 205 will be described. Figure 12(a) schematically shows a composite space, which is a combination of the real space and the second virtual space, as viewed from above. In Figure 12(a), as in Figure 3(b), real objects in the real space, such as walls and furniture, are shown, and furthermore, a second virtual image X, which is a virtual three-dimensional image that includes virtual position information and virtual shape information, is superimposed on the first virtual image. In the example shown in Figure 12(a), the position indicated by the virtual position information is between the simulated vehicle and the desk and chair in the real space, and is set to the right of the front of the virtual vehicle so as to correspond to the orientation of the user's face inside the simulated vehicle. The shape indicated by the virtual shape information is a planar rectangle. 【0076】 As described above, the position indicated by the virtual location information is set between the simulated vehicle and the desk and chair. Therefore, real-world objects that are closer to the user than the position indicated by the virtual location information (e.g., the steering wheel of the simulated vehicle) will have a second virtual image generated that positions them in front of X. Conversely, real-world objects that are further away from the user than the position indicated by the virtual location information (i.e., the desk outside the simulated vehicle) will have a second virtual image generated that positions them outside of X, meaning they will be hidden by X. 【0077】 Figure 12(b) shows the image displayed on the display screen 25 of the virtual space display device 20 to which the second virtual image is output. Unlike Figure 4, real-world objects such as desks outside the simulated vehicle are obscured by X. 【0078】Although Figure 12 illustrates a case where the user is looking forward to the vehicle, this disclosure also allows for the display of a virtual image aligned with the user's face direction even when the user is looking to the left, right, or rear of the simulated vehicle. 【0079】 To compare this disclosure with the prior art, Figure 13 shows the case where a three-dimensional image of the first virtual space generated by the first image generation unit 103 is displayed on the display screen 25 of the virtual space display device 20. In the example in Figure 13, in addition to the virtual image, real objects such as furniture are visible. This is because the distance from the user to the real objects, such as furniture, is shorter than the distance from the user in the virtual vehicle to the virtual objects in the virtual space, such as the tree, traffic light, and virtual preceding vehicle. When an image like that in Figure 13 is displayed on the display screen 25, the sense of reality in the virtual space is diminished. 【0080】 When the first video generation unit 103 generates a first virtual image for the right eye and a first virtual image for the left eye as the first virtual image, the second video generation unit 205 similarly generates a second virtual image for the right eye (corresponding to the "second virtual image for the right eye") and a second virtual image for the left eye (corresponding to the "second virtual image for the left eye") as the second virtual image, based on the first virtual image. 【0081】 The second output unit 206 outputs the second virtual image generated by the second image generation unit 205 to the virtual space display device 20. 【0082】(3) Operation of the Simulation System 10 Next, the operation of the simulation system 10 will be explained with reference to Figure 14. Figure 14 not only shows the overall operation of the simulation system 10, but also the operation of the first virtual image generation device 100 and the second virtual image generation device 200 that constitute the simulation system 10. Furthermore, it shows not only the simulation method executed by the entire simulation system 10, or by each of the first virtual image generation device 100 and the second virtual image generation device 200, but also the processing procedure of the simulation program that can be executed by the entire simulation system 10, or by each of the first virtual image generation device 100 and the second virtual image generation device 200. These processes are not limited to the order shown in Figure 14. That is, the order can be changed unless there is a constraint such as a relationship where a certain step utilizes the result of the preceding step. The same applies to Figure 17 of Embodiment 2, which will be described later. 【0083】 First, the operation of the first virtual image generation device 100 will be described. The first virtual space information acquisition unit 101 acquires virtual object information indicating virtual objects in the first virtual space, and virtual vehicle information indicating the position of a virtual vehicle in the first virtual space (S101). The first virtual direction information acquisition unit 102 acquires first virtual direction information indicating the orientation of the user's face in the first virtual space (S102). The second virtual space information acquisition unit 104 acquires virtual position information indicating a predetermined position in the second virtual space, and virtual shape information indicating the shape of a virtual object in the second virtual space (S103). The first image generation unit 103 generates a first virtual image based on the virtual object information and virtual vehicle information acquired in S101, and the first virtual direction information acquired in S102 (S104). The virtual image deformation unit 105 deforms the first virtual image generated in S104 based on the virtual position information and virtual shape information acquired in S103 (S105). The first output unit 106 outputs the first virtual image that was transformed in S105 (S106). 【0084】Next, the operation of the second virtual image generation device 200 will be described. The second virtual direction information acquisition unit 202 acquires second virtual direction information indicating the orientation of the user's face in the second virtual space (S201). The virtual space information setting unit 203 sets a predetermined position and shape of a virtual object in the second virtual space based on the second virtual direction information acquired in S201 (S202). The virtual space information setting unit 203 outputs virtual position information indicating the predetermined position set in S202 and virtual shape information indicating the shape of the virtual object (S203). The real object information acquisition unit 204 acquires real distance information indicating the real distance to a real object in the real space and video information indicating the real video of a real object in the real space (S204). The virtual image acquisition unit 201 acquires the first virtual image after deformation output in S106 (S205). The second image generation unit 205 generates a second virtual image using the first virtual image after deformation acquired in S205 (S206). The second output unit 206 outputs the second virtual image generated in S206 to the virtual space display device 20 (S207). 【0085】 (4) Summary In summary, according to this embodiment, when displaying a virtual image, which is an image of a virtual mobile object moving within a virtual space, on a virtual space display device, it is possible to provide the user with the virtual image without displaying real objects located closer than the virtual objects within the virtual space. This makes it possible to provide the user with a more realistic virtual space experience. 【0086】 4. Embodiment 2 (1) Configuration of the second virtual image generation device 210 Next, the configuration of the second virtual image generation device 210 of Embodiment 2 will be described with reference to Figure 15. Note that the configuration of the first virtual image generation device 100 in this embodiment is the same as that of Embodiment 1, so its description will be omitted. In addition, the configuration of the second virtual image generation device 200 that has the same processing as Embodiment 1 will be omitted from the description, and the differences from Embodiment 1 will be explained in detail. 【0087】The second virtual image generation device 210 (corresponding to the “simulation device”) further includes, in addition to the configurations of the second virtual image generation device 200 of Embodiment 1, a gaze information acquisition unit 211, an operation information acquisition unit 212, an operation target information acquisition unit 213, a display range identification unit 214, a display necessity determination unit 215, and an image selection unit 216. 【0088】 The gaze information acquisition unit 211 acquires gaze information indicating the user's gaze direction from the eye tracking unit 24 of the virtual space display device 20. 【0089】 The operation information acquisition unit 212 acquires operation information from the simulated vehicle that indicates the user's operating status of the simulated vehicle. For example, if the user operates the steering wheel of the simulated vehicle, the operation information acquisition unit 212 acquires operation information from the simulated vehicle indicating that the user operated the steering wheel. Note that the operation information acquisition unit 212 may only acquire operation information when the user operates the simulated vehicle. 【0090】 The operation target information acquisition unit 213 (corresponding to the "target information acquisition unit") acquires operation video (corresponding to the "target video") related to the operation target unit (corresponding to the "moving target unit"), which is a specific part of the simulated vehicle operated by the user during a virtual driving event. The operation video is, for example, a video that instructs the user on how to operate the operation target unit or what operations the user should perform on the operation target unit. Examples of operation target units include the steering wheel and car navigation system of the simulated vehicle. 【0091】 The operation target information acquisition unit 213 further acquires operation target position information (corresponding to "target position information") which indicates the operation target coordinates (corresponding to "target coordinates"), which are coordinates related to the operation target unit. The operation target coordinates are coordinates in the second virtual space, that is, they correspond to coordinates in the real space. Coordinates related to the operation target unit include, for example, the coordinates of the position of the operation target unit and the coordinates of the position where the operation video is displayed. 【0092】 The second virtual direction information acquisition unit 202 (corresponding to the "virtual direction information acquisition unit") acquires second virtual direction information (corresponding to the "virtual direction information") that indicates the "face orientation" of the user in the second virtual space. 【0093】The display range identification unit 214 identifies the display range, which is the range of the second virtual space that can be displayed on the display screen 25 of the virtual space display device 20, based on the second virtual direction information acquired by the second virtual direction information acquisition unit 202. 【0094】 The display necessity determination unit 215 determines whether or not to display the operation video based on the operation information acquired by the operation information acquisition unit 212. Specifically, if the operation information indicates that the target unit has been operated, the display necessity determination unit 215 determines not to display the operation information. Conversely, if the operation information indicates that the target unit has not been operated, the display necessity determination unit 215 determines to display the operation information. It should be noted that determining whether or not to display the operation video based on the operation information includes not only making a decision based on the content of the operation information, but also making a decision based on whether or not the operation information has been acquired. 【0095】 If the display necessity determination unit 215 determines that operation information should be displayed, the display necessity determination unit 215 further determines whether or not to display guidance video prompting the user to look at the operation target, based on the distance between a predetermined coordinate included in the display range determined by the display range identification unit 214 and the operation target coordinate. Specifically, the display necessity determination unit 215 determines to display guidance video if the distance between the predetermined coordinate included in the display range and the operation target coordinate is "greater than or equal to" a predetermined distance. "Greater than or equal to" includes both cases where the same value as the comparison target is included and cases where it is not included. 【0096】 Here, the predetermined coordinates included in the display range are, for example, the coordinates of the center of the display range. Alternatively, the predetermined coordinates may be the coordinates of the direction of gaze indicated by the gaze information acquired by the gaze information acquisition unit 211. 【0097】The video selection unit 216 selects an operation video to display from among multiple operation videos when the display necessity determination unit 215 determines that an operation video should be displayed. For example, the video selection unit 216 selects an operation video to display from among multiple operation videos according to the "time" from a reference "time". Here, "time" refers to any point in the time axis, and includes not only time in the narrow sense but also cases where time is indirectly indicated, such as timers and clocks. "Time" refers to any length of time, and includes not only time in the narrow sense but also cases where time is indirectly indicated, such as the start and end times, the number of timers and clocks, the period, etc. 【0098】 As an example, the video selection unit 216 selects an operation video to display from among multiple operation videos according to the elapsed time since the time when the video selection unit 216 last selected an operation video. When selecting an operation video at this time, it is desirable to select an operation video that is emphasized in terms of size, color, etc., compared to the operation video that was selected last time. As another example, in a driving simulation, the video selection unit 216 may select an operation video to display from among multiple operation videos according to the elapsed time since an event related to the target unit occurred, or the time until the event is scheduled to occur. 【0099】 The video selection unit 216 further selects a guidance video to display from among multiple guidance videos when the display necessity determination unit 215 determines that a guidance video should be displayed. For example, the video selection unit 216 selects a guidance video to display from among multiple guidance videos according to the length of the distance between a predetermined coordinate and the coordinate of the control target. Specifically, if the distance is long, it selects a guidance video that more strongly attracts the user's attention. Then, as the user turns their face towards the control target and the length of the distance between the predetermined coordinate and the coordinate of the control target decreases, the guidance video is changed as needed by selecting a new guidance video. 【0100】The video selection unit 216 may further select a guidance video to display from among multiple guidance videos according to the "time" from a reference "time". For example, the video selection unit 216 selects a guidance video to display from among multiple guidance videos according to the elapsed time since the time when the video selection unit 216 last selected a guidance video. In this case, it is desirable to select a guidance video that is emphasized in size, color, and dimensions compared to the guidance video that was last selected. As another example, in a driving simulation, the video selection unit 216 may select a guidance video to display from among multiple guidance videos according to the elapsed time since an event related to the target unit occurred, or the time until the event is scheduled to occur. 【0101】 The video selection unit 216 appropriately selects operation videos and guidance videos according to the user's operation status on the target unit, thereby encouraging the user to direct their attention to the target unit and experience operating the simulated vehicle. 【0102】 Furthermore, if only one operation video or one guidance video has been created in advance, the video selection process by the video selection unit 216 can be omitted. 【0103】 The second video generation unit 205 (corresponding to the “video generation unit”) of this embodiment generates a second virtual video that includes virtual position information indicating coordinates included in the display range within the second virtual space identified by the display range identification unit 214, in addition to the guidance video selected by the video selection unit 216. 【0104】 The second video generation unit 205 of this embodiment further generates a second virtual video that includes operation target position information in the operation video selected by the video selection unit 216. 【0105】 The second output unit (corresponding to the "output unit") outputs the second virtual image generated by the second video generation unit 205 to the virtual space display device 20. 【0106】The second virtual image generation process of this embodiment will be explained with reference to Figure 16. All images in Figure 16 show the images displayed on the display screen 25 of the virtual space display device 20 to which the second virtual image of this embodiment is output, and the parts shown in gray are parts that are not displayed on the display screen 25. 【0107】 In the example shown in Figure 16(a), the car navigation system, which is the target of operation, is not displayed on the display screen 25 using the virtual space display device 20. Therefore, to encourage the user to look at the target of operation, a guidance video represented by three triangles is displayed in the lower left corner of the display screen 25. Figure 16(b) shows a state where only a part of the target of operation is displayed in the lower left corner of the display screen 25. Part of the operation video is displayed in the lower left corner of the display screen 25, but it may not be within the user's field of view. Therefore, as in Figure 16(a), a guidance video is displayed in the center of the display screen 25. Figure 16(c) illustrates a different guidance video than that in Figure 16(b). In the example shown in Figure 16(c), as in Figure 16(b), a guidance video represented by one triangle is displayed in the center of the display screen 25. Figure 16(d) shows a state where the operation video is displayed near the center of the display screen 25. In this state, since the operation video is likely to be within the user's field of view, the guidance video is not displayed. 【0108】 Figure 16 illustrates an example where the selected guidance video, the size of the guidance video, and the position where the guidance video is displayed are changed according to the distance between a predetermined coordinate and the coordinate of the target being operated. However, the system is not limited to this example. For example, the color of the guidance video may be changed, the guidance video may be made to blink, or the blinking frequency may be changed. The operation video may also be changed in a similar manner. 【0109】 (2) Operation of the second virtual image generation device 210 Next, the operation of the second virtual image generation device 210 of this embodiment will be described with reference to Figures 17 to 19. Note that Figure 17 is a flowchart that explains only the operations related to the second virtual image generation device 210 of this embodiment, and the same processes as the second virtual image generation device 200 of Embodiment 1 are omitted from the explanation. Each process shown in Figure 17 should be executed between S201 and S206 in Figure 14. 【0110】 The operation information acquisition unit 212 acquires operation information from the in-vehicle device 30 (S211). The display necessity determination unit 215 determines whether or not to display the operation video based on the operation information acquired in S211 (S212). Here, if the operation information indicates that the target unit has been operated, the display necessity determination unit 215 determines not to display the operation video (S212:N). On the other hand, if the operation information indicates that the target unit has not been operated, the display necessity determination unit 215 determines to display the operation video (S212:Y). The video selection unit 216 selects the operation video to display (S213). Details of the process in S213 will be described later with reference to Figure 18. The gaze information acquisition unit 211 acquires gaze information indicating the direction of the user's gaze (S214). The target unit information acquisition unit 213 acquires target unit position information indicating the coordinates of the target unit (S215). The display range identification unit 214 identifies a display range, which is the range of the second virtual space that can be displayed on the display screen 25, based on second virtual direction information indicating the orientation of the user's face in the second virtual space (S216). The display necessity determination unit 215 determines whether or not to display the guidance video (S217). Here, the display necessity determination unit 215 determines whether or not the distance between a predetermined coordinate within the display range identified in S217 and the target coordinate indicated by the target operation position information acquired in S215 is greater than or equal to a predetermined distance (S218). If the distance between the coordinates is greater than or equal to the predetermined distance (S218: Y), the display necessity determination unit 215 determines to display the guidance video. On the other hand, if the distance between the coordinates is less than or equal to the predetermined distance (S218: N), the display necessity determination unit 215 determines not to display the guidance video. The video selection unit 216 selects the guidance video to display (S219). Details of the process in S219 will be described later with reference to Figure 19. 【0111】Next, with reference to Figure 18, the process of selecting the operation video in S213 will be explained. The video selection unit 216 determines whether the time from the reference time is less than a predetermined threshold 1 (S221). If the time from the reference time is less than the predetermined threshold 1 (S221: Y), operation video 1 is selected (S222). On the other hand, if the time from the reference time is greater than or equal to the predetermined threshold 1 (S221: N), the video selection unit 216 further determines whether the time from the reference time is less than a predetermined threshold 2 (S223). If the time from the reference time is less than the predetermined threshold 2 (S223: Y), operation video 2 is selected (S224). The video selection unit 216 performs the same process from S224 onwards. Here, operation video 2 is an operation video in which the size, color, and dimensions are emphasized more than operation video 1, and it is desirable that operation video n and operation video n-1 are operation videos in which the emphasis is greater than that of operation video 2. 【0112】 Next, with reference to Figure 19, the process of selecting a guidance video in S219 will be explained. The video selection unit 216 determines whether the time from the reference time is less than a predetermined threshold 1 (S231). If the time from the reference time is less than the predetermined threshold 1 (S231: Y), guidance video 1 is selected (S232). On the other hand, if the time from the reference time is greater than or equal to the predetermined threshold 1 (S231: N), the video selection unit 216 further determines whether the time from the reference time is less than a predetermined threshold 2 (S233). If the time from the reference time is less than the predetermined threshold 2 (S233: Y), guidance video 2 is selected (S234). The video selection unit 216 performs the same process from S234 onwards. Here, guidance video 2 is a guidance video in which the size, color, and dimensions are emphasized more than guidance video 1, and it is desirable that guidance video n and guidance video n-1 are guidance videos in which the emphasis is greater than that of guidance video 2. 【0113】 Figure 19 illustrates an example of selecting guidance images based on the time elapsed from a reference time. However, instead of selecting guidance images based on the time elapsed from a reference time, or in addition to selecting guidance images based on the distance from a predetermined coordinate to the target coordinate, guidance images may also be selected based on the time elapsed from a reference time. 【0114】In the above-described embodiment 2, a configuration was described in which an operation video is displayed according to the user's operation status on the target part, and a guidance video is displayed to guide the user's gaze to the target part. However, this embodiment may also display an image on a target part of a simulated vehicle that does not require user operation, and a guidance video is displayed to guide the user to that target part. 【0115】 For example, the instrument panel, rearview mirrors, etc., of a simulated vehicle are not operated by the user. However, during a driving event, it may be desirable for the user to see these in-vehicle components. In this case, a target video related to the vehicle target (corresponding to the "moving target"), which is a specific part of the simulated vehicle that is not operated by the user, may be displayed, or a guidance video that directs the user's gaze to the vehicle target may be displayed. In this configuration, since the user does not operate the target, the display necessity determination unit 215 does not determine whether to display the target video based on operation information. Instead, the display necessity determination unit 215 may determine to display the target video while a driving event occurs in which the user should look at the vehicle target. 【0116】 (3) If the field of view of the small virtual space display device is narrow, the user may not be able to see the controllable parts, causing the user to lose the opportunity to operate the simulated vehicle, and consequently, the user may not be able to fully experience the virtual vehicle simulation. In contrast, in this embodiment, guidance videos that prompt the user to look at the controllable parts that the user operates during the vehicle driving simulation, and operation videos related to the controllable parts can be displayed in the virtual space as appropriate, making it possible to reliably provide the user with the opportunity to operate the simulated vehicle. 【0117】 (4) Modifications In the above-described embodiment 2, an embodiment was described in which a second virtual image is generated using guidance images and operation images, based on the configuration of the simulation system of embodiment 1. However, embodiment 2 does not necessarily have to be based on the configuration of embodiment 1. Therefore, in this modification, a simulation device 220 that does not have to be based on the configuration of embodiment 1 will be described. 【0118】Figure 20 shows the simulation device 220 of this modified example. Unlike the second virtual image generation device 210 shown in Figure 15, the simulation device 220 in Figure 20 is not connected to the first virtual image generation device 100. The configuration of the simulation device 220 of this modified example will be explained below, focusing on the differences from the second virtual image generation device 210. 【0119】 The simulation device 220 shown in Figure 20 includes a real object information acquisition unit 204, a gaze information acquisition unit 211, an operation information acquisition unit 212, an operation target information acquisition unit 213, a display range identification unit 214, a display necessity determination unit 215, an image selection unit 216, a virtual space information acquisition unit 221, a virtual direction information acquisition unit 222, an image generation unit 225, and an output unit 226. 【0120】 Of the various components of the simulation device 220, the configurations and processing of the real object information acquisition unit 204, the gaze information acquisition unit 211, the operation information acquisition unit 212, the operation target information acquisition unit 213, the display range identification unit 214, the display necessity determination unit 215, and the video selection unit 216 correspond to the configurations of each component of Embodiment 2. Therefore, the description of Embodiment 2 will be referenced, and a detailed explanation will be omitted. 【0121】 The virtual space information acquisition unit 221 acquires virtual object information that indicates virtual objects within the virtual space, which is a virtual three-dimensional space. The virtual object information includes the coordinates, shape, and image of the virtual object. In this modified example, the "virtual space" corresponds to the second virtual space in Embodiment 2. That is, the "virtual space" in this modified example is a virtual space having the same coordinate system as the coordinate system of real space. 【0122】 The virtual direction information acquisition unit 222 acquires virtual direction information indicating the orientation of the user's face in the virtual space. The virtual direction information in this modified example corresponds to the second virtual direction information of Embodiment 2. 【0123】 The video generation unit 225 generates a virtual image. In this modified example, the video generation unit 225 generates a virtual image using the virtual object information acquired by the virtual space information acquisition unit 221 and the operation video and guidance video selected by the video selection unit 216. 【0124】The output unit 226 outputs the virtual image generated by the image generation unit 225 to the virtual space display device 20. 【0125】 Similar to Embodiment 2, the simulation device 220 in this modified example does not necessarily have to simulate user operations. 【0126】 5. Summary The features of the simulation systems, etc., in each embodiment of this disclosure (including examples; the same applies hereinafter) have been described above. 【0127】 The terms used in each embodiment are illustrative and may be replaced with synonymous terms or terms that include synonymous functions. 【0128】 The block diagram used in describing the embodiment classifies and organizes the device configuration by function. Each block representing a function can be realized by any combination of hardware or software. Furthermore, since it represents a function, such a block diagram can also be understood as a disclosure of a method invention and a program invention that realizes said method. 【0129】 The functional blocks that can be understood as processes, flows, and methods described in each embodiment may be reordered, unless there are constraints such as a relationship where one step utilizes the results of other preceding steps. 【0130】 The terms first, second, through N (where N is an integer) used in each embodiment and claim are used to distinguish between two or more configurations or methods of the same kind, and do not imply any order or hierarchy. 【0131】 Furthermore, examples of the forms of the simulation system and virtual image generation device disclosed herein include the following: Examples of component forms include semiconductor elements, electronic circuits, modules, and microcomputers. Examples of semi-finished products include ECUs and system boards. Examples of finished products include mobile phones, smartphones, tablets, personal computers (PCs), workstations, and servers. 【0132】In addition, this disclosure can be implemented not only with dedicated hardware having the configuration and functions described in each embodiment, but also as a combination of a program for implementing this disclosure recorded on a recording medium such as memory or a hard disk, and general-purpose hardware having a dedicated or general-purpose CPU and memory capable of executing this program. 【0133】 Programs stored on non-transitional physical recording media of dedicated or general-purpose hardware (e.g., external storage devices (hard disks, USB memory, CDs / BDs, etc.), or internal storage devices (RAM, ROMs, etc.)) can also be provided to the dedicated or general-purpose hardware via the recording media, or via a communication line from a server without using the recording media. This allows for the provision of the latest functionality through program upgrades. 【0134】 The simulation system disclosed herein primarily targets vehicle simulation systems that virtually provide an automobile driving experience, but may also target simulation systems that provide a mobility experience using a mobile object other than an automobile.

Claims

1. A simulation system (10) that outputs images to a virtual space display device (20) attached to the head of a user inside a simulated mobile body, the simulation system comprising a first virtual image generation device (100) and a second virtual image generation device (200), the first virtual image generation device comprising: a first virtual space information acquisition unit (101) that acquires virtual object information indicating virtual objects in a first virtual space which is a virtual three-dimensional space, and virtual mobile body information indicating the position of a virtual mobile body corresponding to the simulated mobile body in the first virtual space; a first image generation unit (103) that generates a three-dimensional image of the first virtual space on which the virtual mobile body moves based on the virtual object information and the virtual mobile body information, and converts the three-dimensional image into a virtual two-dimensional image visible to the user inside the virtual mobile body to generate a first virtual image; and a first output unit (106) that outputs the first virtual image, the second virtual image generation device, A simulation system comprising: a virtual image acquisition unit (201) that acquires the first virtual image; a second image generation unit (205) that acquires virtual position information indicating a predetermined position in a second virtual space which is a virtual three-dimensional space having the same coordinate system as the coordinate system of the real space, and generates a second virtual image which is a virtual three-dimensional image that includes the virtual position information in the first virtual image; and a second output unit (206) that outputs the second virtual image to the virtual space display device.

2. The simulation system according to claim 1, wherein the second virtual image generation device further comprises a real object information acquisition unit (204) that acquires real distance information indicating the real distance to a real object in the real space, and the second image generation unit generates the second virtual image using the real distance information.

3. The simulation system according to claim 2, wherein the real object information acquisition unit further acquires video information showing a real image of the real object, and the second video generation unit synthesizes the real image showing the real object, whose real distance is shorter than the distance to the predetermined position, with the first virtual image to generate the second virtual image.

4. The simulation system according to claim 1, wherein the first virtual image generation device further comprises a first virtual direction information acquisition unit (102) that acquires first virtual direction information indicating the orientation of the user's face in the first virtual space corresponding to the orientation of the user's face in the simulated moving body, and the first image generation unit generates the first virtual image based on the first virtual direction information in addition to the virtual object information and the virtual moving body information.

5. The simulation system according to claim 4, wherein the second virtual image generation device further comprises: a second virtual direction information acquisition unit (202) that acquires second virtual direction information indicating the orientation of the user's face in the second virtual space corresponding to the orientation of the user's face in the simulated moving body; and a virtual space information setting unit that sets the predetermined position based on the second virtual direction information.

6. The simulation system according to claim 1, wherein the first virtual image generation device further comprises a virtual position information acquisition unit (104) for acquiring the virtual position information, and a virtual image deformation unit (105) for deforming the first virtual image based on the virtual position information, and the first output unit outputs the first virtual image deformed by the virtual image deformation unit.

7. The simulation system according to claim 1, wherein the second virtual image generation device further comprises a virtual space information setting unit (203) for setting the shape of virtual objects in the second virtual space, and the second image generation unit generates the second virtual image including the virtual position information and virtual shape information indicating the shape in addition to the first virtual image.

8. The simulation system according to claim 7, wherein the first virtual image generation device further comprises a virtual shape information acquisition unit for acquiring the virtual shape information, and a virtual image deformation unit (105) for deforming the first virtual image based on the virtual shape information, and the first output unit outputs the first virtual image deformed by the virtual image deformation unit.

9. The simulation system according to claim 7, wherein the shape indicated by the virtual shape information is spherical or cylindrical, with the virtual moving body inside.

10. The simulation system according to claim 1, wherein the first image generation unit generates a first virtual image for the right eye and a first virtual image for the left eye as the first virtual image, the second image generation unit generates a second virtual image for the right eye and a second virtual image for the left eye as the second virtual image, and the virtual space display device displays the second virtual image for the right eye on the right eye side display unit of the virtual space display device and displays the second virtual image for the left eye on the left eye side display unit of the virtual space display device.

11. A simulation method performed in a simulation system that outputs images to a virtual space display device (20) attached to the head of a user inside a simulated mobile body, comprising: acquiring virtual object information indicating virtual objects in a first virtual space which is a virtual three-dimensional space, and virtual mobile body information indicating the position of a virtual mobile body corresponding to the simulated mobile body in the first virtual space (S101); generating a three-dimensional image of the first virtual space in which the virtual mobile body moves based on the virtual object information and the virtual mobile body information; converting the three-dimensional image into a virtual two-dimensional image visible to the user inside the virtual mobile body to generate a first virtual image (S104); outputting the first virtual image (S106); acquiring the first virtual image (S205); acquiring virtual position information indicating a predetermined position in a second virtual space which is a virtual three-dimensional space having the same coordinate system as the coordinate system of the real space (S202); A simulation method comprising: generating a second virtual image, which is a virtual three-dimensional image, by including the virtual position information indicating the predetermined position in the first virtual image (S206); and outputting the second virtual image to the virtual space display device (S207).

12. A simulation program executable on a simulation device that outputs images to a virtual space display device (20) attached to the head of a user inside a simulated mobile body, the simulation program acquires (S101) virtual object information indicating virtual objects in a first virtual space which is a virtual three-dimensional space, and virtual mobile body information indicating the position of a virtual mobile body corresponding to the simulated mobile body in the first virtual space, generates a three-dimensional image of the first virtual space on which the virtual mobile body moves based on the virtual object information and the virtual mobile body information, converts the three-dimensional image into a virtual two-dimensional image visible to the user inside the virtual mobile body to generate a first virtual image (S104), outputs the first virtual image (S106), acquires the first virtual image (S205), acquires (S202) virtual position information indicating a predetermined position in a second virtual space which is a virtual three-dimensional space having the same coordinate system as the coordinate system of real space, A simulation program that causes the simulation device to perform the following processes: generating a second virtual image which is a virtual three-dimensional image that includes the virtual position information indicating the predetermined position in the first virtual image (S206); and outputting the second virtual image to the virtual space display device (S207).

Images