Environment generation method, environment generation system, environment generation device, program, and recording medium

Abstract

The present invention provides an environment generation method and the like capable of executing calibration of the predetermined parameter of a virtual object in a virtual space or a mixed reality space under an appropriate condition. In this environment generation method, a virtual space VS, avatars A1, A2, and virtual objects VO1-VO3 are generated. Further, when the calibration processing is not executed, the environment of the virtual space is generated using the virtual space VS, the avatars A1, A2, and the virtual objects VO1-VO3, and when the calibration processing is executed, the environment of the virtual space to be recognized by a user is generated using the virtual space VS, the avatars A1, A2, and the calibrated VO1-VO3.

Description

Environment generation method, environment generation system, environment generation device, program, and recording medium

[0001] The present invention relates to an environment generation method and the like for generating an environment in a virtual space and a composite reality space.

[0002] Conventionally, as an environment generation method in a virtual space, the one described in Patent Document 1 is known. This environment generation method generates an environment of a virtual space where an avatar exists, and causes the user to recognize this virtual space environment via a head-mounted device. In this environment generation method, when the expression of the avatar in the virtual space collapses due to, for example, a shift of the head-mounted device, calibration of the avatar's expression is performed so that the avatar's expression returns to the standard state.

[0003] Japanese Patent Application Laid-Open No. 2021-114036

[0004] In recent years, when generating a virtual space environment corresponding to the real space, a method of generating a virtual object in the virtual space environment corresponding to a real object in the real space has been used. In this method, there may be a state where the coordinates, posture, and shape of the real object in the real space do not match the coordinates, posture, and shape of the virtual object in the virtual space. When such a state occurs, for example, there is a risk of hindering the progress of a game when executing a game using the virtual space. For the above reasons, it is desired to perform calibration of parameters (hereinafter referred to as "predetermined parameters") such as the coordinates, posture, and shape of the virtual object under appropriate conditions.

[0005] On the other hand, according to the above conventional environment generation method, although it is possible to perform calibration of the expression of the avatar in the virtual space, there is a problem that calibration of the predetermined parameters of the virtual object cannot be performed under appropriate conditions. This problem occurs not only when generating a virtual space but also when generating a composite reality space in which a virtual object is placed in the real space.

[0006] The present invention has been made to solve the above problems and aims to provide an environment generation method that can perform calibration of predetermined parameters of virtual objects in a virtual space or mixed reality space under appropriate conditions.

[0007] To achieve the above objective, the invention according to claim 1 is an environment generation method that is executed by a computer system and generates a virtual space environment that is recognized by a user via an environment output device, comprising: a virtual space generation step in which a virtual space generation unit generates a virtual space corresponding to the real space in which the user exists; an avatar generation step in which an avatar corresponding to the user is generated in the virtual space by an avatar generation unit; a virtual object generation step in which a virtual object generation unit generates virtual objects in the virtual space corresponding to real objects existing in the real space; a user state recognition step in which a user state recognition unit recognizes the movement of the user's coordinates and changes in posture; an avatar state control step in which an avatar state control unit controls the movement of the avatar's coordinates and changes in posture based on the movement of the user's coordinates and changes in posture; and a calibration processing unit, The system is characterized by comprising: a calibration process step in which, when the execution condition is met that the coordinates of the user are located within a predetermined range determined based on the coordinates of a predetermined trigger object existing in real space, a calibration process is performed to make a first predetermined parameter, which is at least one of the coordinates, orientation, and shape of a virtual object in virtual space, correspond to a second predetermined parameter, which is at least one of the coordinates, orientation, and shape of a real object; and an environment generation step in which, if the calibration process has not been performed, the environment generation unit generates a virtual space environment for the user to recognize using the virtual space, avatar, and virtual object, and if the calibration process has been performed, the environment generation unit generates a virtual space environment for the user to recognize using the virtual space, avatar, and calibrated virtual object.

[0008] According to this environment generation method, a virtual space is generated to correspond to the real space in which the user resides, an avatar corresponding to the user is generated in the virtual space, and virtual objects are generated in the virtual space to correspond to real objects existing in the real space. Furthermore, the movement of the avatar's coordinates and changes in its posture are controlled based on the user's movement of coordinates and changes in its posture. When the execution condition is met that the user's coordinates are located within a predetermined range determined based on the coordinates of a predetermined trigger object existing in the real space, a calibration process is performed to make the first predetermined parameter of the virtual object correspond to a second predetermined parameter, which is at least one of the coordinates, posture, and shape of the real object. If the calibration process is not performed, a virtual environment for the user to perceive is generated using the virtual space, avatar, and virtual objects. If the calibration process is performed, a virtual environment for the user to perceive is generated using the virtual space, avatar, and calibrated virtual objects.

[0009] In this way, under the condition that the user's coordinates are located within a predetermined range, a calibration process is performed to make the first predetermined parameter of the virtual object correspond to the second predetermined parameter of the real object, thereby appropriately corresponding the relationship between the user and the real object to the relationship between the avatar and the virtual object. For example, it is possible to suppress the discrepancy between the timing when the user moves toward and reaches the real object and the timing when the user's avatar reaches the virtual object.

[0010] Therefore, by appropriately setting a predetermined trigger object, the calibration process for the first predetermined parameter of the virtual object can be performed under the appropriate condition that the user's coordinates are located within a predetermined range. In this specification, the "calibration process for corresponding the first predetermined parameter of the virtual object to the second predetermined parameter of the real object" is not limited to the process of matching the first predetermined parameter of the virtual object to the second predetermined parameter of the real object, but also includes the process of adjusting the first predetermined parameter of the virtual object to a state that brings it closer to the second predetermined parameter of the real object.

[0011] In the present invention, it is preferable that the trigger object is a real object.

[0012] According to this environment generation method, since the trigger object is a real object, a calibration process can be performed to map a first predetermined parameter of a virtual object to a second predetermined parameter of a real object, under appropriate conditions where the user's coordinates are located within a predetermined range determined based on the coordinates of the real object.

[0013] In the present invention, it is preferable that the user includes a first user and a second user, the trigger object is one of the first user and the second user, and the execution condition is that the coordinates of the other of the first user and the second user are located within a predetermined range determined based on the coordinates of one of the first user and the second user.

[0014] According to this environment generation method, under appropriate conditions where the coordinates of the other user (the first user and the second user) are located within a predetermined range determined based on the coordinates of one of the users (the first user and the second user), a calibration process can be performed to associate a first predetermined parameter of a virtual object with a second predetermined parameter of a real object.

[0015] In the present invention, it is preferable that the calibration process involves calibrating the first predetermined parameters of a virtual object so that they correspond to the second predetermined parameters of a real object, and that at least one of the orientation, shape, and relative position of the virtual space with respect to real space is calibrated based on the calibration of the first predetermined parameters of the virtual object.

[0016] According to this environment generation method, the first predetermined parameter of a virtual object is calibrated so that it corresponds to the second predetermined parameter of a real object, and in response, at least one of the orientation, shape, and relative position of the virtual space with respect to the real space is calibrated. As a result, when the first predetermined parameter of the virtual object is calibrated, it is possible to avoid discrepancies between the positional relationship of the avatar, the virtual object, and the virtual space and the positional relationship of the user, the real object, and the real space.

[0017] In the present invention, the real-space recognition unit further comprises the step of recognizing the orientation and shape of the real space, and preferably, in the calibration process, the first predetermined parameter of the virtual object is calibrated, and at least one of the orientation, shape and relative position of the virtual space with respect to the real space is calibrated based on the orientation and shape of the real space.

[0018] According to this environment generation method, when the first predetermined parameters of a virtual object are calibrated, at least one of the orientation, shape, and relative position of the virtual space to the real space can be calibrated in accordance with this calibration and the orientation and shape of the real space. As a result, when the first predetermined parameters of a virtual object are calibrated, it is possible to more accurately avoid discrepancies between the positional relationship of the avatar, the virtual object, and the virtual space and the positional relationship of the user, real objects, and the real space.

[0019] Furthermore, in this specification, "at least one of the orientation, shape, and relative position of the virtual space is calibrated" means that at least one of the orientation and shape of the virtual space is calibrated to match at least one of the orientation and shape of the real space, and / or that the relative position of the virtual space to the real space is calibrated to match a position in the real space, and / or that at least one of the orientation and shape of the virtual space is adjusted to approach at least one of the orientation and shape of the real space, and / or that the relative position of the virtual space to the real space is adjusted to approach a position in the real space.

[0020] In the present invention, during the calibration process step, if the user's coordinates are moving towards the coordinates of a predetermined trigger object, it is preferable that the predetermined range is determined to be larger the greater the speed at which the user's coordinates move.

[0021] According to this environment generation method, when the user's coordinates are moving towards a predetermined trigger object's coordinates, the predetermined range is determined to be larger the faster the user's coordinates move. As a result, if the user is moving at a high speed towards the predetermined trigger object, and the avatar is moving at a high speed, the relationship between the user and the real object can be quickly matched to the relationship between the avatar and the virtual object.

[0022] To achieve the aforementioned objectives, another environment generation method of the present invention is an environment generation method that is executed by a computer system and generates an environment of a mixed reality space that is perceived by a user via an environment outputter, comprising: a reality space recognition step in which a reality space recognition unit recognizes the orientation and shape of the reality space; a virtual object generation step in which a virtual object generation unit generates virtual objects so as to correspond to real objects existing in the reality space; a user state recognition step in which a user state recognition unit recognizes the user's coordinates and orientation; a calibration processing step in which a calibration processing unit performs a calibration process to make a first predetermined parameter, which is at least one of the coordinates, orientation and shape of a virtual object in the mixed reality space, correspond to a second predetermined parameter, which is at least one of the coordinates, orientation and shape of a real object, when the execution condition is met that the user's coordinates are located within a predetermined range determined based on the coordinates of a predetermined trigger object existing in the reality space; and an environment generation step in which, if the calibration process has not been executed, the environment generation unit generates an environment of a mixed reality space that is perceived by a user using the reality space and virtual objects, and if the calibration process has been executed, the environment generation unit generates an environment of a mixed reality space that is perceived by a user using the reality space and calibrated virtual objects.

[0023] According to this environment generation method, the real space is recognized, and virtual objects are generated so that they correspond to real objects existing in the real space. Furthermore, if the execution condition is met that the user's coordinates are located within a predetermined range determined based on the coordinates of a predetermined trigger object existing in the real space, a calibration process is performed to make a first predetermined parameter, which is at least one of the coordinates, orientation, and shape of the virtual object in the mixed reality space, correspond to a second predetermined parameter, which is at least one of the coordinates, orientation, and shape of the real object. If the calibration process is not performed, a mixed reality environment for the user to perceive is generated using the real space and virtual objects. If the calibration process is performed, a mixed reality environment for the user to perceive is generated using the real space and the calibrated virtual objects.

[0024] In this way, under the condition that the user's coordinates are located within a predetermined range, a calibration process is performed so that the first predetermined parameter of the virtual object corresponds to the second predetermined parameter of the real object, thereby making the relationship between the user and the real object correspond to the relationship between the user and the virtual object. For example, it is possible to suppress the discrepancy between the timing when the user moves toward the real object and reaches the real object, and the timing when the user reaches the virtual object. Therefore, by appropriately setting a predetermined trigger object, the calibration of the first predetermined parameter of the virtual object can be performed under the appropriate condition that the user's coordinates are located within a predetermined range.

[0025] To achieve the aforementioned objectives, the environment generation system of the present invention is characterized by being configured to perform any of the above-described environment generation methods.

[0026] To achieve the aforementioned objectives, the environment generation device of the present invention is characterized by being configured to perform any of the above-described environment generation methods.

[0027] To achieve the aforementioned objectives, the program of the present invention is characterized by causing a computer system to execute any of the environment generation methods described above.

[0028] To achieve the aforementioned objectives, the recording medium of the present invention is characterized by recording the above-mentioned program and making the program readable by a computer system.

[0029] This is a perspective view showing the configuration of an environment generation system and the real space, etc., for executing the environment generation method according to the first embodiment of the present invention. This is a block diagram showing the functional configuration of the server. This is a perspective view showing the virtual space environment generated in correspondence with the real space environment in Figure 1. This is a flowchart showing the control processing executed by the server. This is a plan view showing an example of the state of the real space. This is a plan view showing the state of the virtual space generated in correspondence with the real space in Figure 5. This is a plan view showing the state of the real space when the execution conditions for the calibration process are met. This is a plan view showing the state of the virtual space generated in correspondence with the real space in Figure 7 when the calibration process is executed. This is a block diagram showing the functional configuration of the server in the environment generation system according to the second embodiment of the present invention. This is a flowchart showing the control processing executed by the server of the second embodiment. This is a plan view showing the state of the mixed reality space before the execution of the calibration process. This is a plan view showing the state of the mixed reality space after the execution of the calibration process. Forms for carrying out the invention

[0030] The environment generation method and environment generation system that implement the first embodiment of the present invention will be described below with reference to the drawings.

[0031] As shown in Figure 1, the environment generation system S of this embodiment (hereinafter referred to as "System S") is intended to allow a first user U1 and a second user U2, who are both present in a predetermined area of ​​the real space RS (for example, a room), to experience the virtual space VS. In the following description, the first user U1 and the second user U2 will be collectively referred to as "User U".

[0032] Here, "virtual space" refers to a virtual space that the user perceives in place of the real world in which they exist; it is a so-called virtual reality space (VR space), and virtual objects, avatars that move in response to the user's actions, and so on are placed in this virtual space VS.

[0033] In this embodiment, for the sake of ease of understanding, the users are assumed to be two people, a first user U1 and a second user U2. However, the number of users who use the environment generation system of the present invention is not limited to this configuration; it may be one person or three or more people.

[0034] First, the general configuration of System S will be explained with reference to Figure 1. As shown in Figure 1, System S comprises a plurality of signs 1 attached to User U in the real space RS, a camera 2 that photographs User U (more precisely, the signs 1 attached to User U), a server 3 that determines the environment of the virtual space VS that User U will experience, and a head-mounted display 4 (hereinafter referred to as "HMD4") that allows the user to recognize the determined environment.

[0035] In this embodiment, the server 3 corresponds to a computer system, and the head-mounted display 4 corresponds to an environmental output device.

[0036] In System S, the camera 2, server 3, and HMD 4 are capable of wirelessly sending and receiving information from each other via the Internet, public network, short-range wireless communication, etc. However, they may also be configured to send and receive information from each other via wired connections.

[0037] Multiple markers 1 are attached to the user U's head, both hands, and both feet via the HMD 4, gloves, and shoes worn by the user U. The multiple markers 1 are used to recognize the user U's movements in the real-world RS, as will be described later. Therefore, the position and number of markers 1 attached may be changed as appropriate depending on the other devices constituting the system S.

[0038] Camera 2 is installed so as to be able to photograph from multiple directions the range in which user U can move (i.e., the range in which user U can move and act) within the real space RS in which user U exists.

[0039] Server 3 recognizes (e.g., detects or calculates) the marker 1 from the image captured by camera 2, and recognizes the user U's actions (e.g., changes in posture and coordinates) based on the position of the recognized marker 1 in the real space RS. Server 3 also determines the virtual space VS environment that user U should perceive based on these actions.

[0040] The HMD4 is worn on the user U's head and outputs the virtual environment VS (e.g., images and sounds) to the user for recognition. As shown in Figure 2, the HMD4 has a monitor 40 and a speaker 41. The monitor 40 is configured to allow the user U to recognize images of the virtual environment VS determined by the server 3, and the speaker 41 is configured to allow the user U to recognize sounds of the virtual environment VS determined by the server 3.

[0041] When user U experiences the virtual space VS using system S, user U recognizes that they are present in the virtual space VS by perceiving only the images and sounds of the virtual space VS through the HMD4. In other words, system S is configured as a so-called immersive system.

[0042] Furthermore, the environment generation system of the present invention is not limited to the configuration using signs and cameras as described above (so-called motion capture device), but is not limited to any configuration that can recognize the real space and the user's movements.

[0043] Therefore, for example, a configuration in which the number and arrangement of markers and cameras differ from that shown in Figure 1 may be used. Specifically, in order to recognize at least one of the feature points of the user and the feature points of the real space, markers may be attached not only to the user but also to real objects that exist in the real space. Alternatively, feature points may be recognized from the image itself without using markers.

[0044] Further, for example, instead of the identifier and the camera, the HMD may be equipped with sensors such as GPS, and the user's actions may be recognized based on the output from the sensors. Also, such sensors and the motion capture device as described above may be used in combination.

[0045] Further, the environment generation system of the present invention is not limited to being constituted by one server, and it may be configured such that any of the devices constituting the environment generation system includes the processing unit described later.

[0046] Therefore, for example, the entire environment generation system may be constituted by a plurality of servers. Also, at least one of the processing units or at least a part of the functions of the processing unit may be implemented in the camera, the HMD, or other devices, and the environment generation system may be constituted by the cooperation of these devices and the server or by only these devices.

[0047] Next, the configuration of the system S will be described. The system S is constituted by the camera 2, the server 3, and the HMD 4, and these camera 2, server 3, and HMD 4 include one or more electronic circuit units including a CPU, a RAM, a ROM, an interface circuit, and the like.

[0048] As shown in FIG. 2, the server 3 has functions as a real space recognition unit 30, an environment generation unit 31, a user state recognition unit 32, and an avatar state control unit 33, and these functions are realized by at least one of the hardware configuration and the program implemented in the server 3.

[0049] The real space recognition unit 30 recognizes the image data of the real space RS captured by the camera 2, and based on the image data, recognizes the situation of the real space RS. The situation of the real space RS is, for example, the posture and coordinates of the user U and the real objects (in this embodiment, the box RO1, the bookshelf RO2, and the drawer RO3 shown in FIG. 1) existing in the real space RS. In the following description, the box RO1, the bookshelf RO2, and the drawer RO3 are collectively referred to as "real object RO".

[0050] The environment generation unit 31 generates a virtual environment VS that is perceived by the user U via the monitor 40 and speaker 41 of the HMD 4. In this case, the "environment perceived" by the user U refers to the virtual space VS environment that the user U experiences through their five senses. For example, this environment is composed of images of virtual objects that exist in the virtual space VS perceived by the user U, sounds generated based on those virtual objects, and so on.

[0051] The environment generation unit 31 includes a virtual space generation unit 31a, an avatar generation unit 31b, a virtual object generation unit 31c, and a calibration processing unit 31d.

[0052] Based on the recognition results of the real-world space recognition unit 30, the virtual space generation unit 31a generates a virtual space VS corresponding to the real-world space RS (see Figure 1) where user U exists, as shown in Figure 3. Specifically, the virtual space generation unit 31a generates the background of the virtual space VS, images that will become avatars in the virtual space VS, and sounds associated with those images.

[0053] Although this embodiment does not include system S, if the environment generation system includes a configuration that realizes a predetermined feel (for example, a cushion that changes hardness), a configuration that generates a predetermined scent, etc., the virtual space generation unit may generate the virtual space VS using the feel and scent in addition to images and sounds.

[0054] Here, as shown in Figure 3, the shape of the virtual space VS generated by the virtual space generation unit 31a is configured to be generated according to the shape of the real space RS. Here, "shape of the real space" is recognized based on the shape of the region where the virtual space VS is generated, and the coordinates, orientation, and shape of real objects RO that exist in that real space.

[0055] Furthermore, in an immersive system like this embodiment, the "shape of the virtual space VS" refers to the area where the avatar corresponding to the user can operate. Specifically, it is the area defined by the background image of the virtual space VS and the virtual objects placed in the virtual space VS.

[0056] The avatar generation unit 31b generates an avatar to be placed in the virtual space VS based on the posture and coordinates of the user U recognized by the user state recognition unit 32, which will be described later.

[0057] The avatar includes a first avatar A1 corresponding to the first user U1, and a second avatar A2 corresponding to the second user U2 (see Figure 3). The first avatar A1 and the second avatar A2 operate in the virtual space VS in accordance with the actions of the corresponding user U in the real space RS. Hereinafter, the first avatar A1 and the second avatar A2 will be collectively referred to as "Avatar A".

[0058] In this embodiment, when the user U experiences the virtual space VS using system S, the user U recognizes only the images and sounds of the virtual space VS and is made to believe that user U is present in the virtual space VS. In other words, system S is configured as a so-called immersive system.

[0059] The virtual object generation unit 31c generates virtual objects (in this embodiment, the treasure chest VO1, pillar VO2, and rock wall VO3 shown in Figure 3) that are to be recognized by the user U, corresponding to the coordinates, orientation, and shape of the real object RO. In the following description, the treasure chest VO1, pillar VO2, and rock wall VO3 are collectively referred to as "virtual object VO".

[0060] The calibration processing unit 31d performs a calibration process when the execution conditions are met, as described below. The execution conditions for this calibration process are that at least one of the following conditions (c1) and (c2) is met.

[0061] (c1) The coordinates of at least one of the first user U1 and the second user U2 are located within a first predetermined range X1 (the range shown by dashed lines in Figures 5 and 7) set based on the coordinates of box RO1. (c2) The coordinates of the first user U1 are located within a second predetermined range X2 (the range shown by dashed lines in Figures 5 and 7) set based on the coordinates of the second user U2.

[0062] In this case, the coordinates of the first user U1, the second user U2, and the box RO1 are set as two-dimensional coordinates with the floor plane of the real space as the coordinate plane. Furthermore, the first predetermined range X1 is determined so that the faster the movement of the coordinates of the first user U1 or the second user U2, the larger the range becomes, as the first user U1 or the second user U2 moves so that their coordinates approach the coordinates of the box RO1.

[0063] Furthermore, in the case of condition (c2) above, the second predetermined range X2 is set based on the coordinates of the second user U2. Alternatively, the second predetermined range X2 may be set based on the coordinates of the first user U1. In that case, the condition "the coordinates of the second user U2 are located within the second predetermined range X2" can be used instead of condition (c2).

[0064] In the calibration process, if the above execution conditions are met, that is, if at least one of conditions (c1) and (c2) is met, the calibration value is first calculated by, for example, the method described below. Specifically, multiple points where the positions of the real space RS and the virtual space VS have been calibrated are set as multiple calibration points in the real space RS, and the position sensed by a sensing device (e.g., VR goggles) and the calibration value at each calibration point are stored in the server 3 in advance.

[0065] Then, based on the point distance, which is the distance between user U and each calibration point, a weighted calculation is performed on the calibration value of each calibration point and the default calibration value, thereby calculating the calibration value.

[0066] Next, using these calibration values, the first predetermined parameters of the virtual object VO are calibrated so that the first predetermined parameters, which are the coordinates, orientation, and shape of the virtual object VO, correspond to the second predetermined parameters, which are the coordinates, orientation, and shape of the real object RO. Furthermore, based on the calibration of the first predetermined parameters and the orientation and shape of the real space RS, the orientation, shape, and relative position of the entire virtual space VS with respect to the real space RS are calibrated. For example, the virtual space VS is calibrated from the state shown in Figure 6 (described later) to the state shown in Figure 8 (described later).

[0067] Furthermore, in the calibration process, the first predetermined parameter may be set to at least one of the coordinates, orientation, and shape of the virtual object VO, and the second predetermined parameter may be set to at least one of the coordinates, orientation, and shape of the real object RO. In this case, it is sufficient that the first predetermined parameter and the second predetermined parameter are set to correspond to each other. For example, if the first predetermined parameter is set to the coordinates of the virtual object VO, the second predetermined parameter may be set to the coordinates of the real object RO.

[0068] Furthermore, the calibration process may be configured such that at least one of the orientation, shape, and relative position of the entire virtual space VS with respect to the real space RS is calibrated based on the calibration of the first predetermined parameter and the orientation and shape of the real space RS, or at least one of the orientation, shape, and relative position of the entire virtual space VS with respect to the real space RS is calibrated based on the calibration of the first predetermined parameter.

[0069] Furthermore, the first predetermined range X1 and the second predetermined range X2 are not limited to circular ranges as shown in Figures 5 and 7, but may also be elliptical ranges, polygonal ranges, or ranges enclosed by curves.

[0070] In the environment generation unit 31, if the calibration process described above has not been performed, the virtual environment VS is generated using the virtual object VO, avatar A, and virtual space VS generated as described above. On the other hand, if the calibration process described above has been performed, the virtual environment VS is generated using the calibrated virtual object VO, calibrated virtual space VS, and avatar A.

[0071] In addition, the calibration process described above may be configured to perform only the calibration of the first predetermined parameter of the virtual object VO. In this case, the environment generation unit 31 should be configured to generate the virtual environment VS using the calibrated virtual object VO, virtual space VS, and avatar A.

[0072] The user state recognition unit 32 recognizes image data of user U captured by camera 2 and recognizes the state of user U in real space RS based on that image data, and includes a user posture recognition unit 32a and a user coordinate recognition unit 32b. Here, the state of user U in real space RS refers to user U's posture and coordinates, and by extension, user U's actions, which are indicated by the amount of change therein.

[0073] The user posture recognition unit 32a extracts feature points of the user U's body from the input user U image data, and recognizes the user U's posture in the real space RS based on the extraction results.

[0074] The user coordinate recognition unit 32b recognizes the coordinates of user U in the real space RS based on the extraction results of feature points such as user U's body extracted from the input user U image data, and the situation of the real space RS recognized by the real space recognition unit 30 (for example, the coordinates of real objects).

[0075] The avatar state control unit 33 controls the state of avatar A in the virtual space VS corresponding to user U, based on the state of user U in the real space RS (i.e., posture and coordinates) recognized by the user state recognition unit 32.

[0076] Next, referring to Figure 4, we will explain the control process performed by the server 3 of system S to allow user U to experience the virtual space VS.

[0077] In this process, first, a real-space recognition process is performed (Figure 4 / STEP 1). In this real-space recognition process, the posture and shape of the real-space RS in which user U is located are recognized (see Figure 1). In this embodiment, the real-space recognition process corresponds to the real-space recognition step.

[0078] Next, the virtual space generation process is executed (Figure 4 / STEP 2). In this virtual space generation process, the virtual space VS (see Figure 3) is generated based on the orientation and shape of the real space RS. In this embodiment, the virtual space generation process corresponds to the virtual space generation step.

[0079] Next, the avatar generation process is executed (Figure 4 / STEP 3). In this avatar generation process, avatar A corresponding to user U is generated in the virtual space VS. In this embodiment, the avatar generation process corresponds to the avatar generation step.

[0080] After the avatar generation process is executed, the virtual object generation process is performed (Figure 4 / STEP 4). In this virtual object generation process, the virtual object VO is generated in the virtual space VS so that it corresponds to the real object R0. In this embodiment, the virtual object generation process corresponds to the virtual object generation step.

[0081] Next, user state recognition processing is performed (Figure 4 / STEP 5). In this user state recognition processing, the movement of user U's coordinates and changes in posture are recognized. Specifically, a marker 1 is recognized from the image captured by camera 2, and based on the position of the recognized marker 1 in real space RS, the movement of user U's coordinates and changes in posture are recognized. In this embodiment, the user state recognition processing corresponds to the user state recognition step.

[0082] After the user state recognition process is executed, the avatar state control process is performed (Figure 4 / STEP 6). In this avatar state recognition process, the movement of the coordinates and changes in the posture of avatar A corresponding to user U are controlled based on the movement of the coordinates and changes in the posture of user U recognized by the user state recognition process described above. In this embodiment, the avatar state control process corresponds to the avatar state control step.

[0083] Next, it is determined whether the conditions for executing the calibration process are met (Figure 4 / STEP 7). In this determination, if at least one of the aforementioned conditions (c1) and (c2) is met, it is determined that the conditions for executing the calibration process are met; otherwise, it is determined that the conditions for executing the calibration process are not met.

[0084] If this determination is negative (Figure 4 / STEP 7...NO) and the conditions for executing the calibration process are not met, proceed to STEP 9, which will be described later. On the other hand, if this determination is positive (Figure 4 / STEP 7...YES) and the conditions for executing the calibration process are met, the calibration process will be executed (Figure 4 / STEP 8).

[0085] In this calibration process, calibration values ​​are calculated using the method described above, and these calibration values ​​are used to calibrate the first predetermined parameter of the virtual object VO so that it corresponds to the second predetermined parameter of the real object.

[0086] Simultaneously, the orientation, shape, and relative position of the entire virtual space VS with respect to the real space RS are calibrated based on the calibration of the first predetermined parameter and the orientation and shape of the real space RS. In this embodiment, the calibration process corresponds to the calibration step.

[0087] After the calibration process is executed, or if the calibration process is not executed because the conditions for execution of the calibration process are not met, the virtual environment generation process is executed (Figure 4 / STEP 9). In this virtual environment generation process, if the calibration process is not executed, the virtual space VS, avatar A, and virtual object VO generated as described above are used to create the virtual space VS environment for recognition by user U. On the other hand, if the calibration process is executed, the virtual space VS environment is created using the calibrated virtual space VS, avatar A, and calibrated virtual object VO. In this embodiment, the virtual environment generation process corresponds to the virtual environment generation step.

[0088] Next, the virtual environment output process is executed (Figure 4 / STEP 10). In this virtual environment output process, a signal is output to the HMD4 to allow user U to recognize the virtual space VS environment that has been generated as described above. As a result, the virtual space VS environment that has been generated as described above is recognized by user U.

[0089] Next, with reference to Figures 5 to 8, the effects of the calibration process when the above control processes are executed will be explained. For example, suppose that the environment of the real space RS is in the state shown in Figure 5, and the environment of the virtual space VS that user U is experiencing through the HMD4 is in the state shown in Figure 6. In the virtual space VS environment shown in Figure 6, the attitude of the virtual object VO relative to avatar A is out of sync with the attitude of the real object RO relative to user U.

[0090] When this posture discrepancy occurs, and the first user U1 moves from the position shown in Figure 5 to the position shown in Figure 7, and the coordinates of the first user U1 are located within the first predetermined range X1, the calibration process described above is executed, and as shown in Figure 8, the posture of the virtual object VO relative to avatar A in the virtual space VS environment is calibrated to match the posture of the real object RO relative to user U.

[0091] In other words, by performing a calibration process, for example, the orientation of a real object RO relative to user U can be made to match the orientation of a virtual object VO relative to avatar A. Simultaneously, the orientation, shape, and relative position of the entire virtual space VS with respect to the real space RS are calibrated. As a result, in the environment of the virtual space VS, the positional relationship of avatar A relative to the virtual object VO and the virtual space VS can be made to match the positional relationship of the real object RO and the real space RS with respect to user U.

[0092] As described above, according to the environment generation method of the first embodiment, the virtual space VS is generated to correspond to the real space RS in which user U exists, an avatar A corresponding to user U is generated in the virtual space VS, and a virtual object VO is generated to correspond to the real object RO existing in the real space RS. Furthermore, the movement of the coordinates and changes in the posture of avatar A are controlled based on the movement of the coordinates and changes in the posture of user U.

[0093] Then, when the conditions for executing the calibration process are met, that is, when at least one of the aforementioned conditions (c1) and (c2) is met, the calibration process is executed. In this calibration process, the first predetermined parameter of the virtual object VO is calibrated so that it corresponds to the second predetermined parameter of the real object RO, and the posture, shape, and relative position of the entire virtual space VS with respect to the real space RS are calibrated based on the calibration of the first predetermined parameter and the posture and shape of the real space RS. This prevents user U from feeling any discomfort with the positional relationship between the virtual object VO and the virtual space VS with respect to avatar A when the first predetermined parameter of the virtual object VO is calibrated.

[0094] Furthermore, the first predetermined range X1 in condition (c1) is determined such that, when the first user U1 or the second user U2 is moving so that their coordinates approach the coordinates of box RO1, the range becomes larger the faster the speed at which the coordinates of the first user U1 or the second user U2 move. As a result, if the speed at which user U moves toward the real object RO1 is high, and the speed at which avatar A moves toward the virtual object VO1 is high, the relationship between user U and the real object RO1 can be quickly mapped to the relationship between avatar A and the virtual object VO1.

[0095] In the first embodiment, the case in which the environment generation method is executed by a single computer system was described, but the present invention is not limited to a computer system; any system for executing the environment generation method of the present invention is acceptable.

[0096] Therefore, for example, the environment generation device may be composed of one computer (server 3) as described in the first embodiment above. Alternatively, it may be a calibration program for executing the aforementioned environment generation method on any one or more computers, and a recording medium that stores the program and allows the program to be read by a computer used by a user or the like.

[0097] Next, an environment generation method and an environment generation system that implements the same according to the second embodiment of the present invention will be described. In the case of the environment generation system of this embodiment, the mechanical and electrical configurations are the same as those of the environment generation system S of the first embodiment, and only the functional configuration and control processing content of the server 3 are different. Therefore, the following description will focus on the differences. Also, the same reference numerals are used for components that are the same as in the first embodiment, and their descriptions will be omitted.

[0098] In this embodiment, the environment generation system Sx shown in Figure 9 (hereinafter referred to as "system Sx") is used to allow user U to experience the mixed reality space MRS (see Figures 11 and 12). This mixed reality space MRS corresponds to a space in which real objects of the real space RS are superimposed on virtual objects. In this embodiment, box RO1 in Figure 5 is a real object, and treasure chest VO1 in Figures 11 and 12 is a virtual object. In the following description, box RO1 will be referred to as "real object RO1," and treasure chest VO1 will be referred to as "virtual object VO1."

[0099] As shown in Figure 9, the server 3 of this embodiment is equipped with the functions of a real-space recognition unit 30, a user state recognition unit 32, and an environment generation unit 35, and these functions are realized by at least one of the hardware configuration and program implemented in the server 3.

[0100] As described above, the real-space recognition unit 30 recognizes the state of the real-space RS, and as described above, the user state recognition unit 32 recognizes the posture and coordinates of the user U in the real-space RS.

[0101] Furthermore, the environment generation unit 35 generates the environment of the mixed reality space MRS that is perceived by the user U via the monitor 40 and speaker 41 of the HMD4, and includes a virtual object generation unit 35a and a calibration processing unit 35b.

[0102] In the virtual object generation unit 35a, a virtual object VO1 (see Figures 11 and 12) is generated to be recognized by the user U, corresponding to the coordinates, orientation, and shape of the real object RO1 (see Figure 5).

[0103] Furthermore, the calibration processing unit 35b executes the calibration process when the aforementioned execution conditions are met, that is, when at least one of the aforementioned conditions (c1) and (c2) is met. In this calibration process, a calibration value is calculated using the method described above, and the first predetermined parameter of the virtual object VO1 is calibrated using this calibration value.

[0104] In the environment generation unit 35, if the calibration process described above is not performed, the environment of the mixed reality space MRS is generated using the real space RS and the virtual object VO1 generated as described above. On the other hand, if the calibration process described above is performed, the environment of the mixed reality space MRS is generated using the calibrated virtual object VO1.

[0105] Next, referring to Figure 10, we will explain the control process performed by the server 3 of system S to allow user U to experience the mixed reality space MRS.

[0106] In this process, first, a real-space recognition process is performed (Figure 10 / STEP 21). In this real-space recognition process, the posture and shape of the real-space RS in which user U is located are recognized (see Figure 1). In this embodiment, the real-space recognition process corresponds to the real-space recognition step.

[0107] Next, the virtual object generation process is executed (Figure 10 / STEP 22). In this virtual object generation process, the virtual object VO1 is generated so that it corresponds to the real object R01. In this embodiment, the virtual object generation process corresponds to the virtual object generation step.

[0108] Next, the user state recognition process is executed (Figure 10 / STEP 23). In this user state recognition process, the movement of user U's coordinates and changes in posture are recognized using the method described above. In this embodiment, the user state recognition process corresponds to the user state recognition step.

[0109] Subsequently, it is determined whether or not the conditions for executing the calibration process are met (Figure 10 / STEP 24). In this determination, if at least one of the aforementioned conditions (c1) and (c2) is met, it is determined that the conditions for executing the calibration process are met; otherwise, it is determined that the conditions for executing the calibration process are not met.

[0110] If this determination is negative (Figure 10 / STEP 24...NO) and the conditions for executing the calibration process are not met, proceed to STEP 26, which will be described later. On the other hand, if this determination is positive (Figure 10 / STEP 24...YES) and the conditions for executing the calibration process are met, the calibration process will be executed (Figure 10 / STEP 25).

[0111] In this calibration process, calibration values ​​are calculated using the method described above, and these calibration values ​​are used to calibrate the first predetermined parameter of the virtual object VO1 so that it corresponds to the second predetermined parameter of the real object RO1. In this embodiment, the calibration process corresponds to the calibration step.

[0112] After the calibration process is executed, or if the conditions for executing the calibration process are not met, the environment generation process is executed (Figure 10 / STEP 26). In this environment generation process, if the calibration process described above was not executed, the environment of the mixed reality space MRS for user U to recognize is generated using the recognition results of the virtual object VO1 and the real space RS generated as described above. On the other hand, if the calibration process is executed, the environment of the mixed reality space MRS is generated using the recognition results of the calibrated virtual object VO1 and the real space RS. In this embodiment, the environment generation process corresponds to the environment generation step.

[0113] Next, the environment output process is executed (Figure 10 / STEP 27). In this environment output process, a signal is output to the HMD4 to allow user U to recognize the environment of the mixed reality space MRS that has been generated as described above.

[0114] As mentioned above, the monitor 40 mounted on the HMD4 is configured to allow observation of the real-world space RS through the monitor 40. Therefore, when the above signal is output to the HMD4, the HMD4 allows the user U to perceive the real-world space RS through the monitor 40 and displays the virtual object VO1 superimposed on the real-world space RS. Simultaneously, sound is generated from the speaker 41 mounted on the HMD4, thereby creating the environment of the mixed-reality space MRS. As a result, the user U can perceive the environment of the mixed-reality space MRS that has been generated as described above.

[0115] Next, with reference to Figures 5, 7, and 11-12 mentioned above, the effects of the calibration process when the above control process is executed will be explained. For example, suppose that the environment of the real space RS is in the state shown in Figure 5, and the environment of the mixed reality space MRS that user U experiences through the HMD4 is in the state shown in Figure 11. In the mixed reality space MRS environment shown in Figure 11, the orientation of the virtual object VO relative to user U is misaligned with the orientation of the real object RO relative to user U.

[0116] When this pose discrepancy occurs, and the first user U1 moves from the position shown in Figure 5 to the position shown in Figure 7, and the coordinates of the first user U1 are located within the first predetermined range X1, the calibration process described above is executed, and as shown in Figure 12, the pose of the virtual object VO relative to user U is calibrated to match the pose of the real object RO relative to user U in the mixed reality space MRS environment.

[0117] In other words, by performing a calibration process, for example, the orientation of the real object RO1 relative to user U can be made to match the orientation of the virtual object VO1 relative to user U.

[0118] As described above, according to the environment generation method of the second embodiment, the real space RS is recognized, and the virtual object VO1 is generated so as to correspond to the real object RO1 that exists in the real space RS. Furthermore, if the execution conditions described above are met, that is, if at least one of conditions (c1) and (c2) is met, a calibration process is performed to make the first predetermined parameter of the virtual object VO1 correspond to the second predetermined parameter of the real object RO1. If the execution conditions are not met, an environment of the mixed reality space MRS to be recognized by the user U is generated using the real space RS and the virtual object VO1. If the execution conditions are met, an environment of the mixed reality space MRS to be recognized by the user U is generated using the real space RS and the calibrated virtual object VO1.

[0119] Thus, under conditions where the execution conditions for the calibration process are met, the calibration process is performed so that the first predetermined parameter of the virtual object VO1 corresponds to the second predetermined parameter of the real object RO1. This makes it possible to make the relationship between user U and real object RO1 correspond to the relationship between user U and virtual object VO1. For example, it is possible to suppress the discrepancy between the timing when user U moves toward real object RO1 and reaches real object RO1, and the timing when user U reaches virtual object VO1. Therefore, the calibration of the first predetermined parameter of virtual object VO1 can be performed under appropriate conditions where at least one of conditions (c1) and (c2) is met.

[0120] S Environment Generation System 3 Server (Computer System) 30 Real Space Recognition Unit 31 Environment Generation Unit 31a Virtual Space Generation Unit 31b Avatar Generation Unit 31c Virtual Object Generation Unit 31d Calibration Processing Unit 32 User State Recognition Unit 33 Avatar State Control Unit 4 Head-Mounted Display (Environment Output Device) VS Virtual Space RS Real Space A1 First Avatar (Avatar) A2 Second Avatar (Avatar) U1 First User (User) U2 Second User (User) VO1 Treasure Chest (Virtual Object) VO2 Rock Wall (Virtual Object) VO3 Pillar (Virtual Object) RO1 Box (Real Object) RO2 Bookshelf (Real Object) RO3 Drawer (Real Object) X1 First Determined Range (Determined Range) X2 Second Determined Range (Determined Range) Sx Environment Generation System 35 Environment Generation Unit 35a Virtual object generation unit 35b Calibration processing unit

Claims

1. An environment generation method for generating a virtual space environment that is executed by a computer system and made recognizable to a user via an environment output device, comprising: a virtual space generation step in which a virtual space generation unit generates the virtual space to correspond to the real space in which the user resides; an avatar generation step in which an avatar generation unit generates an avatar corresponding to the user in the virtual space; a virtual object generation step in which a virtual object generation unit generates virtual objects in the virtual space to correspond to real objects existing in the real space; a user state recognition step in which a user state recognition unit recognizes the movement of the user's coordinates and changes in their posture; and an avatar state control step in which an avatar state control unit controls the movement of the avatar's coordinates and changes in their posture based on the movement of the user's coordinates and changes in their posture. An environment generation method comprising: a calibration processing step in which, when the execution condition is met that the user's coordinates are located within a predetermined range determined based on the coordinates of a predetermined trigger object existing in the real space, a calibration processing step in which a calibration processing unit performs a calibration process to make a first predetermined parameter, which is at least one of the coordinates, orientation, and shape of a virtual object in the virtual space, correspond to a second predetermined parameter, which is at least one of the coordinates, orientation, and shape of a real object; and an environment generation step in which, if the calibration processing has not been performed, an environment generation unit generates the environment of the virtual space to be recognized by the user using the virtual space, the avatar, and the virtual object, and if the calibration processing has been performed, an environment generation unit generates the environment of the virtual space to be recognized by the user using the virtual space, the avatar, and the calibrated virtual object.

2. The environment generation method according to claim 1, characterized in that the trigger object is the existing object.

3. The environment generation method according to claim 1, wherein the user includes a first user and a second user, the trigger object is one of the first user and the second user, and the execution condition is that the coordinates of the other of the first user and the second user are located within the predetermined range determined based on the coordinates of one of the first user and the second user.

4. The environment generation method according to claim 1, wherein in the calibration process, the first predetermined parameter of the virtual object is calibrated so that the first predetermined parameter of the virtual object corresponds to the second predetermined parameter of the real object, and at least one of the orientation, shape and relative position of the virtual space with respect to the real space is calibrated based on the calibration of the first predetermined parameter of the virtual object, and in the environment generation step, if the calibration process is not performed, the environment of the virtual space to be recognized by the user is generated using the virtual space, the avatar and the virtual object, and if the calibration process is performed, the environment of the virtual space to be recognized by the user is generated using the calibrated virtual space, the avatar and the calibrated virtual object.

5. The environment generation method according to claim 4, further comprising a real space recognition step in which a real space recognition unit recognizes the orientation and shape of the real space, wherein the calibration process is characterized in that at least one of the orientation, shape and relative position of the virtual space with respect to the real space is calibrated based on the calibration of the first predetermined parameter of the virtual object and the orientation and shape of the real space.

6. The environment generation method according to claim 1, characterized in that, in the calibration processing step, when the user's coordinates are moving closer to the coordinates of the predetermined trigger object, the predetermined range is determined to be larger the greater the speed at which the user's coordinates move.

7. An environment generation method for generating a mixed reality environment that is executed by a computer system and made recognizable to a user via an environment outputter, comprising: a real-space recognition step in which a real-space recognition unit recognizes the orientation and shape of the real space; a virtual object generation step in which a virtual object generation unit generates virtual objects corresponding to real objects existing in the real space; a user state recognition step in which a user state recognition unit recognizes the coordinates and orientation of the user; and a calibration processing step in which a calibration processing unit performs a calibration process to make a first predetermined parameter, which is at least one of the coordinates, orientation and shape of a virtual object in the mixed reality space, correspond to a second predetermined parameter, which is at least one of the coordinates, orientation and shape of a real object, when the execution condition is met that the user's coordinates are located within a predetermined range determined based on the coordinates of a predetermined trigger object existing in the real space. An environment generation method characterized by comprising: an environment generation step in which, if the calibration process has not been performed, the environment generation unit generates the environment of the mixed reality space to be recognized by the user using the real space and the virtual objects; and if the calibration process has been performed, the environment generation unit generates the environment of the mixed reality space to be recognized by the user using the real space and the calibrated virtual objects.

8. An environment generation system characterized by being configured to perform the environment generation method described in any one of claims 1 to 7.

9. An environment generating device characterized by being configured to perform the environment generating method described in any one of claims 1 to 7.

10. A program characterized by causing a computer system to execute the environment generation method described in any one of claims 1 to 7.

11. A recording medium characterized by recording the program described in claim 10, and the program being readable by the computer system.

Images