Information processing device, information processing method, and computer program

Abstract

An information processing device (100) comprises: an acquiring unit (121) that acquires image information representing a two-dimensional image; a determining unit (122) that determines a reference object to serve as a reference from among one or more image objects included in the two-dimensional image represented by the acquired image information, and, on the basis of the dimensions of the determined reference object, determines display information representing a display region in which the two-dimensional image is displayed in a virtual space; and an output unit (123) that outputs the determined display information.

Description

Information Processing Apparatus, Information Processing Method, and Computer Program

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a computer program.

[0002] Patent Document 1 discloses a technique of superimposing a generated virtual image and a captured image (captured image) captured by a camera in the real space.

[0003] Japanese Patent Application Laid-Open No. 2019-212062

[0004] However, the technique disclosed in Patent Document 1 cannot provide a highly immersive viewing experience.

[0005] Therefore, the present disclosure provides an information processing apparatus, an information processing method, and a computer program that can provide a highly immersive viewing experience.

[0006] An information processing apparatus according to an aspect of the present disclosure includes an acquisition unit that acquires image information indicating a two-dimensional image, and among one or more image objects included in the two-dimensional image indicated by the acquired image information, a reference object to be used as a reference is determined, and based on the dimensions of the determined reference object, a display information indicating a display area in which the two-dimensional image is displayed in a virtual space is determined, and an output unit that outputs the determined display information.

[0007] An information processing method according to an aspect of the present disclosure is an information processing method executed by an information processing apparatus, including acquiring image information indicating a two-dimensional image, determining a reference object to be used as a reference among one or more image objects included in the two-dimensional image indicated by the acquired image information, determining display information indicating a display area in which the two-dimensional image is displayed in a virtual space based on the dimensions of the determined reference object, and outputting the determined display information.

[0008] A computer program according to an aspect of the present disclosure is a computer program for causing a computer to execute the above information processing method.

[0009] These comprehensive or specific embodiments may be implemented as a system, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM (Compact Disc-Read Only Memory), or as any combination of a system, method, integrated circuit, computer program, and recording medium.

[0010] According to one aspect of this disclosure, it is possible to realize an information processing device, an information processing method, and a computer program that can provide a highly immersive viewing experience.

[0011] Figure 1 is a block diagram showing the functional configuration of the information processing system according to this embodiment. Figure 2 is a hardware configuration diagram of the information processing device according to this embodiment. Figure 3 is a flowchart showing an example of the operation of the information processing device according to this embodiment. Figure 4A is a flowchart detailing step S22 of the example of the operation of the information processing device according to this embodiment. Figure 4B is a flowchart showing other details of step S22 of the example of the operation of the information processing device according to this embodiment. Figure 5 is a diagram showing an example of processing in the first display method according to this embodiment. Figure 6 is a diagram for explaining the positional relationship between the user and the 2D image in the virtual space in Figure 5(b). Figure 7 is a flowchart detailing step S20 when the first processing is performed in the example of the operation of the information processing device according to this embodiment. Figure 8 is a diagram showing an example of image augmentation processing according to this embodiment. Figure 9 is a flowchart detailing step S20 when the second processing is performed in the example of the operation of the information processing device according to this embodiment. Figure 10 is a diagram showing an example of conversion processing according to this embodiment. Figure 10 shows the screen of the display unit. Figure 11 is a flowchart detailing step S20 when the third processing is performed in the example of the operation of the information processing device according to this embodiment.

[0012] (Knowledge forming the basis of this disclosure) In recent years, services provided through virtual spaces using computer simulations have attracted attention.

[0013] A virtual space can be represented, for example, by three-dimensional shape data that shows the shape of the terrain of the space or objects such as buildings placed within the space (spatial objects). Furthermore, the background of a virtual space realized by VR (Virtual Reality) (hereinafter sometimes referred to as VR virtual space) is represented by a 360-degree image (also called a spherical image). In a VR virtual space, the user (viewer) can see the scenery in all directions regardless of which direction they are facing, enabling an immersive viewing experience. Additionally, an immersive viewing experience is also possible through AR (Augmented Reality), which merges the real and virtual spaces by displaying real-world images as background images. Thus, virtual spaces are realized by XR (Extended Reality), which includes VR, AR, and MR (Mixed Reality).

[0014] One example of a technology that provides a virtual space to a user is the technology disclosed in Patent Document 1. Patent Document 1 discloses a conventional technology that superimposes a generated virtual image with an image captured by a camera in the real world.

[0015] Incidentally, in such conventional technologies, as mentioned above, 360-degree images are commonly used as background images for VR virtual spaces, which are an example of virtual images.

[0016] Such 360-degree images can be acquired using a 360-degree camera. One example of a 360-degree image format is the equirectangular projection. In this format, the image captured by the 360-degree camera is converted into a rectangle with an aspect ratio of 1:2 to generate the 360-degree image.

[0017] Other formats include, for example, cubic map format, spherical format, cylindrical map format, or equidistant azimuthal projection format.

[0018] Here, we consider the case where a user views an image different from the background image of a VR virtual space (for example, a 2D image) when a VR virtual space is provided to the user. Conventionally, when a user views a 2D image using VR goggles, for example, the user selects one 2D image from multiple images stored in the VR goggles' photo app, and the selected 2D image is displayed in the VR virtual space.

[0019] In the VR display area shown on the display unit of a device such as VR goggles, a browser application that displays, for example, a selected 2D image is superimposed on the background image of the VR virtual space, and the 2D image is displayed on the browser application. In other words, the selected 2D image is superimposed on the background image.

[0020] In this case, while the user can see the 2D image they wish to view, the 2D image is placed as part of the background image of the VR virtual space. For example, in this case, the 2D image is placed in a predetermined position and of a predetermined size within the background image of the VR virtual space. Depending on the position where the 2D image is displayed, the distance between the user and the 2D image, or the size of the 2D image, the user may feel uncomfortable, or the 2D image and the background image may not harmonize. In these cases, the user may not be able to have a highly immersive viewing experience. Furthermore, in these cases, the user may experience motion sickness, among other issues.

[0021] Therefore, it is necessary to provide a highly immersive viewing experience.

[0022] Therefore, the information processing device according to the first aspect of the present disclosure includes: an acquisition unit that acquires image information showing a two-dimensional image; a determination unit that determines a reference object from among one or more image objects included in the two-dimensional image shown by the acquired image information, and determines display information indicating a display area in which the two-dimensional image is displayed in a virtual space based on the dimensions of the determined reference object; and an output unit that outputs the determined display information.

[0023] This allows, for example, display information to be determined that indicates a display area where the dimensions of a reference object in virtual space are displayed at a size equal to the dimensions of the reference object in real space. As a result, users can view a two-dimensional image including the reference object without feeling any discomfort, enabling a viewing experience as if the reference object were in the same space as the user. In other words, an information processing device that can provide a highly immersive viewing experience is realized.

[0024] For example, in the information processing apparatus according to the second aspect of the present disclosure, in the information processing apparatus according to the first aspect, the acquisition unit acquires object information indicating a spatial object displayed in the virtual space, and the determination unit determines the display information indicating the display area in which the two-dimensional image is displayed in the virtual space, based on the dimensions of the determined reference object and the dimensions of the spatial object indicated by the acquired object information.

[0025] This allows, for example, display information to be determined that indicates a display area where the dimensions of a reference object in virtual space are displayed at a size equal to the dimensions of the reference object in real space. As a result, users can view a two-dimensional image including the reference object without feeling any sense of incongruity. Furthermore, the use of object information makes it easier for the dimensions of spatial objects and the dimensions of the reference object to match the dimensional relationship in real space, so users are less likely to feel any incongruity regarding the relationship between the apparent size of the reference object and the surrounding environment such as spatial objects. Consequently, users can have a viewing experience as if the reference object were in the same space as them. In other words, an information processing device that can provide a highly immersive viewing experience is realized.

[0026] For example, in the information processing apparatus according to the third aspect of this disclosure, in the information processing apparatus according to the first or second aspect, the determination unit determines the display information in which the center coordinates of the reference object are the center of the display area.

[0027] This allows the reference object to be placed at the center of the display area, enabling users to become more immersed in the virtual space. In other words, it realizes an information processing device that can provide a more realistic viewing experience.

[0028] For example, in the information processing apparatus according to the fourth aspect of the present disclosure, in the information processing apparatus according to any one of the first to third aspects, the determination unit determines the display information that indicates the display area that shows the display dimensions and display position of the two-dimensional image in the virtual space.

[0029] This allows the dimensions of reference objects in virtual space to be displayed at a size closer to the dimensions of reference objects in real space. As a result, users can view 2D images including reference objects without feeling any discomfort, enabling a viewing experience as if the reference objects were in the same space as the user. In other words, an information processing device that can provide a more immersive viewing experience is realized.

[0030] For example, in the information processing apparatus according to the fifth aspect of the present disclosure, in the information processing apparatus according to any one of the first to fourth aspects, the determination unit performs image augmentation processing on the two-dimensional image and determines the display information indicating the display area in which the two-dimensional image that has undergone image augmentation processing is displayed in the virtual space.

[0031] As a result, for example, if a reference object is magnified and photographed, an image augmentation process is applied to a 2D image in which the background, objects or buildings around the reference object are not captured. Therefore, the user can view the augmented 2D image, for example, a 2D image in which the surrounding area is magnified, or a 2D image in which a background has been generated in the surrounding area. In other words, an information processing device is realized that can provide a more immersive viewing experience.

[0032] For example, in the information processing apparatus according to the sixth aspect of the present disclosure, in the information processing apparatus according to any one of the first to fourth aspects, the determination unit performs a conversion process to convert the two-dimensional image into a 360-degree image, and determines the display information indicating the display area in which the converted 360-degree image is displayed in the virtual space.

[0033] This allows users to view 360-degree images as background images in the virtual space, rather than 2D images with frames. In other words, it enables the realization of an information processing device that can provide a more immersive viewing experience.

[0034] For example, in the information processing apparatus according to the seventh aspect of the present disclosure, in the information processing apparatus according to any one of the first to fourth aspects, the determination unit determines the display information indicating the display area in which the three-dimensional model shown by the three-dimensional data based on the two-dimensional image is displayed in the virtual space.

[0035] This allows users to view a three-dimensional model displayed on a background image in a virtual space. In other words, it enables the realization of an information processing device that can provide a more immersive viewing experience.

[0036] For example, in the information processing apparatus according to the eighth aspect of the present disclosure, in the information processing apparatus according to the seventh aspect, the determination unit performs a generation process to generate the three-dimensional data based on the two-dimensional image, and determines the display information that indicates the display area in which the three-dimensional model shown by the generated three-dimensional data is displayed in the virtual space.

[0037] This enables the realization of an information processing device that can display a three-dimensional model represented by three-dimensional data it has generated itself.

[0038] An information processing method according to the ninth aspect of this disclosure is an information processing method performed by an information processing device, which acquires image information showing a two-dimensional image, determines a reference object from among one or more image objects included in the two-dimensional image shown by the acquired image information, determines display information indicating a display area in which the two-dimensional image is displayed in a virtual space based on the dimensions of the determined reference object, and outputs the determined display information.

[0039] This allows, for example, display information to be determined that indicates a display area where the dimensions of a reference object in virtual space are displayed at a size equal to the dimensions of the reference object in real space. As a result, users can view a 2D image including the reference object without feeling any discomfort, enabling a viewing experience as if the reference object were in the same space as the user. In other words, an information processing method is realized that can provide a highly immersive viewing experience.

[0040] The computer program relating to the tenth aspect of this disclosure is a computer program that causes a computer to execute the information processing method relating to the ninth aspect.

[0041] This allows a computer to execute the above information processing method according to a computer program.

[0042] These comprehensive or specific embodiments may be implemented as a system, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, method, integrated circuit, computer program, or recording medium.

[0043] The embodiments will be described in detail below with reference to the drawings.

[0044] Note that all the embodiments described below show comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, the components not described in the independent claims are described as optional components.

[0045] Note that each figure is a schematic diagram and is not necessarily drawn precisely. Also, in each figure, the same reference numerals are given to substantially the same configurations, and redundant descriptions may be omitted or simplified.

[0046] Also, in this specification, ordinal numbers such as "first" and "second" do not mean the number or order of components unless otherwise specified, and are used for the purpose of avoiding confusion and distinguishing between the same type of components.

[0047] (Embodiment) [Configuration] Hereinafter, the information processing system 1000 according to the present embodiment will be described.

[0048] First, the information processing system 1000 will be described with reference to FIG. 1.

[0049] FIG. 1 is a block diagram showing the functional configuration of the information processing system 1000 according to the present embodiment.

[0050] The information processing system 1000 is a system including an information processing device 100, a camera 200, and a server 300.

[0051] The information processing device 100 is a device used to provide a virtual space for the user. When the virtual space is provided to the user, the information processing device 100 can cause the user to view an image (for example, a two-dimensional image) different from the background image of the virtual space. The information processing device 100 can display the image superimposed on the background image.

[0052] The information processing device 100 comprises a communication unit 110, an information processing unit 120, a display unit 130, a reception unit 140, a sensor unit 150, and a storage unit 160. The information processing device 100 may also be connected to the camera 200 and the server 300, respectively.

[0053] In Figure 1, an example is shown in which a single device, the information processing device 100, is equipped with functional units such as a communication unit 110, an information processing unit 120, a display unit 130, a reception unit 140, a sensor unit 150, and a storage unit 160, but the invention is not limited to this. In other words, each functional unit may be provided in multiple devices. For example, the information processing device 100 may be equipped with an image processing device that handles image processing functions and an image display device that handles image display, with the image processing device and the image display device each sharing the processing of functional units such as the communication unit 110. Alternatively, the image processing device, the image display device, and the server 300 may perform some or all of the processing performed by the functional units of the information processing device 100.

[0054] Image processing devices include, for example, PCs (Personal Computers), smartphones, tablets, or game consoles. Image display devices are devices that display images in the user's field of view. Image display devices include, for example, head-mounted displays (HMDs) and AR glasses, which are worn so that the display is positioned in front of the user's eyes, and are often shaped like goggles or glasses, but are not limited to these. Furthermore, image display devices are typically used to provide XR experiences, including VR, AR, and MR.

[0055] In this embodiment, the information processing device 100 is an image display device, and more specifically, a head-mounted display.

[0056] Next, the functional units of the information processing device 100, such as the communication unit 110, information processing unit 120, display unit 130, reception unit 140, sensor unit 150, and storage unit 160, will be described. These functional units consist of a CPU, GPU, memory, or communication interface, etc.

[0057] The communication unit 110 is a processing unit that enables the information processing device 100 to communicate with the camera 200, the server 300, and other devices. The communication unit 110 performs communication using, for example, a wide-area communication network such as the Internet. The communication performed by the communication unit 110 may be wireless communication, but it may also be wired communication. There are no particular limitations on the communication standards used for communication.

[0058] The information processing unit 120 is a processing unit that, when a virtual space is provided to the user, performs processing to allow the user to view an image different from the background image of the virtual space. The information processing unit 120 includes an acquisition unit 121, a determination unit 122, and an output unit 123.

[0059] The acquisition unit 121 acquires image information. The acquisition unit 121 may also acquire object information. For example, the acquisition unit 121 acquires both image information and object information stored in the storage unit 160. Here, we will first explain the object information.

[0060] Object information is information that indicates a spatial object displayed in the virtual space. For example, object information is information that indicates a spatial object displayed in the background image of the virtual space. The background image of the virtual space may be a 360-degree image. In this case, the spatial object is an object displayed in the background image (i.e., the 360-degree image). Furthermore, the background image is not limited to an image acquired by a 360-degree camera, but may also be a pass-through image of the real world acquired by a camera mounted on the HMD.

[0061] Spatial objects may be natural objects such as trees, mountains, seas, or rivers displayed in the background image of the virtual space, or they may be artificial objects such as buildings, vehicles, or household appliances. Furthermore, spatial objects may also be animals or people. Here, "spatial object" is a general term for objects included in the background image of the virtual space, and also includes objects such as trees, buildings, people, and other objects that actually exist in the real world and are displayed in the virtual space as pass-through images of the HMD.

[0062] Object information includes dimension data indicating the dimensions of the spatial object. The dimensions of the spatial object indicated by the dimension data are the dimensions of the spatial object in real space. These dimensions indicate the height or width of the spatial object in real space, etc. The dimension data may include, but is not limited to, the overall dimensions of the spatial object. The dimension data may include the dimensions of a part of the spatial object, or, if the spatial object is divided into multiple sub-objects, it may include the overall dimensions of each of the multiple sub-objects.

[0063] Object information may include placement data that indicates the arrangement of spatial objects. Placement data is data that shows the relative or absolute positional relationship between the user and spatial objects within the virtual space. More specifically, placement data is data that shows where a spatial object is positioned relative to the user within the virtual space.

[0064] Object information may be information that represents the entire background image of the virtual space. For example, object information is information used to display the background image of the virtual space on the display unit 130. The display unit 130 acquires object information and displays the background image in the virtual space based on the acquired object information. In other words, object information is image information. Also, when a background image is displayed in the virtual space, spatial objects are naturally also displayed. Object information may include, for example, the three-dimensional shape data mentioned above. That is, object information may include data that indicates at least one of the shape, color, and arrangement of the spatial object.

[0065] Next, I will explain the image information.

[0066] Image information is information that represents a two-dimensional image, and more specifically, information that represents a two-dimensional image displayed in a virtual space. The image information should preferably include the scene depicted in the two-dimensional image. The scene depicted in the two-dimensional image is not particularly limited, but could include, for example, landscapes, portraits, macros, sports, fashion, food, or architecture.

[0067] A two-dimensional image contains one or more objects. For example, if the scene depicted in a two-dimensional image is a portrait, one example of one or more objects is the person included in the two-dimensional image. Another example of one or more objects may be trees in a landscape, people playing sports or sports equipment (e.g., a ball), clothing, food, or buildings.

[0068] The two-dimensional image shown in the image information may include attribute information. This attribute information is information about the camera that took the two-dimensional image (for example, camera 200). The attribute information may include the camera's manufacturer, model, shutter speed, sensitivity, angle of view, focal length, image sensor size, or shooting distance. The attribute information may also include information indicating the location where the two-dimensional image was taken (GPS information), or the date the two-dimensional image was taken.

[0069] The determination unit 122 determines a reference object from among one or more image objects included in the two-dimensional image indicated by the image information acquired by the acquisition unit 121. Then, based on the dimensions of the determined reference object and the dimensions of the spatial object indicated by the object information acquired by the acquisition unit 121, the determination unit 122 determines display information that indicates the display area in which the two-dimensional image will be displayed in the virtual space.

[0070] The display area indicates the display dimensions and position of a 2D image in virtual space. In other words, the display area shows how large and where the 2D image is positioned and displayed in virtual space. The display position of the 2D image may also be indicated by the distance between the user and the 2D image. Display information is information that indicates such a display area. Furthermore, the display area indicates the position of the virtual screen on which the 2D image is placed in virtual space.

[0071] As described above, the method in which the determination unit 122 determines the display information based on the dimensions of the reference object, and the method in which the determination unit 122 determines the display information based on the dimensions of the reference object and the dimensions of the spatial object, may be described as the first display method.

[0072] Furthermore, as explained above in (the knowledge on which this disclosure is based), a method in which a two-dimensional image is displayed such that it is placed at a predetermined position and of a predetermined size in a background image of a virtual space may be described as a second display method. In other words, in the second display method, the determination unit 122 does not determine the display information based on the dimensions of the reference object and the dimensions of the spatial object. It is preferable that the predetermined position information indicating the predetermined position and the predetermined size information indicating the predetermined size are stored in the storage unit 160.

[0073] Furthermore, in the first display method, the determination unit 122 may perform any one of the following first, second, and third processes.

[0074] For example, the first process is the following image augmentation process. The determination unit 122 applies image augmentation processing (first process) to the two-dimensional image indicated by the image information acquired by the acquisition unit 121. In this case, the determination unit 122 determines display information that indicates the display area in which the two-dimensional image that has undergone image augmentation processing (first process) is displayed in the virtual space.

[0075] For example, the second process is the following conversion process. The determination unit 122 may perform a conversion process (second process) to convert the two-dimensional image indicated by the image information acquired by the acquisition unit 121 into a 360-degree image. In this case, the determination unit 122 determines display information indicating the display area in which the 360-degree image that has undergone the conversion process (second process) will be displayed in the virtual space.

[0076] As mentioned above, two-dimensional images and 360-degree images are preferably used, but in this embodiment, the determination unit 122 may determine display information indicating a display area in which the three-dimensional model shown by the three-dimensional data based on the two-dimensional image is displayed in the virtual space. That is, for example, the third process is the following generation process. The determination unit 122 performs a generation process (third process) to generate three-dimensional data based on the two-dimensional image shown by the image information acquired by the acquisition unit 121. In this case, the determination unit 122 determines display information indicating a display area in which the three-dimensional model shown by the three-dimensional data that has undergone the generation process (third process) is displayed in the virtual space. In this case, the display area indicates the display dimensions and display area of ​​the three-dimensional model in the virtual space.

[0077] When the first to third processes are performed, the decision unit 122 may decide to acquire necessary additional data. In this case, according to the decision of the decision unit 122, the communication unit 110 acquires the necessary additional data from the server 300 or website and outputs the acquired additional data to the information processing unit 120 or storage unit 160. The first to third processes may also be collectively referred to as the creation process. As described above, the creation process is performed to determine the display information indicating the display area.

[0078] The output unit 123 outputs the display information determined by the determination unit 122. The output unit 123 also outputs the image information acquired by the acquisition unit 121. The output unit 123 outputs the display information and image information to the display unit 130.

[0079] The display unit 130 displays an image. The display unit 130 is implemented by a display panel such as a liquid crystal panel or an organic EL (Electroluminescence) panel. The display unit 130 acquires object information from the storage unit 160 and displays a background image of the virtual space based on the acquired object information. That is, the display unit 130 applies image processing to the object information and displays the processed video signal to the user as a visual representation.

[0080] Furthermore, the display unit 130 acquires the display information and image information output from the output unit 123 (information processing unit 120). The display unit 130 overlays the two-dimensional image indicated by the image information onto the background image of the virtual space, according to the display area indicated by the display information. In other words, the display unit 130 places the reference object indicated by the two-dimensional image in the virtual space (background image) according to the display area indicated by the display information and outputs (displays) it.

[0081] The display unit 130 may, for example, have one display, or it may have two displays, each positioned to correspond to one of the user's eyes.

[0082] The reception unit 140 is a processing unit that receives operations from the user. The reception unit 140 may receive a first operation from the user that instructs how to display a two-dimensional image. The first operation indicates which of the first display method and the second display method was instructed. The reception unit 140 may output first operation information indicating the received first operation to the information processing unit 120. That is, the first operation information is information indicating which of the first display method and the second display method was instructed.

[0083] Furthermore, the reception unit 140 may receive a second operation from the user instructing whether or not to perform the creation process. The second operation indicates whether or not to perform the creation process, and if so, which of the first to third processes to perform. The reception unit 140 may output second operation information indicating the received second operation to the information processing unit 120. That is, the second operation information indicates whether or not to perform the creation process, and if so, which of the first to third processes to perform.

[0084] The sensor unit 150 is, for example, an IMU (Inertial Measurement Unit), an accelerometer, a magnetometer, a depth sensor, or a gyroscope, and is a device for estimating the user's position and orientation. Specifically, the sensor unit 150 may, for example, detect the amount of rotation of the user's head, with at least one of the three mutually orthogonal axes in the video space as the rotation axis, or it may detect the amount of displacement with at least one of the three axes as the displacement direction.

[0085] The sensor unit 150 may estimate at least one of the user's position and orientation based on the detection result of one or more of the position, orientation, movement, velocity, angular velocity, and acceleration of part or all of the user's body, and generate position / orientation information (hereinafter sometimes simply referred to as position information) that indicates at least one of the user's position and orientation. The position / orientation information may be information that indicates at least one of the user's position and orientation in real space, or it may be information that indicates the displacement of at least one of the user's position and orientation relative to at least one of the user's position and orientation at a predetermined point in time.

[0086] The sensor unit 150 may be, for example, a camera or other imaging device, or a LiDAR (Laser Imaging Detection and Ranging) or other distance measuring device. The sensor unit 150 may detect the user's head movement by capturing images of the user's head movement and processing the captured images. Alternatively, a device that performs position estimation using wireless communication in an arbitrary frequency band, such as millimeter waves, may be used as the sensor unit 150. The user's position may be detected by a GPS (Global Positioning System) receiver or the like. Position information obtained by performing self-position estimation using LiDAR or the like as the sensor unit 150 may also be used. For example, if the information processing device 100 is implemented as a smartphone, the sensor unit 150 is built into the smartphone.

[0087] Furthermore, the sensor unit 150 may include a temperature sensor such as a thermocouple for detecting the temperature of the information processing device 100. The sensor unit 150 may also include a sensor for detecting the remaining charge of the battery in the information processing device 100, or a battery connected to the information processing device 100.

[0088] The memory unit 160 is a storage device that stores control programs used for information processing, such as determining display information by the information processing unit 120, and various types of information used in said information processing. The memory unit 160 stores object information and image information. In addition, the memory unit 160 stores multiple image information, and each of the multiple image information displays one two-dimensional image. In other words, the memory unit 160 stores multiple two-dimensional images.

[0089] Camera 200 is a device that captures two-dimensional images. The two-dimensional image captured by camera 200 may be a two-dimensional image included in image information. In this case, when camera 200 captures an image, camera 200 outputs image information indicating the two-dimensional image to information processing device 100, the communication unit 110 of information processing device 100 acquires the outputted image information, and the storage unit 160 stores the acquired image information.

[0090] Camera 200 may record video, or it may capture a 360-degree image or a depth image showing the distance (depth) of each pixel in the image from the camera.

[0091] The server 300 is a device installed in a remote location away from where the information processing device 100 is installed, and is, for example, a cloud server. The server 300 may have a storage unit such as an HDD or semiconductor memory, and may store information used for processing performed by the information processing device 100.

[0092] Figure 2 is a hardware configuration diagram of the information processing device 100 according to this embodiment.

[0093] The information processing device 100 in Figure 2 comprises a processor 401, a memory 402, a communication interface 403, and an input device 404. The information processing device 100 may also include a display (not shown) and sensors (not shown).

[0094] The processor 401 is, for example, a CPU, a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit). The CPU, DSP, or GPU may execute a program stored in the memory 402 to realize the processing performed by the information processing unit 120. Alternatively, the processor 401 may be, for example, a circuit that performs information processing. The processor 401 may also be a dedicated circuit that performs the processing performed by the information processing unit 120.

[0095] The memory 402 is composed of, for example, RAM (Random Access Memory) or ROM (Read-Only Memory). The memory 402 may also include a magnetic recording medium such as a hard disk or a semiconductor memory such as an SSD. Alternatively, the memory 402 may be internal memory built into the CPU or GPU.

[0096] The communication interface 403 is, for example, a communication module compatible with communication methods such as Bluetooth® or WiGig®. The information processing device 100 communicates with other communication devices via the communication interface 403 and obtains the bitstream to be decoded. The obtained bitstream is stored in, for example, the memory 402.

[0097] The communication IF403 consists, for example, of a signal processing circuit and an antenna corresponding to the communication method. The communication method is not limited to Bluetooth® and WiGig®, but may also be LTE (Long-Term Evolution), NR (New Radio), or Wi-Fi®, etc.

[0098] Furthermore, the communication method is not limited to the wireless communication method described above. The communication method may also be a wired communication method such as Ethernet®, USB (Universal Serial Bus), or HDMI® (High Definition Multimedia Interface).

[0099] The input device 404 is an interface for receiving user input. The input device 404 may be a pointing device such as a mouse, touchpad, touch panel, or trackball, or it may be a keyboard.

[0100] The communication unit 110 is implemented, for example, by a communication IF 403. The information processing unit 120 is implemented, for example, by a processor 401 and a memory 402. The display unit 130 is implemented, for example, by a display (not shown). The reception unit 140 is implemented by an input device 404. The sensor unit 150 is implemented, for example, by a sensor (not shown). The storage unit 160 is implemented, for example, by a memory 402.

[0101] The following describes an example of the operation of the information processing method performed by the information processing device 100.

[0102] [Operation Example] Figure 3 is a flowchart of an operation example of the information processing device 100 according to this embodiment.

[0103] The acquisition unit 121 of the information processing device 100 acquires image information. The acquisition unit 121 may also acquire object information as needed. More specifically, the acquisition unit 121 acquires object information, image information, first operation information, second operation information, and position information (S10).

[0104] It should be noted that the acquisition unit 121 does not need to acquire object information, image information, first operation information, second operation information, and position information simultaneously. For example, the acquisition unit 121 may first acquire object information from the storage unit 160, and while the display unit 130 is displaying the background image of the virtual space based on the acquired object information, the acquisition unit 121 may acquire image information, first operation information, second operation information, and position information.

[0105] The location information is generated by the sensor unit 150, the generated location information is output to the communication unit 110, the communication unit 110 acquires the outputted location information, and the acquisition unit 121 acquires the location information acquired by the communication unit 110.

[0106] The procedure for the acquisition unit 121 to acquire image information will be explained further. While the display unit 130 is displaying a background image of the virtual space, the user selects one two-dimensional image from multiple images (two-dimensional images) stored in the photo application of the information processing device 100. That is, the user performs a selection operation to select one two-dimensional image from multiple images by operating the reception unit 140. The reception unit 140 receives the selection operation and outputs selection information indicating the two-dimensional image indicated by the selection operation to the information processing unit 120, and the acquisition unit 121 acquires the outputted selection information. The acquisition unit 121 then acquires image information indicating the two-dimensional image indicated by the acquired selection information from the storage unit 160.

[0107] Furthermore, in step S10, it is desirable that other information necessary for processing by the information processing device 100 be acquired.

[0108] Next, the determination unit 122 determines whether the display method is the first display method based on the first operation information acquired by the acquisition unit 121 (S12).

[0109] If the display method indicated by the first operation information is the first display method (Yes in S12), the determination unit 122 decides whether or not to perform the creation process based on the second operation information acquired by the acquisition unit 121 (S14).

[0110] If the second operation information indicates that a creation process should be performed (Yes in S14), the decision unit 122 performs the creation process (S20). In step S20, the decision unit 122 decides whether or not to acquire any additional data (S16).

[0111] If the decision unit 122 decides to acquire the necessary additional data (Yes in S16), the communication unit 110 acquires the necessary additional data from the server 300 or website (S18). The communication unit 110 outputs the acquired additional data to the information processing unit 120.

[0112] Then, the decision unit 122 acquires the additional data output in step S18. In step S18, the acquisition unit 121 may acquire the additional data stored in the storage unit 160. The decision unit 122 may perform the creation process based on the acquired additional data. If the decision unit 122 decides not to acquire the necessary additional data (No in S16), the communication unit 110 or the acquisition unit 121 will not acquire the additional data, and in step S20, the decision unit 122 may perform the creation process without using the additional data.

[0113] Then, after the creation process is performed in step S20, the determination unit 122 determines the display information (S22). If the second operation information indicates that the creation process will not be performed (No in S14), then in step S22, the determination unit 122 determines the display information without performing the creation process.

[0114] Then, the output unit 123 outputs the display information determined by the determination unit 122 (S24). Here, the output unit 123 outputs the display information and image information to the display unit 130, and the display unit 130 displays the two-dimensional image indicated by the image information superimposed on the background image of the virtual space, according to the display area indicated by the display information. In other words, the display unit 130 places the reference object indicated by the two-dimensional image in the virtual space (background image) according to the display area indicated by the display information and outputs (displays) it.

[0115] If the display method indicated by the first operation information is the second display method (No in S12), the determination unit 122 determines predetermined information indicating that the display method is the second display method (S26).

[0116] Then, the output unit 123 outputs predetermined information determined by the determination unit 122 (S28). Here, the output unit 123 outputs the predetermined information and image information to the display unit 130, and the display unit 130 displays the two-dimensional image indicated by the image information superimposed on the background image of the virtual space at a predetermined position and size.

[0117] <Details of the First Display Method 1> Next, we will explain the first display method. First, we will explain the details of the method by which the determination unit 122 determines the display information based on the dimensions of the reference object. In other words, the display information is determined without using object information. Here, we will explain the process of step S22 when the second operation information indicates that no creation process will be performed (No in S14), that is, when the determination unit 122 determines the display information without performing a creation process, using Figure 4A.

[0118] Figure 4A is a flowchart detailing step S22 of an example of the operation of the information processing device 100 according to this embodiment.

[0119] First, the determination unit 122 determines the scene represented by the two-dimensional image indicated by the image information acquired in step S10 (S220). If the image information indicates the scene represented by the two-dimensional image, the determination unit 122 may determine the scene represented by the two-dimensional image based on the image information. That is, if the image information indicates that the scene represented by the two-dimensional image is a portrait, the determination unit 122 determines that the scene represented by the two-dimensional image is a portrait based on the image information. If the image information does not indicate the scene represented by the two-dimensional image, the determination unit 122 may use Artificial Intelligence (Generative AI) to determine the scene represented by the two-dimensional image. For example, this Generative AI determines the scene by applying image classification technology.

[0120] Next, the determination unit 122 determines a reference object from among one or more image objects included in the two-dimensional image indicated by the acquired image information (S222). The determination unit 122 may determine the reference object based on the scene determined in step S220. For example, a table linking a scene (e.g., a portrait) and a reference object corresponding to that scene (e.g., a person shown in the portrait) is stored in the storage unit 160, and the determination unit 122 determines the reference object based on the determined scene and the table. The determination unit 122 refers to the table and determines the reference object corresponding to the determined scene. For example, if the scene is a portrait, the corresponding reference object is determined to be a person shown in the two-dimensional image. Even if step S220 has not been performed, the determination unit 122 may use generation AI to determine the reference object included in the two-dimensional image based on image recognition technology. For example, if image recognition technology determines that a person is shown in the two-dimensional image, the person is determined to be the reference object.

[0121] Furthermore, at this time, the determination unit 122 may determine the center coordinates of the determined reference object based on image recognition technology. For example, the coordinates of a rectangular area surrounding the reference object (e.g., a person) may be identified, and the physical center point of the rectangle may be calculated from the coordinates of two opposite corners of the identified rectangular area, and this center point may be used as the center coordinate.

[0122] Furthermore, the determination unit 122 determines the dimensions of the determined reference object (S224). The determination unit 122 may determine the dimensions of the reference object based on attribute information contained in the two-dimensional image. That is, the determination unit 122 may determine the dimensions of the reference object based on parameters of the camera 200 such as shutter speed, sensitivity, angle of view, focal length, image sensor size, or shooting distance. More specifically, in step S224, the determination unit 122 determines the dimensions of the reference object in real space. These dimensions represent the height or width of the reference object in real space, etc.

[0123] As described above, the reference object and its dimensions are determined in steps S222 and S224. Subsequently, the determination unit 122 determines the display area based on the dimensions of the reference object. More specifically, the determination unit 122 determines the display area based on the dimensions of the reference object, the position of the user's viewpoint, and the distance between the user and the display position on the virtual screen (S226). More specifically, the determination unit 122 may determine the display area based on the dimensions of the reference object, position information, height information indicating the user's eye level, distance information indicating the distance between the user and the display position on the virtual screen, and predetermined size information. That is, the determination unit 122 may determine a display area such that the reference object has a predetermined viewing angle when viewed from the user's viewpoint, based on the user's position indicated by the position information, the distance between the user's viewpoint and the display position on the virtual screen on which the 2D image is pasted (for example, the distance from the user's viewpoint to the display position on the virtual screen), and the dimensions of the display area (for example, the number of pixels on the display, or the height and width of the virtual screen). In this display area, the scaling factor of the 2D image is often indicated; that is, the determination unit 122 may determine the scaling factor of the 2D image. Specifically, the determination unit 122 determines the display dimensions (display area) of the 2D image so that the reference object is visible from the user's viewpoint at a size close to its actual size (the actual size of the reference object) based on the dimensions of the reference object in real space, the distance from the user's viewpoint to the virtual screen, and the size of the display area. The scaling factor is preferably a multiplier to a predetermined size indicated by predetermined size information.

[0124] Here, we will also explain height information. During initial calibration, the information processing device 100 may determine the direction of gravity based on the sensor results of the mounted sensor unit 150 (accelerometer, gyro sensor, and depth sensor, etc.) and estimate the eye level, which represents how high the information processing device 100 being used by the user is above the floor. At this time, the information processing device 100 may define the position of the floor in the virtual space based on the direction of gravity and the eye level, and store the relationship between the user's viewpoint position and the floor (for example, a value corresponding to the user's height) in the storage unit 160. In pass-through AR display, the determination unit 122 may determine the display position of the 2D image based on the floor position and eye level, so that the reference object is considered to be standing on the floor. The information indicating the estimated eye level is the height information and is stored in the storage unit 160 as described above. Here, for example, in step S10, it is preferable for the acquisition unit 121 to acquire the height information from the storage unit 160.

[0125] This process allows the reference object to be displayed at a size close to its actual size, based on the distance between the user and the virtual screen and / or the user's eye level, even when object information is not used. As a result, the user can perceive the reference object as being close to its actual size. In other words, because the dimensions of the reference object in the virtual space are displayed at a size equal to the dimensions of the reference object in the real space, the user can view a 2D image including the reference object without feeling any discomfort, enabling a viewing experience where the reference object appears to be in the same space as the user.

[0126] Furthermore, distance information indicating the distance between the user and the display position on the virtual screen will be explained. When determining the display area of ​​a two-dimensional image, the distance between the user and the display position on the virtual screen, more specifically, the distance from the user's viewpoint to the display position on the virtual screen where the two-dimensional image is placed (hereinafter also referred to as the display distance), may be determined as follows. The display distance may be determined based on the predetermined position information described above, that is, it may be determined in advance as a predetermined value. The display distance may also be determined as variable during execution, for example, when step S10 is executed. In other words, in this embodiment, it is not limited to the case where the display distance is constant, but can be executed regardless of where the display position is set. In this embodiment, it may be assumed that the display distance (or display position) is variable rather than fixed.

[0127] However, in order for the reference object to be visible at a size close to its actual size from the user's perspective, it is better to determine the display distance. Therefore, if the display distance is not predetermined as a fixed value, the determination unit 122 may determine the display distance by, for example, one of the following methods (1) to (4), or a combination thereof.

[0128] (1) User operation-based determination The determination unit 122 may move the display position of the two-dimensional image in the forward and backward directions based on user operation input (e.g., pinch operation, drag operation, slider operation, etc.) and determine the display distance based on the amount of movement. That is, the determination unit 122 may determine the display distance based on operation input received by the reception unit 140.

[0129] (2) Determination based on comfortable viewing angle The determination unit 122 may determine the display distance to satisfy a predetermined viewing angle range (comfortable viewing angle) that is easily visible to the user. The determination unit 122 may, for example, calculate the display distance at which the reference object is at a predetermined viewing angle based on the dimensions of the reference object, the size of the display area, and the display conditions (viewing angle, resolution, etc.), and determine the display distance according to the calculation result.

[0130] (3) Determination based on floor surface and eye level The determination unit 122 may determine the display position as a predetermined point on the floor surface and the distance to said predetermined point as the display distance, based on the position of the floor surface estimated by initial calibration or recalibration during execution and the user's eye level (the height of the information processing device 100 used by the user from the floor surface). For example, the determination unit 122 may determine the display position as a point on the floor surface that is a predetermined distance (e.g., 2 m) forward from a point on the floor surface vertically below the user's viewpoint, and determine the distance to said display position as the display distance. This makes it easier to achieve a natural display in pass-through AR display where the reference object appears to be standing on the floor surface.

[0131] (4) Interference avoidance determination The determination unit 122 may adjust the display distance to avoid interference with spatial objects included in the background image of the virtual space (e.g., unnatural overlapping, embedding, occlusion, etc.). For example, if object information is available, the determination unit 122 may correct the display position based on the arrangement or dimensions of the spatial objects so that the reference object does not overlap the spatial objects in an unnatural way.

[0132] The display distance is determined in this way, and distance information indicating the determined display distance is determined. Subsequently, the determination unit 122 determines the display dimensions (scaling ratio) of the 2D image and the display area (display dimensions and display position) based on the display distance indicated by the distance information, as described above.

[0133] Then, the determination unit 122 determines the display information indicating the display area determined in step S226 (S228).

[0134] In this manner, the process in step S22 is performed, and then the process in step S24 is performed.

[0135] <Details of the First Display Method 2> The following describes other first display methods. Here, we will explain detail 2 of the method in which the determination unit 122 determines the display information based on the dimensions of the reference object and the dimensions of the spatial object. In this detail 2, a natural appearance is achieved in terms of the scale relationship between the reference object and the background image in the virtual space.

[0136] Here, we will explain the process in step S22 when the second operation information indicates that no creation process will be performed (No in S14), that is, when the determination unit 122 determines the display information without performing the creation process, using Figure 4B.

[0137] Figure 4B is a flowchart showing other details of step S22 of an example of the operation of the information processing device 100 according to this embodiment.

[0138] As shown in Figure 4B, the processes in steps S220 to S224 are carried out.

[0139] Then, the determination unit 122 determines the display area in which the 2D image will be displayed in the virtual space based on the dimensions of the determined reference object and the dimensions of the spatial object indicated by the object information (S226a). More specifically in step S226a, the determination unit 122 determines the display area based on the dimensions of the reference object, the dimensions of the spatial object, and the user's position indicated by the position information.

[0140] The determination unit 122 determines the display area so that the dimensions of the reference object in the virtual space are displayed at the same size as the dimensions of the reference object in the real space. In other words, the determination unit 122 determines the display area so that the reference object appears to be the actual size of the reference object from the user's viewpoint in the virtual space. More specifically, the determination unit 122 determines the display area so that the ratio of the dimensions of the spatial object in the background image to the dimensions of the reference object is the same as the ratio of the dimensions of the spatial object in the real space to the dimensions of the reference object.

[0141] As described above, the display area indicates the display dimensions and position of the 2D image in the virtual space.

[0142] The display dimensions of a two-dimensional image should be indicated by a magnification ratio relative to the display dimensions (i.e., a predetermined size) that would be displayed if the two-dimensional image were displayed using the second display method. The magnification ratio may be greater than or equal to 1 (i.e., enlargement), or greater than 0 and less than 1 (i.e., reduction).

[0143] When a virtual space is represented by a three-dimensional orthogonal coordinate system using mutually orthogonal x, y, and z axes, the display position of a two-dimensional image should be expressed using the values ​​of each axis (x, y, z). For example, the display position should be the values ​​of each axis indicating the center of the two-dimensional image. If the display position is indicated by the distance between the user and the two-dimensional image, this distance should more specifically be the distance between the user's viewpoint (the position of the user's eyes) and the two-dimensional image.

[0144] For example, if the reference object is a person, the display area is determined such that when the person is displayed in the virtual space, their dimensions are the same as the dimensions of the person in the real space. For example, consider the case where the person is displayed with a height of 100 cm when displayed using the second display method.

[0145] If the dimensions of the person determined in S224 (height in this case) are 150 cm, the display area is determined so that the 2D image is enlarged by 1.5 times in both the vertical and horizontal directions. In this case, the display area is also determined so that the ratio of the dimensions of the spatial object in the background image to the dimensions of the person is the same as the ratio of the dimensions of the spatial object in real space to the dimensions of the person.

[0146] The above example shows a 2D image displayed at 1.5 times its original size, but this is not the only example.

[0147] For example, a two-dimensional image may be displayed at its initial display dimensions, i.e., the display dimensions shown when displayed using the second display method. In this case, a portion of the two-dimensional image is cropped and displayed, while the remaining portion of the uncropped two-dimensional image is enlarged to its initial display dimensions.

[0148] Furthermore, the determination unit 122 may determine the display area such that the center coordinates of the reference object are at the center of the display area. The center coordinates of the reference object are determined, for example, based on image recognition technology as described above.

[0149] Furthermore, the process in step S226a will be explained using Figure 5.

[0150] Figure 5 shows an example of processing in the first display method according to this embodiment. Figure 5 shows the screen 131 of the display unit 130.

[0151] In Figure 5, the screen 131 shows a 2D image superimposed on a background image (360-degree image) of the virtual space, representing the image information acquired in step S10. The frame f1 of the 2D image is also shown on the screen 131. In the 2D image shown on the screen 131 of Figure 5, a person, which is an example of a reference object, is simply indicated by dots. Similarly, in subsequent figures, a person, which is an example of a reference object, may also be indicated by dots.

[0152] Figure 5(a) shows the screen 131 before the process of step S226a is performed, and Figure 5(b) shows the screen 131 after the process of step S226a is performed.

[0153] Figure 5(a) shows a two-dimensional image of the display dimensions (i.e., predetermined size) that are displayed when shown using the second display method, for example. For example, even if the dimensions of a person determined in step S224, i.e., the dimensions (height) of the person in real space, are 150 cm, the user may perceive the dimensions (height) of the person shown in Figure 5(a) as being 100 cm.

[0154] On the other hand, in Figure 5(b), a two-dimensional image is displayed in the display area determined in step S226a. When the user views the screen 131 shown in Figure 5(b), the user perceives the person's dimensions (height) as being 150 cm.

[0155] Figure 6 is a diagram illustrating the positional relationship between the user and the 2D image in the virtual space in Figure 5(b). The 2D image is displayed according to the display dimensions and display position of the 2D image shown in the display area determined in step S226a. The process in step S226a is also called the scaling process.

[0156] Furthermore, if the process in step S226 shown in Figure 4A is performed, the screen 131 will change as shown in Figure 5. In this case, Figure 5(a) shows the screen 131 before the process in step S226 is performed, and Figure 5(b) shows the screen 131 after the process in step S226 is performed. The process in step S226 is also called the scaling process.

[0157] Then, the determination unit 122 determines the display information indicating the display area determined in step S226a (S228).

[0158] In this manner, the process in step S22 is performed, and then the process in step S24 is performed.

[0159] In the process shown in Figure 4B, the display area is determined by considering the dimensional relationships of objects such as trees or buildings included in the background image of the virtual space. As a result, the dimensions of the spatial objects in the background image and the dimensions of the reference object match the dimensional relationships in real space, making it less likely for the user to feel any incongruity regarding the relationship between the apparent size of the reference object and the surrounding environment, such as spatial objects.

[0160] The information processing device 100 according to this embodiment includes an acquisition unit 121 that acquires image information showing a two-dimensional image, a determination unit 122 that determines a reference object from among one or more image objects included in the two-dimensional image shown by the acquired image information, and determines display information indicating a display area in which the two-dimensional image is displayed in a virtual space based on the dimensions of the determined reference object, and an output unit 123 that outputs the determined display information.

[0161] As a result, for example, display information is determined that indicates a display area where the dimensions of a reference object in virtual space are displayed at a size equal to the dimensions of the reference object in real space. This allows the user to view a two-dimensional image including the reference object without feeling any discomfort, enabling a viewing experience as if the reference object were in the same space as the user. In other words, an information processing device 100 is realized that can provide a highly immersive viewing experience.

[0162] In this embodiment, the information processing device 100 includes an acquisition unit 121 which acquires object information indicating a spatial object to be displayed in a virtual space, and a determination unit 122 which determines display information indicating a display area in which a two-dimensional image is displayed in a virtual space, based on the dimensions of the determined reference object and the dimensions of the spatial object indicated by the acquired object information.

[0163] As a result, for example, display information is determined that indicates a display area where the dimensions of a reference object in virtual space are displayed at a size equal to the dimensions of the reference object in real space. This allows the user to view a two-dimensional image including the reference object without feeling any discomfort. Furthermore, the use of object information makes it easier for the dimensions of spatial objects and the dimensions of the reference object to match the dimensional relationship in real space. Therefore, the user is less likely to feel any discomfort regarding the relationship between the apparent size of the reference object and the surrounding environment, such as spatial objects. Consequently, the user can have a viewing experience as if the reference object were in the same space as the user. In other words, an information processing device 100 that can provide a highly immersive viewing experience is realized.

[0164] Furthermore, in the information processing device 100 according to this embodiment, the determination unit 122 determines display information in which the center coordinates of the reference object are the center of the display area.

[0165] This allows the reference object to be placed at the center of the display area, enabling the user to become more immersed in the virtual space. In other words, an information processing device 100 is realized that can provide a more realistic viewing experience.

[0166] Furthermore, in the information processing device 100 according to this embodiment, the determination unit 122 determines display information that indicates the display area showing the display dimensions and display position of a two-dimensional image in a virtual space.

[0167] This allows the dimensions of a reference object in virtual space to be displayed at a size closer to the dimensions of the reference object in real space. As a result, users can view 2D images including the reference object without feeling any discomfort, enabling a viewing experience as if the reference object were in the same space as the user. In other words, an information processing device 100 is realized that can provide a more immersive viewing experience.

[0168] Furthermore, the determination of the dimensions of the reference object in step S224 is not limited to the above, and may be derived based on spatial objects and other objects in the background image. For example, the number of floors of a building, which is an example of other objects, may be determined, and the dimensions of the reference object may be determined based on the number of floors.

[0169] For example, if the reference object is a person, the dimensions of the reference object (i.e., the person's height) may be inferred from the person's characteristics or attributes (whether they are an adult or a child, whether they are male or female, their nationality or skeletal structure).

[0170] In step S224, the dimensions of the reference object do not need to be precisely determined. In other words, the dimensions should be determined to be within a certain range that does not feel unnatural to the user and allows them to experience a sense of realism, and the object should be displayed with the determined dimensions. For example, the dimensions of the reference object in the virtual space should be determined to be between 80% and 120% of the dimensions of the reference object in the real space.

[0171] It should be noted that the display methods exemplified in this specification as "life-size" or "actual size" are merely examples of ways to give the user a sense of presence or immersion, and this embodiment is not limited to these. For example, depending on the type of content, scene, user preference, or presentation purpose (emphasis, comprehension promotion, presentation, etc.), the reference object may be displayed to appear larger than its actual size, or smaller than its actual size. Specific examples include displaying an object enlarged to emphasize a subject in art appreciation, or displaying an object reduced to its actual size to create a miniature diorama-like presentation. Furthermore, the determination unit 122 may decide to change the display magnification (or display distance) over time based on user input or predetermined rules, for example, to create a sense of approaching the subject. Thus, the determination of the display area is not necessarily aimed at reproducing actual size, but can be performed to realize a display method that achieves a desired sense of presence or immersion.

[0172] While Figure 5 illustrates an example where a single person is depicted in a two-dimensional image, the method is not limited to this. If multiple people are depicted in the two-dimensional image, one of the people may be selected, and the dimensions of that person may be determined using the selected person as the reference object. In this case, a person with easily measurable attributes (for example, a person whose entire body is visible in the two-dimensional image, or a person who overlaps with other people the least in the two-dimensional image) may be selected. Alternatively, multiple people may be selected as the reference object, the dimensions of each of the multiple people may be determined, and the dimensions of the reference object may be determined using a predetermined method (for example, average or weighted average).

[0173] Furthermore, selecting one person or one object reduces the processing load on the information processing device 100. Additionally, selecting multiple people or multiple objects allows for the deriving of highly accurate dimensions of reference objects.

[0174] <Details 3 of the First Display Method (First Processing)> Below, we will explain Details 3, which is an example of the first processing being performed in the first display method. Specifically, we will explain the processing when the second operation information indicates that the creation process is performed (Yes in S14) and the second operation information indicates that the first processing is performed, using Figure 7.

[0175] Figure 7 is a flowchart detailing step S20 when the first processing is performed in an example of the operation of the information processing device 100 according to this embodiment.

[0176] In the example shown in Figure 7, the determination unit 122 applies image augmentation processing (first processing) to the two-dimensional image indicated by the image information acquired by the acquisition unit 121. As a result, the two-dimensional image is augmented, which can provide the user with a greater sense of realism.

[0177] For example, if a 2D image is a magnified image of a reference object, the 2D image may be localized and therefore may not include the background, objects, or buildings surrounding the reference object. In such cases, magnifying the area surrounding the 2D image or generating a background for the 2D image can enable a more immersive viewing experience.

[0178] In the example shown in Figure 7, steps S200 to S206 are performed first. Steps S200 to S206 are the same as steps S220 to S226, respectively. Note that step S206 may be the same as step S226a.

[0179] Then, the determination unit 122 performs image augmentation processing (first processing) on ​​the two-dimensional image indicated by the image information acquired by the acquisition unit 121 in step S10 (S208).

[0180] Image augmentation processing may be performed using existing methods or using a generation AI. When using a generation AI, the determination unit 122 may instruct the generation AI by including its input, output, or method in the prompt.

[0181] In the image augmentation process, the image augmentation process may be performed based on the two-dimensional image (more specifically, a single two-dimensional image) indicated by the image information acquired from the storage unit 160 in step S10, but is not limited to this.

[0182] In other words, in the image enhancement process shown in step S208, additional related images (multiple related images) may be acquired as additional data. The determination unit 122 decides whether or not to acquire the additional data (S16). For example, the second operation performed by the user may include an operation indicating whether or not to acquire additional data in the creation process. If the second operation includes an operation indicating whether or not to acquire additional data, the second operation information includes information indicating whether or not to acquire additional data in step S16.

[0183] In step S16, the decision unit 122 decides whether or not to acquire additional data based on the acquired second operation information.

[0184] If the second operation information indicates that additional data should be acquired (Yes in step S16), the decision unit 122 decides to acquire the additional data, and the acquisition unit 121 or the communication unit 110 acquires the relevant image as additional data (S18).

[0185] Thus, in the image augmentation process, in addition to the single two-dimensional image, multiple related images may be acquired, and the image augmentation process may be performed based on the single two-dimensional image and the multiple related images. In this case, the acquisition unit 121 may acquire multiple related images stored in the storage unit 160, or the communication unit 110 may search a website and acquire multiple related images from that website. Alternatively, the communication unit 110 may acquire multiple related images stored in the server 300. Each of the multiple related images is a two-dimensional image that is related to the two-dimensional image indicated by the image information.

[0186] The acquisition unit 121 or the communication unit 110 may acquire a related image based on the shooting location information, camera information, or date information included in the two-dimensional image indicated by the image information, and the shooting location information, camera information, or date information included in the related image. The shooting location information is information indicating the location or direction in which the image was taken, the camera information is information indicating the manufacturer, model, shutter speed, sensitivity, angle of view, focal length, image sensor size, or shooting distance of the camera that took the image, and the date information is information indicating the year, month, day, and time in which the image was taken.

[0187] For example, it is desirable to acquire related images that are the same as the two-dimensional image indicated by the image information, and that are taken at the same location or in the surrounding area. Alternatively, it is desirable to acquire related images that are the same as the two-dimensional image indicated by the image information, and that are taken in the same direction or within a predetermined range. Furthermore, it is desirable to acquire related images that are the same as the two-dimensional image indicated by the image information, or taken at the same time, season, or weather. However, it is also possible to acquire related images from different times, seasons, or weather conditions for the two-dimensional image indicated by the image information and perform image augmentation processing.

[0188] Furthermore, if related images are obtained from a website, the time period, season, or weather of the region to be expanded by the image augmentation process may be indicated by a prompt.

[0189] The determination unit 122 then determines the display area in the virtual space where the two-dimensional image that has undergone image augmentation processing will be displayed (S210). This display area indicates the display dimensions and the display position of the two-dimensional image that has undergone image augmentation processing within the virtual space.

[0190] Thus, the process in step S20 is performed, and then the process in step S22 is performed. In step S22, display information indicating the display area determined in step S210, as shown in Figure 7, is determined.

[0191] Figure 8 shows an example of image augmentation processing according to this embodiment. Figure 8 shows the screen 131 of the display unit 130. Figure 8(a) shows the screen 131 before the processing of step S206 is performed, and Figure 8(b) shows the screen 131 after the processing of step S206 is performed. Figure 8(c) shows the screen 131 after step S208 (image augmentation processing) is performed. In Figures 8(a) and (b), the screen 131 shows a two-dimensional image indicated by the image information acquired in step S10 superimposed on the background image (360-degree image) of the virtual space. Also, in Figures 8(a) and (b), the frame f1 of the two-dimensional image is shown on the screen 131. In Figure 8(c), the screen 131 shows a two-dimensional image that has undergone image augmentation processing superimposed on the background image (360-degree image) of the virtual space. In Figure 8(c), screen 131 shows the frame f2 of a two-dimensional image that has undergone image augmentation processing.

[0192] In this case, a boundary line may be displayed that distinguishes the region of the 2D image before image augmentation processing from the region of the 2D image after image augmentation processing. For example, in Figure 8(c), this boundary line is shown as a dashed line.

[0193] As described above, in the information processing device 100 according to this embodiment, the determination unit 122 performs image augmentation processing on a two-dimensional image and determines display information indicating a display area in which the two-dimensional image that has undergone image augmentation processing is displayed in a virtual space.

[0194] As a result, for example, if a reference object is magnified and photographed, an image augmentation process is applied to a 2D image in which the background, objects or buildings around the reference object are not captured. Therefore, the user can view the augmented 2D image, for example, a 2D image in which the surrounding area is magnified, or a 2D image in which a background has been generated in the surrounding area. In other words, an information processing device 100 is realized that can provide a more immersive viewing experience.

[0195] Furthermore, in the example shown in Figure 7, the processing in step S206, which corresponds to the scaling process in step S226 (or step S226a), is performed, and then the image augmentation process is carried out in step S208. As a result, the reference object included in the two-dimensional image after the image augmentation process does not cause the user to feel any sense of incongruity, and the user can have a viewing experience as if the reference object were in the same space as the user. In other words, an information processing device 100 is realized that can provide a viewing experience with a higher sense of realism.

[0196] <Details 4 of the First Display Method (Second Processing)> Below, we will explain Details 4, which is an example of the second processing being performed in the first display method. Specifically, we will explain the processing when the second operation information indicates that creation processing is performed (Yes in S14) and the second operation information indicates that the second processing is performed, using Figure 9.

[0197] Figure 9 is a flowchart detailing step S20 when the second processing is performed in an example of the operation of the information processing device 100 according to this embodiment.

[0198] In the example shown in Figure 9, the determination unit 122 performs a conversion process (second process) to convert the two-dimensional image indicated by the image information acquired by the acquisition unit 121 into a 360-degree image. For example, compared to the examples shown in Figures 4A, 4B, and 7, the frame surrounding the two-dimensional image (e.g., frame f1 or frame f2) is not displayed, which can give the user a greater sense of realism.

[0199] In the example shown in Figure 9, the processing in steps S200 to S206 is performed as in Figure 7. After step S206, the determination unit 122 performs a conversion process (second processing) to convert the two-dimensional image indicated by the image information acquired by the acquisition unit 121 in step S10 into a 360-degree image (S308).

[0200] More specifically, the determination unit 122 performs a conversion process to convert a two-dimensional image from a two-dimensional image format to a 360-degree image format. This conversion process may include format conversion and image expansion processing (first process) to extend the area around the two-dimensional image. At this time, the image expansion processing may be performed using the same method as in step S208 in Figure 7, and the output format after conversion may also be specified. The output format may specify that it is a 360-degree image and include formatting such as the dimensions of the 360-degree image.

[0201] The 360-degree image, which has been converted into a 2D image, may be displayed in place of the background image of the virtual space, or it may be displayed superimposed on the background image of the virtual space. Furthermore, the process in step S16 may be performed, similar to the first process. If the answer to step S16 is Yes, then the process in step S18 may be performed. If the answer to step S16 is No, then the process in step S18 is not performed.

[0202] The determination unit 122 then determines the display area in the virtual space where the converted 360-degree image will be displayed (S310). This display area indicates the display dimensions and the display position of the converted 360-degree image within the virtual space.

[0203] In this way, the process of step S20 is performed, and then the process of step S22 is performed. In step S22, display information indicating the display area determined in step S310, as shown in Figure 9, is determined.

[0204] Figure 10 is a diagram showing an example of the conversion process according to this embodiment. Figure 10 shows the screen 131 of the display unit 130. Figure 10(a) shows the screen 131 before the process of step S206 is performed, and Figure 10(b) shows the screen 131 after the process of step S206 is performed. Figure 10(c) shows the screen 131 after step S310 (conversion process) is performed. In Figures 10(a) and (b), the screen 131 shows a 2D image indicated by the image information acquired in step S10 superimposed on the background image (360-degree image) of the virtual space. Also, in Figures 10(a) and (b), the frame f1 of the 2D image is shown on the screen 131. In Figure 10(c), the screen 131 shows the 360-degree image obtained by performing the conversion process on the 2D image.

[0205] In this case, a boundary line may be displayed that distinguishes the area of ​​the 2D image before the conversion process from the area of ​​the 360-degree image after the conversion process. For example, in Figure 10(c), the boundary line is shown as a dashed line.

[0206] As described above, in the information processing device 100 according to this embodiment, the determination unit 122 performs a conversion process to convert a two-dimensional image into a 360-degree image, and determines display information indicating a display area in which the converted 360-degree image is displayed in a virtual space.

[0207] This allows users to view a 360-degree image as the background image of the virtual space, rather than a 2D image with frames f1 and f2 attached. In other words, an information processing device 100 is realized that can provide a more immersive viewing experience.

[0208] <Details 5 of the First Display Method (Third Processing)> Below, we will explain detail 5, which is an example of the third processing being performed in the first display method. Specifically, we will explain the processing when the second operation information indicates that creation processing is performed (Yes in S14) and the second operation information indicates that the third processing is performed, using Figure 11.

[0209] Figure 11 is a flowchart detailing step S20 when the third process is performed in an example of the operation of the information processing device 100 according to this embodiment.

[0210] In the example shown in Figure 11, the determination unit 122 performs a generation process (third process) to generate three-dimensional data based on the two-dimensional image indicated by the image information acquired by the acquisition unit 121. As a result, the user can view a three-dimensional image, providing the user with a greater sense of realism.

[0211] In the example shown in Figure 11, the same processes as in Figure 9 are performed as in steps S200 to S206. After step S206, the determination unit 122 performs a generation process (third process) to generate three-dimensional data based on the two-dimensional image indicated by the image information acquired by the acquisition unit 121 in step S10 (S408).

[0212] The generated 3D data includes, for example, point clouds, meshes, NeRF (Neural Radiance Fields), and Gaussian Platting data representing 3D models. Furthermore, these 3D models are 3D models of reference objects.

[0213] Furthermore, the process in step S16 may be performed in the same manner as in the first and second processes. If the answer in step S16 is Yes, then the process in step S18 may be performed. If the answer in step S16 is No, then the process in step S18 is not performed.

[0214] The determination unit 122 then determines a display area in the virtual space where the 3D model represented by the generated 3D data will be displayed (S410). This display area indicates the display dimensions and display position of the generated 3D model within the virtual space.

[0215] In this case, the 3D data is generated after the processing in step S206 has been performed. Therefore, when the 3D model represented by this 3D data, more specifically the 3D model of the reference object, is displayed in the virtual space, the display area is determined so that the dimensions of the 3D model of the reference object in the virtual space are displayed at the same size as the dimensions of the reference object in the real space.

[0216] In this way, the process of step S20 is performed, and then the process of step S22 is performed. In step S22, display information indicating the display area determined in step S410 shown in Figure 11 is determined.

[0217] As described above, in the information processing device 100 according to this embodiment, the determination unit 122 determines display information that indicates a display area in which a three-dimensional model represented by three-dimensional data based on a two-dimensional image is displayed in a virtual space.

[0218] This allows users to view a three-dimensional model displayed on a background image in a virtual space. In other words, an information processing device 100 is realized that can provide a more immersive viewing experience.

[0219] Furthermore, in the information processing device 100 according to this embodiment, the determination unit 122 performs a generation process to generate 3D data based on a 2D image, and determines display information that indicates a display area in which the 3D model represented by the generated 3D data is displayed in a virtual space.

[0220] This enables the realization of an information processing device 100 that can display a three-dimensional model represented by three-dimensional data that it has generated itself.

[0221] In the example shown in Figure 11, the determination unit 122 performs a generation process (third process) to generate 3D data, thereby obtaining a 3D model represented by the 3D data. However, this is not the only example. For instance, the communication unit 110 acquires 3D data based on a 2D image from a server 300 or a website and outputs the acquired 3D data to the information processing unit 120. The acquisition unit 121 then acquires the outputted 3D data, and the determination unit 122 determines the display area in which the 3D model represented by the acquired 3D data will be displayed in the virtual space. In other words, the information processing device 100 does not need to generate 3D data itself. In this case, the determination unit 122 may determine what kind of data the 3D data based on the 2D image is, and the communication unit 110 may acquire the 3D data according to this determination.

[0222] (Other Embodiments) Although the information processing methods, etc., have been described above based on the embodiments, this disclosure is not limited to these embodiments. Various modifications of these embodiments that a person skilled in the art could conceive of may also be included in this disclosure, as long as they do not depart from the spirit of this disclosure.

[0223] In this embodiment, the configuration described in Figure 2 is provided in the information processing device 100 (image display device), but some of the configuration described in Figure 2 may be provided in another terminal, such as an image processing device.

[0224] Furthermore, although the sensor and display are not shown in Figure 2, the information processing device 100 may be equipped with them, or another terminal such as an image processing device may be equipped with them. Alternatively, for example, the sensor and display may be included in an image processing device separate from the information processing device 100, and the information processing device 100 may be realized by communicating with the image processing device via the communication IF 403. Any of these configurations may be included in any terminal, and there may be one or more terminals that realize the information processing device 100. The sensor may be a wearable sensor attached to a part of the body, such as the viewer's head, but a sensor other than a wearable sensor may also be used.

[0225] In the case where the information processing device 100 is an AR glasses, which is an example of an image display device, a transparent display may be used as the display unit 130. For realizing the superposition of virtual space and real space (AR), optical see-through or pass-through methods may be used. In optical see-through, a semi-transparent display or holographic optical element is used to allow the light of the real space to be seen directly, and digital information is superimposed and displayed on top of it. In the pass-through method, a high-quality camera is used to capture the real space, and the captured image is displayed on the display, so that an image combining the real space and digital information is displayed to the user. In this way, the display unit 130, which is a display that displays according to the video signal, generates an image, and the viewer perceives the image as the image is transmitted to the viewer's eyes.

[0226] An example has been shown in which the information processing device 100, which is an image display device, may be equipped with a display, but it is not limited to this. An example may also be used in which the image processing device is equipped with a display and displays the processed video signal via the display. The means for displaying the video signal is not limited to the above configuration. For example, the processed video signal may be output to an external image processing device connected by a communication module. The communication performed by the communication module may be wired or wireless.

[0227] As another example, the image processing device may have a terminal for outputting an analog video signal, and a cable for AR glasses or the like may be connected to the terminal to display the video signal from the AR glasses or the like. In this case, the image display device may be AR glasses, a head-mounted display, smart glasses, or a wearable display, etc., worn on the user's head or part of their body. Alternatively, the image display device may be a multi-display system, etc., consisting of multiple fixed displays. The image display device may then reproduce the video signal.

[0228] The sensor unit 150 may be, for example, an external tracking camera. In this case, the information processing device 100 does not have a sensor unit 150, and the communication unit 110 communicates with the sensor unit 150 which is located outside the information processing device 100.

[0229] Furthermore, the location information may also be information indicating at least one of the relative position and orientation between the user and the information processing system 1000 or an external device equipped with a sensor unit 150.

[0230] The information processing device 100 may acquire location information from an external device equipped with a sensor unit 150 via a communication interface. In this case, the information processing device 100 does not need to include the sensor unit 150. Here, the external device is, for example, an image display device or a device worn on the user's head.

[0231] As described above, in determining the reference object, the determination unit 122 may process based on the scene of the 2D image. For example, if the scene is a portrait or sports scene, the determination unit 122 will determine a person as the reference object, and if the scene is a macro scene, it will determine an insect, a flower, or a dish as the reference object. The table above shows how such scenes are linked to the reference objects corresponding to those scenes.

[0232] In the explanation of the process shown in Figure 7, it is stated that prompts may be used for image augmentation processing. These prompts may be stored in advance, for example, in the memory unit 160. The information processing unit 120 may retrieve the pre-stored prompts and perform image augmentation processing using the retrieved prompts.

[0233] In the examples shown in Figures 4A to 6, when a 2D image is reduced in size, the dimensions of the display area may be reduced from the original display dimensions, or they may be displayed without reduction. If the dimensions of the display area are not reduced, there is no image outside the display area, and therefore no data to display. In such cases, it is advisable to perform image augmentation processing as described in Figure 7, etc.

[0234] Furthermore, the information processing unit 120 may acquire the user's parallax information, decide to display images (3D images) with different parallaxes for the right and left eyes, and output them.

[0235] Furthermore, in the above embodiment, each component may be implemented by being composed of dedicated hardware or by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0236] Furthermore, the order in which each step in the flowchart is performed is illustrative for the purpose of specifically illustrating this disclosure, and may be in a different order. Also, some of the above steps may be performed simultaneously (in parallel) with other steps, and some of the above steps may not be performed.

[0237] Furthermore, the division of functional blocks in the block diagram is just one example; multiple functional blocks can be implemented as a single functional block, one functional block can be divided into multiple parts, or some functions can be moved to other functional blocks. In addition, the functions of multiple functional blocks with similar functions can be processed in parallel or time-sharing by a single piece of hardware or software.

[0238] Furthermore, the information processing device 100, etc. according to the above embodiment may be implemented as a single device or as a plurality of devices. When the information processing device 100, etc. is implemented as a plurality of devices, the individual components of the information processing device 100, etc. may be distributed among the plurality of devices in any manner. When the information processing device 100, etc. is implemented as a plurality of devices, the communication method between the plurality of devices is not particularly limited and may be wireless communication or wired communication. In addition, wireless communication and wired communication may be combined between the devices.

[0239] Furthermore, each component described in the above embodiment may be implemented as software, or typically as an integrated circuit (LSI). These may be individually integrated onto a single chip, or some or all of them may be integrated onto a single chip. Here, we refer to it as an LSI, but depending on the degree of integration, it may also be called an IC, system LSI, super LSI, or ultra LSI. Moreover, the method of integrated circuit implementation is not limited to LSIs; it may also be implemented using a dedicated circuit (a general-purpose circuit that executes a dedicated program) or a general-purpose processor. After LSI manufacturing, a programmable FPGA (Field Programmable Gate Array) or a reconfigurable processor that can reconfigure the connections or settings of circuit cells inside the LSI may be used. Furthermore, if an integrated circuit implementation technology that replaces LSIs emerges due to advances in semiconductor technology or other derived technologies, it is natural that the components may be integrated using that technology.

[0240] A system LSI is a highly functional LSI manufactured by integrating multiple processing units onto a single chip. Specifically, it is a computer system composed of a microprocessor, ROM, RAM, and other components. The ROM stores the computer program. The system LSI achieves its function by having the microprocessor operate according to the computer program.

[0241] Furthermore, one aspect of this disclosure may be a computer program that causes a computer to perform each of the characteristic steps included in the above-described information processing method.

[0242] Furthermore, for example, the program may be a program to be executed by a computer. Also, in one aspect of this disclosure, such a program may be recorded on a computer-readable non-temporary recording medium. For example, such a program may be recorded on a recording medium and distributed or made available. For example, by installing the distributed program on a device having another processor and having that processor execute the program, it becomes possible to have that device perform the above-mentioned processes.

[0243] This disclosure is useful for information processing devices and the like that can provide a highly immersive viewing experience.

[0244] 100 Information Processing Device 110 Communication Unit 120 Information Processing Unit 121 Acquisition Unit 122 Determination Unit 123 Output Unit 130 Display Unit 131 Screen 140 Reception Unit 150 Sensor Unit 160 Storage Unit 200 Camera 300 Server 401 Processor 402 Memory 403 Communication IF 404 Input Device 1000 Information Processing System f1, f2 Frame

Claims

1. An information processing device comprising: an acquisition unit that acquires image information representing a two-dimensional image; a determination unit that determines a reference object from among one or more image objects included in the two-dimensional image represented by the acquired image information, and determines display information indicating a display area in which the two-dimensional image is displayed in a virtual space based on the dimensions of the determined reference object; and an output unit that outputs the determined display information.

2. The information processing apparatus according to claim 1, wherein the acquisition unit acquires object information indicating a spatial object to be displayed in the virtual space, and the determination unit determines the display information indicating the display area in which the two-dimensional image is displayed in the virtual space, based on the dimensions of the determined reference object and the dimensions of the spatial object indicated by the acquired object information.

3. The information processing apparatus according to claim 1, wherein the determination unit determines the display information in which the center coordinates of the reference object are the center of the display area.

4. The information processing apparatus according to claim 1, wherein the determination unit determines the display information indicating the display area that shows the display dimensions and display position of the two-dimensional image in the virtual space.

5. The information processing apparatus according to claim 1, wherein the determination unit performs image augmentation processing on the two-dimensional image and determines the display information indicating the display area on which the two-dimensional image subjected to the image augmentation processing is displayed in the virtual space.

6. The information processing apparatus according to claim 1, wherein the determination unit performs a conversion process to convert the two-dimensional image into a 360-degree image, and determines the display information indicating the display area in which the converted 360-degree image is displayed in the virtual space.

7. The information processing apparatus according to claim 1, wherein the determination unit determines the display information indicating the display area in which the three-dimensional model shown by the three-dimensional data based on the two-dimensional image is displayed in the virtual space.

8. The information processing apparatus according to claim 7, wherein the determination unit performs a generation process to generate the three-dimensional data based on the two-dimensional image, and determines the display information indicating the display area in which the three-dimensional model represented by the generated three-dimensional data is displayed in the virtual space.

9. An information processing method performed by an information processing device, comprising: acquiring image information showing a two-dimensional image; determining a reference object from among one or more image objects included in the two-dimensional image shown by the acquired image information; determining display information indicating a display area in which the two-dimensional image is displayed in a virtual space based on the dimensions of the determined reference object; and outputting the determined display information.

10. A computer program for causing a computer to execute the information processing method described in claim 9.

Images