Information processing apparatus, screen generation method, recording medium, and information processing system

Abstract

An information processing apparatus includes: a processing unit to associate position information indicating a first position in a captured image of an object with position information indicating a second position in a three-dimensional image including a three-dimensional area corresponding to the object. The three-dimensional image having the position information indicating the second position associated with the first position, includes a first three-dimensional image being generated based on design information and a second three-dimensional image being generated based on point cloud information. The information processing apparatus further includes a generation unit to generate a screen, which includes: a first display area for displaying a predetermined-area image representing a predetermined area in the captured image; a second display area for displaying at least a portion of the first three-dimensional image; and a third display area for displaying at least a portion of the second three-dimensional image.

Description

FN202501296[DESCRIPTION][Title of Invention]INFORMATION PROCESSING APPARATUS, SCREEN GENERATION METHOD, RECORDING MEDIUM, AND INFORMATION PROCESSING SYSTEM[Technical Field]

[0001] The present disclosure relates to an information processing apparatus, a screen generation method, a recording medium, and an information processing system.[Background Art]

[0002] A wide-field-of-view image having a wide viewing angle and captured in an imaging range including even an area that is difficult for a normal angle of view to cover has been known in recent years. The wide-field-of-view image is hereinafter referred to as a "wide-view image". Examples of the wide-view image include a 360-degree image that is a captured image of an entire 360-degree view. The 360-degree image is also referred to as a spherical image, an omnidirectional image, or an "all-around" image.If the entire wide-view image is displayed by a display terminal, the wide-view image is curved and difficult to view. Accordingly, such a display terminal displays a predetermined-area image indicating a predetermined area in the wide-view image, and a user views the predetermined-area image (see Patent Literature (PTL) 1).[Citation List][Patent Literature]

[0003] [PTL 1]Japanese Unexamined Patent Application Publication No. 2023-140923[Summary of Invention][Technical Problem]FN202501296

[0004] However, by viewing the predetermined-area image corresponding to a viewable range of the wide-view image, the user can hardly recognize the content of the predetermined-area image and the location where the predetermined-area image was captured.

[0005] Further, an image capturing device captures a 360-degree view of objects around an image capturing position in a single imaging process. In other words, the image capturing device does not capture each object from all 360-degree directions relative to the object. Even an object within the imaging range does not have its back captured, and the user is unable to know the condition of the back of the object.

[0006] In addition, by viewing the predetermined-area image corresponding to a viewable range of the wide-view image, the user can hardly recognize whether the predetermined-area image is appropriate as a predetermined-area image corresponding to time information.

[0007] The above-described problem also occurs when the captured image is not a wide-view image that is a curved image.[Solution to Problem]

[0008] The present disclosure described herein provides an information processing apparatus including a processing unit and a generation unit. The processing unit associates position information indicating a first position in a captured image of an object with position information indicating a second position in a three-dimensional image including a three-dimensional area corresponding to the object. The three-dimensional image has the position information indicating the second position associated with the first position. The three-dimensional image includes a first three-dimensional image and a second three-dimensional image. The first three-dimensional image is generated based on design information. TheFN202501296second three-dimensional image is generated based on point cloud information. The generation unit generates a screen. The screen includes a first display area, a second display area, and a third display area. The first display area displays a predetermined-area image representing a predetermined area in the captured image. The second display area displays at least a portion of the first three-dimensional image. The third display area displays at least a portion of the second three-dimensional image.The present disclosure described herein provides a screen generation method performed by an information processing apparatus. The screen generation method includes associating position information indicating a first position in a captured image of an object with position information indicating a second position in a three-dimensional image including a three-dimensional area corresponding to the object, the three-dimensional image having the position information indicating the second position associated with the first position, the three-dimensional image including a first three-dimensional image and a second three-dimensional image, the first three-dimensional image being generated based on design information, the second three-dimensional image being generated based on point cloud information; and generating a screen. The screen includes a first display area, a second display area, and a third display area. The first display area displays a predetermined-area image representing a predetermined area in the captured image. The second display area displays at least a portion of the first three-dimensional image. The third display area displays at least a portion of the second three-dimensional image.The present disclosure described herein provides a recording medium carrying computer readable code for controlling a computer system to carry out a screen generation method. The screen generation method includes associating position information indicating a first position in a captured image of an object with position information indicating a second position in a three-dimensional image including a three-dimensional area corresponding to the object, the three-dimensional image having the position information indicating the second position associated with the first position, the three-dimensional image including a first three-FN202501296dimensional image and a second three-dimensional image, the first three-dimensional image being generated based on design information, the second three-dimensional image being generated based on point cloud information; and generating a screen. The screen includes a first display area, a second display area, and a third display area. The first display area displays a predetermined-area image representing a predetermined area in the captured image. The second display area displays at least a portion of the first three-dimensional image. The third display area displays at least a portion of the second three-dimensional image.The present disclosure described herein provides an information processing system including an information processing apparatus and a display terminal. The information processing apparatus includes a processing unit and a generation unit. The processing unit associates position information indicating a first position in a captured image of an object with position information indicating a second position in a three-dimensional image including a three-dimensional area corresponding to the object. The three-dimensional image has the position information indicating the second position associated with the first position. The three-dimensional image includes a first three-dimensional image and a second three-dimensional image. The first three-dimensional image is generated based on design information. The second three-dimensional image is generated based on point cloud information. The generation unit generates a screen. The screen includes a first display area, a second display area, and a third display area. The first display area displays a predetermined-area image representing a predetermined area in the captured image. The second display area displays at least a portion of the first three-dimensional image. The third display area displays at least a portion of the second three-dimensional image. The display terminal includes a display to display the screen.[Advantageous Effects of Invention]

[0009] According to one aspect of the present disclosure, generation of an image that complements a captured image obtained by image capturing enhances user convenience.FN202501296[Brief Description of Drawings]

[0010] A more complete appreciation of embodiments of the present disclosure and many of the attendant advantages and features thereof can be readily obtained and understood from the following detailed description with reference to the accompanying drawings.FIG. 1 A is a left side view of an image capturing device according to an embodiment of the present disclosure.FIG. IB is a front view of the image capturing device according to the embodiment of the present disclosure.FIG. 1C is a plan view of the image capturing device according to the embodiment of the present disclosure.FIG. 2 is an illustration of an example of how the image capturing device is used.FIG. 3 A is a diagram illustrating an example of a hemispherical image (front side) captured by the image capturing device.FIG. 3B is a diagram illustrating an example of a hemispherical image (back side) captured by the image capturing device.FIG. 3C is a diagram illustrating an example of an image represented by Mercator projection. FIG. 4A is a conceptual diagram illustrating an example of how a Mercator image is mapped to a sphere.FIG. 4B is a diagram illustrating an example of a spherical image.FIG. 5 is a diagram illustrating an example of the position of a virtual camera and the position of a predetermined area in a case where the spherical image is of a three-dimensional sphere. FIG. 6A is a perspective view of the virtual camera and the predetermined area illustrated in FIG. 5.FIG. 6B is a diagram illustrating an example of a predetermined-area image obtained in the state illustrated in FIG. 6A and displayed on a display.FN202501296FIG. 6C is a diagram illustrating an example of a predetermined area obtained by changing the viewpoint of the virtual camera illustrated in FIG. 6A.FIG. 6D is a diagram illustrating an example of a predetermined-area image obtained in the state illustrated in FIG. 6C and displayed on the display.FIG. 7 is a diagram illustrating an example of a point in a three-dimensional Euclidean space defined in spherical coordinates.FIG. 8 is a conceptual diagram illustrating an example of the relationship between the predetermined area and a point of interest.FIG. 9 is a schematic diagram of an example of a communication system.FIG. 10 is a block diagram illustrating an example hardware configuration of the image capturing device.FIG. 11 is a block diagram illustrating an example hardware configuration of a relay device. FIG. 12 is a block diagram illustrating an example hardware configuration of each of a communication control apparatus and a communication terminal.FIG. 13 is a block diagram illustrating an example functional configuration of the communication system.FIG. 14 is a schematic diagram of an example of a user device management table.FIG. 15 is a schematic diagram of an example of a virtual room management table.FIG. 16Ais a schematic diagram of an example of a three-dimensional image management table.FIG. 16B is a schematic diagram of another example of the three-dimensional image management table.FIG. 17 is a sequence diagram illustrating an example of a process of communicating content data in the communication system.FIG. 18 is a sequence diagram illustrating an example of a process of starting image and sound recording in the communication system.FN202501296FIG. 19 is a sequence diagram illustrating an example of a process of stopping image and sound recording in the communication system.FIG. 20 is a sequence diagram illustrating an example of a process of playing back recorded images and sounds in the communication system.FIG. 21 is a diagram illustrating an example of a recorded data selection screen.FIG. 22 is a flowchart of a part of an example of a screen display process performed by the communication control apparatus illustrated in FIG. 12.FIG. 23 is a flowchart of another part of the example of the screen display process performed by the communication control apparatus illustrated in FIG. 12.FIG. 24 is a flowchart of still another part of the example of the screen display process performed by the communication control apparatus illustrated in FIG. 12.FIG. 25 is a diagram illustrating an example of an initial display screen displayed on the communication terminal illustrated in FIG. 12.FIG. 26 is a diagram illustrating an example of a screen displaying a first three-dimensional image in which the position and the field of view of a virtual camera have been changed from those in a first three-dimensional image on the screen illustrated in FIG. 25.FIG. 27A is a diagram illustrating an example of a first three-dimensional image displayed for a position of the virtual camera.FIG. 27B is a diagram illustrating an example of a first three-dimensional image displayed for another position of the virtual camera.FIG. 27C is a diagram illustrating an example of a first three-dimensional image displayed for another position of the virtual camera.FIG. 27D is a diagram illustrating an example of a first three-dimensional image displayed for another position of the virtual camera.FIG. 28 is a diagram illustrating an example of a screen displaying a predetermined-area image that is enlarged in accordance with an enlargement of a first three-dimensional image.FN202501296FIG. 29 is a diagram illustrating an example of a screen displaying a predetermined-area image that is reduced in size in accordance with a reduction of the size of a first three-dimensional image.FIG. 30 is a diagram illustrating an example of a screen on which a corresponding predetermined area, an icon of a virtual camera, and the line of sight of the virtual camera are displayed in a display area.FIG. 31 is a diagram illustrating an example of a screen displaying a first three-dimensional image and a second three-dimensional image that have changed in accordance with a change in time information.FIG. 32 is a diagram illustrating an example of a screen displaying a first three-dimensional image and a second three-dimensional image that have changed in accordance with a change in time information.FIG. 33 is a diagram illustrating an example of a screen displaying a first three-dimensional image and a second three-dimensional image that have changed in accordance with a change in time information.FIG. 34 is a diagram illustrating an example of a screen on which progress identification information is displayed in a superimposed manner.FIG. 35 is a diagram illustrating an example of a screen before first difference identification information and second difference identification information are displayed in a superimposed manner.FIG. 36 is a diagram illustrating an example of a screen after the first difference identification information is displayed in a superimposed manner.FIG. 37 is a diagram illustrating an example of a screen after the first difference identification information and the second difference identification information are displayed in a superimposed manner.FIG. 38A is a diagram illustrating an example of a screen displaying a three-dimensional model that changes differently even when the change in time information is the same.FN202501296FIG. 38B is a diagram illustrating an example of a change in construction date information in the three-dimensional image management table illustrated in FIG. 16B.The accompanying drawings are intended to depict embodiments of the present disclosure and should not be interpreted to limit the scope thereof. The accompanying drawings are not to be considered as drawn to scale unless explicitly noted. Also, identical or similar reference numerals designate identical or similar components throughout the several views.[Description of Embodiments]

[0011] In describing embodiments illustrated in the drawings, specific terminology is employed for the sake of clarity. However, the disclosure of this specification is not intended to be limited to the specific terminology so selected and it is to be understood that each specific element includes all technical equivalents that have a similar function, operate in a similar manner, and achieve a similar result.Referring now to the drawings, embodiments of the present disclosure are described below. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0012] Overview of Spherical ImageA method for generating a spherical image according to one or more embodiments will be described with reference to FIGs. lAto 8. The spherical image is also referred to as a spherical panoramic image or a 360-degree panoramic image, and is an example of a wide-view moving image having a wide range of viewing angles. Examples of the wide-view image include a panoramic image with a viewing angle of about 180 degrees.

[0013] The external appearance of an image capturing device 10 will be described with reference to FIGs. lAto 1C. The image capturing device 10 is a digital camera for capturing images fromFN202501296which a spherical image is generated. FIGs. 1 A, IB, and 1C are a left side view, a front view, and a plan view, respectively, of the image capturing device 10.

[0014] As illustrated in FIG. 1 A, the image capturing device 10 has a size such that a person can hold the image capturing device 10 with one hand. As illustrated in FIGs. 1A, IB, and 1C, the image capturing device 10 includes an imaging element 103a and an imaging element 103b in an upper portion thereof. Specifically, the imaging element 103a is disposed on the front side of the image capturing device 10, and the imaging element 103b is disposed on the back side of the image capturing device 10. As illustrated in FIG. IB, the image capturing device 10 further includes an operation unit 115 such as a shutter button on the back side of the image capturing device 10.

[0015] The usage scenario of the image capturing device 10 will be described with reference to FIG.2. FIG. 2 illustrates how a user uses the image capturing device 10. As illustrated in FIG. 2, the image capturing device 10 is communicably connected to a relay device 3 placed on a table 2. The image capturing device 10 is used to capture images of surrounding objects and scenery. The imaging elements 103a and 103b illustrated in FIGs. 1 A and 1C capture objects surrounding the user to obtain two hemispherical images. If the image capturing device 10 does not transmit the spherical image, which is generated from the captured hemispherical images, to another communication terminal or system, the relay device 3 is not used.

[0016] An overview of a process for generating the spherical image from the images captured by the image capturing device 10 will be described with reference to FIGs. 3 A to 3C and FIGs. 4A and 4B. FIG. 3 A illustrates a hemispherical image (front side) captured by the image capturing device 10. FIG. 3B illustrates a hemispherical image (back side) captured by the image capturing device 10. FIG. 3C illustrates an image in equirectangular projection, which is hereinafter referred to as an "equirectangular projection image" (or equidistant cylindricalFN202501296projection image). The equirectangular projection image may be an image represented by Mercator projection. The image represented by Mercator projection is hereinafter referred to as a "Mercator image". FIG. 4 A conceptually illustrates an example of how the equirectangular projection image is mapped to a sphere. FIG. 4B illustrates a spherical image. The term "equirectangular projection image" refers to a spherical image in equirectangular projection format, which is an example of the wide-view image described above.

[0017] As illustrated in FIG. 3 A, an image obtained by the imaging element 103a is a curved hemispherical image (front side) captured through a wide-angle lens 102a such as a fisheye lens described below. As illustrated in FIG. 3B, an image obtained by the imaging element 103b is a curved hemispherical image (back side) captured through a wide-angle lens 102b such as a fisheye lens described below. The image capturing device 10 combines the hemispherical image (front side) and the hemispherical image (back side), which are flipped 180 degrees, to generate an equirectangular projection image EC as illustrated in FIG. 3C.

[0018] The image capturing device 10 uses software, such as Open Graphics Library for Embedded Systems (OpenGL ES), to map the equirectangular projection image EC to the sphere so as to cover the surface of the sphere in a manner illustrated in FIG. 4A to generate the spherical image CE as illustrated in FIG. 4B. The spherical image CE is represented as the equirectangular projection image EC, which corresponds to a surface facing the center of the sphere. OpenGL ES is a graphics library used for visualizing two-dimensional (2D) data and three-dimensional (3D) data. OpenGL ES is an example of software for executing image processing. Any other software may be used to create the spherical image CE. The spherical image CE may be either a still image or a moving image. Although the image capturing device 10 generates a spherical image in the above description, a communication control apparatus 5, a communication terminal 7, or a communication terminal 9 may performFN202501296substantially the same image processing or a part of the image processing instead of the image capturing device 10.

[0019] OpenGL ES is used to map the Mercator image to a sphere in such a manner as to cover the surface of the sphere as illustrated in FIG. 4A. As a result, as illustrated in FIG. 4B, a spherical image is generated.That is, the spherical image is represented as the Mercator image, which corresponds to a surface facing the center of the sphere.

[0020] As described above, since the spherical image CE is an image mapped to a sphere in such a manner as to cover the surface of the sphere, part of the image may look distorted when viewed by a user, providing a strange feeling. Accordingly, an image of a predetermined area that is part of the spherical image CE is displayed as a less distorted planar image having fewer curves on the communication terminal 7 or 9 to make the user feel comfortable. The image of the predetermined area is hereinafter referred to as a "predetermined-area image". The display of the predetermined-area image will be described with reference to FIGs. 5 to 8.

[0021] FIG. 5 is a diagram illustrating the position of a virtual camera IC1 and the position of a predetermined area T in a case where the spherical image CE is of a three-dimensional sphere CS. The position of the virtual camera IC1 corresponds to the position of the virtual viewpoint of the user viewing the spherical image CE represented as a surface area of the three-dimensional solid sphere CS. FIG. 6A is a perspective view of the virtual camera IC1 and the predetermined area T illustrated in FIG. 5. FIG. 6B illustrates a predetermined-area image Q obtained in the state illustrated in FIG. 6A and displayed on a display. FIG. 6C illustrates a predetermined area T' obtained by changing the viewpoint of the virtual camera IC1 illustrated in FIG. 6A. FIG. 6D illustrates a predetermined-area image Q' obtained in the state illustrated in FIG. 6C and displayed on the display.FN202501296

[0022] Assuming that the spherical image CE generated in the way described above is a surface area of the sphere CS, the virtual camera IC1 is inside the spherical image CE as illustrated in FIG.5. The predetermined area T in the spherical image CE is an imaging area of the virtual camera ICE Specifically, the predetermined area T is specified by field-of-view information indicating an imaging direction and a field of view of the virtual camera IC1 in a three-dimensional virtual space including the spherical image CE.

[0023] Zooming in or out of the predetermined area T may be implemented by bringing the virtual camera IC1 closer to or farther away from the spherical image CE. The predetermined-area image Q is an image of the predetermined area T in the spherical image CE. The predetermined area T is defined by a field of view a and a distance f from the virtual camera IC1 to the spherical image CE.

[0024] When the virtual viewpoint of the virtual camera IC1 is shifted, or changed, from the state illustrated in FIG. 6Ato the right (left in the drawing) as illustrated in FIG. 6C, the predetermined area T in the spherical image CE is shifted to the predetermined area T'.Accordingly, the predetermined-area image Q displayed on a predetermined display is changed to the predetermined-area image Q'. As a result, the image displayed on the predetermined display changes from the image illustrated in FIG. 6B to the image illustrated in FIG. 6D.

[0025] The relationship between the field-of-view information and the image of the predetermined area T will be described with reference to FIGs. 7 and 8.FIG. 7 illustrates a point in a three-dimensional Euclidean space defined in spherical coordinates. FIG. 8 conceptually illustrates the relationship between the predetermined area T and a point of interest (e.g., a center point CP).FN202501296

[0026] In FIG. 7, the center point CP is represented by a spherical polar coordinate system to obtain position coordinates (r, 9, cp). The position coordinates (r, 9, cp) represent a radius vector, a polar angle, and an azimuth angle, respectively. The radius vector r is a distance from the origin of a three-dimensional virtual space including the spherical image CE to any point (in FIG. 8, the center point CP). Accordingly, the radius vector r is equal to the distance f illustrated in FIG. 8.

[9927] As illustrated in FIG. 8, when the center of the predetermined area T, which is the imaging area of the virtual camera IC1, is assumed to be the center point CP illustrated in FIG. 7, a trigonometric function equation typically expressed by Formula (1) below is satisfied.

[9928] (L / f) = tan(a / 2) (1)

[9929] In Formula (1), f denotes the distance from the virtual camera IC1 to the center point CP of the predetermined area T. L denotes the distance between the center point CP and a given vertex of the predetermined area T, while 2L is a diagonal line, and a denotes the angle of view. In this case, the field-of-view information for specifying the predetermined area T can be represented by pan (9), tilt (cp), and field of view (fov) (a) values. Zooming in or out of the predetermined area T may be determined by increasing or decreasing the range (arc) of the angle of view a.

[9939] Overview of Communication SystemAn overview of a communication system 1 according to an embodiment of the present disclosure will be described with reference to FIG. 9. FIG. 9 is a schematic diagram illustrating an example configuration of the communication system 1.

[9931] FN202501296As illustrated in FIG. 9, the communication system 1 includes the communication control apparatus 5, the image capturing device 10, the relay device 3, the communication terminal 7, and communication terminals 9a and 9b. The communication terminals 9a and 9b are individually referred to as a "communication terminal 9" or collectively referred to as "communication terminals 9". The communication control apparatus 5, the communication terminal 7, and the communication terminal 9 are examples of an information processing apparatus. Each of the communication terminals 7 and 9 may be referred to as a "display terminal" for displaying, for example, an image.

[0032] As described above, the image capturing device 10 is a digital camera for capturing a wide-view image (such as a spherical image). The relay device 3 has a cradle function for charging the image capturing device 10 and transmitting and receiving data to and from the image capturing device 10. The relay device 3 can communicate with the image capturing device 10 via a contact point and can communicate with the communication control apparatus 5 via a communication network 100. Examples of the communication network 100 include the Internet, a local area network (LAN), and a (wireless) router.

[0033] The communication control apparatus 5 is, for example, a computer, and can communicate with the relay device 3 and the communication terminals 7 and 9 via the communication network 100. The communication control apparatus 5 manages, for example, field-of-view information, and thus may be referred to as an "information management apparatus".

[0034] The communication terminals 7 and 9 are computers such as notebook personal computers (PCs), and can communicate with the communication control apparatus 5 via the communication network 100. Each of the communication terminals 7 and 9 is installed with OpenGL ES and generates the predetermined-area image (see FIGs. 6A to 6D) from the spherical image received from the communication control apparatus 5.FN202501296The communication control apparatus 5 may be implemented by a single computer or a plurality of computers.

[0035] The image capturing device 10 and the relay device 3 are placed at predetermined positions by, for example, an organizer X in a site Sa such as a construction site, an exhibition site, an education site, or a medical site. The communication terminal 7 is operated by the organizer X. The communication terminal 9a is operated by a participant A such as a viewer at a remote location from the site Sa. The communication terminal 9b is operated by a participant B such as a viewer at a remote location from the site Sa. The participant A and the participant B may be located in the same location or different locations.

[0036] The communication control apparatus 5 transmits (distributes) the wide-view image, which is obtained from the image capturing device 10 via the relay device 3, to the communication terminals 7 and 9. The communication control apparatus 5 further transmits (distributes) a captured image, which is obtained from the communication terminal 7 or 9 via the relay device 3, to the communication terminal 9 or 7. The captured image transmitted from the image capturing device 10 via the relay device 3 is a wide-view image. In a case where a single-lens reflex camera is used instead of the image capturing device 10, the captured image is a standard narrow-field-of-view image. The captured image may be a moving image or a still image.

[0037] Hardware ConfigurationNext, the hardware configurations of the image capturing device 10, the relay device 3, the communication control apparatus 5, and the communication terminals 7 and 9 according to the present embodiment will be described in detail with reference to FIGs. 10 to 12.

[0038] Hardware Configuration of Image Capturing DeviceFN202501296FIG. 10 is a block diagram illustrating an example hardware configuration of the image capturing device 10. As illustrated in FIG. 10, the image capturing device 10 includes an imaging unit 101, an image processor 104, an imaging controller 105, a microphone 108, an audio processor 109, a central processing unit (CPU) 111, a read only memory (ROM) 112, a static random access memory (SRAM) 113, a dynamic random access memory (DRAM) 114, the operation unit 115, an input / output interface (I / F) 116, a short-range communication circuit 117, an antenna 117a for the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network I / F 121.

[0039] The imaging unit 101 includes two wide-angle lenses (so-called fish-eye lenses) 102a and 102b (collectively referred to as lens 102 unless distinguished), each having an angle of view of equal to or greater than 180 degrees so as to form a hemispherical image. The imaging unit 101 further includes two imaging elements 103a and 103b corresponding to the lenses 102a and 102b, respectively.

[0040] Each of the imaging elements 103 a and 103b includes an image sensor such as a complementary metal oxide semiconductor (CMOS) sensor or a charge-coupled device (CCD) sensor, a timing generation circuit, and a group of registers. The image sensor converts an optical image formed by the lens 102a or 102b into an electric signal and outputs image data. The timing generation circuit generates horizontal or vertical synchronization signals, pixel clocks, and the like for the image sensor. In the group of registers, various commands, parameters, and the like for an operation of the imaging element 103 a or 103b are set. As a non-limiting example, the imaging unit 101 includes two wide-angle lenses. The imaging unit 101 may include one wide-angle lens or three or more wide-angle lenses.

[0041] Each of the imaging elements 103a and 103b of the imaging unit 101 is connected to the image processor 104 via a parallel I / F bus. Further, each of the imaging elements 103 a andFN202501296103b of the imaging unit 101 is connected to the imaging controller 105 via a serial I / F bus such as an inter-integrated circuit (I2C) bus.

[0042] The image processor 104, the imaging controller 105, and the audio processor 109 are connected to the CPU 111 via a bus 110. The ROM 112, the SRAM 113, the DRAM 114, the operation unit 115, the input / output I / F 116, the short-range communication circuit 117, the electronic compass 118, the gyro sensor 119, the acceleration sensor 120, and the network I / F 121 are also connected to the bus 110.

[0043] The image processor 104 acquires respective items of image data output from the imaging elements 103a and 103b via the parallel I / F buses and performs predetermined processing on the items of image data. Thereafter, the image processor 104 combines the items of image data to generate data of an equirectangular projection image (an example of a wide-view image) described below.

[0044] The imaging controller 105 usually functions as a master device while each of the imaging elements 103a and 103b usually functions as a slave device. The imaging controller 105 sets commands and the like in the group of registers of each of the imaging elements 103a and 103b via the I2C bus. The imaging controller 105 receives various commands from the CPU 111. The imaging controller 105 further acquires status data and the like of the group of registers of each of the imaging elements 103a and 103b via the I2C bus. The imaging controller 105 sends the acquired status data and the like to the CPU 111.

[0045] The imaging controller 105 instructs the imaging elements 103a and 103b to output the image data at the time when a shutter button of the operation unit 115 is pressed. In some cases, the image capturing device 10 displays a preview image on a display (e.g., a display of an external terminal such as a smartphone that performs short-range communication with theFN202501296image capturing device 10 through the short-range communication circuit 117) or displays a moving image (movie). In the case of displaying a movie, image data are continuously output from the imaging elements 103a and 103b at a predetermined frame rate (expressed in frames per minute).

[0046] As described below, the imaging controller 105 operates in cooperation with the CPU 111 to synchronize the time when the imaging element 103a outputs image data and the time when the imaging element 103b outputs image data. In the present embodiment, the image capturing device 10 does not include a display unit (or display). In some embodiments, the image capturing device 10 may include a display unit. The microphone 108 converts sound to audio data (signals).The audio processor 109 acquires the audio data output from the microphone 108 via an I / F bus and performs predetermined processing on the audio data.

[0047] The CPU 111 controls entire operation of the image capturing device 10 and performs predetermined processing.The ROM 112 stores various programs for execution by the CPU 111. Each of the SRAM 113 and the DRAM 114 operates as a work memory to store programs to be executed by the CPU 111 or data being currently processed. More specifically, in one example, the DRAM 114 stores image data currently processed by the image processor 104 and data of the equirectangular projection image on which processing has been performed.

[0048] The operation unit 115 collectively refers to various operation buttons such as a shutter button, a power switch, a touch panel having both the display and operation functions, and the like. The user operates the operation unit 115 to input various image capturing modes or image capturing conditions.

[0049] FN202501296The input / output I / F 116 collectively refers to an interface circuit such as a universal serial bus (USB) I / F that allows the image capturing device 10 to communicate with an external medium such as a Secure Digital (SD) card or an external personal computer. The input / output I / F 116 may be either wired or wireless. The data of the equirectangular projection image, which is stored in the DRAM 114, is stored in the external medium via the input / output I / F 116 or transmitted to an external terminal (apparatus) via the input / output I / F 116, as desired.

[0050] The short-range communication circuit 117 communicates with the external terminal (apparatus) via the antenna 117a of the image capturing device 10 by short-range wireless communication technology such as near field communication (NFC), Bluetooth®, or Wi-Fi®. The short-range communication circuit 117 can transmit the data of the equirectangular projection image to the external terminal (apparatus).

[0051] The electronic compass 118 calculates an orientation of the image capturing device 10 from the Earth's magnetism and outputs orientation information. The orientation information is an example of related information (metadata) in compliance with exchangeable image file format (Exif). The orientation information is used for image processing such as image correction of a captured image. The related information also includes data of a date and time when the image was captured, and data of a data size of image data.

[0052] The gyro sensor 119 detects a change in tilt (roll, pitch, and yaw) of the image capturing device 10 with movement of the image capturing device 10. The change in tilt is one example of related information (metadata) in compliance with Exif. This information is used for image processing such as image correction of a captured image.

[0053] The acceleration sensor 120 detects acceleration in three axial directions.FN202501296

[0054] The image capturing device 10 may also calculate the position (an angle with respect to the direction of gravity) of the image capturing device 10 using, for example, the electronic compass 118 and the acceleration sensor 120. The acceleration sensor 120 of the image capturing device 10 improves the accuracy of image correction.

[0055] The network I / F 121 is an interface for performing data communication using the communication network 100, such as the Internet, via a router or the like. The hardware elements of the image capturing device 10 are not limited to the illustrated ones as long as the functional configuration of the image capturing device 10 can be implemented. At least some of the hardware elements described above may reside on the relay device 3 or the communication network 100.

[0056] Hardware Configuration of Relay DeviceFIG. 11 is a block diagram illustrating an example hardware configuration of the relay device 3. In FIG. 11, the relay device 3 is a cradle having a wireless communication function.

[0057] As illustrated in FIG. 11, the relay device 3 includes a CPU 301, a ROM 302, a RAM 303, an electrically erasable and programmable ROM (EEPROM) 304, a CMOS sensor 305, a bus line 310, a communication device 313, an antenna 313a, a positioning device 314, and an input / output I / F 316.

[0058] The CPU 301 controls entire operation of the relay device 3. The ROM 302 stores an initial program loader (IPL) or any other program used for booting the CPU 301. The RAM 303 is used as a work area for the CPU 301.

[0059] FN202501296The EEPROM 304 reads and writes data under the control of the CPU 301. The EEPROM 304 stores an operating system (OS) to be executed by the CPU 301, other programs, and various types of data.

[0060] The CMOS sensor 305 is a solid-state imaging element that captures an image of an object under the control of the CPU 301 to obtain image data.

[0061] The communication device 313 performs communication with the communication network 100 through the antenna 313a by using a wireless communication signal.

[0062] The positioning device 314 receives a positioning signal including position information (latitude, longitude, and altitude) of the relay device 3 using a global navigation satellite system (GNSS) satellite such as a global positioning system (GPS) satellite or using an Indoor Messaging System (IMES) serving as an indoor GPS.

[0063] The input / output I / F 316 is an interface circuit, such as a USB I / F, which is electrically connected to the input / output I / F 116 of the image capturing device 10. The input / output I / F 316 may be either wired or wireless.

[0064] The bus line 310 is, for example, an address bus or a data bus for electrically connecting the components such as the CPU 301 to one another.

[0065] Hardware Configuration of Communication Control Apparatus and Communication Terminal FIG. 12 is a block diagram illustrating an example hardware configuration of the communication control apparatus 5. The hardware configuration of each of the communication terminals 7 and 9 is substantially the same as that of the communication control apparatus 5, and thus the description thereof will be omitted.FN202501296

[0066] As illustrated in FIG. 12, the communication control apparatus 5 includes, as a computer, a CPU 501, a ROM 502, a RAM 503, a solid-state drive (SSD) 504, an external device connection I / F 505, a network I / F 506, a display 507, an operation device 508, a medium I / F 509, a bus line 510, a CMOS sensor 511, a speaker 512, and a positioning device 514.

[0067] The CPU 501 controls entire operation of the communication control apparatus 5. The ROM 502 stores an IPL or any other program used for booting the CPU 501. The RAM 503 is used as a work area for the CPU 501.

[0068] The SSD 504 reads and writes various types of data under the control of the CPU 501. In one example, the communication terminals 7 and 9 are smartphones or the like, and each of the communication terminals 7 and 9 does not include the SSD 504. In one example, the communication control apparatus 5 includes a hard disk drive (HDD) in place of the SSD 504. The same applies to the communication terminals 7 and 9.

[0069] The external device connection I / F 505 is an interface for connecting the communication control apparatus 5 to various external devices. The external devices include, but are not limited to, a display, a speaker, a keyboard, a mouse, a USB memory, and a printer.

[0070] The network I / F 506 is an interface for performing data communication via the communication network 100.

[0071] The display 507 is a type of display device such as a liquid crystal display or an organic electroluminescent (EL) display that displays various images.

[0072] FN202501296The operation device 508 is an input means operated by the user to select or execute various instructions, select a target for processing, or move a cursor being displayed. Examples of the input means include various operation buttons, a power switch, a shutter button, and a touch panel.

[0073] The medium I / F 509 controls reading or writing (storing) of data from or to a storage medium 509m such as a flash memory. Examples of the storage medium 509m include a digital versatile disc (DVD) and a Blu-ray Disc®.

[0074] The CMOS sensor 511 is a type of imaging means for capturing an image of an object under the control of the CPU 501 to obtain image data. The communication control apparatus 5 may include a CCD sensor in place of the CMOS sensor 511.

[0075] The speaker 512 is a circuit that converts an electric signal into physical vibration to generate sound such as music or voice.

[0076] The positioning device 514 receives a positioning signal including position information (latitude, longitude, and altitude) of each of the communication terminals 7 and 9 using a GNSS satellite such as a GPS satellite or using an IMES serving as an indoor GPS.

[0077] The bus line 510 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 501 to each other.

[0078] Functional ConfigurationsNext, an example functional configuration of the communication system 1 according to the present embodiment will be described with reference to FIG. 13.

[0079] FN202501296Functional Configuration of Image Capturing DeviceAs illustrated in FIG. 13, the image capturing device 10 includes a reception unit 12, a detection unit 13, an imaging unit 16, a sound collection unit 17, a connection unit 18, and a storing / reading unit 19. The components of the image capturing device 10 are functions or means implemented by any one of the hardware elements illustrated in FIG. 10 operating in accordance with instructions from the CPU 111 according to a program for the image capturing device 10 loaded onto the DRAM 114 from the SRAM 113.

[0080] The image capturing device 10 further includes a storage unit 1000, which is implemented by the ROM 112, the SRAM 113, and the DRAM 114 illustrated in FIG. 10.

[0081] Functional Elements of Image Capturing DeviceThe reception unit 12 of the image capturing device 10 is implemented by the operation unit 115 operating in accordance with instructions from the CPU 111. The reception unit 12 receives an operation input from the user.

[0082] The detection unit 13 is implemented by, for example, the electronic compass 118, the gyro sensor 119, and the acceleration sensor 120 operating in accordance with instructions from the CPU 111. The detection unit 13 detects the position of the image capturing device 10 to obtain position information.

[0083] The imaging unit 16 is implemented by the imaging unit 101, the image processor 104, and the imaging controller 105 operating in accordance with instructions from the CPU 111. The imaging unit 16 obtains a captured image of scenery and objects.

[0084] FN202501296The sound collection unit 17 is implemented by the audio processor 109 operating in accordance with instructions from the CPU 111. The sound collection unit 17 picks up sounds around the image capturing device 10.

[0085] The connection unit 18 is implemented by the input / output I / F 116 operating in accordance with instructions from the CPU 111. The connection unit 18 performs data communication with the relay device 3.

[0086] The storing / reading unit 19 is implemented by operation of the CPU 111. The storing / reading unit 19 stores various types of data (or information) in the storage unit 1000 or reads various types of data (or information) from the storage unit 1000.

[0087] Functional Configuration of Relay DeviceAs illustrated in FIG. 13, the relay device 3 includes a communication unit 31 and a connection unit 38. The components of the relay device 3 are functions or means implemented by any one of the hardware elements illustrated in FIG. 11 operating in accordance with instructions from the CPU 301 according to a program for the relay device 3 loaded onto the RAM 303 from the EEPROM 304.

[0088] Functional Elements of Relay DeviceThe communication unit 31 of the relay device 3 is implemented by the communication device 313 operating in accordance with instructions from the CPU 301 illustrated in FIG. 11. The communication unit 31 performs data communication with the image capturing device 10 and the communication control apparatus 5 via the communication network 100.

[0089] FN202501296The connection unit 38 is implemented by the input / output I / F 316 operating in accordance with instructions from the CPU 301. The connection unit 38 performs data communication with the image capturing device 10.

[0090] Functional Configuration of Communication Control ApparatusNext, the functional configuration of the communication control apparatus 5 will be described in detail with reference to FIG. 13. The communication control apparatus 5 includes a communication unit 51, a reception unit 52, a generation unit 53, a processing unit 54, an authentication unit 55, and a storing / reading unit 59. The components of the communication control apparatus 5 are functions or means implemented by any one of the hardware elements illustrated in FIG. 12 operating in accordance with instructions from the CPU 501 according to a program for the communication control apparatus 5 loaded onto the RAM 503 from the SSD 504.

[0091] The communication control apparatus 5 further includes a storage unit 5000, which is implemented by the RAM 503 and the SSD 504 illustrated in FIG. 12. The storage unit 5000 includes a user device management database (DB) 5001, a virtual room management DB 5002, and a three-dimensional image management DB 5003.

[0092] User Device Management DBFIG. 14 schematically illustrates a user device management table. The user device management DB 5001 includes the user device management table illustrated in FIG. 14. The user device management table stores a user ID (or device ID), a password, a name, a user image, and an Internet protocol (IP) address in association with one another as data items to be managed.

[0093] FN202501296The user ID is an example of user identification information for identifying a user, such as the organizer X, the participant A, or the participant B. The device ID is an example of device identification information for identifying a device, such as the image capturing device 10. In one embodiment, a head-mounted display or the like other than the image capturing device 10 is used. In this case, the head-mounted display or the like is also identified as a device.

[0094] The name is the name of the user or the device. The name of the user may be the name of the communication terminal used by the user.

[0095] The user image is registered in advance by the user. Examples of the user image include a schematic image of the face of the user, and a photograph of the face of the user.

[0096] The IP address is an example of information for specifying the address of a device used by the user, such as the image capturing device 10 or the communication terminal 7 or 9.

[0097] Virtual Room Management DBFIG. 15 schematically illustrates a virtual room management table. The virtual room management DB 5002 includes the virtual room management table illustrated in FIG. 15. The virtual room management table stores a virtual room ID, a virtual room name, a device ID, an organizer ID, a participant ID, a content ID, a content uniform resource locator (URL) (storage location information of content data including images and sounds), and a three-dimensional image ID in association with one another as data items to be managed.

[0098] The virtual room ID is an example of virtual room identification information for identifying a virtual room.

[0099] The virtual room name is the name of the virtual room and is assigned by, for example, a user.FN202501296

[0100] The device ID is the same as the device ID illustrated in FIG. 14 and is the ID of a device that has joined the virtual room indicated by the virtual room ID in the same record.

[0101] The organizer ID is the ID of an organizer participating in the virtual room indicated by the virtual room ID in the same record. The organizer ID is an example of organizer identification information for identifying an organizer among users indicated by user IDs illustrated in FIG.14.

[0102] The participant ID is the ID of a participant participating in the virtual room indicated by the virtual room ID in the same record. The participant ID is an example of participant identification information for identifying a participant among the users indicated by the user IDs illustrated in FIG. 14.

[0103] The content ID is an example of content identification information for identifying content data including images and sounds. The images include a wide-view image that has been captured, and the sounds include a sound (including voice) obtained during the capture of the wide-view image.

[0104] The content URL is an example of content storage location information indicating a location where content (wide-view image and audio information) data is stored. The content URL is stored in association with the content data, the time of image capturing (or recording) and audio capturing (or recording), and the image capturing position (absolute position on the earth).

[0105] The three-dimensional image ID is an example of three-dimensional image identification information for identifying a first three-dimensional image displayed in a display area 620AFN202501296described below and a second three-dimensional image displayed in a display area 620B described below. The first three-dimensional image is a three-dimensional image displayed in the display area 620A based on design data. The second three-dimensional image is a three-dimensional image displayed in the display area 620B based on, for example, point cloud data acquired by, for example, a 3D scanner. The first three-dimensional image and the second three-dimensional image may be managed by separate IDs.

[0106] Three-dimensional Image Management DBFIG. 16A is a schematic diagram of a three-dimensional image management table. The three-dimensional image management DB 5003 includes the three-dimensional image management table illustrated in FIG. 16 A. The three-dimensional image management table illustrated in FIG. 16A stores, for each three-dimensional image ID, a model ID, position information, and a point cloud acquisition date in association with one another as data items to be managed.

[0107] The model ID is an example of three-dimensional model identification information for identifying a three-dimensional model. The three-dimensional model is generated based on a point cloud acquired by, for example, a time-of-flight (TOF) camera.

[0108] The position information indicates the position of the three-dimensional model in a three-dimensional virtual space using three-dimensional coordinates of XYZ. The position information is indicated by, for example, the three-dimensional coordinates of eight points defining a rectangular parallelepiped space occupied by the three-dimensional model. The position information is managed in association with an absolute position on the earth, based on, for example, a positioning signal including position information (latitude, longitude, and altitude) of the TOF camera and three-dimensional coordinates of a point cloud with respect to a reference position of the TOF camera. The position information of the TOF camera is received from a GNSS satellite such as a GPS satellite or an IMES serving as an indoor GPS.FN202501296

[0109] The point cloud acquisition date is an example of information indicating the date on which a point cloud (point cloud data) is acquired by, for example, the TOF camera. In FIG. 16 A, as an example, the date on which point cloud data is acquired is indicated by year, month, and day. The point cloud acquisition date may be information indicating the date and time at which point cloud data is acquired.

[0110] FIG. 16B is a schematic diagram of another example of the three-dimensional image management table. The three-dimensional image management DB 5003 includes the three-dimensional image management table illustrated in FIG. 16B. The three-dimensional image management table illustrated in FIG. 16B is a table for managing attribute information indicating attributes of components (three-dimensional models) of a structure in a three-dimensional virtual space. The three-dimensional image management table illustrated in FIG.16B stores, for each three-dimensional image ID, a component number (component No.), component information, dimension information, color information, material information, position information, and construction date information in association with one another as attribute information to be managed. The component number, the component information, the dimension information, the color information, the material information, the position information, and the construction date information are acquired from, for example, design data.

[0111] The component information is information for identifying a component that is made up of various parts. Examples of the component include a wall, a floor, a ceiling, a window, a pipe, and a door.

[0112] FN202501296The dimension information is information for identifying the dimensions of the component in the virtual space and is indicated by, for example, numerical values in three-axis directions of XYZ.

[0113] The color information is information for identifying the color of the component. The material information is information for identifying the material of the component.

[0114] The position information is information for identifying the position of the component in the virtual space and is indicated by, for example, coordinates in three-axis directions of XYZ. The position information may be used to determine whether multiple components are adjacent to each other. The position information is acquired from, for example, design data such as computer-aided design (CAD) data or building information modeling (BIM) data.

[0115] The construction date information is information indicating a scheduled date when the component is scheduled to be constructed in the real world. With the construction date information, a structure excluding an unconstructed component at a certain point in time can be identified. In a case where the scheduled date when which the component is scheduled to be constructed is changed in the real world, the construction date information may be changed to the changed scheduled date. In the three-dimensional image management table illustrated in FIG. 16B, construction period information may further be stored in association with the construction date information. The construction period information indicates a period during which the component is to be constructed and is acquired from, for example, the design data.

[0116] As described above, the position information illustrated in FIG. 16A and the position information illustrated in FIG. 16B are stored in association with the absolute position on the earth or the indoor position. For example, by associating the origin (X = 0, Y = 0, Z = 0) of the position information illustrated in FIG. 16A and the position information illustrated inFN202501296FIG. 16B with the absolute positions (latitudes, longitudes, and altitudes) on the earth, all coordinates in the first three-dimensional image and the second three-dimensional image, including the three-dimensional models or components, are associated with the absolute positions on the earth.

[0117] In the three-dimensional image management DB 5003, for example, a three-dimensional point cloud, a mesh object, and a textured mesh object may be managed instead of the three-dimensional models.

[0118] Functional Elements of Communication Control ApparatusNext, the functional elements of the communication control apparatus 5 will be described in detail with reference to FIG. 13.

[0119] The communication unit 51 of the communication control apparatus 5 is implemented by the network I / F 506 operating in accordance with instructions from the CPU 501 illustrated in FIG. 12. The communication unit 51 performs data communication with other devices, such as the relay device 3 and the communication terminals 7 and 9, via the communication network 100. The communication unit 51 also functions as an acquisition unit and acquires instruction information indicating an instruction transmitted from the communication terminals 7 and 9.

[0120] The reception unit 52 is implemented by the operation device 508 operating in accordance with instructions from the CPU 501. The reception unit 52 receives an operation input from the user (e.g., a system administrator or the like).

[0121] The generation unit 53 is implemented by operation of the CPU 501. The generation unit 53 generates a screen to be transmitted to each of the communication terminals 7 and 9, using,FN202501296for example, the data stored in the storage unit 5000. For example, the generation unit 53 generates a screen 600 (see FIG. 25). The screen 600 includes a display area 610, a display area 620A, and a display area 620B. The display area 610 displays a predetermined-area image representing a predetermined area in a wide-view image (an example of a captured image) captured by the image capturing device 10. The display area 620A displays at least a portion of a first three-dimensional image. The display area 620B displays at least a portion of a second three-dimensional image. In the first three-dimensional image and the second three-dimensional image, a second position is associated with a first position in the wide-view image by the processing unit 54.

[0122] The processing unit 54 is implemented by operation of the CPU 501. The processing unit 54 performs a process for associating position information indicating a position (an example of the first position) in a captured image of an object, which is captured by the image capturing device 10, with position information indicating a position (an example of the second position) in the first three-dimensional image and the second three-dimensional image, each of which includes a three-dimensional area corresponding to the object. The processing unit 54 may also be referred to as an "association unit".

[0123] The authentication unit 55 performs authentication to determine, for example, whether each user is authorized to use the virtual room.

[0124] The storing / reading unit 59 is implemented by operation of the CPU 501. The storing / reading unit 59 stores various types of data (or information) in the storage unit 5000 or reads various types of data (or information) from the storage unit 5000.

[0125] Functional Configuration of Communication Terminal 7FN202501296Next, the functional configuration of the communication terminal 7 will be described in detail with reference to FIG. 13. The communication terminal 7 includes a communication unit 71, a reception unit 72, a generation unit 73, a display control unit 74, a sound input / output control unit 75, a connection unit 78, and a storing / reading unit 79. The components of the communication terminal 7 are functions or means implemented by any one of the hardware elements illustrated in FIG. 12 operating in accordance with instructions from the CPU 501 according to a program for the communication terminal 7 loaded onto the RAM 503 from the SSD 504.

[0126] The communication unit 71 of the communication terminal 7 is implemented by the network I / F 506 operating in accordance with instructions from the CPU 501 illustrated in FIG. 12. The communication unit 71 performs data communication with other devices, such as the communication control apparatus 5, via the communication network 100.

[0127] The reception unit 72 is implemented by the operation device 508 operating in accordance with instructions from the CPU 501. The reception unit 72 receives an operation input from the user (i.e., the organizer X). The reception unit 72 also functions as an acquisition unit and acquires an instruction provided through a user operation.

[0128] The generation unit 73 is implemented by operation of the CPU 501. The generation unit 73 generates a screen to be displayed on the communication terminal 7, using, for example, data stored in a storage unit 7000. In a case where the generation unit 53 of the communication control apparatus 5 generates a screen to be displayed on the communication terminal 7, the communication terminal 7 may or may not include the generation unit 73.

[0129] FN202501296The display control unit 74 is implemented by operation of the CPU 501. The display control unit 74 controls the display 507 of the communication terminal 7 or an external display connected to the external device connection I / F 505 to display various images.

[0130] The sound input / output control unit 75 is implemented by operation of the CPU 501 of the communication terminal 7. The sound input / output control unit 75 controls an external microphone connected to the external device connection I / F 505 to collect sound. In one example, the communication terminal 7 includes a microphone. In this case, the sound input / output control unit 75 controls the microphone to collect sound. The sound input / output control unit 75 further controls the speaker 512 of the communication terminal 7 or an external speaker connected to the external device connection I / F 505 to output sound.

[0131] The generation unit 73 is implemented by operation of the CPU 501. The generation unit 73 adds a voice-over or subtitles to video and audio content data recorded by the communication terminal 7 to create content data such as for teaching materials.

[0132] The storing / reading unit 79 is implemented by operation of the CPU 501. The storing / reading unit 79 stores various types of data (or information) in the storage unit 7000 or reads various types of data (or information) from the storage unit 7000.

[0133] Functional Configuration of Communication Terminal 9Next, the functional configuration of the communication terminal 9 will be described in detail with reference to FIG. 13.

[0134] The communication terminal 9 includes a communication unit 91, a reception unit 92, a generation unit 93, a display control unit 94, a sound input / output control unit 95, a connection unit 98, and a storing / reading unit 99. The components of the communicationFN202501296terminal 9 are functions or means implemented by any one of the hardware elements illustrated in FIG. 12 operating in accordance with instructions from the CPU 501 according to a program for the communication terminal 9 loaded onto the RAM 503 from the SSD 504.

[0135] The communication terminal 9 further includes a storage unit 9000, which is implemented by the RAM 503 and the SSD 504 illustrated in FIG. 12.

[0136] The communication unit 91 of the communication terminal 9 is implemented by the network I / F 506 operating in accordance with instructions from the CPU 501. The communication unit 91 performs data communication with other devices, such as the communication control apparatus 5, via the communication network 100.

[0137] The reception unit 92 is implemented by the operation device 508 operating in accordance with instructions from the CPU 501. The reception unit 92 receives an operation input from the user (i.e., a participant). The reception unit 92 also functions as an acquisition unit and acquires an instruction provided through a user operation.

[0138] The generation unit 93 is implemented by operation of the CPU 501. The generation unit 93 generates a screen to be displayed on the communication terminal 9, using, for example, data stored in the storage unit 9000. In a case where the generation unit 53 of the communication control apparatus 5 generates a screen to be displayed on the communication terminal 9, the communication terminal 9 may or may not include the generation unit 93.

[0139] The display control unit 94 is implemented by operation of the CPU 501. The display control unit 94 controls the display 507 of the communication terminal 9 or an external display connected to the external device connection I / F 505 to display various images.

[0140] FN202501296The sound input / output control unit 95 is implemented by operation of the CPU 501 of the communication terminal 9. The sound input / output control unit 95 controls an external microphone connected to the external device connection I / F 505 to collect sound. In one example, the communication terminal 9 includes a microphone. In this case, the sound input / output control unit 95 controls the microphone to collect sound. The sound input / output control unit 95 further controls the speaker 512 of the communication terminal 9 or an external speaker connected to the external device connection I / F 505 to output sound.

[0141] The connection unit 98 is implemented by the external device connection I / F 505 operating in accordance with instructions from the CPU 501. The connection unit 98 performs data communication with an external device connected to the communication terminal 9 in a wired or wireless way.

[0142] The storing / reading unit 99 is implemented by operation of the CPU 501. The storing / reading unit 99 stores various types of data (or information) in the storage unit 9000 or reads various types of data (or information) from the storage unit 9000.

[0143] Processes or OperationsNext, processes or operations according to an embodiment of the present disclosure will be described with reference to FIGs. 17 to 38B. The following processes are performed after the image capturing device 10 and the communication terminals 7 and 9 have joined the same virtual room.

[0144] Process for Communicating Content Data in Communication SystemA process for communicating content data in the communication system 1 will be described with reference to FIG. 17. FIG. 17 is a sequence diagram illustrating a process of communicating a wide-view image and field-of-view information in the communicationFN202501296system 1. In the present embodiment, the image capturing device 10, the communication terminal 7 of the organizer X, the communication terminal 9a of the participant A, and the communication terminal 9b of the participant B are in the same virtual room. In response to the creation of a virtual room, the storing / reading unit 59 adds one record to the virtual room management DB 5002 (see FIG. 15) and manages a virtual room ID, a virtual room name, a device ID, an organizer ID, and a participant ID in association with one another. A content ID, a content URL, and a three-dimensional image ID are stored later. The processing of steps Sil to S15 of FIG. 17 is repeated, for example, about 30 or 60 times per second.

[0145] In step S 11, the image capturing device 10 captures a spherical image of an area in the site Sa using the imaging unit 16, and collects sounds around the area in the site Sa using the sound collection unit 17, to obtain content (wide-view image and audio information) data. After that, the connection unit 18 transmits the content data to the relay device 3. In this case, the connection unit 18 further transmits the virtual room ID for identifying the virtual room that the image capturing device 10 has joined, and the device ID for identifying the image capturing device 10. Accordingly, the connection unit 38 of the relay device 3 acquires the content data, the virtual room ID, and the device ID.

[0146] In step SI 2, the communication unit 31 of the relay device 3 transmits the content data, the virtual room ID, and the device ID, which are received by the connection unit 38 in step S 11, to the communication control apparatus 5 via the communication network 100. Accordingly, the communication unit 51 of the communication control apparatus 5 receives the content data, the virtual room ID, and the device ID.

[0147] The image capturing device 10 may transmit the content data, the virtual room ID, and the device ID to the communication terminal 7 instead of the relay device 3 (step Slid). In thisFN202501296case, the communication terminal 7 transmits the content data, the virtual room ID, and the device ID to the communication control apparatus 5 (step S12d).

[0148] In step S 13, in the communication control apparatus 5, the storing / reading unit 59 searches the virtual room management DB 5002 based on the virtual room ID received in step S12 to read the user IDs (the organizer ID and the participant IDs) of the users who are participating in the same virtual room as the image capturing device 10. The storing / reading unit 59 also searches the user device management DB 5001 based on the read organizer ID and participant IDs to read the corresponding user images of the organizer X, the participant A, and the participant B and the corresponding IP addresses of the communication terminal 7, the communication terminal 9a, and the communication terminal 9b. The communication unit 51 refers to the IP address of the communication terminal 7 and transmits screen data including the content data received in step S12 to the communication terminal 7. Accordingly, the communication unit 71 of the communication terminal 7 receives the screen data. This processing (screen display process) will be described in detail below.

[0149] In step SI 4, the communication unit 51 of the communication control apparatus 5 refers to the IP address of the communication terminal 9a and transmits the content data received in step S12 to the communication terminal 9a. Accordingly, the communication unit 91 of the communication terminal 9a receives the content data. This processing (screen display process) will be described in detail below.

[0150] In step SI 5, similarly, the communication unit 51 of the communication control apparatus 5 refers to the IP address of the communication terminal 9b and transmits the content data received in step S12 to the communication terminal 9b. Accordingly, the communication unit 91 of the communication terminal 9b receives the content data. This processing (screen display process) will be described in detail below.FN202501296

[0151] Process for Starting Image and Sound Recording in Communication SystemA process for starting image and sound recording in the communication system 1 will be described with reference to FIG. 18. FIG. 18 is a sequence diagram illustrating a process of starting image and sound recording in the communication system 1.

[0152] In step S31, in the communication terminal 7 of the organizer X, the reception unit 72 receives an operation for starting image and sound recording from the organizer X.

[0153] Before starting image and sound recording, in step S32, the communication unit 71 of the communication terminal 7 transmits an instruction to share field-of-view information (sharing instruction) to the communication control apparatus 5. The sharing instruction includes the virtual room ID of the virtual room that the communication terminal 7 has joined and the device ID of the image capturing device 10. Accordingly, the communication unit 51 of the communication control apparatus 5 receives the sharing instruction to share the field-of-view information.

[0154] In step S33, the storing / reading unit 59 of the communication control apparatus 5 stores a content URL and a field-of-view information URL in the virtual room management DB 5002 (see FIG. 15). The communication unit 51 transmits an instruction to start recording and a request to upload field-of-view information to the communication terminal 7. The instruction to start recording includes information on the content URL indicating a location where the communication terminal 7 stores the content data after recording. The request to upload field-of-view information includes information on the field-of-view information URL indicating a location where the field-of-view information is to be stored. Accordingly, in the communication terminal 7, the communication unit 71 receives the instruction to start recording and the request to upload field-of-view information.FN202501296

[0155] In step S34, the communication unit 51 further transmits a request to upload field-of-view information to the communication terminal 9a. The request to upload field-of-view information includes information on a URL indicating a location where the field-of-view information is to be stored. Accordingly, in the communication terminal 9a, the communication unit 91 receives the request to upload field-of-view information.

[0156] In step S35, similarly, the communication unit 51 transmits a request to upload field-of-view information to the communication terminal 9b. The request to upload field-of-view information includes information on a URL indicating a location where the field-of-view information is to be stored. Accordingly, in the communication terminal 9b, the communication unit 91 receives the request to upload field-of-view information.

[0157] In step S36, in the communication terminal 7, the storing / reading unit 79 serves as an image recording unit and a sound recording unit, and starts recording the content data received in step S13 illustrated in FIG. 17. In the case where the processing of step SI 2d is performed instead of step S12 of FIG. 17, the communication terminal 7 starts recording images and sounds, which are included in the content data received from the image capturing device 10 in step Slid, instead of the content data received from the communication control apparatus 5 in step S13.

[0158] In the communication terminal 7, for example, in response to the reception unit 72 receiving a change in the field of view from the organizer X while the predetermined-area image (see FIG. 6B), which is an image of a predetermined area (see FIG. 6A) in the wide-view image received in step S13, is being displayed, in step S37, the display control unit 74 displays a predetermined-area image (see FIG. 6D) indicating a predetermined area (see FIG.6C) obtained by changing the field of view of the same wide-view image. In this case, theFN202501296reception unit 72 also serves as an acquisition unit. In response to receiving the display of a predetermined area in the wide-view image from the user (i.e., the organizer X), the reception unit 72 acquires field-of-view information (pan, tilt, and fov) for specifying the predetermined area in the wide-view image to be displayed on the display 507. The communication unit 71 transmits the field-of-view information for specifying the changed predetermined area to the communication control apparatus 5, so that the field-of-view information URL received in step S33 is updated. The field-of-view information includes the user ID of the organizer X, who is the user of the communication terminal 7 from which the field-of-view information is transmitted. Accordingly, the communication unit 51 of the communication control apparatus 5 receives the field-of-view information. Then, the storing / reading unit 59 stores the user ID of the organizer X, the field-of-view information, and the receiving date and time of the information for the field-of-view URL stored in the virtual room management DB 5002 (see FIG. 15). The receiving date and time in this case is the date and time when the reception unit 72 of the communication terminal 7 receives a user operation, performed by the organizer X, for changing the field of view.

[0159] In step S38, processing that is substantially the same as the processing of step S37 is also performed by the communication terminal 9a, independently of the processing of step S37, so that the communication control apparatus 5 receives the field-of-view information. The user ID transmitted in step S38 is the user ID of the participant A.

[0160] In step S39, processing that is substantially the same as the processing of step S37 is also performed by the communication terminal 9b, independently of the processing of steps S37 and S38, so that the communication control apparatus 5 receives the field-of-view information. The user ID transmitted in step S39 is the user ID of the participant B.

[0161] FN202501296The processing of steps S37 to S39 may be collectively executed on the communication control apparatus 5 at the end of the recording.

[0162] Process for Stopping Image and Sound Recording in Communication SystemA process for stopping image and sound recording in the communication system 1 will be described with reference to FIG. 19. FIG. 19 is a sequence diagram illustrating a process of stopping image and sound recording in the communication system 1.

[0163] In step S51, in the communication terminal 7 of the organizer X, the reception unit 72 receives an operation for stopping image and sound recording from the organizer X.

[0164] In step S52, the storing / reading unit 79 stops recording the images and sounds included in the content data.

[0165] In step S53, the communication unit 71 transmits the content data including the recorded images and sounds to the communication control apparatus 5, so that the content data is uploaded to the storage location indicated by the content URL received in step S33. The content data includes times (timestamps) each indicating a time when the image and the sound are recorded, from the start to the end of the recording. Accordingly, the communication unit 51 of the communication control apparatus 5 receives the content data.

[0166] In step S54, the storing / reading unit 59 of the communication control apparatus 5 stores the content data and the timestamps in the storage location indicated by the content URL. Further, the storing / reading unit 59 converts the timestamps into an elapsed playback time in accordance with the total recording time of the content data of which the recording is stopped, and stores the timestamps and the elapsed playback time in association with each other.

[0167] FN202501296In step S55, the communication unit 51 transmits a notification of the end of recording images and sounds (a recording completion notification) to the communication terminal 7. The recording completion notification includes information indicating the content URL.Accordingly, the communication unit 71 of the communication terminal 7 receives the recording completion notification.

[0168] In step S56, similarly, the communication unit 51 transmits a notification of the end of recording images and sounds (a recording completion notification) to the communication terminal 9a. The recording completion notification includes information indicating the content URL. Accordingly, the communication unit 91 of the communication terminal 9a receives the recording completion notification.

[0169] In step S57, similarly, the communication unit 51 transmits a notification of the end of recording images and sounds (a recording completion notification) to the communication terminal 9b. The recording completion notification includes information indicating the content URL. Accordingly, the communication unit 91 of the communication terminal 9b receives the recording completion notification.

[0170] In one embodiment, in step S55, the recording completion notification does not include the content URL.

[0171] Process for Playing back Recorded Images and Sounds in Communication SystemA process for playing back recorded images and sounds in the communication system 1 will be described with reference to FIGs. 20 to 26. FIG. 20 is a sequence diagram illustrating a process of playing back recorded images and sounds in the communication system 1. FIG. 21 is a diagram illustrating a recorded data selection screen. In this example, the participant A uses the communication terminal 9a to play back recorded content data.FN202501296

[0172] In response to the reception unit 92 of the communication terminal 9a receiving a login operation of inputting a user ID and a password from the participant A, in step S71, the communication unit 91 transmits a login request to the communication control apparatus 5. The login request includes the user ID and password of the participant A. Accordingly, in the communication control apparatus 5, the communication unit 51 receives the login request, and the authentication unit 55 performs authentication by referring to the user device management DB 5001 (see FIG. 14). The following description is given assuming that the participant A is determined to be an authorized accessor by the login authentication.

[0173] In step S72, the generation unit 53 of the communication control apparatus 5 generates a recorded data selection screen 940 as illustrated in FIG. 21. In this case, the storing / reading unit 59 searches the virtual room management DB 5002 (see FIG. 15) using the user ID received in step S71 as a search key, and reads all the corresponding virtual room IDs, virtual room names, and content URLs. The generation unit 53 generates thumbnails 941, 942, and 943 using images included in the content data (with timestamps) stored in the storage locations indicated by the content URLs. As a result, the generation unit 53 assigns each thumbnail a virtual room name, such as "construction site a", and a recording time, such as "2022 / 10 / 31 15:00" (meaning 3 p.m. on October 31, 2022), indicating a predetermined time (e.g., the recording start time) of the timestamp.

[0174] In step S73, the communication unit 51 transmits selection screen data of the recorded data selection screen 940 generated in step S72 to the communication terminal 9a. The selection screen data includes, for each thumbnail, a content ID for identifying a wide-view image from which the thumbnail is generated. Accordingly, the communication unit 91 of the communication terminal 9a receives the selection screen data.

[0175] FN202501296In step S74, the display control unit 94 of the communication terminal 9a displays the recorded data selection screen 940 as illustrated in FIG. 21 on the display 507 of the communication terminal 9a. The reception unit 92 receives an operation for specifying (selecting) a particular thumbnail from the participant A. The following description will be given assuming that the thumbnail 941 is specified (selected).

[0176] In step S75, the communication unit 91 transmits a request to download the content data from which the selected thumbnail 941 is generated to the communication control apparatus 5. The request includes the content ID associated with the thumbnail 941. Accordingly, the communication unit 51 of the communication control apparatus 5 receives the request to download the content data.

[0177] In step S76, the storing / reading unit 59 of the communication control apparatus 5 searches the virtual room management DB 5002 (see FIG. 15) using the content ID received in step S75 as a search key, and reads the content data from the corresponding content URL. The content data also includes position information described below indicating the position of the image capturing device 10 at the time of image capturing. Further, the storing / reading unit 59 searches the three-dimensional image management DB 5003 using, as a search key, the three-dimensional image ID associated with the content ID received in step S75. The storing / reading unit 59 reads the parameters of the model IDs, the position information, and the point cloud acquisition date (see FIG. 16A) associated with the three-dimensional ID, or the parameters of the component numbers, the position information, and the construction date information (see FIG. 16B) associated with the three-dimensional ID.

[0178] The communication unit 51 transmits data of a first three-dimensional image and a second three-dimensional image together with the requested content data to the communication terminal 9a. Accordingly, the communication unit 91 of the communication terminal 9aFN202501296receives the content data and the data of the first three-dimensional image and the second three-dimensional image.

[0179] In step S77, the communication terminal 9a performs a playback process. The display control unit 94 of the communication terminal 9a displays a screen including the recorded images on the display 507 of the communication terminal 9a. The sound input / output control unit 95 outputs the recorded sounds through the speaker 512 of the communication terminal 9a.

[0180] Details of Screen Display ProcessSubsequently, of steps S13 to S15, the processing of step S14 will be described with reference to FIGs. 22 to 38B. When the participant A operates the communication terminal 9a, the reception unit 92 receives the operation, and the communication unit 91 transmits operation information indicating the content of the operation to the communication control apparatus 5. Accordingly, in the communication control apparatus 5, the communication unit 51, which serves as an acquisition unit, acquires the operation information, and the generation unit 53 generates a screen 600 based on the content of the operation indicated by the operation information. The communication unit 51 transmits data of the screen 600 to the communication terminal 9a. The communication unit 91 of the communication terminal 9a receives the data of the screen 600. The display control unit 94 of the communication terminal 9a displays the screen 600 on, for example, the display 507 of the communication terminal 9a. In this example, the communication control apparatus 5 functions as an information processing apparatus.

[0181] The communication unit 91 of the communication terminal 9a may receive data used for generating the screen 600 from the communication control apparatus 5. The reception unit 92, which serves as an acquisition unit, may acquire the operation information indicating the content of the operation performed by the participant A. In this case, the generation unit 93 ofFN202501296the communication terminal 9a generates the screen 600 based on the data and the operation information. In this example, the communication terminal 9a functions as an information processing apparatus. Each of the communication terminals 7 and 9b can perform substantially the same process or have substantially the same functions as the communication terminal 9a.

[0182] A process for generating the screen 600 to be displayed on the display 507 of the communication terminal 9a, performed by the communication control apparatus 5, will be described. In steps S13 and SI 5, the communication control apparatus 5 performs the same processing as the processing of step S14, except that the generated screen is displayed on different communication terminals (i.e., the communication terminals 7 and 9b). Thus, descriptions of the processing of steps S13 and S15 will be omitted. FIGs. 22 to 24 illustrate a flowchart of the operation performed by the communication control apparatus 5 in a screen display process.

[0183] The generation unit 53 of the communication control apparatus 5 generates the screen 600 to be displayed on the communication terminal 9a, as illustrated in FIG. 25. FIG. 25 is a diagram illustrating an example of an initial display screen displayed on the communication terminal 9a. The screen 600 includes a display area 610, a display area 620A, and a display area 620B. The display area 610 is an example of a first display area. The display area 620A is an example of a second display area. The display area 620B is an example of a third display area. In FIG. 25, the display area 610, the display area 620A, and the display area 620B are displayed simultaneously in the same size on the screen 600. Alternatively, the display area 610, the display area 620A, and the display area 620B may be displayed simultaneously in different sizes. Alternatively, one of the display area 610, the display area 620A, and the display area 620B may be selectively displayed in accordance with the selection made by theFN202501296participant A. Alternatively, the display area 610, the display area 620A, and the display area 620B on the screen 600 may be displayed separately on a plurality of displays.

[0184] The display area 610 displays a predetermined-area image (see FIG. 6B) representing a predetermined area (see FIG. 6A) in a wide-view image. The wide-view image is an example of a captured image and is obtained by capturing an image of objects such as a desk, a pillar, and a window.

[0185] When the captured image is not a curved image such as a wide-view image, the display area 610 displays a predetermined-area image representing a predetermined area that is the same area as the imaging area of the captured image.

[0186] The display area 620A displays a portion of or all of a first three-dimensional image that includes a three-dimensional model representing the shape of an object in three dimensions based on design data. The first three-dimensional image includes coordinates (an example of second coordinates) associated with coordinates (an example of first coordinates) in the wide-view image. The display area 620B displays a portion of or all of a first three-dimensional image that includes a three-dimensional model representing the shape of an object in three dimensions based on, for example, point cloud data. The second three-dimensional image includes coordinates (an example of second coordinates) associated with coordinates (an example of first coordinates) in the wide-view image.

[0187] As described with reference to FIG. 15, the image capturing position, which is the position of the image capturing device 10, is associated with the absolute position on the earth. As described with reference to FIGs. 16A and 16B, all coordinates in the first three-dimensional image and the second three-dimensional image including the three-dimensional models andFN202501296components managed in the three-dimensional image management DB 5003 are also associated with the absolute position on the earth.

[0188] Accordingly, the processing unit 54 performs alignment processing. For example, the processing unit 54 associates coordinates in the wide-view image and coordinates in the first three-dimensional image with each other. The coordinates in the wide-view image indicate, as a first position, the image capturing position in the wide-view image of the content data received by the communication control apparatus 5 in step S12 (or S12d). The coordinates in the first three-dimensional image indicate, as a second position, the position in the first three-dimensional image that has the same absolute position as the coordinates in the wide-view image. The coordinates are an example of the position information.

[0189] As another example of alignment processing, the processing unit 54 may associate the coordinates in the wide-view image with the coordinates indicating the second position in the first three-dimensional image by image processing such as matching features of the wide-view image and the first three-dimensional image, without using the absolute position on the earth or indoors or in order to complement the absolute position.

[0190] As described above, the generation unit 53 generates the screen 600 including the display area 610, the second display area, and the third display area. The display area 610 displays the predetermined-area image, which represents a predetermined area in the wide-view image. The second display area and the third display area display at least a portion of the first three-dimensional image and the second three-dimensional image, respectively, in which the second position (second coordinates) is associated with the first position (first coordinates) in the wide-view image by the processing unit 54.

[0191] FN202501296When the three-dimensional image management DB 5003 illustrated in FIG. 16A or 16B stores, as a data item to be managed, for example, a three-dimensional point cloud, a mesh object, or a textured mesh object instead of the three-dimensional model, the display area 620B displays at least a portion of the second three-dimensional image including the three-dimensional point cloud, the mesh object, or the textured mesh object as a three-dimensional area corresponding to an object included in a wide-view image.

[0192] The screen 600 also includes a time information change field 630 for receiving an operation to change time information of the first three-dimensional image displayed in the display area 620 A. For example, when the time information change field 630 receives an operation from the user to move a display component 632 horizontally along a time bar 631, the time information of the first three-dimensional image displayed in the display area 620A is changed. In FIG. 25, the time bar 631 represents time information, with the oldest time information (past time) on the left, the current time information (current time) in the middle, and the latest time information (future time) on the right.

[0193] The user performs an operation of positioning the display component 632 at the "Present" position on the time bar 631 to display the present (or current) first three-dimensional image in the display area 620A. The user performs an operation of positioning the display component 632 at the " 1.10" position on the time bar 631 to display the previous first three-dimensional image obtained as of "1.10", meaning January 10, which is earlier than the current time, in the display area 620A. The user performs an operation of moving the display component 632 to the right of the "Present" position on the time bar 631 to display the future first three-dimensional image in the display area 620A. The dates, such as "12.20" (December 20) and "1.10" (January 10), which are displayed to the left of the "Present" position on the time bar 631, may be displayed in association with the "point cloud acquisition date" in theFN202501296three-dimensional image management table illustrated in FIG. 16A or the "construction date information" in the three-dimensional image management table illustrated in FIG. 16B.

[0194] The screen 600 also includes a progress confirmation button 640, a first difference confirmation button 650A, and a second difference confirmation button 650B. The progress confirmation button 640 is used to receive a progress confirmation operation from the user. The first difference confirmation button 650A and the second difference confirmation button 650B are used to receive a difference confirmation operation from the user. The progress confirmation button 640, the first difference confirmation button 650A, and the second difference confirmation button 650B may be constantly displayed on the screen 600, or may be invoked and displayed by the user as appropriate. The progress confirmation operation and the difference confirmation operation will be described in detail below.

[0195] The screen 600 further includes a close button 609. The close button 609 is pressed to close the screen 600.

[0196] The content displayed on the screen 600 will be described in detail below. The virtual camera IC1 is used to specify a predetermined area depicted in the predetermined-area image displayed in the display area 610 (see FIG. 8). A virtual camera IC2 described below is used to specify a first three-dimensional image and a second three-dimensional image to be displayed in the display area 620A and the display area 620B, respectively.

[0197] In step Sill, the generation unit 53 of the communication control apparatus 5 specifies, in the wide-view image of the content data received by the communication unit 51, a predetermined-area image to be displayed in the display area 610 as illustrated in FIG. 25, based on the field-of-view information. The field-of-view information indicates a preset virtual field of view of the virtual camera IC1.FN202501296

[0198] In step SI 12, the generation unit 53 aligns the position of the virtual camera IC2 with the image capturing position included in the wide-view image obtained by the image capturing device 10. The generation unit 53 further aligns the virtual field of view of the virtual camera IC2 (an example of a second field of view) with the preset virtual field of view of the virtual camera IC1 (an example of a first field of view). Accordingly, the generation unit 53 generates a first three-dimensional image and a second three-dimensional image to be displayed in the display area 620A and the display area 620B, respectively.

[0199] In step S 113, the generation unit 53 determines whether the field of view of the virtual camera IC1 in the display area 610 has been changed (see FIG. 6C). For example, when the participant A performs an operation of changing the predetermined-area image in the display area 610 illustrated in FIG. 25, the reception unit 92 receives the changed field of view, and the communication unit 91 transmits field-of-view information indicating the changed field of view to the communication control apparatus 5. When the communication unit 51 receives the field-of-view information indicating the changed field of view, the generation unit 53 determines that the field of view of the virtual camera IC1 has been changed. When the field of view of the virtual camera IC1 has been changed (YES in step SI 13), the process returns to step Sill.

[0200] On the other hand, when the field of view of the virtual camera IC1 has not been changed (NO in step S 113), in step S 114, the generation unit 53 determines whether the position (viewpoint) of the virtual camera IC2 has been changed. For example, when the participant A changes the first three-dimensional image in the display area 620A illustrated in FIG. 25, the reception unit 92 receives the changed position of the virtual camera IC2, and the communication unit 91 transmits position information indicating the changed position to the communication control apparatus 5. When the communication unit 51 receives the positionFN202501296information indicating the changed position, the generation unit 53 determines that the position of the virtual camera IC2 has been changed.

[0201] The change in the position of the virtual camera IC2 will be described with reference to FIGs.26 and 27Ato 27D. FIG. 26 is a diagram illustrating a screen displaying a first three-dimensional image in which the position and the field of view of the virtual camera IC2 have been changed from those in the first three-dimensional image on the screen 600 illustrated in FIG. 25. FIGs. 27Ato 27D are diagrams illustrating first three-dimensional images displayed for different positions of the virtual camera IC2.

[0202] When the participant A operates a cursor Cl on the communication terminal 9a as illustrated in FIG. 26, the generation unit 53 generates a screen 600 as illustrated in FIG. 26, and then, the display control unit 94 displays the screen 600 as illustrated in FIG. 26. FIG. 26 corresponds to FIGs. 27A and 27D. That is, in FIG. 27A, when the virtual camera IC2 is oriented to the south, a first three-dimensional image as illustrated in the lower part of FIG.27Ais displayed in the display area 620A. As illustrated in FIGs. 27B, 27C, and 27D, when the position of the virtual camera IC2 is moved and the virtual camera IC2 is oriented to the east, the north, and the west, respectively, first three-dimensional images as illustrated in the lower parts of FIGs. 27B, 27C, and 27D are displayed in the display area 620, respectively. Further, a second three-dimensional image is displayed in the display area 620B in accordance with the changed position of the virtual camera IC2 and the first three-dimensional image. As described above, the participant A operates the display area 620A with the cursor Cl to change the first three-dimensional image and the second three-dimensional image to be displayed in the display area 620A and the display area 620B, respectively. This allows the participant A to view a three-dimensional model (e.g., a plurality of stacked boxes) that is not displayed in FIG. 27A and is invisible to the participant A. This also allows the participant AFN202501296to view the back of a three-dimensional model (e.g., a pillar) that is not displayed in FIG. 27 A and is invisible to the participant A.

[0203] Returning to FIG. 22, when the position of the virtual camera IC2 has not been changed (NO in step S 114), as illustrated in FIG. 23, in step S 115, the generation unit 53 determines whether the field of view of the virtual camera IC2 has been changed. For example, when the participant A changes the first three-dimensional image in the display area 620A illustrated in FIG. 25, the reception unit 92 receives the changed field of view, and the communication unit 91 transmits field-of-view information indicating the changed field of view to the communication control apparatus 5. When the communication unit 51 receives the field-of-view information indicating the changed field of view, the generation unit 53 determines that the field of view of the virtual camera IC2 has been changed.

[0204] When the field of view of the virtual camera IC2 has been changed (YES in step SI 15), in step SI 16, the generation unit 53 enlarges the first three-dimensional image in the display area 620A and the second three-dimensional image in the display area 620B as illustrated in FIG.28, or reduces the size of the first three-dimensional image and the second three-dimensional image as illustrated in FIG. 29 in accordance with the change in the field of view of the virtual camera IC2.

[0205] In step SI 17, simultaneously with or immediately after the processing of step SI 16, the generation unit 53 changes the field of view of the virtual camera IC1 in accordance with (or in conjunction with) the change in the field of view of the virtual camera IC2 to change the predetermined-area image displayed in the display area 610.

[0206] FIG. 28 is a diagram illustrating a screen displaying a predetermined-area image that is enlarged in accordance with an enlargement of the first three-dimensional image. FIG. 29 is aFN202501296diagram illustrating a screen displaying a predetermined-area image that is reduced in size in accordance with a reduction of the size of the first three-dimensional image. As an example of changing the predetermined-area image, the generation unit 53 enlarges the predetermined-area image displayed in the display area 610 as illustrated in FIG. 28, or reduces the size of the predetermined-area image displayed in the display area 610 as illustrated in FIG. 29.

[0207] On the other hand, in FIG. 22, when the position (viewpoint) of the virtual camera IC2 has been changed (YES in step SI 14), in step SI 18, the generation unit 53 changes the first three-dimensional image and the second three-dimensional image in accordance with the change in the position of the virtual camera IC2. Since the first three-dimensional image displayed in the display area 620A and the second three-dimensional image displayed in the display area 620B are images in the virtual space, the position of the virtual camera IC2 is freely changeable (YES in step SI 14). In the real world, however, the image capturing position of the image capturing device 10 is not necessarily freely changeable.

[0208] In the present embodiment, the image capturing position of the image capturing device 10 is fixed or manually movable. Thus, the image capturing position of the image capturing device 10 is not automatically changeable in accordance with the change in the position of the virtual camera IC2. In some embodiments, the image capturing position of the image capturing device 10 may be automatically changeable in accordance with the change in the position of the virtual camera IC2.

[0209] After the processing of step SI 18, as illustrated in FIG. 23, in step SI 19, the generation unit 53 determines whether at least a portion of the predetermined area depicted in the predetermined-area image being displayed in the display area 610 is included in the first three-dimensional image.

[0210] FN202501296When at least a portion of the predetermined area depicted in the predetermined-area image being displayed in the display area 610 is included in the first three-dimensional image (YES in step S 119), in step S120, as illustrated in FIG. 30, the generation unit 53 superimposes a corresponding predetermined area 621 on the display area 620A. The corresponding predetermined area 621 corresponds to the field of view of the virtual camera IC1. FIG. 30 is a diagram illustrating an example of a screen on which the corresponding predetermined area 621, an icon of the virtual camera IC1, and the line of sight of the virtual camera IC1 are displayed in the display area 620A. The corresponding predetermined area 621 is an area corresponding to the predetermined area depicted in the predetermined-area image displayed in the display area 610. In FIG. 30, the corresponding predetermined area 621 is outlined by a broken line as an example of a display mode. The display mode may include a broken line and a solid line of a specified color. The broken line and the solid line may have different thicknesses from other parts of the display.

[0211] The generation unit 53 superimposes an icon 622 of the virtual camera IC1 on the display area 620A at a position corresponding to the image capturing position of the image capturing device 10. The icon 622 is placed so that the virtual camera IC1 is directed to a center point CPI of the corresponding predetermined area 621. The icon 622 is an example of a schematic diagram (image) of the virtual camera ICE The schematic diagram may include, for example, characters such as "camera" or a figure including the characters, in addition to the icon. The icon 622 may be simply represented by a shape such as a circle to indicate the image capturing position of the image capturing device 10 and may be arranged in the vicinity of the area corresponding to the field of view of the virtual camera IC1 instead of being arranged at the exact position corresponding to the image capturing position of the image capturing device 10. As a variation of the screen illustrated in FIG. 30, the screen 600 may initially include the display areas 620A and 620B alone without including the display area 610, and the display area 610 may be displayed on the screen 600 in response to the reception unit 92FN202501296receiving a user operation on the icon 622. At this time, the display areas 620A and 620B are not included in the screen 600, but may be displayed on another screen.

[0212] Further, the generation unit 53 superimposes a line 623 on the display area 620Ato indicate the line of sight of the virtual camera IC1 from the icon 622 towards the center point CPI. The line 623 may be a solid line or a broken line or may be displayed with a thickness or color different from those of other lines.

[0213] FIG. 30 illustrates an operation of receiving an operation to change time information in the time information change field 630 and superimposing the corresponding predetermined area 621, the icon 622, and the line 623 on the display area 620A, by way of example but not limitation. In another example, an operation to change time information in the time information change field 630 may be received, and the corresponding predetermined area 621, the icon 622, and the line 623 may be superimposed on the display area 620B.

[0214] When the field of view of the virtual camera IC2 has not been changed (NO in step SI 15), after the processing of step SI 17, when at least a portion of the predetermined area depicted in the predetermined-area image being displayed in the display area 610 is not included in the first three-dimensional image (NO in step SI 19), or after the processing of step SI 20, then in Step S 121 , the generation unit 53 determines whether the communication unit 51 has received an instruction to change the time information of the first three-dimensional image displayed in the display area 620A.

[0215] When the communication unit 51 acquires (or receives) an instruction to change the time information of the first three-dimensional image displayed in the display area 620A (YES in step S 121), in step S122, the generation unit 53 refers to the three-dimensional imageFN202501296management DB 5003 illustrated in FIG. 16A and determines whether there is any point cloud data.In FIG. 16A, the point cloud acquisition date is information indicating the date on which the point cloud data was acquired. Thus, if a date before the time indicated by the changed time information is stored as a point cloud acquisition date, the generation unit 53 determines that point cloud data is present.

[0216] When the communication unit 51 determines in step S122 that point cloud data is present (YES in step S122), in step S123, the generation unit 53 displays a first three-dimensional image based on the design data in the display area 620A in accordance with the time information changed in step S 121 , and displays a second three-dimensional image based on, for example, the point cloud data in the display area 620B.

[0217] For example, as illustrated in FIG. 31, the user performs an operation of moving the display component 632 to the "1.10" position on the time bar 631. The first three-dimensional image and the second three-dimensional image, which are obtained as of "1.10" (January 10), which is earlier than the current time, are displayed in the display area 620A and the display area 620B, respectively. FIG. 31 is a diagram illustrating an example of a screen that displays a state in which the first three-dimensional image and the second three-dimensional image have changed in accordance with a change in time information. In FIG. 31, the size of a three-dimensional model ml A included in the first three-dimensional image in the display area 620 A and the size of a three-dimensional model mlB included in the second three-dimensional image in the display area 620B have changed with respect to the size of the three-dimensional model ml A included in the first three-dimensional image in the display area 620A and the size of the three-dimensional model mlB included in the second three-dimensional image in the display area 620B illustrated in FIG. 26, respectively.

[0218] FN202501296As illustrated in FIG. 32, the user performs an operation of moving the display component 632 to the "12.20" position on the time bar 631. The first three-dimensional image and the second three-dimensional image, which are obtained as of "12.20" (December 20), which is earlier than the current time and " 1.10", are displayed in the display area 620A and the display area 620B, respectively. FIG. 32 is a diagram illustrating an example of a screen that displays a state in which the first three-dimensional image and the second three-dimensional image have changed in accordance with a change in time information. In FIG. 32, the three-dimensional model ml A included in the first three-dimensional image in the display area 620A illustrated in FIGs. 26 and 31 and the three-dimensional model mlB included in the second three-dimensional image in the display area 620B illustrated in FIGs. 26 and 31 have changed to a hidden state.

[0219] As described above, the generation unit 53 can generate the display area 620Afor displaying at least a portion of the first three-dimensional image and the display area 620B for displaying at least a portion of the second three-dimensional image in accordance with the time information indicating the time (current time) at which the captured image that displays the predetermined-area image in the display area 610 is captured and the time (past time) before the time at which the captured image is captured.

[0220] When the communication unit 51 determines in step S122 that no point cloud data is present (NO in step S122), in step S124, the generation unit 53 displays a first three-dimensional image based on the design data in the display area 620A in accordance with the time information changed in step S 121 , and does not display a second three-dimensional image based on the point cloud data in the display area 620B.

[0221] For example, as illustrated in FIG. 33, the user performs an operation of moving the display component 632 to the "+1M (one month later)" position on the time bar 631. A first three-FN202501296dimensional image corresponding to a time one month later than the current time is displayed in the display area 620A. FIG. 33 is a diagram illustrating an example of a screen that displays a state in which the first three-dimensional image and the second three-dimensional image have changed in accordance with a change in time information. The screen 600 illustrated in FIG. 33 has changed such that a three-dimensional model m2, which is not displayed in the display area 620A illustrated in FIG. 26, is included in the first three-dimensional image in the display area 620 A, and the second three-dimensional image is not displayed in the display area 620B.

[0222] The generation unit 53 refers to the construction date information in the three-dimensional image management table illustrated in FIG. 16B. The components for which the construction date is one month later are visible, whereas the components for which the construction date is not reached by one month are invisible or partially visible. Thus, a first three-dimensional image that will be constructed one month later (from the current time) can be displayed in the display area 620 A.

[0223] As described above, the generation unit 53 can generate the display area 620Afor displaying at least a portion of the first three-dimensional image in accordance with the time information indicating the time (future time) after the time at which the captured image that displays the predetermined-area image in the display area 610 is captured.

[0224] After the processing of step S123 or S124, in step S125, the generation unit 53 determines whether the communication unit 51 has acquired an instruction to confirm the progress. For example, in response to receipt of an operation from the user to press the progress confirmation button 640, the generation unit 53 determines that an instruction to confirm the progress has been acquired.

[0225] FN202501296When it is determined in step S125 that an instruction to confirm the progress has been acquired (YES in step S125), in step S126, the generation unit 53 displays progress identification information 641, which indicates a difference from the first three-dimensional image obtained before the time information is changed, in a superimposed manner on the first three-dimensional image obtained after the time information is changed.FIG. 34 is a diagram illustrating a screen on which progress identification information is displayed in a superimposed manner. In FIG. 34, the progress identification information 641, such as hatching, is displayed in a superimposed manner on the three-dimensional model m2, which is not displayed in the display area 620A illustrated in FIG. 26. The display of the progress identification information 641 makes it easier to confirm the progress from the current time, which is before the time information is changed, to one month later, which is after the time information is changed.

[0226] As described above, the generation unit 53 can generate the display area 620Aby superimposing the progress identification information 641, which identifies a portion different from that in a first three-dimensional image corresponding to first time information (e.g., the current time), on a first three-dimensional image corresponding to second time information (e.g., one month later). The first time information and the second time information may refer to the past and the present, or may both refer to the past.

[0227] When it is determined in step S125 that an instruction to confirm the progress has not been acquired (NO in step S125) or after the processing of step SI 26, in step SI 27, the generation unit 53 determines whether the communication unit 51 has acquired a first difference confirmation instruction. For example, in response to receipt of an operation from the user to press the first difference confirmation button 650A, the generation unit 53 determines that a first difference confirmation instruction has been acquired.

[0228] FN202501296When it is determined in step S127 that a first difference confirmation instruction has been acquired (YES in step S127), in step S128, the generation unit 53 displays first difference identification information 651 A, which indicates a difference between the first three-dimensional image in the display area 620A and the second three-dimensional image in the display area 620B, in a superimposed manner on the second three-dimensional image in the display area 620B. FIG. 35 is a diagram illustrating a screen before first difference identification information and second difference identification information are displayed in a superimposed manner. FIG. 36 is a diagram illustrating a screen after the first difference identification information is displayed in a superimposed manner. FIG. 35 illustrates an example in which the three-dimensional model ml A included in the first three-dimensional image in the display area 620Ais different from the three-dimensional model mlB included in the second three-dimensional image in the display area 620B and an image Ml of an object included in the predetermined-area image in the display area 610.

[0229] In FIG. 36, the first difference identification information 651 A, such as hatching, is displayed in a superimposed manner on the display area 620B at a position different from the three-dimensional model ml A included in the first three-dimensional image in the display area 620A. The display of the first difference identification information 651 A makes it easier to confirm the difference between the first three-dimensional image in the display area 620A and the second three-dimensional image in the display area 620B.

[0230] When it is determined in step S127 that a first difference confirmation instruction has not been acquired (NO in step S127) or after the processing of step S128, in step S129, the generation unit 53 determines whether the communication unit 51 has acquired a second difference confirmation instruction. For example, in response to receipt of an operation from the user to press the second difference confirmation button 650B, the generation unit 53 determines that a second difference confirmation instruction has been acquired.FN202501296

[0231] When it is determined in step S129 that a second difference confirmation instruction has been acquired (YES in step S129), in step S130, the generation unit 53 displays second difference identification information 65 IB, which indicates a difference between the predetermined-area image in the display area 610 and the first three-dimensional image in the display area 620A, in a superimposed manner on the predetermined-area image in the display area 610. FIG. 37 is a diagram illustrating a screen after the first difference identification information and the second difference identification information are displayed in a superimposed manner.

[0232] In FIG. 37, the second difference identification information 65 IB, such as hatching, is displayed in a superimposed manner on the display area 610 at a position different from the three-dimensional model ml A included in the first three-dimensional image in the display area 620 A. The display of the second difference identification information 65 IB makes it easier to confirm the difference between the predetermined-area image in the display area 610 and the first three-dimensional image in the display area 620 A.

[0233] In the display area 610, at the position different from the three-dimensional model ml A included in the first three-dimensional image in the display area 620A, the first three-dimensional image may be converted into a two-dimensional image having the same field of view as the predetermined-area image, and the difference between the predetermined-area image and the converted first three-dimensional image may be recognized by image recognition artificial intelligence (Al). In another example, in the display area 610, at the position different from the three-dimensional model ml A included in the first three-dimensional image in the display area 620 A, the first three-dimensional image and the predetermined-area image may be aligned in position on a pixel -by-pixel basis, and the difference between the predetermined-area image and the first three-dimensional image may be recognized in the form of three-dimensional data.FN202501296The same applies to the position in the display area 620B different from the three-dimensional model ml A included in the first three-dimensional image in the display area 620 A.

[0234] The screen illustrated in FIG. 36 is an example. In another example, the difference between the first three-dimensional image in the display area 620A and the second three-dimensional image in the display area 620B may be displayed as first difference identification information in a superimposed manner on the first three-dimensional image in the display area 620 A. The screen illustrated in FIG. 37 is an example. In another example, the difference between the predetermined-area image in the display area 610 and the first three-dimensional image in the display area 620Amay be displayed as second difference identification information in a superimposed manner on the first three-dimensional image in the display area 620A.

[0235] As described above, the generation unit 53 can generate a third display area by superimposing first difference identification information for identifying a portion where the first three-dimensional image and the second three-dimensional image are different from each other on the second three-dimensional image or generate a second display area by superimposing the first difference identification information on the first three-dimensional image. Further, the generation unit 53 can generate a second display area by superimposing second difference identification information for identifying a portion where the first three-dimensional image and the captured image are different from each other on the first three-dimensional image or generate a first display area by superimposing the second difference identification information on the captured image.

[0236] When it is determined in step S129 that a second difference confirmation instruction has not been acquired (NO in step S129), when an instruction to change the time information of the first three-dimensional image displayed in the display area 620A has not been acquired (NOFN202501296in step S 121), or after the processing of step S130, the screen display process illustrated in FIG. 24 ends.

[0237] FIG. 38Ais a diagram illustrating an example of a screen on which a three-dimensional model changes differently even when the change in time information is the same. FIG. 38B illustrates that the construction date information in the three-dimensional image management table illustrated in FIG. 16B is changed from "2024.5.1" to "2024.5.10".

[0238] In one example, the construction date information indicates "2024.5.1", which means May 1, 2024, and the date one month after the current time is "2024.5.1 ". In this case, as illustrated in FIG. 33, the three-dimensional model m2 with the "component No." being "5" is displayed so as to be included in the first three-dimensional image in the display area 620A. In another example, the construction date information indicates "2024.5.10", which means May 10, 2024, and the date one month after the current time is "2024.5.1". In this case, as illustrated in FIG. 38 A, the three-dimensional model m2 with the "component No." being "5" is not displayed in the first three-dimensional image in the display area 620A.

[0239] When the three-dimensional image management DB 5003 illustrated in FIG. 16B stores, as a data item to be managed, construction period information indicating a period during which a component is to be constructed, in FIG. 38 A, a portion of the three-dimensional model m2 with the "component No." being "5" may be displayed in the first three-dimensional image in the display area 620A.

[0240] Specifically, the construction period information of the three-dimensional model m2 with the "component No." being "5" has the construction period information "20 days" and the construction date information "2024.5.10". In this case, the three-dimensional model m2 is calculated to be approximately 50% complete as of "2024.5.1 " (May 1, 2024), which is oneFN202501296month after the current time, based on a daily calculation. Thus, half of the three-dimensional model m2 may be displayed in the first three-dimensional image in the display area 620A.

[0241] As described above, in a case where the construction date information in the three-dimensional image management table illustrated in FIG. 16B is changed, the generation unit 53 can generate the display area 620A in which at least one of the arrangement and the size of the three-dimensional model corresponding to the time information is changed even when the change in time information is the same (e.g., one month later).

[0242] As described above, according to the present embodiment, the processing unit 54 associates position information indicating a first position in a captured image of an object with position information indicating a second position in a three-dimensional image including a three-dimensional area corresponding to the object. The generation unit 53 (or the generation unit 73 or 93) generates a screen 600 including a display area 610, a display area 620A, and a display area 620B. The display area 610 displays a predetermined-area image, which represents a predetermined area in the captured image of the object. The display area 620 A displays at least a portion of a first three-dimensional image in which the second position is associated with the first position in the captured image by the processing unit 54. The first three-dimensional image is generated based on design data. The display area 620B displays at least a portion of a second three-dimensional image generated based on point cloud data.

[0243] Accordingly, by viewing the screen 600, the user can recognize the content of the predetermined-area image and the location where the predetermined-area image was captured. By viewing the screen 600, the user can determine whether the predetermined-area image is appropriate through the comparison with the first three-dimensional image and the second three-dimensional image. Even when the back of an object is not captured, the user can understand the state of the object, such as the back of the object, by viewing the screen 600.FN202501296Accordingly, generation of an image that complements a captured image obtained by image capturing has the effect of enhancing user convenience.

[0244] The above-described embodiments are illustrative and do not limit the present invention. Thus, numerous additional modifications and variations are possible in light of the above teachings. For example, elements and / or features of different illustrative embodiments may be combined with each other and / or substituted for each other within the scope of the present invention. Any one of the above-described operations may be performed in various other ways, for example, in an order different from the one described above.

[0245] The present invention can be implemented in any convenient form, for example using dedicated hardware, or a mixture of dedicated hardware and software. The present invention may be implemented as computer software implemented by one or more networked processing apparatuses. The processing apparatuses include any suitably programmed apparatuses such as a general purpose computer, a personal digital assistant, a Wireless Application Protocol (WAP) or third-generation (3G)-compliant mobile telephone, and so on. Since the present invention can be implemented as software, each and every aspect of the present invention thus encompasses computer software implementable on a programmable device. The computer software can be provided to the programmable device using any conventional carrier medium (carrier means). The carrier medium includes a transient carrier medium such as an electrical, optical, microwave, acoustic or radio frequency signal carrying the computer code. An example of such a transient medium is a Transmission Control Protocol / Internet Protocol (TCP / IP) signal carrying computer code over an IP network, such as the Internet. The carrier medium may also include a storage medium for storing processor readable code such as a floppy disk, a hard disk, a compact disc read-only memory (CD-ROM), a magnetic tape device, or a solid state memory device.

[0246] FN202501296The functionality of the elements disclosed herein may be implemented using circuitry or processing circuitry which includes general purpose processors, special purpose processors, integrated circuits, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or combinations thereof which are configured or programmed, using one or more programs stored in one or more memories, to perform the disclosed functionality. Processors are considered processing circuitry or circuitry as they include transistors and other circuitry therein. In the disclosure, the circuitry, units, or means are hardware that carry out or are programmed to perform the recited functionality. The hardware may be any hardware disclosed herein which is programmed or configured to carry out the recited functionality.There is a memory that stores a computer program which includes computer instructions. These computer instructions provide the logic and routines that enable the hardware (e.g., processing circuitry or circuitry) to perform the method disclosed herein. This computer program can be implemented in known formats as a computer-readable storage medium, a computer program product, a memory device, a record medium such as a CD-ROM or DVD, and / or the memory of an FPGA or ASIC.

[0247] The programs described above may be stored in (non-transitory) recording media such as digital versatile disc-read only memories (DVD-ROMs), and such (non-transitory) recording media may be provided in the form of program products to domestic or foreign users.

[0248] Each of the CPUs 111, 301, and 501 may be implemented using a processor or a plurality of processors.

[0249] This patent application is based on and claims priority to Japanese Patent Application No. 2025-006666, filed on January 17, 2025, in the Japan Patent Office, the entire disclosure of which is hereby incorporated by reference herein.FN202501296

[0250] Aspects of the present disclosure are as follows.In a first aspect, an information processing apparatus (5) includes a processing unit (54) and a generation unit (53). The processing unit (54) associates position information indicating a first position in a captured image of an object with position information indicating a second position in a three-dimensional image including a three-dimensional area corresponding to the object. The generation unit (53) generates a screen (600) including a first display area (610), a second display area (620A), and a third display area (620B). The first display area (610) displays a predetermined-area image representing a predetermined area in the captured image. The second display area (620A) displays at least a portion of a first three-dimensional image in which the second position is associated with the first position in the captured image by the processing unit (54). The first three-dimensional image is generated based on design data. The third display area (620B) displays at least a portion of a second three-dimensional image generated based on point cloud information.According to a second aspect, in the information processing apparatus (5) of the first aspect, the generation unit (53) superimposes first difference identification information on the first three-dimensional image to generate the second display area, or superimposes the first difference identification information on the second three-dimensional image to generate the third display area. The first difference identification information indicates a portion where the first three-dimensional image and the second three-dimensional image are different from each other.According to a third aspect, in the information processing apparatus (5) of the first aspect or the second aspect, the generation unit (53) superimposes second difference identification information on the first three-dimensional image to generate the second display area, or superimposes the second difference identification information on the captured image to generate the first display area. The second difference identification information indicates aFN202501296portion where the first three-dimensional image and the captured image are different from each other.According to a fourth aspect, in the information processing apparatus (5) of any one of the first to third aspects, the generation unit (53) generates the second display area or the third display area such that a position image indicating the captured image or an image capturing position of the captured image is displayed in a superimposed manner on the first three-dimensional image or the second three-dimensional image at a position corresponding to the captured image or the image capturing position of the captured image.According to a fifth aspect, in the information processing apparatus (5) of the fourth aspect, the generation unit (53) generates the screen including the second display area and the third display area but not including the first display area, and generates the screen including the first display area in response to reception of an operation performed on the position image. According to a sixth aspect, in the information processing apparatus (5) of any one of the first to fifth aspects, the generation unit (53) generates the second display area such that at least one of an arrangement and a size of a three-dimensional model included in the first three-dimensional image changes in accordance with time information indicating a time before or after a time at which the captured image is captured.According to a seventh aspect, in the information processing apparatus (5) of the sixth aspect, the generation unit (53) generates the second display area including the time information. In an eighth aspect, a screen generation method executed by an information processing apparatus includes an association process and a generation process. The association process associates position information indicating a first position in a captured image of an object with position information indicating a second position in a three-dimensional image including a three-dimensional area corresponding to the object. The generation process generates a screen including a first display area, a second display area, and a third display area. The first display area displays a predetermined-area image representing a predetermined area in the captured image. The second display area displays at least a portion of a first three-dimensionalFN202501296image in which the second position is associated with the first position in the captured image in the association process. The first three-dimensional image is generated based on design data. The third display area displays at least a portion of a second three-dimensional image generated based on point cloud information.In a ninth aspect, a program causes a computer to execute an association process and a generation process. The association process associates position information indicating a first position in a captured image of an object with position information indicating a second position in a three-dimensional image including a three-dimensional area corresponding to the object. The generation process generates a screen including a first display area, a second display area, and a third display area. The first display area displays a predetermined-area image representing a predetermined area in the captured image. The second display area displays at least a portion of a first three-dimensional image in which the second position is associated with the first position in the captured image in the association process. The first three-dimensional image is generated based on design data. The third display area displays at least a portion of a second three-dimensional image generated based on point cloud information.In a tenth aspect, an information processing system includes an information processing apparatus and a display terminal. The information processing system includes a processing unit and a generation unit. The processing unit associates position information indicating a first position in a captured image of an object with position information indicating a second position in a three-dimensional image including a three-dimensional area corresponding to the object. The generation unit generates a screen including a first display area, a second display area, and a third display area. The first display area displays a predetermined-area image representing a predetermined area in the captured image. The second display area displays at least a portion of a first three-dimensional image in which the second position is associated with the first position in the captured image by the processing unit. The first three-FN202501296dimensional image is generated based on design data. The third display area displays at least a portion of a second three-dimensional image generated based on point cloud information. [Reference Signs List]

[0251] 1 Communication system (example of information processing system)3 Relay device5 Communication control apparatus (example of information processing apparatus)7 Communication terminal (example of display terminal, example of information processing apparatus)9, 9a, 9b Communication terminal (example of display terminal, example of information processing apparatus)10 Image capturing device51 Communication unit (example of acquisition unit)52 Reception unit53 Generation unit71 C ommuni cati on unit72 Reception unit (example of acquisition unit)73 Generation unit74 Display control unit75 Sound input / output control unit78 Connection unit91 C ommuni cati on unit92 Reception unit (example of acquisition unit)93 Generation unit94 Display control unit95 Sound input / output control unit98 Connection unitFN202501296507 Display (example of display unit)600 Screen610 Display area (example of first display area) 620A Display area (example of second display area) 620B Display area (example of third display area) 640 Progress confirmation button641 Progress identification information650A First difference confirmation button650B Second difference confirmation button651 A First difference identification information 65 IB Second difference identification information 5001 User device management DB5002 Virtual room management DB5003 Three-dimensional image management DB

Claims

FN202501296[CLAIMS]1. An information processing apparatus comprising:a processing unit configured to associate position information indicating a first position in a captured image of an object with position information indicating a second position in a three-dimensional image including a three-dimensional area corresponding to the object,the three-dimensional image having the position information indicating the second position associated with the first position,the three-dimensional image including a first three-dimensional image and a second three-dimensional image, the first three-dimensional image being generated based on design information, the second three-dimensional image being generated based on point cloud information; anda generation unit configured to generate a screen,the screen including:a first display area for displaying a predetermined-area image representing a predetermined area in the captured image;a second display area for displaying at least a portion of the first three-dimensional image; anda third display area for displaying at least a portion of the second three-dimensional image.

2. The information processing apparatus according to claim 1, wherein the generation unit is configured to superimpose first difference identification information on the first three-dimensional image to generate the second display area, the first difference identification information indicating a portion where the first three-dimensional image and the second three-dimensional image are different from each other.FN2025012963. The information processing apparatus according to claim 1, wherein the generation unit is configured to superimpose first difference identification information on the second three-dimensional image to generate the third display area, the first difference identification information indicating a portion where the first three-dimensional image and the second three-dimensional image are different from each other.

4. The information processing apparatus according to claim 1, wherein the generation unit is configured to superimpose second difference identification information on the first three-dimensional image to generate the second display area, the second difference identification information indicating a portion where the first three-dimensional image and the captured image are different from each other.

5. The information processing apparatus according to claim 1, wherein the generation unit is configured to superimpose second difference identification information on the captured image to generate the first display area, the second difference identification information indicating a portion where the first three-dimensional image and the captured image are different from each other.

6. The information processing apparatus according to claim 1, wherein the generation unit is configured to generate the second display area such that a position image indicating an image capturing position of the captured image is displayed while being superimposed on the first three-dimensional image at a position corresponding to the image capturing position of the captured image.

7. The information processing apparatus according to claim 1, wherein the generation unit is configured to generate the third display area such that a position image indicating an image capturing position of the captured image is displayed while beingFN202501296superimposed on the second three-dimensional image at a position corresponding to the image capturing position of the captured image.

8. The information processing apparatus according to claim 6 or 7, wherein before generating the screen, the generation unit is configured to generate another screen including the second display area and the third display area but not including the first display area, and generate the screen further including the first display area in response to reception of an operation performed on the position image.

9. The information processing apparatus according to claim 1, wherein the generation unit is configured to generate the second display area such that at least one of an arrangement and a size of a three-dimensional model included in the first three-dimensional image changes in accordance with time information indicating at least one of a time before a time at which the captured image is captured and a time after the time at which the captured image is captured.

10. The information processing apparatus according to claim 9, wherein the generation unit is configured to generate the second display area including the time information.

11. A screen generation method performed by an information processing apparatus, the screen generation method comprising:associating position information indicating a first position in a captured image of an object with position information indicating a second position in a three-dimensional image including a three-dimensional area corresponding to the object,the three-dimensional image having the position information indicating the second position associated with the first position,FN202501296the three-dimensional image including a first three-dimensional image and a second three-dimensional image, the first three-dimensional image being generated based on design information, the second three-dimensional image being generated based on point cloud information; andgenerating a screen,the screen including:a first display area for displaying a predetermined-area image representing a predetermined area in the captured image;a second display area for displaying at least a portion of the first three-dimensional image; anda third display area for displaying at least a portion of the second three-dimensional image.

12. A recording medium carrying computer readable code for controlling a computer system to carry out a screen generation method, the screen generation method comprising:associating position information indicating a first position in a captured image of an object with position information indicating a second position in a three-dimensional image including a three-dimensional area corresponding to the object,the three-dimensional image having the position information indicating the second position associated with the first position,the three-dimensional image including a first three-dimensional image and a second three-dimensional image, the first three-dimensional image being generated based on design information, the second three-dimensional image being generated based on point cloud information; andgenerating a screen,the screen including:FN202501296a first display area for displaying a predetermined-area image representing a predetermined area in the captured image;a second display area for displaying at least a portion of the first three-dimensional image; anda third display area for displaying at least a portion of the second three-dimensional image.

13. The recording medium according to claim 12, wherein the generating includes:superimposing first difference identification information on the second three-dimensional image to generate the third display area, the first difference identification information indicating a portion where the first three-dimensional image and the second three-dimensional image are different from each other.

14. An information processing system comprising:an information processing apparatus; anda display terminal,the information processing apparatus including:a processing unit configured to associate position information indicating a first position in a captured image of an object with position information indicating a second position in a three-dimensional image including a three-dimensional area corresponding to the object,the three-dimensional image having the position information indicating the second position associated with the first position,the three-dimensional image including a first three-dimensional image and a second three-dimensional image, the first three-dimensional image being generated based on designFN202501296information, the second three-dimensional image being generated based on point cloud information; anda generation unit configured to generate a screen,the screen including:a first display area for displaying a predetermined-area image representing a predetermined area in the captured image;a second display area for displaying at least a portion of the first three-dimensional image; anda third display area for displaying at least a portion of the second three-dimensional image,the display terminal including a display to display the screen.

Images