Information processing system, information processing method, and program

Abstract

[Problem] To make it possible to increase accuracy of three-dimensional reconstruction. [Solution] An information processing system comprising a control unit that: estimates, for each captured image of a first captured image group of an object captured by a first imaging device and a second captured image group of the object captured by a second imaging device, the position and orientation of imaging on the basis of a common reference; generates, on the basis of a ranging data group of the object measured by a ranging device attached to the second imaging device, an integrated point cloud obtained by integrating each piece of ranging data so as to be consistent with the positions and orientations estimated from each captured images of the second captured image group; and performs three-dimensional reconstruction of the object on the basis of the integrated point cloud and the first captured image group.

Description

Information Processing System, Information Processing Method, and Program

[0001] The present disclosure relates to an information processing system, an information processing method, and a program.

[0002] In recent years, a technique for performing three-dimensional reconstruction (so-called 3D modeling) of a subject using a plurality of captured images obtained by capturing the subject from a number of viewpoints has been known. The output three-dimensional reconstruction results include various forms such as free viewpoint video and mesh data.

[0003] In Patent Document 1 below, based on the point cloud of an object obtained by a three-dimensional measurement device that measures a three-dimensional space and outputs position information of objects in the space as point cloud data, and a plurality of captured images obtained by capturing the object from a number of viewpoints by an imaging device, a technique for generating a free viewpoint image is disclosed.

[0004] Japanese Unexamined Patent Application Publication No. 2023 - 47882

[0005] However, when the accuracy of a sensor that acquires distance measurement data indicating the distance to an object is not sufficiently high, there is a problem that the accuracy of three-dimensional reconstruction using the point cloud indicating the position information of the object generated from the distance measurement data becomes low.

[0006] Therefore, in the present disclosure, an information processing system, an information processing method, and a program capable of improving the accuracy of three-dimensional reconstruction are proposed by using data on the position and orientation estimated from captured images different from those for three-dimensional reconstruction during the integration of a plurality of distance measurement data.

[0007] According to the present disclosure, for each captured image of a first captured image group of an object captured by a first imaging device and a second captured image group of the object captured by a second imaging device, an estimation of the position and orientation of imaging in a common reference is performed, and based on a distance measurement data group of the object distance-measured by a distance measurement device attached to the second imaging device, an integrated point cloud obtained by integrating each distance measurement data is generated so as to match the position and orientation estimated from each captured image of the second captured image group, and a three-dimensional reconstruction of the object is performed based on the integrated point cloud and the first captured image group. An information processing system including a control unit is provided.

[0008] Furthermore, the present disclosure provides an information processing method in which a processor estimates the position and orientation of each image of a first group of images of an object captured by a first imaging device and a second group of images of the object captured by a second imaging device, based on a common reference; generates an integrated point cloud by integrating the distance measurement data of the object, based on a group of distance measurement data of the object measured by a distance measuring device attached to the second imaging device, so as to match the position and orientation estimated from each image of the second group of images; and performs a three-dimensional reconstruction of the object based on the integrated point cloud and the first group of images.

[0009] Furthermore, the present disclosure provides a program for causing a computer to function as a control unit that estimates the position and orientation of each image, based on a common reference, for a first group of images of an object captured by a first imaging device and a second group of images of the object captured by a second imaging device; generates an integrated point cloud by integrating the distance measurement data of the object, based on a group of distance measurement data of the object measured by a distance measuring device attached to the second imaging device, so as to match the position and orientation estimated from each image of the second group of images; and performs a three-dimensional reconstruction of the object based on the integrated point cloud and the first group of images.

[0010] This figure shows the configuration of an information processing system 1 according to one embodiment of the present disclosure. This is a block diagram showing an example of the functional configuration of an information processing device 30 according to this embodiment. This figure is for explaining the alignment of a multi-view integrated point cloud according to this embodiment. This is a sequence diagram showing an example of the operation flow of 3D reconstruction according to this embodiment. This figure is for explaining the attachment of an imaging distance measuring device 20 to an imaging device 10 according to a first modification of this embodiment. This is a block diagram showing a configuration example according to a second modification of this embodiment. This is a block diagram showing a configuration example according to a third modification of this embodiment. This is a block diagram showing a hardware configuration example of an information processing device 900 according to one embodiment of the present disclosure.

[0011] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0012] Furthermore, the explanation will be given in the following order: 1. Overview 2. Configuration 2-1. System Configuration 2-2. Functional Configuration of Information Processing Device 30 3. Operational Processing 4. Variations 4-1. First Variation 4-2. Second Variation 4-3. Third Variation 4-4. Fourth Variation 4-5. Fifth Variation 5. Hardware Configuration 6. Supplementary Information

[0013] <1. Overview> This disclosure proposes an information processing system that can improve the accuracy of 3D reconstruction using multiple captured images of an object taken from multiple viewpoints.

[0014] Conventionally, to improve the accuracy of 3D reconstruction, imaging images (RGB images) and distance measurement data are used. Specifically, multiple distance measurement data are integrated to generate a single point cloud before 3D reconstruction, and this integrated point cloud is aligned with the RGB image and used for 3D reconstruction.

[0015] However, if the accuracy and resolution of the sensor that acquires the distance measurement data are not high, there is a problem in that the accuracy of integrating multiple distance measurement data is low. For example, in the case of sensors that are not sufficiently accurate, such as those used in mobile devices, even if an integrated point cloud can be generated, the accuracy is only enough to understand the general shape, and alignment with the RGB image cannot be performed with sufficient accuracy. In addition, if the accuracy and resolution of the sensor that acquires the distance measurement data are not high, there is also a problem in that the alignment accuracy between the captured image for 3D reconstruction and the integrated distance measurement data is low. For example, when aligning the integrated point cloud to the coordinate system of the RGB image, if the fine shape cannot be captured in the integrated point cloud, alignment is difficult whether done automatically or manually.

[0016] The problems described above lead to a decrease in the accuracy of 3D reconstruction using integrated ranging data. It is also conceivable to improve alignment quality by fixing the positional relationship between the optical sensor (imaging unit) that acquires RGB images and the depth sensor (ranging unit) that acquires ranging data. However, in this case, it is desirable to shorten the distance between the imaging unit and the ranging unit to reduce the parallax between them, which requires miniaturization of the imaging unit. Specifically, this can lead to limitations in the design of the imaging unit and the size of the lens, potentially resulting in a decrease in the quality of the important RGB images used for 3D reconstruction.

[0017] Therefore, in this disclosure, a separate imaging unit is provided from the imaging unit that acquires the image for 3D reconstruction. In this disclosure, the accuracy of 3D reconstruction is improved by using the position and orientation data of the image estimated from the image acquired from this separate imaging unit for alignment when integrating multiple distance measurement data.

[0018] <2. Configuration> <<2-1. System Configuration>> Figure 1 is a diagram showing the configuration of an information processing system 1 according to one embodiment of the present disclosure. The information processing system 1 according to this embodiment includes an imaging device 10, an imaging distance measuring device 20, and an information processing device 30 (an example of an information processing device).

[0019] (Imaging device 10) The imaging device 10 has an imaging unit 12. The imaging device 10 (specifically the imaging unit 12) is an example of a first imaging device. The imaging device 10 captures images of an object from multiple viewpoints using the imaging unit 12. The multiple captured images obtained (hereinafter also referred to as image group A) are used in the information processing device 30 for three-dimensional reconstruction of the object (i.e., generation of a 3D model of the object). Since the captured images obtained by the imaging unit 12 are used for three-dimensional reconstruction, the imaging device 10 is implemented with a configuration that can obtain high-quality captured images, such as a single-lens reflex camera.

[0020] The imaging device 10 may further have sensors for detecting the position and orientation of the image being taken. Specifically, the imaging device 10 may have a position sensor, an IMU (Inertial Measurement Unit), and a geomagnetic sensor. The position sensor is a receiver that receives data used to calculate the position. As an example, the position sensor can be implemented by a GNSS receiver that receives data from GPS (Global Positioning System), quasi-zenith satellites, GLONASS, Galileo, BeiDou, and other navigation satellites. The IMU detects information on the three-dimensional inertial motion of the imaging device 10 (three-axis acceleration information and three-axis angular velocity information). More specifically, the IMU has an acceleration sensor and an angular velocity sensor and detects the acceleration and angular velocity of the imaging device 10 (output of three-axis acceleration information and three-axis angular velocity information). The geomagnetic sensor detects the Earth's magnetism and determines the direction. Various data acquired by the sensors may be input to the information processing device 30 together with the image group A.

[0021] (Image-based distance measuring device 20) The image-based distance measuring device 20 has an imaging unit 22 and a distance measuring unit 24. The image-based distance measuring device 20 (specifically the imaging unit 22) is an example of a second imaging device. The distance measuring unit 24 is an example of a distance measuring device. In the example shown in Figure 1, the image-based distance measuring device 20 is used as an example of a device in which the second imaging device and the distance measuring device are integrated. The image-based distance measuring device 20 is implemented by a mobile terminal such as a smartphone. For example, it is assumed that the imaging unit 22 and the distance measuring unit 24 are located as close together as possible in the image-based distance measuring device 20 in order to minimize parallax, and that their positional relationship is fixed.

[0022] The imaging distance measuring device 20 captures images of an object from multiple viewpoints using the imaging unit 22. At the same time, the imaging distance measuring device 20 also acquires the distance to the object from multiple viewpoints using the distance measuring unit 24. The distance measuring unit 24 may output a depth image as depth data, for example. The imaging distance measuring device 20 may simultaneously acquire RGB images using the imaging unit 22 and depth images using the distance measuring unit 24. The multiple captured images (hereinafter also referred to as image group B) and the multiple depth data (hereinafter also referred to as depth data group) obtained by the imaging distance measuring device 20 are used in the information processing device 30 when generating a multi-viewpoint integrated point cloud.

[0023] The imaging distance measuring device 20 may further have sensors for detecting the position and orientation of the image. Specifically, the imaging distance measuring device 20 may have a position sensor, an IMU, and a geomagnetic sensor. Various data acquired by the sensors may be input to the information processing device 30 together with the image group B and the depth data group.

[0024] The imaging device 10 and imaging distance measuring device 20 described above each have a communication unit (not shown), and the data acquired by each device may be transmitted to the information processing device 30 via the communication unit. Furthermore, the data acquired by the imaging device 10 and the imaging distance measuring device 20 may be input to the information processing device 30 via a storage medium.

[0025] Furthermore, the imaging of an object by the imaging device 10 and the imaging distance measuring device 20 may be performed simultaneously or sequentially. For example, the user may first image the object with the imaging device 10, and then image the same object with the imaging distance measuring device 20.

[0026] (Information Processing Device 30) The information processing device 30 includes a communication unit 310, a control unit 320, and a storage unit 330. The information processing device 30 can be implemented, for example, by a PC (Personal Computer), a smartphone, a tablet terminal, an HMD (Head Mounted Display), or a server on a network.

[0027] The communication unit 310 includes a transmitting unit that transmits data to an external device and a receiving unit that receives data from an external device. The communication unit 310 according to this embodiment communicates with an external device or the Internet using, for example, a wired / wireless LAN (Local Area Network), Wi-Fi®, Bluetooth®, a mobile communication network, etc. For example, the communication unit 310 may receive image group A from the imaging device 10, and image group B and depth data group from the imaging distance measuring device 20.

[0028] The control unit 320 functions as both an arithmetic processing unit and a control device, controlling the overall operation of the information processing device 30 according to various programs. The control unit 320 is implemented by electronic circuits such as a CPU (Central Processing Unit) or a microprocessor. The control unit 320 may also include a ROM (Read Only Memory) for storing programs and arithmetic parameters, and a RAM (Random Access Memory) for temporarily storing parameters that change as needed.

[0029] The functional configuration of the control unit 320 will be described later with reference to Figure 2.

[0030] The storage unit 330 is implemented by a storage medium that stores programs used in the processing of the control unit 320, calculation parameters, parameters that change as needed, etc. In this embodiment, the storage unit 330 may store, for example, image group A, image group B, and depth data group.

[0031] Although the configuration of the information processing device 30 has been described in detail above, the configuration of the information processing device 30 according to this disclosure is not limited to the example shown in Figure 1. For example, the information processing device 30 may further have an operation display unit.

[0032] <<2-1. Functional Configuration of Information Processing Device 30>> Figure 2 is a block diagram showing an example of the functional configuration of the information processing device 30 according to this embodiment. As shown in Figure 2, the control unit 320 of the information processing device 30 functions as a camera parameter estimation unit 321, a multi-view integration unit 322, and a three-dimensional reconstruction unit 323.

[0033] The camera parameter estimation unit 321 estimates camera parameters for each captured image in image group A and image group B. Camera parameters are an example of device parameters. Camera parameters include internal camera parameters and external camera parameters.

[0034] The camera intrinsic parameters include at least one of the focal length, optical center point, and lens distortion parameters. The camera parameter estimation unit 321 estimates the camera intrinsic parameters for each captured image in image group A and image group B. In some cases, information regarding the camera intrinsic parameters may be output as metadata along with the image group from the imaging device 10 and the imaging distance measuring device 20. The camera parameter estimation unit 321 may appropriately utilize the metadata obtained from the imaging device 10 and the imaging distance measuring device 20 when estimating the camera intrinsic parameters for each captured image.

[0035] Camera external parameters include information about the position and orientation of the device at the time of imaging. The camera parameter estimation unit 321 estimates the position and orientation of each image in image group A and image group B. Here, the camera parameter estimation unit 321 estimates the position and orientation of the image based on a common reference. More specifically, the common reference is a coordinate system, and the camera parameter estimation unit 321 estimates the three-dimensional position and orientation of each image in image group A and image group B based on the same coordinate system.

[0036] The camera parameter estimation unit 321 can analyze each captured image and estimate its three-dimensional position and orientation relative to the same coordinate system. If each captured image is associated with information regarding the camera's position and orientation acquired by the imaging device 10 or the imaging distance measuring device 20, the camera parameter estimation unit 321 may appropriately refer to this information to estimate the three-dimensional position and orientation relative to the same coordinate system for each captured image. Information regarding the camera's position and orientation includes, for example, information acquired by the position sensor (e.g., GPS information), information acquired by the IMU, and information acquired by the geomagnetic sensor. Furthermore, if a marker is placed near an object and the object is captured together with the marker, the camera parameter estimation unit 321 may use the marker in the captured image to estimate the three-dimensional position and orientation relative to the same coordinate system.

[0037] The camera parameter estimation unit 321 outputs image group A and camera parameters A of image group A (internal camera parameters and external camera parameters of image group A) to the 3D reconstruction unit 323. The camera parameter estimation unit 321 also outputs camera parameters B of image group B (specifically, external camera parameters (hereinafter also referred to as imaging position and orientation data)) to the multi-view integration unit 322.

[0038] The multi-view integration unit 322 generates a multi-view integrated point cloud based on the depth data set. Specifically, the multi-view integration unit 322 integrates a depth data set consisting of multiple distance measurement data to generate a single point cloud. In doing so, the multi-view integration unit 322 refers to the imaging position and orientation data of the image group B output from the camera parameter estimation unit 321 and generates a multi-view integrated point cloud that matches this imaging position and orientation data.

[0039] To explain in more detail, first, the depth data set consists of multiple depth data points indicating the distance between the object and the distance measuring unit 24. When integrating such multiple depth data to generate a three-dimensional point cloud showing the shape of the object, the position and orientation at the time each depth data was acquired are required. Here, the position and orientation of the imaging unit 22, i.e., the imaging position and orientation of image group B, can be determined by estimating the camera parameters of image group B. In addition, the multi-view integration unit 322 can determine the position and orientation of the distance measuring unit 24, i.e., the distance measuring position and orientation of the depth data set, from the imaging position and orientation of image group B and the calibration results performed in advance between the imaging unit 22 and the distance measuring unit 24. Based on each depth data, the multi-view integration unit 322 can generate a three-dimensional multi-view integrated point cloud showing the shape of the object by repeating the process for multiple viewpoints, placing a point a distance ahead from each distance measuring position, taking into account the orientation, by the distance indicated by the depth data. The three-dimensional position of the multi-view integrated point cloud will match the imaging position and orientation of image group B.

[0040] The imaging position and orientation data for image group B is consistent data, sharing the same coordinate system as the position and orientation data for image group A, which is used for 3D reconstruction. Therefore, the multi-view integration unit 322 can indirectly align the multi-view integrated point cloud with image group A, which is used for 3D reconstruction, by matching the multi-view integrated point cloud with the imaging position and orientation data for image group B. Since the relative position and orientation relationship between the distance measuring unit 24, which acquired the depth data set used for the multi-view integrated point cloud, and the imaging unit 22, which acquired image group B, is known (calibrated), the multi-view integration unit 322 can accurately align the multi-view integrated point cloud with the imaging position and orientation data for image group B.

[0041] Figure 3 is a diagram illustrating the alignment of a multi-view integrated point cloud according to this embodiment. As shown in Figure 3, the imaging position and orientation data 150 of image group A and the imaging position and orientation data 250 of image group B are data that use the same coordinate system. Furthermore, the relative position and orientation relationship between the distance measuring unit 24 that acquires the depth data group 270 and the imaging unit 22 that acquires image group B is known, and the depth data group 270 and the imaging position and orientation data 250 of image group B have been calibrated.

[0042] The multi-view integration unit 322 refers to the imaging position and orientation data 150 of the image group A and the imaging position and orientation data 250 of the image group B with the same coordinate system, and aligns the multi-view integrated point cloud 400 of the object generated from the depth data group 270 (with the image group B). As a result, a highly accurate multi-view integrated point cloud 400 with reduced misalignment with the image group A can be obtained.

[0043] The 3D reconstruction unit 323 performs 3D reconstruction of the object using the image group A and the multi-view integrated point cloud. The 3D reconstruction unit 323 may use the camera parameters A corresponding to the image group A. The camera parameters A used for performing 3D reconstruction of the object are, for example, at least data on the focal length and the imaging position and orientation data (camera extrinsic parameters). Further, the 3D reconstruction unit 323 may further use the depth data group.

[0044] It is assumed that the image group A used for 3D reconstruction is a high-quality image. In the present disclosure, since the distance measuring unit 24 that acquires the depth data group is provided near the imaging unit 22 (so that the parallax is minimized as much as possible), it does not affect the design and size of the imaging device 10 for which acquisition of a high-quality image is desired, and an imaging device 10 that can acquire an image with sufficient image quality can be used.

[0045] As described above, the functional configuration of the information processing device 30 has been specifically described.

[0046] <3. Operation Processing> Fig. 4 is a sequence diagram showing an example of the flow of the operation processing of 3D reconstruction in the present embodiment.

[0047] As shown in Fig. 4, first, the imaging and distance measuring device 20 calibrates the imaging unit 22 and the distance measuring unit 24 (step S103). Note that the calibration does not necessarily have to be performed at this timing. If the imaging and distance measuring device 20 stores the values at the time of factory shipment or the results of past calibrations, these may be used.

[0048] Next, the imaging device 10 performs multi-view imaging of the object by the imaging unit 12 (step S106). For example, the user operates the imaging device 10 to image the object from a plurality of viewpoints.

[0049] Next, the imaging distance measuring device 20 performs multi-viewpoint imaging of the object using the imaging unit 22 (step S109). For example, a user operates the imaging distance measuring device 20 to image the object from multiple viewpoints. Simultaneously with the multi-viewpoint imaging of the object by the imaging unit 22, the imaging distance measuring device 20 measures the distance between the distance measuring unit 24 and the object using the distance measuring unit 24 and acquires depth data.

[0050] Next, the imaging device 10 outputs the image group A, captured from multiple viewpoints, to the information processing device 30 (step S112). The imaging distance measuring device 20 also outputs the image group B, captured from multiple viewpoints, and the depth data group to the information processing device 30 (step S115). Input of each data to the information processing device 30 is not limited to wireless or wired communication; the user may also use a storage medium.

[0051] Next, the information processing device 30 estimates camera parameters in a common coordinate system for each captured image in image group A and image group B (step S118). More specifically, the information processing device 30 estimates internal camera parameters and external camera parameters for each captured image in image group A and image group B. As described above, the external camera parameters are information about the position and orientation of the image being captured, and the information processing device 30 estimates the position and orientation of the image being captured in a common coordinate system for each captured image in image group A and image group B.

[0052] Next, the information processing device 30 generates a multi-view integrated point cloud from the depth data set and adjusts the position and orientation of the multi-view integrated point cloud to match the position and orientation of image group B (step S121). That is, the information processing device 30 generates a multi-view integrated point cloud that matches the position and orientation of image group B based on the depth data set and the camera external parameters of image group B.

[0053] Then, the information processing device 30 uses the image group A, the camera parameters of the image group A, and the multi-view integrated point cloud to perform a three-dimensional reconstruction of the object (i.e., generate a 3D model of the object) (step S124).

[0054] The operation process flow according to this embodiment has been described above. Note that the operation process shown in Figure 4 is just one example, and this disclosure is not limited thereto. For example, not all of the processes shown in Figure 4 are performed, and the order of some of the processes shown in Figure 4 may be different. For example, steps S106 and S112 and steps S109 and S115 shown in Figure 4 may be performed in parallel, or in the reverse order of the order shown in Figure 4.

[0055] <4. Modified Examples> <<4-1. First Modified Example>> In the embodiment described above, it was stated that imaging by the imaging device 10 and imaging by the imaging distance measuring device 20 do not necessarily have to be performed simultaneously. For example, the user may perform imaging by the imaging distance measuring device 20 after imaging by the imaging device 10. On the other hand, it is also possible to shorten the imaging work time by performing imaging by the imaging device 10 and imaging by the imaging distance measuring device 20 simultaneously.

[0056] Figure 5 is a diagram illustrating the attachment of the imaging distance measuring device 20 to the imaging device 10 according to a first modification of this embodiment. As shown in Figure 5, the imaging distance measuring device 20 can be attached to the imaging device 10 by fixing the attachment 3 that supports the imaging distance measuring device 20 to the imaging device 10.

[0057] The imaging device 10 and the imaging distance measuring device 20 may be connected by wired or wireless communication, and imaging and distance measurement in the imaging distance measuring device 20 may be performed simultaneously with imaging by the imaging device 10 in conjunction with the shutter operation performed by the imaging device 10.

[0058] Furthermore, while imaging is being performed by the imaging device 10, imaging and distance measurement may be continuously performed by the imaging distance measuring device 20. For example, the imaging distance measuring device 20 may acquire a video using the imaging unit 22 and generate a group of images B from the video.

[0059] When the imaging rangefinder 20 is attached to the imaging device 10, the relative positional relationship between the imaging device 10 and the imaging rangefinder 20 can be fixed. Calibration may be performed between the imaging device 10 and the imaging rangefinder 20. The calibration results are used when estimating camera parameters for each image in image group A and image group B.

[0060] When imaging is performed simultaneously by the imaging device 10 and the imaging distance measuring device 20, it is also possible to use the fact that imaging is performed simultaneously as a constraint condition for estimating camera parameters.

[0061] <<4-2. Second Modification>> At least a part of each component of the control unit 320 of the information processing device 30 may be performed in another device. For example, each component of the control unit 320 may be performed in the imaging distance measuring device 20A.

[0062] Figure 6 is a block diagram showing a configuration example according to a second modification of this embodiment. As shown in Figure 6, the imaging distance measuring device 20A has an image processing unit 26 (an example of a control unit). The image processing unit 26 can also function as a camera parameter estimation unit 261, a multi-view integration unit 262, and a three-dimensional reconstruction unit 263.

[0063] The functions of the camera parameter estimation unit 261, the multi-view integration unit 262, and the 3D reconstruction unit 263 are the same as those of the camera parameter estimation unit 321, the multi-view integration unit 322, and the 3D reconstruction unit 323, as described with reference to Figure 2.

[0064] The imaging distance measuring device 20A receives the image group A of the object obtained by the imaging unit 12 of the imaging device 10. The camera parameter estimation unit 321 estimates camera parameters for each image, from image group A to image group B of the same object obtained by the imaging unit 22. The multi-view integration unit 322 generates a multi-view integrated point cloud based on the depth data group obtained by the distance measuring unit 24 and matches it to the imaging position and orientation of image group B estimated by the camera parameter estimation unit 321. Then, the 3D reconstruction unit 323 performs a 3D reconstruction of the object using image group A, the camera parameters of image group A, and the multi-view integrated point cloud.

[0065] Here, as an example, a case in which each component of the control unit 320 is provided on the imaging distance measuring device 20A has been described, but the disclosure is not limited thereto, and at least a part of each component of the control unit 320 may be provided on the imaging device 10.

[0066] <<4-3. Third Modification>> The information processing system 1 according to this disclosure may consist of a server (an example of an information processing device) and a user terminal. Figure 7 is a block diagram showing a configuration example according to the third modification of this embodiment.

[0067] As shown in Figure 7, the user terminal 40 and the server 60 communicate via the network 70. The user terminal 40 can be a PC, smartphone, tablet, or HMD, etc.

[0068] The user terminal 40 includes a communication unit 42, an operation display unit 44, a control unit 46, and a storage unit 48. The user terminal 40 acquires image group A obtained by the imaging unit 12 of the imaging device 10, image group B obtained by the imaging unit 22 of the imaging distance measuring device 20, and depth data group obtained by the distance measuring unit 24.

[0069] The control unit 46 of the user terminal 40 may transmit image group A, image group B, and depth data group from the communication unit 42 to the server 60 via the network 70 and request the execution of 3D reconstruction. The control unit 46 of the user terminal 40 may receive the result of the 3D reconstruction from the server 60 and display it on the operation display unit 44.

[0070] The server 60 includes a communication unit 62, a control unit 64, and a storage unit 66. The control unit 64 can also function as a camera parameter estimation unit 641, a multi-view integration unit 642, and a 3D reconstruction unit 643. The functions of the camera parameter estimation unit 641, the multi-view integration unit 642, and the 3D reconstruction unit 643 are the same as those of the camera parameter estimation unit 321, the multi-view integration unit 322, and the 3D reconstruction unit 323, as described with reference to Figure 2. The control unit 64 of the server 60 transmits the results of the 3D reconstruction by the 3D reconstruction unit 323 to the user terminal 40.

[0071] Furthermore, at least some of the functions of the control unit 64 may be provided on the user terminal 40. For example, the functions of the camera parameter estimation unit 641 may be executed by the control unit 46 of the user terminal 40.

[0072] <<4-4. Fourth Modification>> The camera parameter estimation unit 321 may perform preprocessing such as image color correction and image shake removal on each captured image in image group A and image group B before estimating the camera parameters.

[0073] Furthermore, the multi-view integration unit 322 may, for each image captured in image group A and image group B, exclude images whose estimated camera parameter confidence level is lower than a predetermined value, and then use the imaging position and orientation data of image group B for the alignment of the multi-view integrated point cloud. The method for calculating the confidence level is not particularly limited, but for example, outliers may be detected using reprojection error.

[0074] Furthermore, the 3D reconstruction unit 323 may perform preprocessing such as denoising the multi-view integrated point cloud, cropping only the region of interest, calculating and setting the confidence level, removing outliers, smoothing, normal estimation, segmentation preservation, or resampling before performing 3D reconstruction.

[0075] <<4-5. Fifth Modification>> In the embodiments described above, a configuration in which the imaging unit 22 and the distance measuring unit 24 are integrated was given as shown in Figure 1, but the disclosure is not limited thereto, and the imaging unit 22 and the distance measuring unit 24 may be separate. In this case, for example, imaging and distance measurement can be performed with an imaging device having an imaging unit 22 and a distance measuring device having a distance measuring unit 24 detachably attached to the imaging device having an imaging unit 22.

[0076] <5. Hardware Configuration> An embodiment of the present disclosure has been described above. Next, with reference to Figure 8, an example of a hardware configuration used in an imaging device 10, imaging distance measuring device 20, imaging distance measuring device 20A, information processing device 30, user terminal 40, or server 60 according to an embodiment of the present disclosure will be described.

[0077] Figure 8 is a block diagram showing an example of the hardware configuration of an information processing device 900 according to one embodiment of the present disclosure. The information processing device 900 is an example of a hardware configuration applicable to the imaging device 10, imaging distance measuring device 20, imaging distance measuring device 20A, information processing device 30, user terminal 40, or server 60 according to this embodiment. Note that the information processing device 900 does not necessarily have all of the hardware configurations shown in Figure 19.

[0078] As shown in Figure 8, the information processing device 900 includes a processing circuit 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903. The information processing device 900 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925.

[0079] The processing circuit 901 functions as an arithmetic processing unit and control unit, and controls the overall operation or a part of the operation within the information processing unit 900 according to various programs recorded in the ROM 902, RAM 903, storage device 919, or removable recording medium 927. The ROM 902 stores programs and calculation parameters used by the processing circuit 901. The RAM 903 temporarily stores programs used in the execution of the processing circuit 901 and parameters that change as appropriate during its execution. The processing circuit 901, ROM 902, and RAM 903 are interconnected by a host bus 907, which is composed of an internal bus. Furthermore, the host bus 907 is connected to an external bus 911, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 909.

[0080] The input device 915 is a device operated by the user, such as a button. The input device 915 may also include a mouse, keyboard, touch panel, switch, and lever. The input device 915 may also include a microphone that detects the user's voice. The input device 915 may be, for example, a remote control device that uses infrared or other radio waves, or an external connection device 929 such as a mobile phone that is compatible with the operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs it to the processing circuit 901. By operating this input device 915, the user inputs various data to the information processing device 900 or instructs it to perform processing operations.

[0081] The input device 915 may also include an imaging device and sensors. The imaging device is a device that captures real space and generates an image using various components such as an image sensor, such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and a lens for controlling the imaging of a subject onto the image sensor. The imaging device may capture still images or motion images. The sensors are various types of sensors, such as distance sensors, acceleration sensors, gyro sensors, geomagnetic sensors, vibration sensors, light sensors, and sound sensors. The sensors acquire information about the state of the information processing device 900 itself, such as the orientation of the housing of the information processing device 900, and information about the surrounding environment of the information processing device 900, such as the brightness and noise around the information processing device 900. The sensors may also include a GPS sensor that receives GPS (Global Positioning System) signals and measures the latitude, longitude, and altitude of the device.

[0082] The output device 917 is comprised of a device capable of visually or audibly notifying the user of the acquired information. The output device 917 may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, or an audio output device such as a speaker or headphones. The output device 917 may also include a PDP (Plasma Display Panel), a projector, a hologram, a printer, etc. The output device 917 outputs the results obtained from the processing of the information processing device 900 as text or images, or as sound such as voice or acoustics. The output device 917 may also include a lighting device that brightens the surroundings.

[0083] The storage device 919 is a data storage device configured as an example of the storage unit of the information processing device 900. The storage device 919 is composed of, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. This storage device 919 stores programs and various data executed by the processing circuit 901, as well as various data acquired from external sources.

[0084] The drive 921 is a reader / writer for removable recording media 927 such as magnetic disks, optical disks, magneto-optical disks, or semiconductor memory, and is either built into or external to the information processing device 900. The drive 921 reads information recorded on the installed removable recording media 927 and outputs it to the RAM 905. The drive 921 also writes data to the installed removable recording media 927.

[0085] The connection port 923 is a port for directly connecting equipment to the information processing device 900. The connection port 923 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface) port. Alternatively, the connection port 923 may be an RS-232C port, an optical audio terminal, or an HDMI (High-Definition Multimedia Interface) port. By connecting an external device 929 to the connection port 923, various types of data can be exchanged between the information processing device 900 and the external device 929.

[0086] The communication device 925 is a communication interface, for example, consisting of a communication device for connecting to an external network 50. The communication device 925 may be, for example, a communication card for wired or wireless LAN (Local Area Network), Bluetooth®, Wi-Fi®, or WUSB (Wireless USB). Alternatively, the communication device 925 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication. The communication device 925 transmits and receives signals, for example, to the Internet or other communication devices using a predetermined protocol such as TCP / IP. The external network 50 connected to the communication device 925 is a network connected by wire or wireless, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0087] For example, when the information processing device 900 functions as an imaging device 10, imaging distance measuring device 20, imaging distance measuring device 20A, information processing device 30, user terminal 40, or server 60 according to the embodiments of this disclosure, the processing circuit 901 of the information processing device 900 functions as a control unit (not shown) of the imaging device 10, a control unit (not shown) of the imaging distance measuring device 20, a control unit 320, an image processing unit 26, a control unit 46, or a control unit 64 by executing a program loaded on the RAM 903. The storage device 919 stores the information processing program according to this disclosure, as well as various data stored in the storage unit (not shown) of the imaging device 10, the storage unit (not shown) of the imaging distance measuring device 20, the storage unit (not shown) of the imaging distance measuring device 20A, the storage unit 330, the storage unit 48, or the storage unit 66.

[0088] The processing circuit 901 reads and executes program data from the storage device 919, but as an alternative, these programs may be obtained from other devices via the external network 50. In other words, the storage device 919 is not limited to being inside the information processing device 900, but may be located outside the information processing device 900.

[0089] The processing circuit 901 is an example of an integrated circuit, and CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), ASIC (Application Specific Integrated Circuit), and FPGA (Field Programmable Gate Array) can all be considered integrated circuits.

[0090] Furthermore, when the information processing device 900 functions as an imaging device 10, imaging distance measuring device 20, imaging distance measuring device 20A, information processing device 30, user terminal 40, or server 60 according to the embodiments of this disclosure, the communication device 925 corresponds to the communication unit of the imaging device 10 (not shown), the communication unit of the imaging distance measuring device 20 (not shown), the communication unit of the imaging distance measuring device 20A (not shown), the communication unit 310, the communication unit 42, or the communication unit 62. Also, the input device 915 and the output device 917 correspond to the operation display unit 44. Furthermore, the input device 915 corresponds to the imaging unit 12, the imaging unit 22, or the distance measuring unit 24.

[0091] <6. Supplementary Information> Although preferred embodiments of the present disclosure have been described in detail above with reference to the attached drawings, the present technology is not limited to such examples. It is clear that a person with ordinary skill in the art of the present disclosure may conceive of various modifications or alterations within the scope of the technical idea described in the claims, and these will naturally be understood to fall within the technical scope of the present disclosure.

[0092] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0093] The information processing system according to this disclosure may consist of a single device comprising an imaging device 10, an imaging distance measuring device 20, an imaging distance measuring device 20A, an information processing device 30, a user terminal 40, or a server 60. Alternatively, the information processing system according to this disclosure may consist of multiple devices. These multiple devices may be, for example, two or more of the imaging device 10, the imaging distance measuring device 20, the imaging distance measuring device 20A, the information processing device 30, the user terminal 40, and the server 60.

[0094] Furthermore, the embodiments and modifications of this disclosure described above can be combined as appropriate in areas where the processing content is not contradictory. Also, the order of each step shown in the sequence diagram or flowchart of this embodiment can be changed as appropriate. For example, each step may be processed chronologically, repeatedly, or partially in parallel.

[0095] Furthermore, one or more computer programs can be created to enable the functions of the imaging device 10, imaging distance measuring device 20, imaging distance measuring device 20A, information processing device 30, user terminal 40, or server 60, using hardware such as the CPU, ROM, and RAM built into the imaging device 10, imaging distance measuring device 20, imaging distance measuring device 20A, information processing device 30, user terminal 40, or server 60. A computer-readable storage medium storing such one or more computer programs is also provided.

[0096] Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that will be apparent to those skilled in the art from the description herein, in addition to or in lieu of the effects described herein.

[0097] Furthermore, this technology can also be configured as follows: (1) An information processing system comprising a control unit which estimates the position and orientation of each image of an object captured by a first imaging device and a second imaging device, based on a common reference; generates an integrated point cloud by integrating the distance measurement data of the object measured by a distance measuring device attached to the second imaging device, so as to match the position and orientation estimated from each image of the second imaging image group; and performs a three-dimensional reconstruction of the object based on the integrated point cloud and the first imaging image group. (2) The information processing system according to (1), wherein the estimated position and orientation of each image of the first and second imaging image groups is a three-dimensional position and orientation based on the same coordinate system. (3) The information processing system according to (1) or (2), wherein the control unit performs a three-dimensional reconstruction of the object using the first group of captured images, device parameters including estimated position and orientation information of the first group of captured images, and the integrated point cloud. (4) The information processing system according to (3), wherein the device parameters further include at least one of the focal length information, optical center point, and lens distortion parameters of each captured image in the first group of captured images. (5) The information processing system according to any one of (2) to (4), wherein the control unit further uses the distance measurement data group when performing a three-dimensional reconstruction of the object. (6) The information processing system according to any one of (1) to (5), wherein the second imaging device and the distance measurement device are an integrated device. (7) The information processing system according to any one of (1) to (6), wherein imaging by the second imaging device and distance measurement by the distance measurement device are performed simultaneously. (8) The information processing system according to any one of (1) to (7) above, wherein imaging by the first imaging device and imaging by the second imaging device are performed simultaneously. (9) The information processing system according to any one of (1) to (8) above, wherein the control unit uses the position and orientation of each image from the first image group and the second image group, excluding the image whose reliability is lower than a predetermined value, for alignment in the generation of an integrated point cloud.(10) The information processing system according to any one of (1) to (9), wherein the information processing system includes an information processing device having the control unit. (11) The information processing system according to any one of (1) to (9), wherein the information processing system includes the first or second imaging device having the control unit. (12) An information processing method comprising: a processor estimating the position and orientation of each image of a first group of images of an object captured by a first imaging device and a second group of images of the object captured by a second imaging device, based on a common reference; generating an integrated point cloud by integrating the distance measurement data of the object, based on a group of distance measurement data of the object measured by a distance measuring device attached to the second imaging device, so as to match the position and orientation estimated from each image of the second group of images; and performing a three-dimensional reconstruction of the object based on the integrated point cloud and the first group of images. (13) The information processing method according to (12), wherein the estimated position and orientation of each image in the first and second image groups is a three-dimensional position and orientation based on the same coordinate system. (14) The information processing method according to (12) or (13), wherein the processor performs a three-dimensional reconstruction of the object using the first image group, device parameters including the estimated position and orientation information of the first image group, and the integrated point cloud. (15) The information processing method according to any one of (12) to (14), wherein the processor further uses the distance measurement data group when performing the three-dimensional reconstruction of the object. (16) The information processing method according to any one of (12) to (15), wherein imaging by the second imaging device and distance measurement by the distance measuring device are performed simultaneously.(17) A program for causing a computer to function as a control unit, which estimates the position and orientation of each image of an object captured by a first imaging device and a second imaging device, based on a common reference; generates an integrated point cloud by integrating the distance measurement data of the object, based on a group of distance measurement data of the object, measured by a distance measuring device attached to the second imaging device, so as to match the position and orientation estimated from each image of the second imaging image group; and performs a three-dimensional reconstruction of the object based on the integrated point cloud and the first imaging image group.

[0098] 1 Information Processing System 10 Imaging Device 12 Imaging Unit 20 Imaging Distancing Device 22 Imaging Unit 24 Distancing Unit 30 Information Processing Device 310 Communication Unit 320 Control Unit 321 Camera Parameter Estimation Unit 322 Multi-View Integration Unit 323 3D Reconstruction Unit 330 Storage Unit

Claims

1. An information processing system comprising a control unit that estimates the position and orientation of each image of an object captured by a first imaging device and a second imaging device, based on a common reference; generates an integrated point cloud by integrating the distance measurement data of the object, based on a group of distance measurement data of the object measured by a distance measuring device attached to the second imaging device, so as to match the position and orientation estimated from each image of the second imaging image group; and performs a three-dimensional reconstruction of the object based on the integrated point cloud and the first imaging image group.

2. The information processing system according to claim 1, wherein the estimated positional orientation of each image in the first and second image groups is a three-dimensional positional orientation based on the same coordinate system.

3. The information processing system according to claim 1, wherein the control unit performs a three-dimensional reconstruction of the object using the first group of captured images, device parameters including estimated position and orientation information of the first group of captured images, and the integrated point cloud.

4. The information processing system according to claim 3, wherein the device parameters further include at least one of the focal length information, optical center point, and lens distortion parameters of each image in the first group of captured images.

5. The information processing system according to claim 2, wherein the control unit further uses the distance measurement data group when performing a three-dimensional reconstruction of the object.

6. The information processing system according to claim 1, wherein the second imaging device and the distance measuring device are an integrated device.

7. The information processing system according to claim 1, wherein imaging by the second imaging device and distance measurement by the distance measuring device are performed simultaneously.

8. The information processing system according to claim 1, wherein imaging by the first imaging device and imaging by the second imaging device are performed simultaneously.

9. The information processing system according to claim 1, wherein the control unit uses the position and orientation of each image from the first image group and the second image group, excluding the image whose reliability is lower than a predetermined value, for alignment in the generation of an integrated point cloud.

10. The information processing system according to claim 1, wherein the information processing system includes an information processing device having the control unit.

11. The information processing system according to claim 1, wherein the information processing system includes the first or second imaging device having the control unit.

12. An information processing method comprising: a processor estimating the position and orientation of each image of a first group of images of an object captured by a first imaging device and a second group of images of the object captured by a second imaging device, based on a common reference; generating an integrated point cloud by integrating the distance measurement data of the object, based on a group of distance measurement data of the object measured by a distance measuring device attached to the second imaging device, so as to match the position and orientation estimated from each image of the second group of images; and performing a three-dimensional reconstruction of the object based on the integrated point cloud and the first group of images.

13. The information processing method according to claim 12, wherein the estimated positional orientation of each image in the first and second image groups is a three-dimensional positional orientation based on the same coordinate system.

14. The information processing method according to claim 12, wherein the processor performs a three-dimensional reconstruction of the object using the first group of captured images, device parameters including estimated position and orientation information of the first group of captured images, and the integrated point cloud.

15. The information processing method according to claim 12, wherein the processor further uses the distance measurement data set when performing a three-dimensional reconstruction of the object.

16. The information processing method according to claim 12, wherein imaging by the second imaging device and distance measurement by the distance measuring device are performed simultaneously.

17. A program to cause a computer to function as a control unit, which estimates the position and orientation of each image captured by a first imaging device and a second imaging device, based on a common reference; generates an integrated point cloud by integrating the distance measurement data of the object measured by a distance measuring device attached to the second imaging device, so as to match the position and orientation estimated from each image in the second imaging image group; and performs a three-dimensional reconstruction of the object based on the integrated point cloud and the first imaging image group.

Images