Information processing device, method for controlling information processing device, and program for controlling information processing device

Abstract

This information processing device comprises: a shape acquisition unit capable of acquiring three-dimensional shape information of an object to be processed; an image acquisition unit capable of acquiring image data including the object to be processed; and a densification unit capable of densifying the three-dimensional shape information in stages on the basis of the image data.

Description

Information Processing Apparatus, Control Method for Information Processing Apparatus, and Control Program for Information Processing Apparatus

[0001] The present invention relates to an information processing apparatus, a control method for the information processing apparatus, and a control program for the information processing apparatus.

[0002] In recent years, three-dimensional shape information obtained from three-dimensional ranging devices such as LiDAR (Light Detection And Ranging) sensors has been widely used in autonomous driving, object recognition, or generation of 3D models. In Non-Patent Document 1, a method of generating a high-density point cloud by combining an image with a point cloud based on reflected light reflected by a measurement target using a LiDAR sensor is described. In Patent Document 1, a method of generating a high-density three-dimensional shape by repeatedly applying the visual volume intersection method to voxel grids with different densities is described. In Patent Document 2, a method of inferring a high-density three-dimensional shape of a road surface in autonomous driving using a neural network structure is described.

[0003] Xinjing Cheng, Peng Wang and Ruigang Yang 'Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Network'

[0004] Japanese Unexamined Patent Application Publication No. 2021-33682, Japanese Unexamined Patent Application Publication No. 2023-66377

[0005] However, LiDAR sensors emit light rays radially from the sensor center and acquire three-dimensional distance information based on the light rays reflected by the object being measured. Therefore, if the object being measured is located far away, the acquired three-dimensional distance information will be sparse. Similarly, if a low-performance LiDAR is used, the acquired three-dimensional distance information will also be sparse. Thus, in such cases, it is difficult to obtain detailed three-dimensional shape information of the object being measured using the technology described in Non-Patent Document 1. Furthermore, the viewing volume cross-section method in Patent Document 1 is a method based solely on images, so the accuracy of the obtained three-dimensional shape is limited. Moreover, the method described in Patent Document 2 is a method that recovers the surface shape from a point cloud acquired from a LiDAR sensor, so if the point cloud itself is sparse, it is difficult to obtain detailed three-dimensional shape information. Furthermore, its application is limited to road surfaces.

[0006] Therefore, in one aspect, this disclosure aims to provide a technology that can acquire detailed three-dimensional shape information of a measurement target regardless of the distance information from the measurement unit to the measurement target.

[0007] One aspect of this disclosure is an information processing device comprising: a shape acquisition unit capable of acquiring three-dimensional shape information of a target to be processed; an image acquisition unit capable of acquiring image data including the target to be processed; and a density enhancement unit capable of gradually increasing the density of the three-dimensional shape information based on the image data, wherein the image acquisition unit acquires first image data including at least a part of the target to be processed and second image data including a part of the region included in the first image data, or an image included in the first image data with increased resolution; and the density enhancement unit generates first high-density shape information by increasing the density of the three-dimensional shape information corresponding to the target to be processed included in the first image data, and further generates second high-density shape information by increasing the density of the information corresponding to the target to be processed included in the second image data from the first high-density shape information.

[0008] According to this disclosure, it is possible to provide a technology that can acquire detailed three-dimensional shape information of a measurement target regardless of the distance from the measurement position to the measurement target.

[0009] Other features and advantages of the present invention will become apparent from the following description with reference to the accompanying drawings. In the accompanying drawings, the same or similar components are given the same reference numeral.

[0010] The attached drawings are included in the specification and constitute a part thereof, illustrating embodiments of the present invention and are used to explain the principles of the present invention together with the description thereof. Block diagram of a three-dimensional information processing device according to one embodiment Flowchart of three-dimensional information processing according to one embodiment Explanatory diagram of three-dimensional information processing according to one embodiment Explanatory diagram of three-dimensional information processing according to one embodiment Block diagram of a three-dimensional information processing device according to one embodiment Flowchart of three-dimensional information processing according to one embodiment Block diagram of a three-dimensional information processing device according to one embodiment Flowchart of three-dimensional information processing according to one embodiment

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] Using Figure 1, the blocks of the three-dimensional information processing device 100 according to this embodiment will be explained. The three-dimensional information processing device 100 is composed of a three-dimensional distance measuring unit 110, a wide-angle imaging unit 120, a telephoto imaging unit 130, a control unit 140, a non-volatile memory 150, and a system memory 160. In Figure 1, the arrow lines connecting the blocks indicate the flow of data from one block to another, and this data flow is executed by the control unit 140. The control unit 140 is connected to the wide-angle imaging unit 120, the telephoto imaging unit 130, the non-volatile memory 150, and the system memory 160, and controls the operation of these blocks.

[0013] The 3D distance measuring unit 110 (an example of a "shape acquisition unit capable of acquiring 3D shape information") is, for example, a LiDAR sensor, and measures the distance from the 3D distance measuring unit 110 to the object to be measured (an example of a "processing target"). The LiDAR sensor includes a light ray output unit that outputs a light ray and irradiates the surface of the object with the ray, and a receiving unit that receives the reflected light ray reflected from the surface of the object to be measured. The 3D distance measuring unit 110 calculates the distance to the surface of the object to be measured in the direction of irradiation using the time from irradiation of the light ray to reception of the reflected light ray, or the phase difference between the irradiated light ray and the reflected light ray. The 3D distance measuring unit 110 then acquires 3D shape information as a 3D point cloud using the calculated distance and the information of the direction of irradiation. The control unit 140 then stores the 3D point cloud in the 3D point cloud storage unit 152. Note that the 3D shape information is not limited to a 3D point cloud, and may be represented by known methods such as voxels, meshes, or implicit function representations. Furthermore, the 3D distance measuring unit 110 is not limited to a LiDAR sensor, but may also acquire 3D shape information of the object to be measured using, for example, a laser scanner, stereo vision, a moving image of the object to be measured, or CG (Computer Graphics).

[0014] The wide-angle imaging unit 120 and the telephoto imaging unit 130 (each an example of an "image acquisition unit") are, for example, digital still cameras and are capable of photographing the same measurement target. The wide-angle imaging unit 120 and the telephoto imaging unit 130 each include lens units with different angles of view, an image sensor that converts an optical image into an electrical signal, and an A / D converter that converts an analog signal into a digital signal. The wide-angle imaging unit 120 and the telephoto imaging unit 130 acquire an image when an optical image is input to the image sensor via the lens unit and the electrical signal converted by the image sensor is converted into a digital signal. The telephoto lens unit 131 of the telephoto imaging unit 130 has a narrower angle of view than the wide-angle lens unit 121 of the wide-angle imaging unit 120, and although the shooting area is limited, it can acquire a higher resolution image. The control unit 140 stores the image data acquired from the wide-angle imaging unit 120 and the telephoto imaging unit 130 in the image storage unit 153.

[0015] The control unit 140 is configured to include at least one processor and controls the entire 3D information processing device 100. Such a control unit 140 realizes the processes described later by executing a program recorded in the system storage unit 151. The control unit 140 may include a high-density point cloud generation unit 141 and a 3D point cloud extraction unit 142. The high-density point cloud generation unit 141 generates a high-density 3D point cloud from the 3D point cloud recorded in the 3D point cloud storage unit 152 and the image recorded in the image storage unit 153 using a method such as that disclosed in Non-Patent Literature 1. The high-density point cloud generation unit 141 then stores the generated high-density 3D point cloud in the 3D point cloud storage unit 152. The high-density point cloud generation unit 141 is an example of a "densification unit capable of gradually increasing the density of 3D shape information".

[0016] The 3D point cloud extraction unit 142 extracts 3D point clouds from the 3D point clouds recorded in the 3D point cloud storage unit 152 that are included in the region of interest. The region of interest is defined as the region corresponding to the field of view of the telephoto imaging unit 130. The region of interest is set based on the relative position information of the telephoto imaging unit 130 with respect to the 3D distance measuring unit 110. Furthermore, the 3D point cloud extraction unit 142 uses the difference in field of view between the wide-angle imaging unit 120 and the telephoto imaging unit 130 to convert the position information of the extracted 3D point clouds into a coordinate system based on the position of the telephoto imaging unit 130. More specifically, the 3D point cloud extraction unit 142 calculates the focal length of the wide-angle imaging unit 120 from its size and field of view. Similarly, the 3D point cloud extraction unit 142 calculates the focal length of the telephoto imaging unit 130 from its size and field of view. The 3D point cloud extraction unit 142 then uses these focal lengths to calculate the ratio of the size of the measurement target included in the telephoto image (described later) to the size of the measurement target included in the wide-angle image (described later), and uses this ratio to perform an affine transformation on the positional information of the 3D point cloud. Note that the method of coordinate transformation is not limited to using focal lengths. For example, the size of the measurement target included in the images acquired by the wide-angle imaging unit 120 and the telephoto imaging unit 130 may be directly derived. Then, the above ratio may be calculated from that size, and the affine transformation may be performed using this ratio. The 3D point cloud extraction unit 142 then stores the positional information of the sparsely transformed 3D point cloud in the 3D point cloud storage unit 152.

[0017] The non-volatile memory 150 includes a system memory unit 151, a three-dimensional point cloud memory unit 152, and an image memory unit 153. The non-volatile memory 150 is an electrically erasable and recordable memory, such as an EEPROM. The system memory unit 151 stores the operation programs and constants for operation of each block of the control unit 140. The system memory unit 151 also stores the relative position information of the telephoto imaging unit 130 with respect to the three-dimensional distance measuring unit 110. The program referred to here includes programs for executing various flowcharts described later. The three-dimensional point cloud memory unit 152 stores the three-dimensional point cloud acquired by the three-dimensional distance measuring unit 110 and the position information of the three-dimensional point cloud converted by the three-dimensional point cloud extraction unit 142. The image memory unit 153 stores image data captured by the wide-angle imaging unit 120 and the telephoto imaging unit 130, respectively. The system memory 160 is a rewritable volatile memory, such as a DRAM. The system memory 160 stores constants and variables of the control unit 140 during operation, as well as data read from the non-volatile memory 150.

[0018] <Processing Example 1> Using Figures 2, 3A, and 3B, we will explain the process of generating a high-density three-dimensional point cloud representing the measurement target. Figure 2 shows a flowchart of this process, and Figures 3A and 3B are schematic diagrams of the captured image and the three-dimensional point cloud representing the measurement target included in the captured image. In Figures 3A and 3B, dotted lines represent wide-angle images, and dashed lines represent telephoto images. This process is achieved by the control unit 140 loading a program stored in the non-volatile memory 150 into the system memory 160 and executing it, thereby controlling each component.

[0019] In S201, a 3D point cloud and various images are acquired. Specifically, the 3D distance measuring unit 110 measures the distance from the 3D distance measuring unit 110 to the measurement target and outputs a 3D point cloud 320 of the measurement area using the measured distance information. The control unit 140 then stores the 3D point cloud 320 in the 3D point cloud storage unit 152. The wide-angle imaging unit 120 and the telephoto imaging unit 130 each photograph the measurement area and output a wide-angle image 310 (an example of "first image data") and a telephoto image 340 (an example of "second image data"). The control unit 140 then stores the wide-angle image 310 and the telephoto image 340 in the image storage unit 153. In Figure 3AA, the dashed-dot frame 311 included in the wide-angle image 310 indicates the range photographed by the telephoto imaging unit 130. Note that the measurement target measured by the 3D distance measuring unit 110 may include measurement targets not included in the wide-angle image 310. In other words, the wide-angle image 310 only needs to be an image that includes at least a portion of the object to be measured by the 3D distance measuring unit 110.

[0020] In S202, the high-density point cloud generation unit 141 combines the three-dimensional point cloud 320 recorded in the three-dimensional point cloud storage unit 152 with the wide-angle image 310 recorded in the image storage unit 153 to generate a wide-angle high-density point cloud 330 (an example of "first high-density shape information"). More specifically, as shown in Figure 3A, the high-density point cloud generation unit 141 generates the wide-angle high-density point cloud 330 by adding points to the three-dimensional point cloud 320 (an example of "three-dimensional shape information corresponding to the processing target") so as to match the measurement target included in the wide-angle image 310. Point addition is performed by methods such as Depth Completion. The high-density point cloud generation unit 141 then stores the generated wide-angle high-density point cloud 330 in the three-dimensional point cloud storage unit 152. Note that the wide-angle high-density point cloud 330 is not limited to being generated from the wide-angle image 310 and the three-dimensional point cloud 320. The high-density point cloud generation unit 141 may generate a wide-angle high-density point cloud 330 from the 3D point cloud 320 without using the wide-angle image 310 by, for example, other deep learning methods such as CNN or Diffusion generation models, or non-deep learning methods such as bicubic methods. The high-density point cloud generation unit 141 may, before generating the wide-angle high-density point cloud 330 from the 3D point cloud 320, use the relative position information of the wide-angle imaging unit 120 with respect to the 3D distance measuring unit 110 to translate the 3D point cloud 320 in a direction parallel to or perpendicular to the image plane, for example. Such a translation is an example of "converting the first high-density shape information from a coordinate system based on the shape acquisition unit to a coordinate system based on the image acquisition unit."

[0021] In S203, the 3D point cloud extraction unit 142 sets the region of interest 331 using the position information of the dotted frame 311 included in the wide-angle image 310 and the relative position information of the telephoto imaging unit 130 with respect to the 3D distance measuring unit 110. The 3D point cloud extraction unit 142 then extracts the point clouds included in the region of interest 331 from the wide-angle high-density point cloud 330 recorded in the 3D point cloud storage unit 152 as the region of interest point cloud 350. Figure 3A illustrates such a region of interest 331 and region of interest point cloud 350. The 3D point cloud extraction unit 142 then uses the difference in field of view between the wide-angle imaging unit 120 and the telephoto imaging unit 130 to convert the position information of the extracted region of interest point cloud 350 into the coordinate system of the telephoto imaging unit 130. When the position information of the region of interest point cloud 350 is converted in this way, a sparse region of interest point cloud 351 is generated, as shown in Figure 3B.

[0022] The region of interest 331 is set using relative position information of the telephoto imaging unit 130 relative to the 3D distance measuring unit 110, which is stored in the system memory unit 151 in advance, but the setting method is not limited to this. The region of interest 331 may also be set using, for example, the relative position information of the telephoto imaging unit 130 relative to the 3D distance measuring unit 110, which is dynamically calculated by comparing the feature quantities of the wide-angle high-density point cloud 330 and the telephoto image 340. Feature quantities include, for example, SIFT (Scale-Invariant Feature Transform) features or features extracted using CNN (Convolutional Neural Network). Using such features allows for correct matching of the feature quantities of measurement targets contained in images with different scales, such as the wide-angle image 310 and the telephoto image 340, making it possible to calculate relative position information between measurement targets. Alternatively, the region of interest 331 may be set from the features of the telephoto image 340.

[0023] In S204, the high-density point cloud generation unit 141 generates a higher-density telephoto high-density point cloud 360 by combining the point cloud of the region of interest 350, which was extracted in S203 and coordinate-transformed to the coordinate system of the telephoto imaging unit 130, with the telephoto image 340 recorded in the image storage unit 153. More specifically, the high-density point cloud generation unit 141 generates the telephoto high-density point cloud 360 by adding points to the point cloud of interest 350 to match the measurement target included in the telephoto image 340. The generation method is, for example, the same as the method in S202. Figure 3B shows an example of the telephoto high-density point cloud 360 generated in this way. After that, the processing of this flowchart is terminated. Note that the telephoto high-density point cloud 360 is an example of "second high-density shape information".

[0024] <One aspect of operation and effect> When distance information from the 3D distance measuring unit 110 to the measurement target is acquired sparsely, the 3D point cloud 320 of the measurement area becomes sparse, as shown in Figure 3A. However, with the 3D information processing device 100, it is possible to acquire a telephoto high-density point cloud 360 by gradually fitting such a 3D point cloud 320 to the measurement target included in the wide-angle image 310 and the telephoto image 340. Therefore, detailed 3D shape information of the measurement target can be acquired.

[0025] (First Modified Example) The three-dimensional information processing device 100 according to the first modified example is configured to include a single imaging unit equipped with a zoom lens (an example of an "optical lens"). Furthermore, in the three-dimensional information processing device 100, the same imaging unit photographs the object to be measured multiple times, and the positional alignment of the captured images is performed. These points differ from the three-dimensional information processing device 100 according to the embodiment. In the following, the same configuration as in the embodiment will be omitted from the description, and the description will focus on the parts that differ from the embodiment.

[0026] Using Figure 4, the blocks of the three-dimensional information processing device 100 according to the first modified example will be explained. The three-dimensional information processing device 100 according to the first modified example includes an imaging unit 420 as a substitute for the wide-angle imaging unit 120 and telephoto imaging unit 130 according to the embodiment. The three-dimensional information processing device 100 according to the first modified example also includes a three-dimensional distance measuring unit 110, a control unit 140, a non-volatile memory 150, and a system memory 160, similar to the three-dimensional information processing device 100 according to the embodiment.

[0027] The imaging unit 420 is a digital still camera that includes, for example, a zoom drive mechanism (an example of a "zoom mechanism that allows the lens to move in the optical axis direction so that the processing target can be imaged at two different angles of view"). The imaging unit 420 also includes a zoom lens unit 421 in which the shooting angle of view can be changed according to user operation. That is, the zoom lens unit 421 is configured with a zoom lens and a zoom mechanism consisting of a lens drive unit that drives the zoom lens. The zoom function is realized by the lens drive unit moving the zoom lens in the optical axis direction.

[0028] The control unit 140 includes a position difference calculation unit 443 in addition to the high-density point cloud generation unit 141 and the three-dimensional point cloud extraction unit 142 according to the embodiment. The position difference calculation unit 443 compares the feature quantities of the wide-angle image 310 and the telephoto image 340 recorded in the image storage unit 153 to extract matching feature quantities between the two images. Then, using the position information of the extracted feature quantities, it calculates the position difference of the imaging unit 420 between one shooting and another shooting. The feature quantities are, as described above, for example, SIFT feature quantities or feature quantities extracted using CNN. The method for calculating the position difference is not limited to this. For example, if the three-dimensional information processing device 100 is equipped with GPS or IMU (Inertial Measurement Unit), the position difference may be calculated using the information output from these. Alternatively, if time information of when the measurement target was imaged, or feature quantities on the three-dimensional point cloud are recorded in the image data, the position difference may be calculated using this information.

[0029] <Processing Example 2> Using Figures 5 and 6, another example of the process for generating a high-density three-dimensional point cloud representing the measurement target will be explained. This process is achieved by the control unit 140 according to the first modified example loading a program stored in the non-volatile memory 150 into the system memory 160 and executing it, thereby controlling each component. Note that S202, S203, and S204 are the same as in the embodiment and therefore their explanation is omitted.

[0030] In S501, the 3D distance measuring unit 110 measures the distance from the 3D distance measuring unit 110 to the object to be measured and outputs a 3D point cloud 320 of the measurement area using the measured distance information. The control unit 140 then stores the 3D point cloud 320 in the 3D point cloud storage unit 152. The imaging unit 420 also photographs the object to be measured without using the zoom function and stores the captured image as a wide-angle image 310 in the image storage unit 153. The imaging unit 420 also photographs the object to be measured using the zoom function and stores the captured image as a telephoto image 340 in the image storage unit 153. The subsequent processing proceeds in the order of S202 and S502. In S502, the zoom lens unit 421 of the imaging unit 420 drives the zoom lens to photograph the object to be measured with a narrower field of view than in step S501. The control unit 140 then stores the captured image as a telephoto image 340 in the image storage unit 153.

[0031] In S503, the position difference calculation unit 443 calculates the difference between the position captured by the imaging unit 420 at a narrow angle and the position captured by the imaging unit 420 at a wide angle, based on the feature quantities of the wide-angle image 310 and the telephoto image 340 recorded in the image storage unit 153. Then, using this position difference, the telephoto image 340 is aligned with the wide-angle image 310. The position difference calculation unit 443 also stores the calculated position difference in the system storage unit 151 as relative position information of the telephoto imaging unit 130 with respect to the 3D distance measuring unit 110 according to the embodiment. The process then proceeds to S203. In S203, the 3D point cloud extraction unit 142 converts the position information of the extracted point cloud of interest 350 into the coordinate system of the telephoto image 340 captured at a high zoom magnification. The other processing contents are the same as in S203 according to the embodiment. After that, the process proceeds to S204, ending the processing in this flowchart.

[0032] <One aspect of operation and effect> According to the 3D information processing device 100 of the first modified example, the position of a telephoto image taken at a different timing is aligned with the wide-angle image. By performing this process, even when the wide-angle imaging unit and the telephoto imaging unit are not provided as separate units, detailed 3D shape information of the measurement target can be obtained in the same way as the 3D information processing device 100 of the embodiment.

[0033] In the first modified example, the imaging unit 420 equipped with a zoom lens unit generates the wide-angle image 310 and the telephoto image 340. However, for example, a camera with interchangeable lenses may be used, and lenses with different angles of view may be changed each time a picture is taken. The wide-angle image 310 and the telephoto image 340 can also be generated by such a camera with interchangeable lenses.

[0034] (Second Modification) The 3D information processing device 100 according to the second modification differs from the 3D information processing device 100 according to the embodiment and the first modification in that a telephoto image is captured before the wide-angle image is captured. Then, the feasibility of increasing the density of the point cloud is determined using the telephoto image, and the capture of the wide-angle image is performed based on the determination result. In the following, the same configuration as in the embodiment or the first modification will be omitted from the explanation, and the explanation will focus on the differences.

[0035] Using Figure 6, the blocks of the 3D information processing device 100 according to the second modified example will be explained. The 3D information processing device 100 according to the second modified example includes a control unit 140, which includes a high-density point cloud generation unit 141, a 3D point cloud extraction unit 142, and a position difference calculation unit 443 according to the first modified example, as well as a generation determination unit 644 and a zoom control unit 645. The generation determination unit 644 determines whether or not a high-density point cloud can be generated by matching the 3D point cloud 320 recorded in the 3D point cloud storage unit 152 to the measurement target included in the telephoto image 340 recorded in the image storage unit 153. Details of the determination method will be described later. The zoom control unit 645 controls the zoom lens unit 421 based on the determination result of the generation determination unit 644. Furthermore, the 3D information processing device 100 according to the second modified example includes a 3D distance measuring unit 110, an imaging unit 420, a non-volatile memory 150, and a system memory 160, similar to the first modified example.

[0036] <Processing Example 3> Using Figure 7, another example of the process for generating a high-density three-dimensional point cloud representing the measurement target will be explained. This process is achieved by the control unit 140 loading a program stored in the non-volatile memory 150 into the system memory 160 and executing it, thereby controlling each component. Note that S203, S204, S502, and S503 are the same as in the embodiment or the first modification, so their explanation will be omitted.

[0037] In step S701, the 3D distance measuring unit 110 measures the distance from the 3D distance measuring unit 110 to the object to be measured and outputs a 3D point cloud 320 of the measurement area using the measured distance information. The control unit 140 then stores the 3D point cloud 320 in the 3D point cloud storage unit 152. The zoom lens unit 421 of the imaging unit 420 drives the zoom lens to photograph the object to be measured with a narrow field of view. The control unit 140 then stores the captured image data as a telephoto image 340 in the image storage unit 153.

[0038] In S702, the generation determination unit 644 determines whether or not a telephoto high-density point cloud 360 can be generated by matching the 3D point cloud 320 recorded in the 3D point cloud storage unit 152 with the measurement targets included in the telephoto image 340 recorded in the image storage unit 153. More specifically, the generation determination unit 644 calculates the ratio of the number of pixels in the telephoto image 340 to the number of points in the 3D point cloud indicating the measurement targets included in the telephoto image 340. This ratio is calculated based on the angular resolution of the 3D distance measuring unit 110 that measured the 3D point cloud 320, the resolution of the telephoto image 340, and the field of view of the imaging unit 420 that captured the telephoto image 340. The generation determination unit 644 then determines whether or not this ratio is below a predetermined threshold. If the generation determination unit 644 determines that this ratio is below the threshold, it determines that a telephoto high-density point cloud 360 can be generated, and the process proceeds to S703; otherwise, it proceeds to S704.

[0039] In S703, the 3D point cloud extraction unit 142, in response to the determination in S702 that a telephoto high-density point cloud 360 can be generated, sets the region of interest 331 using the relative position information of the telephoto imaging unit 130 with respect to the 3D distance measuring unit 110. This relative position information is stored in the system storage unit 151. Then, from the 3D point cloud 320 recorded in the 3D point cloud storage unit 152, points included in the region of interest 331 are extracted as the region of interest point cloud 350. After that, the process proceeds to S204.

[0040] On the other hand, if it is determined in S702 that a telephoto high-density point cloud 360 cannot be generated, the zoom control unit 645 drives the zoom lens unit 421 to move the lens in S704. The imaging unit 420 then photographs the object to be measured with a wide angle of view. The control unit 140 then stores the captured image data as a wide-angle image 310 in the image storage unit 153. The subsequent processing proceeds in the order of S502, S503, S203, and S204. This completes the processing of this flowchart.

[0041] <One aspect of operation and effect> The flowchart according to the first modified example (Figure 5) has an increased number of steps compared to the flowchart according to the embodiment (Figure 2), thus increasing the processing time. Therefore, in order to shorten the processing time, the 3D information processing device 100 according to the second modified example determines in advance whether or not the 3D point cloud can be made denser before the wide-angle image 310 is taken. If it is determined that the 3D point cloud can be made denser, a telephoto high-density point cloud 360 is generated without taking the wide-angle image 310. With such a 3D information processing device 100, detailed 3D shape information of the measurement target can be efficiently acquired.

[0042] (Other Modifications) In this embodiment, the three-dimensional point cloud is densified twice using two images, a wide-angle image 310 and a telephoto image 340. However, the number of images and the number of densification steps are not limited to those described above. For example, three or more images may be used to perform densification three or more times. In this embodiment, sets of the telephoto image 340 and the wide-angle image 310 used for the two densification steps are each captured, but these sets may be generated from a single image. For example, the three-dimensional information processing device 100 may include an image extraction unit that cuts out a portion of the wide-angle image output (provided) by the wide-angle imaging unit 120 to create a cropped image. That is, the control unit 140 may acquire the wide-angle image from the wide-angle imaging unit 120 and transmit it to the image extraction unit. The control unit 140 may then acquire the cropped image created by the image extraction unit.

[0043] Alternatively, the three-dimensional information processing apparatus 100 may include an image processing unit that generates an image with the same angle of view with reduced resolution by processing a telephoto image. That is, the control unit 140 may acquire the telephoto image from the telephoto imaging unit 130 and transmit it to the image processing unit. Then, the control unit 140 may acquire the low-resolution image created by the image processing unit. Then, the above set may be generated by a combination of the wide-angle image and the crop image (alternate of the telephoto image 340), or a combination of the telephoto image and its low-resolution image (alternate of the wide-angle image 310). Note that each of the various controls described above as being executed by the control unit 140 may be performed by one piece of hardware, or the entire control of the apparatus may be performed by a plurality of pieces of hardware (for example, a plurality of processors or circuits) sharing the processing.

[0044] In addition, a stereo camera may be adopted for the three-dimensional distance measurement unit 110. Also, the imaging unit may be provided separately from the three-dimensional information processing apparatus 100. Further, the measurement target may not be photographed, and the wide-angle image 310 or the telephoto image 340 may be an image artificially synthesized by CG or the like, or may be acquired from another device via a network when the three-dimensional information processing apparatus is connected to the network. Also, the resolution of the telephoto image 340 may not be low.

[0045] Moreover, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope not departing from the gist of this invention are also included in the present invention. Furthermore, each of the above-described embodiments merely shows one embodiment of the present invention, and it is also possible to appropriately combine the embodiments.

[0046] The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media. Then, a computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0047] <Other Embodiments>The present invention can also be realized by supplying a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a recording medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0048] The invention is not limited to the above-described embodiments, and various changes and modifications are possible without departing from the spirit and scope of the invention. Therefore, the claims are appended to disclose the scope of the invention.

[0049] This application claims priority based on Japanese Patent Application No. 2024-190069 filed on October 29, 2024, and incorporates all of the description thereof herein.

Claims

1. An information processing device comprising: a shape acquisition unit capable of acquiring three-dimensional shape information of a target to be processed; an image acquisition unit capable of acquiring image data including the target to be processed; and a density enhancement unit capable of gradually increasing the density of the three-dimensional shape information based on the image data, wherein the image acquisition unit acquires first image data including at least a part of the target to be processed, and second image data including a part of the region included in the first image data, or an image included in the first image data with increased resolution; and the density enhancement unit generates first high-density shape information by increasing the density of the three-dimensional shape information corresponding to the target to be processed included in the first image data, and further generates second high-density shape information by increasing the density of the information corresponding to the target to be processed included in the second image data from the first high-density shape information.

2. The information processing apparatus according to claim 1, wherein the image acquisition unit comprises two optical lenses with different angles of view, the first image data is acquired using the optical lens with a wide angle of view, and the second image data is acquired using the optical lens with a narrow angle of view.

3. The information processing apparatus according to claim 1, wherein the image acquisition unit comprises a zoom mechanism, and captures the processing target at a first zoom magnification of the zoom mechanism, acquires the captured data as first image data, and captures the processing target at a second zoom magnification higher than the first zoom magnification of the first image data, and acquires the captured data as second image data.

4. An information processing apparatus according to any one of claims 1 to 3, wherein the first high-density shape information is converted from a coordinate system based on the shape acquisition unit to a coordinate system based on the image acquisition unit, using the ratio of the size of the processing target contained in the second image data to the size of the processing target contained in the first image data.

5. The information processing apparatus according to any one of claims 1 to 4, further comprising an image extraction unit that extracts a portion from the first image data to obtain the second image data, wherein the image acquisition unit acquires the first image data and provides it to the image extraction unit, and acquires the second image data extracted by the image extraction unit.

6. An information processing apparatus according to any one of claims 1 to 5, further comprising an image processing unit capable of changing the resolution of an image, wherein the image acquisition unit acquires the second image data and provides it to the image processing unit, the image processing unit reduces the resolution of the second image data to generate the first image data, and the image acquisition unit acquires the first image data generated by the image processing unit.

7. The information processing apparatus according to any one of claims 1 to 6, further comprising a determination unit that determines whether or not it is possible to generate second high-density shape information by increasing the density of the three-dimensional shape information corresponding to the processing target contained in the second image data, wherein if the determination unit determines that it is possible to generate it, the image acquisition unit does not acquire the first image data, and the density increasing unit generates the second high-density shape information by increasing the density of the three-dimensional shape information corresponding to the processing target contained in the second image data without using the first image data.

8. The information processing apparatus according to claim 7, wherein the three-dimensional shape information is three-dimensional point cloud information, and the determination unit determines that the second high-density shape information can be generated if the ratio of the number of pixels in the second image data to the number of points in the three-dimensional point cloud corresponding to the processing target included in the second image data is less than or equal to a predetermined threshold.

9. A control method for an information processing device, comprising: a shape acquisition step in which a shape acquisition unit can acquire three-dimensional shape information of a processing target; an image acquisition step in which an image acquisition unit can acquire image data including the processing target; and a density enhancement step in which a density enhancement unit can gradually increase the density of the three-dimensional shape information based on the image data, wherein the image acquisition unit acquires first image data including at least a part of the processing target and second image data including a part of the region included in the first image data, or an image included in the first image data with increased resolution; and the density enhancement unit generates first high-density shape information by increasing the density of the three-dimensional shape information corresponding to the processing target included in the first image data, and further generates second high-density shape information by increasing the density of the information corresponding to the processing target included in the second image data from the first high-density shape information.

10. A program for causing a computer to execute each step in a control method for an information processing device, wherein the control method includes: a shape acquisition step in which a shape acquisition unit can acquire three-dimensional shape information of a processing target; an image acquisition step in which an image acquisition unit can acquire image data including the processing target; and a densification step in which a densification unit can gradually increase the density of the three-dimensional shape information based on the image data, wherein the image acquisition unit acquires first image data including at least a part of the processing target and second image data including a part of the region included in the first image data, or an image included in the first image data with increased resolution; and the densification unit generates first high-density shape information by increasing the density of the three-dimensional shape information corresponding to the processing target included in the first image data, and further generates second high-density shape information by increasing the density of the information corresponding to the processing target included in the second image data from the first high-density shape information.

Images