Information processing method and device, and program

Abstract

The present disclosure relates to an information processing method and device, and a program, that make it easier to suppress a decrease in the accuracy of 3D models. The method involves presenting a recommended orientation of an object corresponding to a deficient region on the surface of a 3D model when imaging a first part of the object, which is set on the basis of the deficient region. The present disclosure is applicable, for example, to information processing methods, image processing methods, 3D model generating methods, imaging guidance methods for 3D model generation, recommended orientation presentation methods, information processing devices, image processing devices, 3D model generating devices, imaging guidance devices for 3D model generation, recommended orientation presenting devices, information processing systems, image processing systems, 3D model generating systems, imaging guidance systems for 3D model generation, recommended orientation presenting systems, and programs, etc.

Description

Information processing method, apparatus, and program

[0001] The present disclosure relates to an information processing method, an information processing apparatus, and a program, and particularly relates to an information processing method, an information processing apparatus, and a program that can more easily suppress a reduction in the accuracy of a 3D model.

[0002] In recent years, photogrammetry technology has begun to be used for generating 3D models for video production and simulation. Photogrammetry technology is a technology that images a target object from a plurality of viewpoints and reconstructs the three-dimensional shape (also referred to as a 3D model) of the object using the obtained plurality of captured images. That is, three-dimensional information is extracted from two-dimensional information by generating depth information of the subject from the common field of view between images.

[0003] In photogrammetry technology, since depth information of the subject is generated from the common field of view between captured images as described above, it has been required that the object remains stationary during each imaging operation. In other words, the object could not be moved during the imaging operation.

[0004] Therefore, when a blind spot occurs in the object, a part of the 3D model may be missing, which may reduce the accuracy of the 3D model. For example, when imaging an object placed on a desk, the portion of the object that is in contact with the desk surface cannot be imaged, so a defect occurs on the surface of that portion in the generated 3D model, which may reduce the accuracy of the 3D model.

[0005] By the way, a method of complementing a defective region of a 3D model by estimation has been proposed (see, for example, Patent Document 1). Such a method can suppress the occurrence of defective regions in a 3D model.

[0006] International Publication No. 2024 / 202718

[0007] However, with this method, it is difficult to accurately estimate complex shapes, textures specific to defective regions, and the like, so the accuracy of complementation decreases, which may reduce the accuracy of the 3D model.

[0008] This disclosure is made in light of these circumstances and aims to make it easier to suppress the reduction in the accuracy of 3D models.

[0009] One aspect of this technology is an information processing method that presents a recommended orientation for an object when imaging a first portion of the object corresponding to a missing area on the surface of a 3D model, which is set based on the missing area on the surface of the object.

[0010] One aspect of this technology is an information processing device that includes a display unit that presents a recommended orientation of an object in imaging a first portion of an object corresponding to a missing region of a 3D model, and the recommended orientation is set based on the missing region.

[0011] One aspect of this technology is a program that causes a computer to perform a process of presenting a recommended orientation for an object when imaging a first portion of the object corresponding to a missing area on the surface of a 3D model, which is set based on the missing area of ​​the object's surface.

[0012] In one aspect of this technology, the information processing method, apparatus, and program present a recommended orientation for an object when imaging a first portion of the object corresponding to a missing area on the surface of a 3D model, which is set based on the missing area of ​​that missing area.

[0013] This figure shows an example of how an object is imaged using photogrammetry technology. This figure shows an example of how a 3D model is generated using photogrammetry technology. This figure shows an example of the procedure for generating a 3D model using this technology. This figure shows an example of the procedure for generating a 3D model using this technology. This block diagram shows an example of the main configuration of an object 3D model generation device. This figure shows an example of a displayed image. This figure shows an example of how a recommended posture is set. This figure shows an example of information indicating a missing region. This figure shows an example of information indicating a missing region. This figure shows an example of information indicating a missing region. This block diagram shows an example of the main configuration of the recommended posture derivation unit. This block diagram shows an example of the main configuration of the missing region normal derivation unit. This block diagram shows an example of the main configuration of the 3D model generation unit. This block diagram shows an example of the main configuration of the clipping unit. This is a flowchart to explain an example of the flow of the object 3D model generation process. This is a flowchart to explain an example of the flow of the 3D model generation process. This is a flowchart to explain an example of the flow of the clipping process. This is a flowchart to explain an example of the flow of the missing region normal derivation process. This is a flowchart to explain an example of the flow of the recommended posture derivation process. This block diagram shows an example of the main configuration of an imaging device. This is a system diagram showing an example of the main configuration of an object 3D model generation system. This is a system diagram showing an example of the main configuration of an object 3D model generation system. This is a block diagram showing common computer configurations.

[0014] The following describes the forms for implementing this disclosure (hereinafter referred to as embodiments). The explanation will be given in the following order: 1. Supporting literature, etc., for technical content and technical terminology 2. Photogrammetry technology 3. Presentation of recommended postures 4. Appendix

[0015] <1. Supporting Documents for Technical Content and Terminology> The scope disclosed in this technology includes not only the contents described in the embodiments, but also the contents described in the following patent documents, which were publicly known at the time of filing, and the contents of other documents referenced in the following patent documents.

[0016] Patent Document 1: (mentioned above)

[0017] In other words, the contents described in the aforementioned patent documents, as well as the contents of other documents referenced in those patent documents, will also serve as a basis for determining the support requirements.

[0018] <2. Photogrammetry Technology> In recent years, photogrammetry technology has begun to be used in the creation of 3D models for video production and simulations. Photogrammetry technology is a technique that takes images of a subject object from multiple viewpoints and uses the obtained images to reconstruct the three-dimensional shape of that object (also called a 3D model). In other words, three-dimensional information is extracted from two-dimensional information by creating depth information of the subject from the common field of view between the captured images.

[0019] In other words, in photogrammetry imaging, as shown in Figure 1, the subject object is imaged from multiple viewpoints, and multiple images from different viewpoints are generated. For example, in Figure 1, the subject object 11 (i.e., the object to be modeled) is placed on a plane 12. This object 11 is imaged from the viewpoints of cameras 13-1, 13-2, and 13-3, respectively. Cameras 13-1, 13-2, and 13-3 indicate the position and orientation (i.e., viewpoint) from which camera 13 performs imaging. In this way, images from each viewpoint (i.e., multiple images that are different from each other) are obtained.

[0020] In photogrammetry, depth information of the subject is created from the common field of view between captured images, so it was required that the object remain stationary between each image capture. In other words, the object could not be moved during the imaging process. Therefore, if a blind spot occurred in the object, part of the 3D model may be missing, potentially reducing the accuracy of the 3D model.

[0021] For example, suppose a 3D model like the one shown in Figure 2A is obtained by reconstructing a 3D model using multiple captured images obtained by imaging as shown in Figure 1. This 3D model consists of a 3D model 21 of object 11 and a 3D model 22 of plane 12. By extracting the portion corresponding to object 11 from this 3D model, a 3D model 21 of object 11 is obtained, as shown in Figure 2B.

[0022] However, as described above, object 11 is placed on the plane 12 (Figure 1), and since object 11 cannot be moved, the contact surface between object 11 and plane 12 could not be imaged (no image was obtained). Therefore, it was not possible to reconstruct the 3D model of this contact surface, and as shown in Figure 2C, a missing region 21A was created on the bottom surface of the 3D model 21 (shaded area in the figure). Thus, the inability to move object 11 during the imaging process may have reduced the accuracy of the 3D model 21.

[0023] Incidentally, as described in Patent Document 1, for example, a method has been considered to interpolate missing regions of a 3D model by estimation. Such a method can suppress the occurrence of missing regions in a 3D model. However, with this method, it is difficult to accurately estimate complex shapes and textures specific to the missing regions, which reduces the accuracy of interpolation and may reduce the precision of the 3D model.

[0024] <3. Presentation of Recommended Posture> Therefore, a technique for aligning and synthesizing multiple 3D models (point cloud data) is applied to separately reconstruct and synthesize the 3D model of the missing region. For example, first, as shown in Figure 3A, the camera 13 is moved to image the object 11 from multiple viewpoints, and the 3D model is reconstructed using the multiple images obtained from different viewpoints. Then, as shown in Figure 3B, the 3D model 31 corresponding to the object 11 is extracted and separated from the 3D model 32 corresponding to the plane 12 (also called clipping). As described above, the missing region 31A exists in this 3D model 31 (Figure 4A), so as shown in Figure 3C, the object 11 is imaged again. At this time, the object 11 is positioned so that the missing region of the 3D model 31 can be imaged. Then, the 3D model is reconstructed using the multiple images obtained from different viewpoints. Furthermore, as shown in Figure 3D, the 3D model 41 corresponding to object 11 is extracted and separated (clipped) from the 3D model 42 corresponding to plane 12. This 3D model 41 has a missing region 41A, as shown in Figure 4A.

[0025] Next, as shown in Figure 4A, the 3D models 31 and 41 generated as described above are combined to produce the 3D model 51 shown in Figures 4B and 4C. Through this combination, the missing regions (missing region 31A and missing region 41A) of each model are filled in by the other 3D model, so no missing regions occur in the 3D model 51 (Figures 4B and 4C). In other words, since the missing regions can be filled in with a high-precision 3D model, the reduction in the accuracy of the 3D model can be suppressed.

[0026] To reconstruct a 3D model of a missing region, it is necessary to properly image the portion of the object corresponding to that region and generate an image of that portion. However, in cases where the object's shape is complex, for example, it can be difficult for the user to accurately identify the occurrence of a missing region. As a result, it may not be possible to properly image the portion corresponding to the missing region, making it difficult to suppress the reduction in the accuracy of the 3D model. Furthermore, there is a risk of reducing work efficiency by unnecessarily increasing the number of imaging attempts in order to more reliably image the portion corresponding to the missing region.

[0027] Therefore, before imaging the portion of object 11 corresponding to the missing region 31A of the 3D model 31, as shown in Figure 3C, the user is presented with a recommended orientation of object 11 for imaging. In other words, the user is presented with a recommended orientation of object 11 that makes it easier to image the portion corresponding to the missing region 31A.

[0028] In this specification, "object" refers to a real-world object (the target of 3D model generation). "3D model" refers to a 3D data representation of that object. "Missing area" refers to a part of the 3D model where information is missing. The information targeted by this missing area (what information is missing and which part is considered a missing area) can be any kind of information. In particular, information related to the surface of an object, such as shape, texture, and gloss, has a greater impact on the accuracy (quality) of the 3D model, so it is desirable to include "information related to the surface of the object" as the information targeted by the missing area. For example, if the shape, texture, gloss, etc. that exist in the object are missing in the 3D model, that part may be considered a missing area.

[0029] For example, in an information processing device or information processing system, a recommended orientation for an object is presented when imaging a first part of the object corresponding to a missing area on the surface of a 3D model, which is set based on the missing area of ​​that missing area.

[0030] For example, the information processing device may include a display unit that presents a recommended orientation for an object when imaging a first portion of the object corresponding to a missing region in a 3D model. The recommended orientation is set based on the missing region.

[0031] For example, suppose the program is a program that causes a computer (e.g., an information processing device or information processing system) to perform a process that presents a recommended orientation for an object when imaging a first part of the object corresponding to a missing area on the surface of a 3D model, which is set based on the missing area of ​​that missing area.

[0032] For example, the information processing system may include a display unit that presents a recommended orientation for an object when imaging a first portion of the object corresponding to a missing region in a 3D model. The recommended orientation shall be set based on the missing region.

[0033] In this way, by presenting a recommended object orientation for imaging the area corresponding to the missing region, the user can set the object to that recommended orientation during imaging. This makes it easier for the user to image the area of ​​the object corresponding to the missing region. Consequently, it becomes easier to generate a 3D model of the missing region and interpolate it into the missing region of the object's 3D model, thus making it easier to suppress the reduction in the accuracy of the 3D model.

[0034] Furthermore, in the information processing method, information processing device, program, or information processing system described above, the recommended orientation of the presented object may be an orientation that allows imaging of the first portion corresponding to the missing area. By doing so, the user can more easily image the first portion. Therefore, the reduction in the accuracy of the 3D model can be more easily suppressed.

[0035] <Object 3D Model Generation Device> Figure 5 is a block diagram showing an example of the configuration of an object 3D model generation device, which is one embodiment of an information processing device or information processing system to which this technology is applied. The object 3D model generation device 100 shown in Figure 5 is a device (or system) that performs processing related to the generation of 3D models of objects. Here, an example of realizing the configuration to which this technology is applied as a single device is described, but this configuration may be realized in any form, for example, as multiple devices (systems).

[0036] <Presentation Unit> As shown in Figure 5, the object 3D model generation device 100 has a presentation unit 111. The presentation unit 111 has a presentation device (e.g., a monitor or speaker) and performs processing related to presenting information to the user. For example, the presentation unit 111 uses its presentation device to present to the user a recommended orientation of the object for imaging a first part of the object corresponding to a missing area of ​​the 3D model. For example, the presentation unit 111 may present an orientation that allows imaging of the first part as the recommended orientation of the object. In this way, the user can set the object in the recommended orientation in the real world and perform imaging, making it easier to image the first part. Therefore, the object 3D model generation device 100 can more easily complete the missing areas of the object's 3D model using a 3D model reconstructed using multiple images obtained by imaging. Therefore, the object 3D model generation device 100 can more easily suppress the reduction in the accuracy of the 3D model.

[0037] <Presentation Image> The method by which the presentation unit 111 presents the recommended posture may be any method. In other words, the presentation device used may be any device to which any presentation method is applied. For example, the presentation unit 111 may present the recommended posture using images or sound. For example, the presentation unit 111 may display a presentation image 130 that represents the recommended posture of an object, as shown in Figure 6. In the example in Figure 6, the presentation image 130 shows the recommended posture of an object with its 3D model 131 placed on a support surface 132. This support surface 132 is a horizontal plane that supports the 3D model 131, and the recommended posture of the object is shown as the posture of the 3D model 131 on this support surface 132. Users can set an object in the real world to its presented recommended posture by placing a real-world object corresponding to the 3D model 131 in the presentation image 130 on a real-world horizontal plane (e.g., the top surface of a table) corresponding to the support surface 132 in the presentation image 130, in the same posture as the 3D model 131 in the presentation image 130. In other words, by referring to this presented image 130, users can more easily set the object to the presented recommended posture.

[0038] In the example shown in Figure 6, the recommended orientation is set so that the missing region 131A of the 3D model 131 is exposed. Therefore, the user can more easily image the portion of the object corresponding to this missing region 131A.

[0039] The recommended orientation is any orientation. For example, as shown in Figure 6, the defective area 131A may be the upper surface, or as shown in Figure 7A and Figure 7B, the defective area 131A may not be the upper surface. Also, as shown in Figure 7A, the support surface 132 may support (contact) the 3D model 131 at one point, or as shown in Figure 7B, the support surface 132 may support (contact) the 3D model 131 at multiple points. In the example of Figure 7A, the 3D model 131 is supported by the support surface 132 at the support portion 141. In other words, the 3D model 131 is in contact with the support surface 132 at the support portion 141. In the example of Figure 7B, the 3D model 131 is supported by the support surface 132 at the support portion 142 and support portion 143. In other words, the 3D model 131 is in contact with the support surface 132 at the support portion 142 and support portion 143.

[0040] <Information indicating missing areas> Note that the content of the presented image 130 (e.g., the method of representing the recommended posture and the image composition) can be anything as long as it presents a recommended posture for the object, and is not limited to the example in Figure 6. For example, the presented image 130 may also present "information indicating missing areas". For example, the presentation unit 111 may further present information indicating missing areas. By doing so, users can more easily (more accurately) grasp the location of the missing areas based on this information. Therefore, users can more easily (more reliably) image the missing areas.

[0041] <Normal Vector of Missing Area> For example, the "information indicating the missing area" may include a normal vector of the missing area, which is a vector indicating the normal of the missing area. In other words, the display unit 111 may display the normal vector of the missing area as information indicating the missing area. In the example of Figure 6, the normal vector of the missing area 133 (arrow), which indicates the normal of the missing area 131A, is displayed. This display allows users to more easily (more accurately) grasp the location of the missing area. This display also allows users to easily grasp the orientation of the missing area. Therefore, users can more easily (more reliably) image the missing area.

[0042] As described above, this missing region normal vector 133 represents the normal to the missing region 131A. Therefore, in the cases of Figure 7A and Figure 7B, since the missing region 131A is not the top surface, the missing region normal vector 133 points in a direction other than vertical.

[0043] <Highlighting of Missing Areas> For example, the information indicating the missing area may include highlighting of the missing area. In other words, the display unit 111 may highlight the missing area as "information indicating a missing area". In the example of A in Figure 8, the missing area 131A is filled in with black, highlighting that it is the missing area 131A. Such highlighting allows users to more easily (more accurately) grasp the location of the missing area. Therefore, users can more easily (more reliably) image the missing area.

[0044] It should be noted that "highlighting" herein means a representation method distinguishable from other portions. As long as a visual effect not present in other portions is applied to the defective region, enabling the defective region to be distinguished from other portions, any specific representation method may be adopted. For example, the defective region may be filled as shown in A of FIG. 8. The color in this case is not limited to black in the example of A of FIG. 8, and may be any color. Furthermore, a predetermined texture (pattern, design, etc.) may be applied to the defective region. Gradation or animation (such as blinking) may be applied to the defective region. The color of the defective region may be lightened or darkened (conversely, the density of other portions may be changed). The defective region may be made translucent. The outer frame of the defective region may be represented by a thick line or the like. In addition, an image surrounding the defective region (for example, a surrounding line, pattern, design, etc.) may be added around the defective region. Furthermore, the defective region may be indicated by characters or the like. For example, the characters "defective region" may be added in or around the defective region. Of course, representation methods other than these may also be applied. It is of course permissible to combine a plurality of methods.

[0045] <Vector Indicating Observation Direction of Defective Region> For example, the "information indicating a defective region" may include a vector indicating an observation direction of the defective region. That is, the presentation unit 111 may present a vector indicating the observation direction of the defective region as information indicating the defective region. In the example of B of FIG. 8, vector 151-1, vector 151-2, and vector 151-3 are presented for defective region 131A. Hereinafter, when there is no need to distinguish between vector 151-1, vector 151-2, and vector 151-3 for description, they are also collectively referred to as vector 151. This vector 151 indicates a representative direction in which the defective region 131A can be observed as the observation direction. With this presentation, a user or the like can grasp the location of the defective region more easily (more accurately). In addition, with this presentation, the user or the like can grasp the observation direction of the defective region 131A (that is, the direction in which the defective region 131A can be imaged) more easily (more accurately). Therefore, the user or the like can image the defective region more easily (more reliably). It should be noted that any number of observation directions (that is, the number of vectors 151) presented in this manner is allowable.

[0046] <Sector-shaped cone indicating a field of view suitable for observing the missing region> For example, the "information indicating the missing region" may include a sector-shaped cone indicating a field of view suitable for observing the missing region. In other words, the display unit 111 may present a sector-shaped cone indicating a field of view suitable for observing the missing region as information indicating the missing region. In the example of C in Figure 8, sector-shaped cones 152-1 and 152-2 are presented for the missing region 131A. In the following, when it is not necessary to distinguish between sector-shaped cones 152-1 and 152-2, they will also be referred to as sector-shaped cone 152. This sector-shaped cone 152 indicates a field of view suitable for observing the missing region 131A. In other words, by making the sector-shaped cone 152 the field of view from the vertex position of this sector-shaped cone 152, the missing region 131A can be included within that field of view. That is, the missing region 131A can be observed. In other words, by imaging from the vertex of the sector-cone 152 with the sector-cone 152 within the field of view, the defective region 131A can be imaged. This means that, with this presentation, users can not only more easily (more accurately) grasp the location of the defective region, but also more easily (more accurately) grasp the position and orientation suitable for observing (imaging) that defective region. Therefore, users can image the defective region more easily (more reliably).

[0047] It should be additionally noted that the phrase "suitable for observing a defective region" means being suitable for "generating a 3D model of the defective region". Therefore, by capturing an image at the position and orientation according to the sector conical shape, a user or the like can obtain a captured image in which the defective region is captured with a size and an angle that are more suitable for generating the 3D model of the defective region. Using the obtained captured image, the 3D model of the defective region can be generated with higher accuracy (more accurately). By performing complementation using such a highly accurate (more accurate) 3D model of the defective region, a reduction in the accuracy of the 3D model of the object can be further suppressed. In other words, through such presentation, a reduction in the accuracy of the 3D model of the object can be further suppressed. Note that the number of sector conical shapes presented as described above (that is, the number of sector conical shapes 152) may be any number.

[0048] <Recommended Imaging Position and Orientation for Defective Region> For example, the "information indicating a defective region" may include a display indicating a recommended imaging position and orientation for the defective region. In other words, a presentation unit 111 may present a display indicating the recommended imaging position and orientation for the defective region as the information indicating the defective region. In the example of part A in FIG. 9, a recommended imaging position and orientation 161-1, a recommended imaging position and orientation 161-2, and a recommended imaging position and orientation 161-3 are presented with respect to a defective region 131A. In the following description, when there is no need to distinguish between the recommended imaging position and orientation 161-1, the recommended imaging position and orientation 161-2, and the recommended imaging position and orientation 161-3, they are also collectively referred to as a recommended imaging position and orientation 161. This recommended imaging position and orientation 161 is a display that indicates a recommended position and orientation as an imaging position and orientation for capturing an image of the defective region 131A. Through this presentation, a user or the like can not only more easily (more accurately) grasp the location of the defective region, but also more easily (more accurately) grasp the recommended imaging position and orientation for the defective region. Therefore, the user or the like can more easily (more reliably) capture an image of the defective region.

[0049] Furthermore, "recommended" means that imaging can be performed in a way that captures the missing region at a size and angle more suitable for generating a 3D model of the missing region (i.e., an image of the missing region captured at a size and angle more suitable for generating a 3D model of the missing region is obtained). In other words, by imaging the missing region at the recommended imaging position and orientation, users can obtain an image in which the missing region is captured at a size and angle more suitable for generating a 3D model of the missing region. Using this obtained image, a 3D model of the missing region can be generated with higher accuracy (more precision). Then, by using this higher accuracy (more precision) 3D model of the missing region to complete the model, the reduction in the precision of the object's 3D model can be further suppressed. In other words, this approach can further suppress the reduction in the precision of the object's 3D model.

[0050] In example A of Figure 9, the recommended imaging position and orientation are indicated by the position and orientation of the camera icon, but the method of representing the recommended imaging position and orientation is not limited to this example and can be any method. For example, the recommended imaging position and orientation may be indicated by a point cloud or marker. Also, as in example B of Figure 9, the recommended imaging position and orientation may be presented as a cluster (set) of candidate points on a sphere. In example B of Figure 9, for the missing region 131A, recommended imaging position and orientation 162-1, recommended imaging position and orientation 162-2, and recommended imaging position and orientation 162-3 are presented as clusters of points on a sphere, indicated by a thick circular line surrounding the 3D model 131 of the object.

[0051] <Multiple Missing Regions> Note that there may be any number of missing regions; there may be one or more. If there are multiple missing regions, information indicating the missing regions may be presented for each missing region, or it may be presented for only some of the missing regions.

[0052] <Information indicating the size of the missing area> For example, the "information indicating the missing area" may include information indicating the size of the missing area. In other words, the display unit 111 may present information indicating the size of the missing area as information indicating the missing area. The size of the missing area may be expressed in any way. For example, the size of the missing area may be expressed by the size, color, pattern, density, transparency, etc. of the information indicating the missing area.

[0053] In the example shown in Figure 10A, missing regions 131A and 131B occur in the 3D model 131. A missing region normal vector 171, which indicates the normal of missing region 131A, and a missing region normal vector 172, which indicates the normal of missing region 131B, are presented. The missing region normal vector 171 is presented as a large arrow, and the missing region normal vector 172 is presented as a small arrow. The difference in the size of these arrows indicates the difference in the size of the corresponding missing regions. In other words, it is shown that the missing region 131A, which corresponds to the missing region normal vector 171 (large arrow), is larger than the missing region 131B, which corresponds to the missing region normal vector 172 (small arrow).

[0054] In the example of Figure 10B, a normal vector 181 representing the normal of the missing region 131A and a normal vector 182 representing the normal of the missing region 131B are presented. The normal vector 181 is presented as a white arrow, and the normal vector 182 is presented as a grid-patterned arrow. The difference in the patterns of these arrows indicates the difference in the size of the corresponding missing regions. In other words, it is shown that the missing region 131A, which corresponds to the normal vector 181 (white arrow), is larger than the missing region 131B, which corresponds to the normal vector 182 (grid-patterned arrow).

[0055] This type of presentation allows users to more easily (and accurately) understand the size of the missing area.

[0056] Generally, larger missing areas have a greater impact on the accuracy (quality) of the object's 3D model. In other words, if multiple missing areas exist, it is desirable to prioritize imaging the larger missing area to minimize the reduction in the object's 3D model's accuracy (quality). Therefore, by prioritizing imaging the larger missing area based on the size of the missing areas as described above, users can further minimize the reduction in the object's 3D model's accuracy (quality). Even if imaging the larger missing area is not prioritized, users can still determine the priority order for imaging each missing area based on the size of the missing areas as described above (i.e., considering the size of each missing area) to further minimize the reduction in the object's 3D model's accuracy (quality). Thus, the reduction in the object's 3D model's accuracy can be more easily minimized.

[0057] <Information indicating the amount of missing images needed to generate a 3D model of the missing region> For example, the "information indicating the missing region" may include information indicating the amount of missing images needed to generate a 3D model (3D data) of the missing region. In other words, the presentation unit 111 may present information indicating the amount of missing images needed to generate a 3D model (3D data) of the missing region as information indicating the missing region. To generate a 3D model of the missing region, a predetermined number (multiple) of images of that missing region are required. With such presentation, users can more easily understand how many more images are needed to generate a 3D model of the missing region (i.e., how many times the missing region needs to be photographed). For example, if there are multiple missing regions, users can determine the priority order of photography based on that information. For example, they may prioritize photographing the missing region with a smaller amount of missing images, or they may prioritize photographing the missing region with a larger amount of missing images.

[0058] This deficiency can be expressed in any way. For example, the "information indicating the deficiency" may include an icon or symbol indicating that deficiency. In other words, the display unit 111 may display an icon or symbol indicating the deficiency of the captured image necessary for generating a 3D model (3D data) of the missing region. For example, an icon or symbol indicating the deficiency of the captured image (e.g., "✓ (sufficient)" or "!" (insufficient)") may be added to the information indicating the missing region (e.g., the normal vector of the missing region). In addition, progress numbers such as "0 / 2" or "1 / 2" may also be displayed.

[0059] Furthermore, the "information indicating the amount of deficiency" may include a color corresponding to the amount of deficiency in the highlighted missing area. In other words, the display unit 111 may use a color corresponding to the amount of missing captured images necessary for generating the 3D model (3D data) when highlighting the missing area. For example, the missing area may be colored according to the amount of deficiency, such as red (0 images), yellow (1 image missing), and green (sufficient). Alternatively, the overall degree of deficiency may be visualized, such as in a heat map.

[0060] Furthermore, the "information indicating the amount needed" may include a semi-transparent icon. In other words, the display unit 111 may display a semi-transparent icon as "information indicating the amount needed." For example, a semi-transparent camera (ghost camera) may be displayed at the location where a photo has already been taken, and a message or mark such as "one more photo needed" may be added. Such a display allows the user to intuitively understand that "an additional photo should be taken here."

[0061] Furthermore, the "information indicating the amount of deficiency" may include an animation corresponding to the amount of deficiency in the highlighting of the missing area. In other words, the display unit 111 may present an animation corresponding to the amount of deficiency as a highlighting of the missing area. This animation can be of any kind, for example, it may be highlighted with blinking, ripples, a halo, etc. For example, the amount of deficiency may be indicated by gradual emphasis, such as making the blinking faster the more severe the imaging deficiency (i.e., the more images that need to be taken). Of course, other animations may be used, or multiple animation methods may be combined and applied.

[0062] Furthermore, the "information indicating the amount of shortage" may include a numerical value indicating the amount of shortage. For example, the display unit 111 may display a numerical value indicating the amount of shortage, such as 0% (0 sheets), 50% (1 sheet), 100% (2 sheets), etc., in or near the missing area. Of course, the location where this numerical value is displayed can be anywhere, as long as the correspondence between the numerical value and the missing area is clear.

[0063] Furthermore, the "information indicating the amount of deficiency" may include a gauge indicating the amount of deficiency. In other words, the amount of deficiency may be represented by a gauge. For example, the display unit 111 may display a circular gauge or a progress bar (bar-shaped gauge) indicating the amount of deficiency in or near the missing area. Of course, the location where this gauge is displayed can be anywhere, as long as the correspondence between the gauge and the missing area is clear. The display unit 111 may also display a gauge indicating the overall coverage rate.

[0064] Furthermore, the "information indicating the amount of deficiency" may include notations for guidance between recommended imaging positions. In other words, the display unit 111 may display notations for guidance between recommended imaging positions as "information indicating the amount of deficiency." For example, the display unit 111 may highlight the imaging position and orientation where the amount is insufficient with a highlighting marker and guide the user with arrows or lines. In addition, the display unit 111 may display a message such as "Next, take additional images here" and sequentially instruct the user to take images at each position.

[0065] Of course, the "information indicating the shortage" can be anything and is not limited to these examples. Furthermore, multiple pieces of the various types of information mentioned above may be combined and applied as "information indicating the shortage." Also, the various types of information mentioned above may be combined and applied as "information indicating the shortage" with other information not mentioned above.

[0066] <Information indicating imaging conditions for the missing region> For example, the "information indicating the missing region" may include information indicating the imaging conditions for the missing region. In other words, the display unit 111 may display information indicating the imaging conditions when imaging the missing region as information indicating the missing region. Such a display allows the user to understand more easily (more accurately) how to image the missing region.

[0067] For example, this "information indicating imaging conditions" may include conditions regarding the position in which the defective area is imaged. In other words, the display unit 111 may present conditions regarding the position in which the defective area is imaged as "information indicating imaging conditions". For example, the display unit 111 may indicate the conditions regarding the position in which the defective area is imaged using coordinates. For example, the display unit 111 may indicate the conditions regarding the position in which the defective area is imaged using a range of coordinates. These coordinates may be, for example, world coordinates, or relative coordinates from the center of the defective area to be imaged. Alternatively, the display unit 111 may indicate the conditions regarding the position in which the defective area is imaged using the distance from the center of the defective area to be imaged. For example, the display unit 111 may indicate the conditions regarding the position in which the defective area is imaged using a range of distances from the center of the defective area to be imaged (for example, 30 to 50 cm).

[0068] Furthermore, this "information indicating imaging conditions" may include conditions related to the orientation for imaging the defective area. In other words, the display unit 111 may present conditions related to the orientation for imaging the defective area as "information indicating imaging conditions." For example, the display unit 111 may indicate the conditions related to the orientation for imaging the defective area using angles or parallax. For example, the display unit 111 may present an angle that ensures sufficient parallax with the existing image. For example, the display unit 111 may present a message (advice) such as "shift by 15 degrees or more from the previous shooting position."

[0069] Furthermore, this "information indicating imaging conditions" may also include conditions related to the number of imaging cycles. In other words, the display unit 111 may present conditions related to the number of imaging cycles as "information indicating imaging conditions." For example, the display unit 111 may present a message indicating the shortage numerically, such as "at least two images are required from the same position." Alternatively, the display unit 111 may present a message expressing relative conditions, such as "one additional image from a different direction."

[0070] Furthermore, this "information indicating imaging conditions" may include conditions related to the amount of light used in imaging the defective area. In other words, the display unit 111 may present conditions related to the amount of light used in imaging the defective area as "information indicating imaging conditions." For example, the display unit 111 may present an imaging direction that minimizes shadows (a recommended imaging direction relative to the light source). The display unit 111 may also specify HDR (High Dynamic Range) imaging or the difference in exposure from the previous imaging.

[0071] Furthermore, this "information indicating imaging conditions" may include field-of-view overlap conditions (overlap threshold). In other words, the display unit 111 may present conditions regarding the degree of overlap between captured images of the missing region as "information indicating imaging conditions." For example, the display unit 111 may specify the degree of overlap with the field of view of the previous image. For example, if the degree of overlap between the current field of view set by the user and an existing captured image is greater than or equal to a threshold, the display unit 111 may present a message indicating that imaging is unnecessary.

[0072] Furthermore, this "information indicating imaging conditions" may include conditions regarding the priority of imaging the defective area. In other words, the display unit 111 may present conditions regarding the priority of imaging the defective area as "information indicating imaging conditions." For example, the display unit 111 may present a priority based on the size of the defective area (for example, a priority order that prioritizes defective areas with a larger area) using characters or images. Alternatively, the importance of imaging directions may be ranked (classified) as "high," "medium," or "low," and the display unit 111 may present that rank (class).

[0073] Of course, the "information indicating imaging conditions" can be anything and is not limited to these examples. Furthermore, multiple pieces of the various types of information mentioned above may be combined and applied as "information indicating imaging conditions." Also, the various types of information mentioned above may be combined and applied as "information indicating imaging conditions" with other information not mentioned above.

[0074] Of course, the "information indicating a missing region" can be anything and is not limited to the examples given above. Furthermore, multiple pieces of the various types of information mentioned above may be combined and applied as "information indicating a missing region." Also, the various types of information mentioned above may be combined and applied as "information indicating a missing region" with other information not mentioned above.

[0075] <Recommended Pose Derivation Unit> As shown in Figure 5, the object 3D model generation device 100 (information processing device or information processing system) may further have a recommended pose derivation unit 112. The recommended pose derivation unit 112 performs processing related to the derivation of the recommended pose of the object. For example, the recommended pose derivation unit 112 may acquire a 3D model of the object (e.g., a mesh), and further acquire information about missing regions and their normals, use that information to derive the recommended pose of the object, and supply information indicating the derived recommended pose to the presentation unit 111.

[0076] In this way, the presentation unit 111 can present a recommended posture derived based on the missing area, etc. By referring to this presentation, the user can set the object in the recommended posture and take an image, making it easier to capture the first part of it. Therefore, the object 3D model generation device 100 can more easily fill in the missing areas of the object's 3D model using the 3D model reconstructed using multiple captured images obtained by the imaging. Therefore, the object 3D model generation device 100 (information processing device or information processing system) can more easily suppress the reduction in the accuracy of the 3D model.

[0077] Figure 11 is a block diagram showing an example of the main configuration of the recommended posture derivation unit 112. As shown in Figure 11, the recommended posture derivation unit 112 has a support surface setting unit 211 and a recommended posture setting unit 212.

[0078] The support surface setting unit 211 performs processing related to setting the support surface. Here, the support surface refers to a horizontal surface that supports the 3D model of the object. For example, the support surface setting unit 211 may acquire the 3D model (mesh) of the object. The support surface setting unit 211 may acquire information about the missing region and its normal vector. Based on this information, the support surface setting unit 211 may set the support surface that supports the 3D model of the object. The support surface setting unit 211 may supply the 3D model (mesh) of the object, information about the missing region in the 3D model, information about the normal vector of the missing region, and information about the set support surface to the recommended posture setting unit 212.

[0079] The recommended posture setting unit 212 performs processing related to setting the recommended posture of an object in imaging of the defective region. For example, the recommended posture setting unit 212 may acquire a 3D model (mesh) of the object supplied from the support surface setting unit 211, information about the defective region in the 3D model, information about the normal of the defective region, and information about the support surface. Based on this information, the recommended posture setting unit 212 may set the recommended posture of the object and supply information indicating the set recommended posture to the presentation unit 111. The presentation unit 111 may then present the set recommended posture.

[0080] For example, the support surface setting unit 211 may set a support surface on which the object (or its 3D model) can be placed in a position that allows imaging of a first portion of the object corresponding to a missing area in the object's 3D model, and the recommended posture setting unit 212 may set the posture of the object (or its 3D model) as it is placed on the support surface as the recommended posture.

[0081] In this case, for example, the support surface setting unit 211 may detect a flat surface (or a substantially flat surface) from the group of faces of the 3D model (mesh) and set a support surface on which the 3D model can be placed (that is, a support surface corresponding to a horizontal plane on which an object can be placed while in contact with the second part corresponding to that flat surface (or substantially flat surface)). Alternatively, the support surface setting unit 211 may detect candidate flat surfaces that are candidates for a flat surface (or substantially flat surface) from the group of faces of the 3D model (mesh), evaluate the flatness (degree of flatness) of the detected candidate flat surface, and set a flat surface (or substantially flat surface) based on the result of that evaluation. Any method can be used to evaluate the flatness. For example, the support surface setting unit 211 may evaluate the similarity of the orientation of the normals of each face constituting the candidate flat surface as the flatness.

[0082] Alternatively, the support surface setting unit 211 may set a plane that is in contact with the detected flat surface (or substantially flat surface) of the 3D model as a candidate support surface, evaluate whether the object can be placed on the second portion corresponding to the flat surface (or substantially flat surface) of that candidate support surface, and set one of the candidate support surfaces as a support surface based on the result of that evaluation.

[0083] As mentioned above, the support surface may support (contact) an object at a single location or at multiple locations. Furthermore, the number of support surfaces and recommended postures that can be set is also unlimited. For example, the support surface setting unit 211 may set a single support surface, and the recommended posture setting unit 212 may set a single recommended posture. Alternatively, the support surface setting unit 211 may set multiple support surfaces, and the recommended posture setting unit 212 may set multiple recommended postures. Alternatively, the support surface setting unit 211 may set multiple candidate support surfaces, select the best support surface (single) from among the candidates, and the recommended posture setting unit 212 may set the recommended posture (single) for that support surface.

[0084] <Defective Region Normal Derivation Unit> As shown in Figure 5, the object 3D model generation device 100 (information processing device or information processing system) may further have a defective region normal derivation unit 113. The defective region normal derivation unit 113 performs processing related to the derivation of the normal of the defective region. For example, the defective region normal derivation unit 113 may acquire a 3D model of the object (e.g., a mesh), detect a defective region occurring in the 3D model, derive a defective region normal vector indicating the normal of the detected defective region, and supply information about the defective region and its normal to the recommended posture derivation unit 112. The recommended posture derivation unit 112 may then derive a recommended posture for the object when imaging the part of the object (first part) corresponding to the defective region based on this information. The defective region normal derivation unit 113 may also supply the object's 3D model (mesh), information about the defective region occurring in the 3D model, and information about the normal of the defective region to the presentation unit 111.

[0085] In this way, the presentation unit 111 can present a recommended posture derived based on the missing regions derived from the 3D model (mesh) of the object. By referring to this presentation, users can set the object in the recommended posture and take images, making it easier to capture the first portion of the object. Therefore, the object 3D model generation device 100 can more easily fill in the missing regions of the object's 3D model using the 3D model reconstructed from multiple captured images obtained by the imaging. Consequently, the object 3D model generation device 100 (information processing device or information processing system) can more easily suppress the reduction in the accuracy of the 3D model.

[0086] Figure 12 is a block diagram showing an example of the main configuration of the missing region normal vector derivation unit 113. As shown in Figure 12, the missing region normal vector derivation unit 113 includes a missing region detection unit 221 and a missing region normal vector derivation unit 222.

[0087] The missing region detection unit 221 performs processing related to the detection of missing regions. For example, the missing region detection unit 221 may acquire a 3D model (mesh) of an object and detect missing regions that have occurred in that 3D model. Any method can be used to detect these missing regions. The missing region detection unit 221 may supply the 3D model (mesh) of the object and information regarding the detected missing regions to the missing region normal vector derivation unit 222.

[0088] The missing region normal vector derivation unit 222 performs processing related to the derivation of a missing region normal vector that indicates the normal of the missing region. For example, the missing region normal vector derivation unit 222 may acquire the 3D model (mesh) of the object and information about the detected missing region, supplied by the missing region detection unit 221. Based on this information, the missing region normal vector derivation unit 222 may derive a missing region normal vector for the missing region that has occurred in the 3D model of the object. In other words, the missing region normal vector derivation unit 222 may derive the normal vector of the missing region detected by the missing region detection unit 221.

[0089] The method for deriving the normal vector of the missing region can be any method. For example, the missing region normal vector derivation unit 222 may detect boundary edges, which are the edges of the missing region detected by the missing region detection unit 221, derive adjacent surface normal vectors, which are the normal vectors of adjacent surfaces that are in contact with the missing region via the boundary edges, derive boundary edge normal vectors, which are the normal vectors of the boundary edges, based on the boundary edges and adjacent surface normal vectors, and then derive the normal vector of the missing region by combining and normalizing the multiple boundary edge normal vectors derived therefrom.

[0090] For example, the missing region normal vector derivation unit 222 may detect edges that do not have adjacent faces as boundary edges e and generate a list of them (ε_boundary). The missing region normal vector derivation unit 222 may also derive adjacent face normal vectors (n_face), which are the normal vectors of adjacent faces that are in contact with the missing region via the boundary edge. For example, if the boundary edge e = (v1, v2), the missing region normal vector derivation unit 222 may use the vector in the direction of that edge (e = v2 - v1) and the adjacent face normal vector (n_face) to derive the boundary edge normal vector (n_missing), which is the normal vector of the boundary edge, as shown in equation (1) below.

[0091] ... (1)

[0092] In other words, the boundary edge normal vector (n_missing) is obtained by the cross product of the vector in the edge direction (e) and the adjacent face normal vector (n_face). That is, the boundary edge normal vector (n_missing) is a vector parallel to the face direction of the adjacent face. The missing region normal vector derivation unit 222 may derive this boundary edge normal vector (n_missing) for each boundary edge. Then, the missing region normal vector derivation unit 222 may derive the missing region normal vector (N_missing) by combining the boundary edge normal vectors (n_missing) corresponding to each boundary edge and normalizing them as shown in equation (2) below. In equation (2), the combined boundary edge is denoted as n'_missing.

[0093] ... (2)

[0094] The missing region normal vector derivation unit 222 may, for example, supply the missing region normal vector derived in this manner to the recommended posture derivation unit 112 as information regarding the normal of the missing region that occurred in the object's 3D model, along with information regarding the detected missing region. Alternatively, the missing region normal vector derivation unit 222 may supply the derived missing region normal vector to the presentation unit 111 as information regarding the normal of the missing region that occurred in the object's 3D model, along with the object's 3D model (mesh) and information regarding the missing region detected in that 3D model.

[0095] Alternatively, instead of the missing region normal vector, a vector indicating the observation direction of the missing region may be derived. For example, the missing region normal vector derivation unit 222 may derive a vector indicating the observation direction of the detected missing region based on the 3D model (mesh) of the supplied object and information regarding the missing region.

[0096] Alternatively, instead of deriving the normal vector for the missing region, the sector-cone shape described above may be derived. For example, the missing region normal vector derivation unit 222 may derive a sector-cone shape that indicates a field of view suitable for observing the detected missing region, based on the 3D model (mesh) of the supplied object and information about the missing region.

[0097] Alternatively, instead of deriving the normal vector for the missing region, the recommended imaging position and orientation described above may be derived. For example, the missing region normal vector derivation unit 222 may derive the recommended imaging position and orientation for the detected missing region based on the 3D model (mesh) of the supplied object and information about the missing region.

[0098] Of course, the defective region normal vector derivation unit 222 may derive any multiple of the following as information indicating the defective region: the defective region normal vector, a vector indicating the observation direction of the defective region, a sector-cone shape, and a recommended imaging position and orientation. Furthermore, the defective region normal vector derivation unit 222 may derive information other than these examples as information indicating the defective region.

[0099] <3D Model Generation Unit and Clipping Unit> As shown in Figure 5, the object 3D model generation device 100 (information processing device or information processing system) may further include a 3D model generation unit 114 and a clipping unit 115. The 3D model generation unit 114 performs processing related to the generation of 3D models. For example, the 3D model generation unit 114 may acquire multiple captured images, use them to generate a 3D model (e.g., point cloud data), and supply the generated 3D model (point cloud) to the clipping unit 115. The clipping unit 115 may extract the 3D model of an object from the supplied 3D model (point cloud) and generate its mesh. In other words, the clipping unit 115 may generate the 3D model (mesh) of an object. The clipping unit 115 may supply the generated 3D model (mesh) to the recommended posture derivation unit 112 and the missing area normal derivation unit 113.

[0100] In this case, the missing region normal vector derivation unit 113 detects the missing region that occurs in the 3D model (mesh) extracted and converted from the 3D model (point cloud) by the clipping unit 115, and derives a missing region normal vector that indicates the normal of the detected region. The recommended posture derivation unit 112 derives a recommended posture based on the 3D model (mesh), information on the missing region, and information on the normal of the missing region. The recommended posture derivation unit 112 may also supply the derived recommended posture to the presentation unit 111.

[0101] In this way, the presentation unit 111 can present a recommended posture derived from missing regions detected from the 3D model generated using multiple captured images. By referring to this presentation, users can set the object in the recommended posture and perform imaging, making it easier to image the first portion of the object. Therefore, the object 3D model generation device 100 can more easily fill in missing regions of the object's 3D model using the 3D model reconstructed using the multiple captured images obtained by imaging. Consequently, the object 3D model generation device 100 (information processing device or information processing system) can more easily suppress the reduction in the accuracy of the 3D model.

[0102] Figure 13 is a block diagram showing an example of the main configuration of the 3D model generation unit 114. As shown in Figure 13, the 3D model generation unit 114 includes a camera position and orientation estimation unit 231, a depth image generation unit 232, and a point cloud generation unit 223.

[0103] The camera position and orientation estimation unit 231 performs processing related to the estimation of the position and orientation of the camera that takes images for generating a 3D model. For example, the camera position and orientation estimation unit 231 may acquire multiple captured images supplied from an imaging device (camera) (not shown) and use those multiple captured images to estimate the position and orientation of the camera that took the images from which those images were generated. The camera position and orientation estimation unit 231 may supply the multiple captured images and information indicating the estimated camera position and orientation to the depth image generation unit 232.

[0104] The depth image generation unit 232 performs processing related to the generation of a depth image in which depth is used as a pixel value. For example, the depth image generation unit 232 may acquire a plurality of captured images supplied from the camera position and orientation estimation unit 231 and information indicating the estimated camera position and orientation. The depth image generation unit 232 may generate a depth image based on this information. The depth image generation unit 232 may supply the generated depth image along with the plurality of captured images and information indicating the camera position and orientation to the point cloud generation unit 223.

[0105] The point cloud generation unit 223 performs processing related to the generation of point cloud data as a 3D model. For example, the point cloud generation unit 223 may acquire multiple captured images supplied from the depth image generation unit 232, as well as the camera's position and orientation and depth image, and generate a 3D model (point cloud) based on this information. The point cloud generation unit 223 may also supply the generated 3D model (point cloud) to the clipping unit 115.

[0106] Figure 14 is a block diagram showing an example of the main configuration of the clipping unit 115. As shown in Figure 14, the clipping unit 115 has an extraction unit 241 and a mesh estimation unit 242.

[0107] The extraction unit 241 performs processing related to the extraction of 3D models of objects. For example, the extraction unit 241 may acquire a 3D model (point cloud) supplied from the 3D model generation unit 114, extract a 3D model of a desired object from the 3D model (point cloud), and supply the extracted 3D model (also referred to as the extracted 3D model (point cloud)) to the Mesh estimation unit 242.

[0108] The method for extracting the 3D model of an object can be any method. For example, plane separation, which searches for large planes within a point cloud using RANSAC (Random Sample Consensus) plane fitting, may be applied. Here, RANSAC is a method applied to regression problems that include outliers. This method can be applied, for example, to separate cars and roads in autonomous driving. Alternatively, clustering, which classifies the point cloud into objects and others using differences in point cloud density, may be applied. Furthermore, semantic segmentation, which classifies what pixels or points represent based on the appearance and shape features of an image or point cloud, may be applied. Of course, other methods may also be applied. In addition, multiple methods may be applied in combination.

[0109] The Mesh estimation unit 242 performs processing related to the generation of a 3D model using a mesh. For example, the Mesh estimation unit 242 may acquire an extracted 3D model (point cloud) supplied from the extraction unit 241, estimate (generate) the mesh of an object based on the extracted 3D model (point cloud), and supply the 3D model (mesh) of the object to the recommended pose derivation unit 112 and the missing region normal derivation unit 113.

[0110] <3D Model Integration Unit, Storage Unit, and Communication Unit> As shown in Figure 5, the object 3D model generation device 100 (information processing device or information processing system) may further include a 3D model integration unit 116, a storage unit 117, and a communication unit 118. The 3D model integration unit 116 performs processing related to the integration of 3D models. For example, the 3D model generation unit 114 may supply the generated 3D model (point cloud) to the 3D model integration unit 116. If the storage unit 117 does not have a 3D model (point cloud) stored in it, the 3D model integration unit 116 may supply the 3D model (point cloud) to the storage unit 117 and have it stored there. If the storage unit 117 has a 3D model (point cloud) stored in it, the 3D model integration unit 116 may integrate the 3D model (point cloud) with the 3D model (point cloud) read from the storage unit 117.

[0111] The method for integrating 3D models can be any method. For example, multiple 3D models may be integrated by global alignment, or by local alignment. Global alignment is a method that extracts characteristic points from the entire point cloud and aligns them by finding points with the same characteristics. Local alignment is a method that compares the local similarities of the point cloud and finds the closest position. Generally, local alignment can integrate with higher accuracy, but it may fail if the initial positions are far apart. In contrast, global alignment can integrate regardless of the initial position, but the alignment accuracy may be low. The 3D model integration unit 116 may appropriately select and apply any of these methods, or may appropriately combine and apply these methods.

[0112] Furthermore, the 3D model integration unit 116 may integrate the entire 3D model (point cloud) generated by the 3D model generation unit 114, or it may integrate the 3D models (point cloud) of objects extracted therefrom. In the following explanation, we will use the case where the entire 3D model (point cloud) generated by the 3D model generation unit 114 is integrated as an example.

[0113] The 3D model integration unit 116 may supply the integrated 3D model (point cloud) (also referred to as the integrated 3D model (point cloud)) to the storage unit 117 for storage. The 3D model integration unit 116 may supply the integrated 3D model (point cloud) to the clipping unit 115. The clipping unit 115 may extract the 3D model of the object from the integrated 3D model (point cloud) and mesh it. In other words, the clipping unit 115 may generate the 3D model (mesh) of the object from the integrated 3D model. The clipping unit 115 may supply the 3D model (mesh) of the object generated from the integrated 3D model to the missing region normal derivation unit 113. The missing region normal vector derivation unit 113 checks for the presence or absence of missing regions in the 3D model. If missing regions exist, as described above, the missing region normal vector derivation unit 113 derives the missing region normal vector, the recommended posture derivation unit 112 derives the recommended posture, and the presentation unit 111 may present the recommended posture of the object, information regarding the missing regions, etc.

[0114] Such a series of processes may be performed on the captured image of the missing region. That is, the 3D model generation unit 114 generates 3D data using multiple captured images of the first portion, the clipping unit 115 extracts a 3D model of an object from the 3D data, the 3D model integration unit 116 integrates the extracted 3D model with an existing 3D model (a 3D model stored in the storage unit 117), the missing region normal vector derivation unit 113 detects the missing region in the integrated 3D model, and if a missing region exists, the recommended pose derivation unit 112 sets a recommended pose, and the presentation unit 111 presents it.

[0115] Furthermore, this series of processes may be repeated until there are no more missing regions. In other words, each processing unit may repeatedly execute the series of processes until no more missing regions are detected in the integrated 3D model.

[0116] Furthermore, if no missing areas exist, the clipping unit 115 may supply the 3D model (mesh) of the object generated from the integrated 3D model to the 3D model integration unit 116. The 3D model integration unit 116 may supply the 3D model (mesh) to the presentation unit 111 for presentation. Alternatively, the 3D model integration unit 116 may supply the 3D model (mesh) to the communication unit 118 for external distribution.

[0117] The storage unit 117 has a storage medium and performs processing related to the storage of 3D models, etc. For example, the storage unit 117 may store 3D models (or integrated 3D models) supplied from the 3D model integration unit 116. Alternatively, the storage unit 117 may supply the stored 3D models (or integrated 3D models) to the 3D model integration unit 116 in response to a request from the 3D model integration unit 116.

[0118] The communication unit 118 has a communication function and performs processing related to communication with an external device (other device). This communication may be wired, wireless, or both. Furthermore, the method of communication may be any method. For example, the communication unit 118 may acquire a 3D model (or integrated 3D model) supplied from the 3D model integration unit 116 and supply it to an external device (other device) via its communication.

[0119] In this way, the display unit 111 can also display a recommended orientation for the integrated 3D model as needed. By referring to this display, users can set the object in the recommended orientation and perform imaging, making it easier to image further missing areas. Therefore, the object 3D model generation device 100 can more easily fill in missing areas of the object's 3D model using the 3D model reconstructed using multiple images obtained from the imaging. Consequently, the object 3D model generation device 100 (information processing device or information processing system) can more easily suppress the reduction in the accuracy of the 3D model.

[0120] <Object 3D Model Generation Process Flow> An example of the flow of the object 3D model generation process (information processing method or program) executed by the object 3D model generation device 100 (information processing device or information processing system) having the above configuration will be explained with reference to the flowchart in Figure 15.

[0121] When the object 3D model generation process is started, in step S101, the 3D model generation unit 114 executes the 3D model generation process as described above and generates a 3D model.

[0122] In step S102, the 3D model integration unit 116 determines whether or not the 3D model is stored in the storage unit 117. If it is determined that it is not stored, the process proceeds to step S103.

[0123] In step S103, the storage unit 117 stores the 3D model generated in step S101.

[0124] In step S104, the clipping unit 115 performs the clipping process as described above, and clips (extracts) the 3D model of the object from the 3D model generated in step S101.

[0125] In step S105, the missing region normal derivation unit 113 performs the missing region normal derivation process as described above and derives the missing region normal for the 3D model of the object clipped in step S104. For example, the missing region normal derivation unit 113 detects the missing region and derives the normal vector of that missing region.

[0126] In step S106, the recommended posture derivation unit 112 performs the recommended posture derivation process as described above, and derives a recommended posture for the 3D model of the object clipped in step S104 based on the missing area detected in step S105.

[0127] In step S107, the display unit 111 presents the recommended orientation derived in step S106 and information indicating the missing region derived in step S105, as described above. In other words, the display unit 111 presents the recommended orientation of the object when imaging the first part of the object corresponding to the missing region, which is set based on the missing region on the surface of the 3D model. The recommended orientation may be an orientation that allows imaging of the first part. The information indicating the missing region is as described above.

[0128] Once the process in step S107 is completed, the process returns to step S101.

[0129] Furthermore, if it is determined in step S102 that the 3D model is stored in the storage unit 117, the process proceeds to step S108.

[0130] In step S108, the 3D model integration unit 116 performs the 3D model integration process as described above, integrating the 3D model generated in step S101 with the 3D models stored in the storage unit 117 (including the integrated 3D model).

[0131] In step S109, the storage unit 117 stores the 3D model that was integrated in step S108.

[0132] In step S110, the clipping unit 115 performs the clipping process as described above, and clips the 3D model of the object from the 3D model integrated in step S108.

[0133] In step S111, the missing region normal derivation unit 113 performs the missing region normal derivation process as described above, and derives the missing region normal for the 3D model of the object clipped in step S110.

[0134] In step S112, the missing region normal derivation unit 113 determines whether or not a missing region exists. If it is determined that a missing region exists, the process returns to step S106. In this case, in step S106, the recommended posture derivation unit 112 performs the recommended posture derivation process as described above and derives a recommended posture for the 3D model of the clipped object from the integrated 3D model in step S108. In step S107, the presentation unit 111 presents information indicating the recommended posture derived in step S106 and the missing region derived in step S111.

[0135] In this case as well, once the processing in step S107 is completed, the process returns to step S101. In other words, these processes are repeatedly executed until it is determined in step S112 that no missing areas exist. If it is determined in step S112 that no missing areas exist, the process proceeds to step S113.

[0136] In step S113, the presentation unit 111 presents a 3D model in which it has been determined that no missing region exists.

[0137] In step S114, the communication unit 118 outputs the 3D model in which it has been determined that no missing region exists to the outside.

[0138] In step S115, the 3D model generation unit 114 determines whether or not to terminate the object 3D model generation process. If it is determined not to terminate, the process returns to step S101.

[0139] If it is determined in step S115 that the process has finished, the object 3D model generation process is terminated.

[0140] By executing the object 3D model generation process (information processing method or program) in this manner using the object 3D model generation device 100 (information processing device or information processing system), users can set the object to the recommended posture based on the presented information and take images of it, making it easier to capture the first part of the object. Therefore, the object 3D model generation process can more easily fill in missing regions of the object's 3D model using the 3D model reconstructed from multiple captured images obtained by the imaging. Consequently, the object 3D model generation process can more easily suppress the reduction in the accuracy of the 3D model.

[0141] <Flow of 3D Model Generation Process> An example of the flow of the 3D model generation process performed in step S101 of Figure 15 will be explained with reference to the flowchart in Figure 16.

[0142] When the 3D model generation process is started, in step S131, the camera position and orientation estimation unit 231 estimates the position and orientation of the camera based on the multiple captured images obtained by the camera, as described above.

[0143] In step S132, the depth image generation unit 232 generates a depth image based on the estimated camera position and orientation, as described above.

[0144] In step S133, the point cloud generation unit 233 generates a 3D model (point cloud) using the depth image, etc., as described above.

[0145] Once the process in step S133 is completed, the 3D model generation process ends, and the process returns to Figure 15.

[0146] <Clip Processing Flow> An example of the clip processing flow performed in step S104 of Figure 15 will be explained with reference to the flowchart in Figure 17.

[0147] When the clipping process is started, in step S151, the extraction unit 241 extracts the 3D model of the object as described above.

[0148] In step S152, the Mesh estimation unit 242 estimates the mesh from the extracted 3D model as described above and generates a 3D model (mesh) of the object.

[0149] Once the process in step S152 is completed, the process returns to Figure 15.

[0150] Furthermore, if this clipping process is performed in step S110 of Figure 15, the same process will be performed on the integrated 3D model (point cloud) generated in step S108.

[0151] <Flowchart for Deriving Normals of Missing Regions> An example of the flowchart for deriving normals of missing regions, which is performed in step S105 of Figure 15, will be explained with reference to the flowchart in Figure 18.

[0152] When the missing region normal derivation process is started, in step S171, the missing region detection unit 221 detects the missing region that occurs in the 3D model of the object clipped in step S104, as described above.

[0153] In step S172, the missing region normal vector derivation unit 222 derives a missing region normal vector that represents the normal of the missing region, as described above.

[0154] Once the process in step S172 is complete, the normal vector derivation process for the missing region is finished, and the process returns to Figure 15.

[0155] Furthermore, if this missing region normal derivation process is performed in step S111 of Figure 15, the same process is performed on the 3D model (mesh) of the object clipped in step S110.

[0156] <Flow of Recommended Posture Derivation Process> An example of the flow of the recommended posture derivation process performed in step S106 of Figure 15 will be explained with reference to the flowchart in Figure 19.

[0157] When the recommended posture derivation process is started, in step S191, the support surface setting unit 211 sets a support surface to support the 3D model of the object clipped in step S104, as described above.

[0158] For example, the support surface setting unit 211 may detect a flat surface in the 3D model and set a support surface on which an object can be placed while in contact with the second portion corresponding to that flat surface. Alternatively, the support surface setting unit 211 may detect candidate flat surfaces from the 3D model, evaluate the flatness of the detected candidate flat surfaces, and set a flat surface based on the result of that evaluation. Alternatively, the support surface setting unit 211 may set a plane in contact with the flat surface as a candidate support surface, evaluate whether an object can be placed on each of the candidate support surfaces with the second portion as the bottom surface, and set one of the candidate support surfaces as a support surface based on the result of that evaluation.

[0159] In step S192, the recommended posture setting unit 212 sets the recommended posture based on its support surface, as described above.

[0160] For example, the support surface setting unit 211 may set a support surface on which an object can be placed in a position that allows imaging of the first portion, and the recommended posture setting unit 212 may set the posture of the object as placed on that support surface as the recommended posture. When the processing in step S192 is completed, the recommended posture derivation process is completed, and the process returns to Figure 15.

[0161] <3D Model> In the above, we have described generating a 3D point cloud model from multiple captured images and clipping the 3D model of the object from that 3D model (point cloud). However, the specifications of this 3D model (3D data) are arbitrary. For example, 3D data other than point clouds may be applied. Also, in the above, we have described converting the 3D model (point cloud) of the object into a 3D model (mesh) to perform missing area detection and recommended pose setting. However, the specifications of this 3D model (3D data) are arbitrary. For example, 3D data other than meshes may be applied. However, since the accuracy (quality) of the 3D model largely depends on the degree to which the object's surface (shape, texture, gloss, etc.) is reproduced, it is desirable to apply a 3D model that can contain information about the object's surface (shape, texture, gloss, etc.), such as a mesh.

[0162] <Imaging Device> This technology is not limited to the example of the object 3D model generation device 100 described above, but can be applied to any configuration. For example, this technology may be applied to an imaging device having an imaging function, as shown in Figure 20. Figure 20 is a block diagram showing an example of the configuration of an imaging device, which is one aspect of an information processing device or information processing system to which this technology is applied. The imaging device 300 shown in Figure 20 is a device (or system) that performs processing related to imaging an object and generating a 3D model of that object.

[0163] As shown in Figure 20, the imaging device 300 has an imaging unit 311 and an object 3D model generation unit 312. The imaging unit 311 has an imaging function, such as an image sensor, and images an object and generates an image. For example, a user operates this imaging device 300 to perform photogrammetry imaging (multiple images taken from different imaging positions and orientations). The imaging unit 311 generates multiple images through such imaging and supplies them to the object 3D model generation unit 312. The object 3D model generation unit 312 has the same configuration as the object 3D model generation device 100 described above and performs the same processing. In other words, the object 3D model generation unit 312 acquires the multiple image supplied from the imaging unit 311 and uses these multiple image to perform object 3D model generation processing to which this technology is applied. Specifically, the object 3D model generation unit 312 presents a recommended orientation of the object when imaging a first part of the object corresponding to a missing area, which is set based on the missing area on the surface of the 3D model.

[0164] In this way, users can set the object in the recommended orientation when imaging the missing area using the imaging unit 311, based on the information provided. This allows users to more easily image the portion of the object corresponding to the missing area. Therefore, the imaging device 300 can more easily generate a 3D model of the missing area and fill in the missing area of ​​the object's 3D model with it, thus more easily suppressing the reduction in the accuracy of the 3D model.

[0165] <Object 3D Model Generation System> Furthermore, this technology may be applied to an object 3D model generation system as shown in Figure 21. Figure 21 is a block diagram showing an example of the configuration of an object 3D model generation system, which is one aspect of an information processing system to which this technology is applied. The object 3D model generation system 400 shown in Figure 21 is a device (or system) that performs processing related to imaging an object and generating a 3D model of that object.

[0166] In the example shown in Figure 21, the object 3D model generation system 400 includes an imaging device 411 and a personal computer 412. The imaging device 411 and the personal computer 412 are connected to each other via a communication medium such as a network (not shown). The specifications of this communication are arbitrary and may be, for example, wired communication or wireless communication.

[0167] As shown in Figure 21, the imaging device 411 has an imaging unit 421 and a communication unit 422. The imaging unit 421, like the imaging unit 311, images an object and generates an image of it. For example, when a user operates the imaging device 411 to perform photogrammetry imaging, the imaging unit 421 generates multiple images and supplies them to the communication unit 422. The communication unit 422 has a communication function and supplies the multiple images supplied from the imaging unit 421 to the personal computer 412 via its communication.

[0168] As shown in Figure 21, the personal computer 412 has an object 3D model generation unit 431. The object 3D model generation unit 431 is a processing unit similar to the object 3D model generation unit 312 and performs the same processing. In other words, the object 3D model generation unit 431 has the same configuration as the object 3D model generation device 100 described above and performs the same processing. Specifically, the object 3D model generation unit 431 acquires multiple captured images supplied from the imaging device 411 and uses these multiple captured images to perform object 3D model generation processing to which this technology is applied. Specifically, the object 3D model generation unit 431 presents a recommended orientation of the object when imaging a first part of the object corresponding to a missing area, which is set based on the missing area on the surface of the 3D model.

[0169] In this way, users can set the object to the recommended orientation presented on the personal computer 412 when imaging the missing region using the imaging device 411, based on the information provided. This allows users to more easily image the portion of the object corresponding to the missing region. Therefore, the object 3D model generation system 400 can more easily generate a 3D model of the missing region and fill in the missing region of the object's 3D model with it, thus more easily suppressing the reduction in the accuracy of the 3D model.

[0170] Furthermore, the object 3D model generation system 400 may have a server or cloud instead of a personal computer 412. In other words, in that case, the server or cloud may have an object 3D model generation unit 431.

[0171] <Object 3D Model Generation System> Furthermore, an object 3D model generation system to which this technology is applied may have the configuration shown in Figure 22. As shown in Figure 22, the object 3D model generation system 500 in this case is a system similar to the object 3D model generation system 400 and performs the same processing, but it has a server 511, an imaging device 512, and a display device 513. The server 511, the imaging device 512, and the display device 513 are connected to each other so as to be able to communicate with each other via a network 510. This network can be any medium that can serve as a communication medium. Furthermore, the specifications of the communication are arbitrary and may be wired communication or wireless communication.

[0172] In this case, the imaging device 512 has the same configuration as the imaging device 411 and performs the same processing. The presentation device 513 has a presentation unit 111. The server 511 has the same configuration as the object 3D model generation device 100, excluding the presentation unit 111.

[0173] In other words, in this system, the imaging device 512 is operated to perform photogrammetry imaging, generating multiple images, which are then supplied to the server 511. The server 511 uses these multiple images to execute an object 3D model generation process applying this technology, sets a recommended orientation for the object, generates information about missing areas, and supplies this information to the presentation device 513. The presentation device 513 then presents the supplied information. Specifically, the presentation device 513 presents a recommended orientation for the object when imaging a first part of the object corresponding to a missing area, which is set based on the missing area on the surface of the 3D model.

[0174] In this way, users can set the object to the recommended orientation presented on the presentation device 513 when imaging the missing area using the imaging device 512, based on the information presented. This allows users to more easily image the portion of the object corresponding to the missing area. Therefore, the object 3D model generation system 500 can more easily generate a 3D model of the missing area and fill in the missing area of ​​the object's 3D model with it, thus more easily suppressing the reduction in the accuracy of the 3D model.

[0175] <4. Addendum> <Computer> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, a computer includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0176] Figure 23 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program.

[0177] In the computer 900 shown in Figure 23, the CPU (Central Processing Unit) 901, ROM (Read-Only Memory) 902, and RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0178] An input / output interface 910 is also connected to the bus 904. An input / output interface 910 is connected to an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915.

[0179] The input unit 911 consists of, for example, a keyboard, mouse, microphone, touch panel, and input terminals. The output unit 912 consists of, for example, a display, speaker, and output terminals. The storage unit 913 consists of, for example, a hard disk, RAM disk, and non-volatile memory. The communication unit 914 consists of, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0180] In a computer configured as described above, the CPU 901 loads, for example, a program stored in the memory unit 913 into the RAM 903 via the input / output interface 910 and the bus 904, and executes it, thereby performing the series of processes described above. The RAM 903 also appropriately stores data necessary for the CPU 901 to perform various processes.

[0181] The program executed by the computer can be recorded and applied, for example, on removable media 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable media 921 into the drive 915.

[0182] Furthermore, this program can also be provided via wired or wireless transmission media such as a local area network, the internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0183] In addition, this program can be pre-installed in ROM 902 or memory unit 913.

[0184] <Applicable Applications of This Technology> Furthermore, this technology can be applied to any configuration. For example, this technology can be applied to various electronic devices.

[0185] Furthermore, this technology can also be implemented as part of a device, such as a processor as a system LSI (Large Scale Integration) (e.g., a video processor), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set with additional functions added to a unit (e.g., a video set).

[0186] Furthermore, this technology can also be applied to network systems composed of multiple devices. For example, this technology may be implemented as cloud computing, where multiple devices share and collaborate on processing via a network. For example, this technology may be implemented in a cloud service that provides image (video) related services to any terminal such as computers, AV (Audio Visual) equipment, portable information processing terminals, and IoT (Internet of Things) devices.

[0187] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules within a single enclosure, are both considered systems.

[0188] <Applicable Fields and Applications of This Technology> Systems, devices, and processing units incorporating this technology can be used in any field, such as transportation, medicine, security, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. Furthermore, the applications are entirely arbitrary.

[0189] For example, this technology can be applied to systems and devices used to provide ornamental content. Furthermore, for example, this technology can be applied to systems and devices used for traffic management, such as traffic condition monitoring and automated driving control. In addition, for example, this technology can be applied to systems and devices used for security. Furthermore, for example, this technology can be applied to systems and devices used for automatic control of machinery, etc. Furthermore, for example, this technology can be applied to systems and devices used for agriculture and livestock farming. Furthermore, for example, this technology can be applied to systems and devices that monitor natural conditions such as volcanoes, forests, and oceans, as well as wildlife. Furthermore, for example, this technology can be applied to systems and devices used for sports.

[0190] <Other> The embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the gist of this technology.

[0191] For example, the configuration described as a single device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as a single device (or processing unit). Furthermore, it is also possible to add configurations other than those described above to the configuration of each device (or each processing unit). In addition, if the overall system configuration and operation are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0192] Furthermore, for example, the program described above may be executed on any device. In that case, the device should have the necessary functions (such as functional blocks) and be able to obtain the necessary information.

[0193] Furthermore, for example, each step of a flowchart may be executed by one device, or it may be divided among multiple devices. Additionally, if a single step includes multiple processes, these processes may be executed by one device, or they may be divided among multiple devices. In other words, multiple processes included in a single step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step.

[0194] Furthermore, for example, a program executed by a computer may be structured so that the steps of the program are executed chronologically in the order described herein, or they may be executed in parallel or individually at necessary times, such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no inconsistencies arise. Moreover, the steps of this program may be executed in parallel with the processing of other programs, or in combination with the processing of other programs.

[0195] Furthermore, for example, multiple technologies relating to this technology can be implemented independently, as long as they do not create a contradiction. Of course, any multiple technologies can also be implemented in combination. For example, some or all of the technologies described in one embodiment can be implemented in combination with some or all of the technologies described in another embodiment. Also, some or all of the above-mentioned technologies can be implemented in combination with other technologies not mentioned above.

[0196] Furthermore, this technology can also be configured as follows: (1) An information processing method for presenting a recommended orientation of an object when imaging a first portion of the object corresponding to a missing area, which is set based on a missing area on the surface of a 3D model. (2) The information processing method according to (1), wherein the recommended orientation is an orientation in which the first portion can be imaged. (3) The information processing method according to (2), wherein the recommended orientation is set based on the missing area and the set recommended orientation is presented. (4) The information processing method according to (3), wherein a support surface is set on which the object can be placed in an orientation in which the first portion can be imaged, and the orientation of the object when placed on the support surface is set as the recommended orientation. (5) The information processing method according to (4), wherein a flat surface is detected in the 3D model and a support surface is set on which the object can be placed in contact with a second portion corresponding to the flat surface. (6) The information processing method according to (5), wherein a candidate flat surface is detected from the 3D model, the flatness of the detected candidate flat surface is evaluated, and the flat surface is set based on the result of the evaluation. (7) The information processing method according to (5) or (6), wherein a plane in contact with the flat surface is set as a candidate support surface, which is a candidate for the support surface, and for each of the candidate support surfaces, it is evaluated whether the object can be placed on the second portion as the bottom surface, and based on the result of the evaluation, one of the candidate support surfaces is set as the support surface. (8) The information processing method according to any one of (3) to (7), wherein the missing area is detected in the 3D model, and the recommended posture is set based on the detected missing area. (9) The information processing method according to (8), wherein 3D data is generated using multiple captured images of the object, the 3D model of the object is extracted from the 3D data, and the missing area is detected in the extracted 3D model.(10) The information processing method according to (9), wherein the information processing method comprises generating 3D data using a plurality of captured images of the first portion, extracting the 3D model of the object from the 3D data, integrating the extracted 3D model with an existing 3D model, detecting the missing region in the integrated 3D model, and if the missing region exists, setting and presenting the recommended posture. (11) The information processing method according to (10), wherein the series of processes are repeatedly executed until the missing region is no longer detected in the integrated 3D model. (12) The information processing method according to any one of (8) to (11), wherein the normal vector of the detected missing region is further derived. (13) The information processing method according to (12), further comprising: detecting a boundary edge which is the edge of the detected missing region; deriving an adjacent surface normal vector which is the normal vector of an adjacent surface tangent to the missing region via the boundary edge; deriving a boundary edge normal vector which is the normal vector of the boundary edge based on the boundary edge and the adjacent surface normal vector; and deriving the normal vector of the missing region by combining and normalizing a plurality of the derived boundary edge normal vectors. (14) The information processing method according to any one of (8) to (13), further comprising: deriving a vector indicating the observation direction of the detected missing region. (15) The information processing method according to any one of (8) to (14), further comprising: deriving a sector-shaped cone which indicates a field of view suitable for observing the detected missing region. (16) The information processing method according to any one of (8) to (15), further comprising: deriving a recommended imaging position and orientation for the detected missing region. (17) The information processing method according to any one of (2) to (16), further comprising presenting information indicating the missing region. (18) The information processing method according to (17), wherein the information indicating the missing region includes a missing region normal vector, which is a vector indicating the normal of the missing region. (19) The information processing method according to (17) or (18), wherein the information indicating the missing region includes highlighting the missing region. (20) The information processing method according to any one of (17) to (19), wherein the information indicating the missing region includes a vector indicating the observation direction of the missing region.(21) The information processing method according to any one of (17) to (20), wherein the information indicating the missing region includes a sector-shaped cone indicating a field of view suitable for observing the missing region. (22) The information processing method according to any one of (17) to (21), wherein the information indicating the missing region includes a display indicating a recommended imaging position and orientation for the missing region. (23) The information processing method according to any one of (17) to (22), wherein the information indicating the missing region includes information indicating the size of the missing region. (24) The information processing method according to any one of (17) to (23), wherein the information indicating the missing region includes information indicating the amount of missing image data necessary for generating 3D data of the missing region. (25) The information processing method according to (24), wherein the information indicating the amount of missing data includes an icon indicating the amount of missing data. (26) The information processing method according to (24) or (25), wherein the information indicating the amount of missing data includes a color corresponding to the amount of missing data for highlighting the missing region. (27) The information processing method according to any one of (24) to (26), wherein the information indicating the amount of deficiency includes a semi-transparent icon. (28) The information processing method according to any one of (24) to (27), wherein the information indicating the amount of deficiency includes an animation corresponding to the amount of deficiency for highlighting the defective area. (29) The information processing method according to any one of (24) to (28), wherein the information indicating the amount of deficiency includes a numerical value indicating the amount of deficiency. (30) The information processing method according to any one of (24) to (29), wherein the information indicating the amount of deficiency includes a gauge indicating the amount of deficiency. (31) The information processing method according to any one of (24) to (30), wherein the information indicating the amount of deficiency includes a notation for guidance between recommended imaging positions. (32) The information processing method according to any one of (17) to (31), wherein the information indicating the defective area includes information indicating the imaging conditions for the defective area. (33) The information processing method according to (32), wherein the information indicating the imaging conditions includes conditions relating to the position for imaging the defective area. (34) The information processing method according to (32) or (33), wherein the information indicating the imaging conditions includes conditions relating to the orientation for imaging the defective region. (35) The information processing method according to any one of (32) to (34), wherein the information indicating the imaging conditions includes conditions relating to the number of times the defective region is imaged.(36) The information processing method according to any one of (32) to (35), wherein the information indicating the imaging conditions includes conditions relating to the amount of light in imaging the missing area. (37) The information processing method according to any one of (32) to (36), wherein the information indicating the imaging conditions includes conditions relating to the degree of overlap between the images of the missing area. (38) The information processing method according to any one of (32) to (37), wherein the information indicating the imaging conditions includes conditions relating to the priority of imaging the missing area. (39) An information processing device comprising a display unit that displays a recommended orientation of an object when imaging a first part of an object corresponding to a missing area of ​​a 3D model, wherein the recommended orientation is set based on the missing area. (40) A program for causing a computer to perform a process of displaying a recommended orientation of an object when imaging a first part of an object corresponding to a missing area, which is set based on the missing area of ​​the surface of a 3D model. (41) An information processing system comprising a display unit that displays a recommended orientation of an object when imaging a first part of an object corresponding to a missing area of ​​a 3D model, wherein the recommended orientation is set based on the missing area.

[0197] 100 Object 3D model generation device, 111 Presentation unit, 112 Recommended posture derivation unit, 113 Missing area normal vector derivation unit, 114 3D model generation unit, 115 Clipping unit, 116 3D model integration unit, 117 Storage unit, 118 Communication unit, 130 Presentation image, 131 Object, 132 Support surface, 133 Missing area normal vector, 211 Support surface setting unit, 212 Recommended posture setting unit, 221 Missing area detection unit, 222 Missing area normal vector derivation unit, 231 Camera position and posture estimation unit, 232 Depth image generation unit, 233 Point cloud generation unit, 241 Extraction unit, 242 Mesh estimation unit, 300 Imaging device, 311 Imaging unit, 312 Object 3D model generation unit, 400 Object 3D model generation system, 411 Imaging device, 412 Personal computer, 421 Imaging unit, 422 Communication unit, 431 Object 3D model generation unit, 500 Object 3D model generation system, 510 Network, 511 Server, 512 Imaging device, 513 Display device, 900 Computer

Claims

1. An information processing method for presenting a recommended orientation of an object when imaging a first portion of the object corresponding to a missing area on the surface of a 3D model, which is set based on the missing area on the surface of the object.

2. The information processing method according to claim 1, wherein the recommended posture is a posture in which the first portion can be imaged.

3. The information processing method according to claim 2, which sets a recommended posture based on the missing region and presents the set recommended posture.

4. The information processing method according to claim 3, wherein a support surface is set on which the object can be placed in a posture that allows imaging of the first portion, and the posture of the object when placed on the support surface is set as the recommended posture.

5. The information processing method according to claim 4, which detects a flat surface of the 3D model and sets a support surface on which the object can be placed while in contact with a second portion corresponding to the flat surface.

6. The information processing method according to claim 3, wherein the missing region is detected in the 3D model, and the recommended posture is set based on the detected missing region.

7. The information processing method according to claim 6, comprising generating 3D data using multiple captured images of the object, extracting the 3D model of the object from the 3D data, and detecting the missing region in the extracted 3D model.

8. The information processing method according to claim 7, comprising: generating 3D data using a plurality of captured images of the first portion; extracting the 3D model of the object from the 3D data; integrating the extracted 3D model with an existing 3D model; detecting the missing region in the integrated 3D model; and, if the missing region exists, setting and presenting the recommended posture.

9. The information processing method according to claim 6, further comprising deriving the normal vector of the detected missing region.

10. The information processing method according to claim 2, further comprising presenting information indicating the missing region.

11. The information processing method according to claim 10, wherein the information indicating the missing region includes a missing region normal vector, which is a vector indicating the normal of the missing region.

12. The information processing method according to claim 10, wherein the information indicating the missing region includes highlighting the missing region.

13. The information processing method according to claim 10, wherein the information indicating the missing region includes a vector indicating the direction of observation of the missing region.

14. The information processing method according to claim 10, wherein the information indicating the missing region includes a sector-shaped cone that indicates a field of view suitable for observing the missing region.

15. The information processing method according to claim 10, wherein the information indicating the defective region includes a display indicating a recommended imaging position and orientation of the defective region.

16. The information processing method according to claim 10, wherein the information indicating the missing region includes information indicating the size of the missing region.

17. The information processing method according to claim 10, wherein the information indicating the missing region includes information indicating the amount of missing captured image necessary for generating 3D data of the missing region.

18. The information processing method according to claim 10, wherein the information indicating the defective region includes information indicating the imaging conditions of the defective region.

19. An information processing device comprising a display unit that displays a recommended orientation of an object in imaging a first portion of an object corresponding to a missing region of a 3D model, wherein the recommended orientation is set based on the missing region.

20. A program for causing a computer to perform a process that presents a recommended orientation of an object when imaging a first portion of the object corresponding to a missing area on the surface of a 3D model, which is set based on the missing area on the surface of the 3D model.

Images