Information processing system and information processing method

Abstract

The present disclosure relates to an information processing system and an information processing method with which it is possible to suitably achieve image capture for 3D modeling. An information processing system according to the present disclosure comprises: an information processing terminal that is provided with a display unit and a sensor group that includes a camera unit; and an image capture device that is connected to the information processing terminal, and executes image acquisition processing for generating three-dimensional information of a subject. The information processing system displays, on the display unit, a monitor image that includes an image capture range of the image capture device and is captured by the camera unit, and presents a UI for assisting image capture as the image acquisition processing by the image capture device. The features according to the present disclosure can be applied to, for example, an image capture navigation system for capturing images for 3D modeling.

Description

Information Processing System and Information Processing Method

[0001] The present disclosure relates to an information processing system and an information processing method, and particularly to an information processing system and an information processing method that can suitably realize shooting for 3D modeling.

[0002] Conventionally, as a method for 3D modeling of a subject (3D object) having a three-dimensional shape, a method called photogrammetry is known, in which the subject is photographed from multiple directions and 3D data is generated based on a plurality of captured images obtained.

[0003] Patent Document 1 discloses a technique for controlling the shooting of a 3D object based on a scoring result that evaluates the accuracy of 3D data that can be generated using captured images obtained by shooting performed so far. According to this technique, higher-definition 3D data can be generated, and thus 3D modeling can be performed more easily.

[0004] International Publication No. 2024 / 080120

[0005] There is a need for a UI that appropriately assists shooting for 3D modeling as described above.

[0006] The present disclosure has been made in view of such a situation, and is intended to suitably realize shooting for 3D modeling.

[0007] The information processing system according to the first aspect of the present disclosure includes an information processing terminal including a sensor group including a camera unit and a display unit, a shooting device connected to the information processing terminal and executing an image acquisition process for generating three-dimensional information of a subject, and a processing circuit that displays, on the display unit, a monitor image that is an image including a shooting range of the shooting device and has been captured by the camera unit, and presents a UI for assisting shooting as the image acquisition process by the shooting device.

[0008] The first aspect of this disclosure is an information processing method, which includes an information processing terminal having a sensor group including a camera unit and a display unit, and a shooting device connected to the information processing terminal and performing image acquisition processing for generating three-dimensional information of a subject, and displays an image including the shooting range of the shooting device, which is a monitor image captured by the camera unit, on the display unit, and presents a UI to assist in shooting as the image acquisition processing by the shooting device.

[0009] The second aspect of the information processing system of this disclosure is an information processing system comprising an information processing terminal equipped with a sensor group including a camera unit and a display unit, and an information processing circuit connected to the information processing terminal that displays a monitor image captured by the camera unit on the display unit, which includes the shooting range of a shooting device that performs image acquisition processing for generating three-dimensional information of a subject, and also presents a UI to assist in shooting as the image acquisition processing by the shooting device.

[0010] In this disclosure, an information processing terminal equipped with a sensor group including a camera unit and a display unit is connected to a shooting device that performs image acquisition processing to generate three-dimensional information of a subject. The monitor image captured by the camera unit, which includes the shooting range of the shooting device, is displayed on the display unit, and a UI is presented to assist in shooting as the image acquisition processing by the shooting device.

[0011] This is a diagram showing an example of the external configuration of the shooting navigation system. This is a diagram showing an example of the external configuration of the shooting navigation system. This is a diagram showing an example of the hardware configuration of the shooting navigation system. This is a block diagram showing an example of the functional configuration of the processing unit. This is a flowchart explaining the flow of the auxiliary UI presentation process. This is a diagram showing an example of the basic configuration of the navigation screen. This is a diagram showing an example of the conventional shooting flow for 3D modeling. This is a diagram showing an example of the shooting for 3D modeling according to this disclosure. This is a flowchart explaining the flow of the two-step shooting process. This is a diagram showing an example of the size selection screen. This is a diagram showing an example of the bounding box. This is a diagram showing an example of the navigation screen when shooting in normal mode. This is a diagram showing an example of the navigation screen when shooting mode switching is determined. This is a diagram showing an example of the GUI on the navigation screen. This is a diagram showing an example of the GUI on the navigation screen. This is a diagram showing an example of the GUI on the navigation screen. This is a diagram explaining the automatic setting of ROI. This is a diagram explaining the relationship between the distance to the subject and the resolution. This is a diagram showing an example of the display form of the AF frame. This is a diagram explaining the arrangement of the GUI on the navigation screen. This is a flowchart explaining the shooting control process for smartphones. This is a flowchart explaining the shooting timing presentation process. This is a flowchart explaining the simultaneous shooting process. This is a flowchart explaining the calibration process. This is a flowchart explaining the 3D model generation process. This is a flowchart explaining the camera parameter update process. This is a diagram showing an example of the navigation screen when calibration is successful. This is a flowchart explaining the haptic feedback process. This is a block diagram showing an example of the computer hardware configuration.

[0012] The following describes the forms for implementing this disclosure (hereinafter referred to as embodiments). The explanation will be given in the following order.

[0013] 1. Background and prior art of the technology disclosed herein 2. Configuration and operation of the shooting navigation system 3. Shooting flow for 3D modeling 4. Two-step shooting 5. UI layout 6. Simultaneous shooting and calibration 7. Haptic feedback 8. Description of a computer to which the technology disclosed herein is applied

[0014] <1. Background of the Technology Related to This Disclosure and Prior Art> (Background of the Technology Related to This Disclosure) Image-based three-dimensional reconstruction using methods such as photogrammetry and Novel View Synthesis (NeRF, 3D Gaussian Splatting) presented challenges for inexperienced users, such as preventing missed shots due to image overlap and occlusion. Therefore, in order to enable users to easily perform three-dimensional reconstruction, it is conceivable to provide an application that supports the capture of 3D objects and 3D scenes by combining a computing device, a sensing device, and a smartphone with a large screen with ILC (Interchangeable Lens Cameras) capable of high-quality capture.

[0015] However, attaching a smartphone to the ILC body means that during shooting, the right hand will be holding the ILC's grip, making it difficult to touch the smartphone, which may cause inconvenience in operation during shooting. In addition, there is a risk of performing unnecessary operations and taking unexpected shots due to unfamiliarity with the operation.

[0016] Furthermore, the nature of the three-dimensional information obtained by smartphones and ILC imaging devices differs, with differences in immediacy and the types of subjects that can be photographed. Therefore, it is necessary to consider the advantages and disadvantages of each and combine them, and it is required to guide the user so that they can operate in the appropriate order and timing.

[0017] (Conventional Technology) In conventional technology, sensors and cameras installed in smartphones were used to assist in capturing 3D objects and 3D scenes. Therefore, a vertical UI (User Interface) specifically designed for smartphones was adopted. Currently, most smartphones do not have a physical button corresponding to the camera shutter button, so the shutter button and start button are placed in the center of the bottom of the screen, and the other controls follow the typical layout of a smartphone screen.

[0018] On the other hand, if we consider attaching a smartphone to the ILC, for reasons such as mounting stability, the smartphone will typically be attached horizontally to the hot shoe (accessory shoe) on the top of the ILC. In this case, the vertical UI layout described above is not appropriate. Also, unlike when using a smartphone alone, it is necessary to hold the heavier ILC body, so it is assumed that the smartphone will be operated with the hand that is not holding the ILC. Taking this into account, a UI layout that prevents accidental operation during actual shooting is required.

[0019] Furthermore, in the combinations described above, there is parallax between the ILC that captures the images and the smartphone that displays them on the screen. As a result, in most cases, the ILC's shooting range will be located at the bottom of the smartphone screen. If the UI is superimposed on that area, it becomes difficult to see the ILC's shooting range.

[0020] Furthermore, in conventional technologies, to make it easier to understand the areas that have already been captured in multi-view photography, an image indicating the direction from which the subject was photographed was projected onto a plane, depending on the subject's orientation. However, this was insufficient to more accurately display the progress of photography or to more precisely indicate areas that had not yet been photographed. User-defined shooting ranges were also limited to subjects with pre-defined shapes, such as objects that could be surrounded by the camera.

[0021] Furthermore, conventional system configurations did not combine a smartphone and an ILC, so there was no need to consider the parallax that occurred between the camera used for capturing images and the camera used for displaying them. On the other hand, in a system configuration to which the technology described herein is applied, parallax occurs, requiring calibration each time it was used. In this case, operations that were not anticipated in conventional system configurations are required. Calibrating the smartphone and the ILC individually becomes very complicated, requiring charts and so on.

[0022] In contrast, the technology disclosed herein provides a UI flow and UI layout suitable for 3D shooting navigation through a shooting navigation system that combines a smartphone and an ILC.

[0023] <2. Configuration and Operation of the Shooting Navigation System> (External Configuration) Figures 1 and 2 show examples of the external configuration of a shooting navigation system as an information processing system to which the technology relating to this disclosure is applied.

[0024] The imaging navigation system 1 shown in Figures 1 and 2 performs imaging for three-dimensional reconstruction.

[0025] The shooting navigation system 1 is configured to include a smartphone 10 and an ILC 20. The smartphone 10 is an information processing terminal equipped with a sensor group including a camera and a display unit. The ILC 20 is a shooting device that is mechanically and electrically connected to the smartphone 10 and performs image acquisition processing to generate three-dimensional information (three-dimensional structure) of the subject. The three-dimensional information referred to here may include not only the three-dimensional model itself, but also shape information and color information for rendering the three-dimensional model. In the shooting navigation system 1, the smartphone 10 is mounted horizontally on the hot shoe on the top of the ILC 20.

[0026] In the shooting navigation system 1, navigation information to assist shooting by the ILC 20 is projected into the real space and displayed on the screen (display unit 11) of the smartphone 10 by estimating the position and orientation using the sensors and camera of the smartphone 10. In addition, the smartphone 10 controls the automatic shooting of the ILC 20 with appropriate shooting conditions and timing. Meanwhile, the ILC 20 takes pictures under the control of the smartphone 10.

[0027] In other words, in the shooting navigation system 1, the screen of the smartphone 10 (display unit 11) and the screen of the ILC 20 (display unit 21) display the same subject that is the target of 3D modeling. With the shooting navigation system 1, the user can take high-quality photos with the ILC 20 while keeping an eye on the navigation information displayed on the display unit 11 of the smartphone 10.

[0028] Furthermore, as shown in Figure 1, in a front view of the ILC20 from the lens side, a gripping section 22 is provided at the left end of the ILC20 body for the photographer to hold with their right hand. In addition, as shown in Figure 2, in a front view of the ILC20 from the display unit 21 side, an operation section 23 is provided on the right side of the hot shoe at the top of the ILC20, which includes a shutter button and custom buttons to which various functions can be assigned. The shutter button and custom buttons are physical buttons that can be operated by the photographer while holding the gripping section 22. The operation section 23 also includes physical buttons and dials provided on the right side of the display unit 21 on the ILC20 body.

[0029] (Hardware Configuration) Figure 3 is a block diagram showing an example of the hardware configuration of the shooting navigation system 1.

[0030] In the shooting navigation system 1 shown in Figure 3, various sensors are provided on the smartphone 10, and functions for spatial recognition, posture calculation, camera control, and shooting assist display are integrated into the application. The ILC 20 also transmits camera information and shooting completion signals to the smartphone 10.

[0031] The smartphone 10 includes a display unit 11, an IMU (Inertial Measurement Unit) 110, a depth sensor 120, a sensor group including a camera unit 130, and an operation unit 140. Furthermore, the smartphone 10 includes a processing unit 150 that can implement various functions by executing a program stored in a memory (not shown).

[0032] In addition to the display unit 21 and operation unit 23 described above, the ILC20 includes a camera unit 210 which consists of interchangeable lenses and an image sensor and has a shooting function for capturing subjects to be 3D modeled. The captured images (also called through images, etc.) captured by the camera unit 210 are displayed on the display unit 21 in real time. The ILC20 also includes a processing unit 230 which can realize various functions by executing a program stored in a memory (not shown).

[0033] The processing unit 150 of the smartphone 10 and the processing unit 230 of the ILC 20 can exchange various types of information via a communication interface (not shown).

[0034] The IMU 110 detects the three-dimensional inertial motion (acceleration, angular velocity) of the smartphone 10 and outputs the detection results to the processing unit 150.

[0035] The depth sensor 120 has a LiDAR (Light Detection And Ranging) sensor (dToF (Direct Time of Flight) module), detects the depth to the subject, and outputs the detection result to the processing unit 150.

[0036] The camera unit 130 consists of a fixed-focus lens and an image sensor, and, like the camera unit 210, has a shooting function for capturing subjects that are the target of 3D modeling. The captured images taken by the camera unit 130 are output to the processing unit 150.

[0037] The operation unit 140 is composed of, for example, a touch panel superimposed on the display unit 11, and accepts instructions and information input in response to user operations. The received instructions and information are output to the processing unit 150.

[0038] The processing unit 150 controls the operation of the smartphone 10 and the ILC 20 based on various sensor data and information from the IMU 110, depth sensor 120, camera unit 130, and operation unit 140.

[0039] (Functional Configuration of the Processing Unit) Figure 4 is a block diagram showing an example of the functional configuration of the processing unit 150.

[0040] The processing unit 150 executes a program stored in memory (not shown) to realize the functional blocks of the position and orientation calculation unit 301, object recognition unit 302, UI presentation unit 303, shooting control unit 304, calibration processing unit 305, and three-dimensional model generation unit 306. At least one of the functional blocks realized in the processing unit 150 may be realized in the processing unit 230 of the ILC 20.

[0041] The position and orientation calculation unit 301 calculates the position and orientation of the smartphone 10 and ILC 20 in three-dimensional space based on various sensor data from the IMU 110, depth sensor 120, and camera unit 130.

[0042] The object recognition unit 302 generates three-dimensional shape information that shows the three-dimensional shape of the object that is the subject of the captured image (hereinafter also referred to as the monitor image) captured by the camera unit 130, which includes the shooting range of the ILC 20. Then, based on the generated three-dimensional shape information, the object recognition unit 302 recognizes the shape and size of the object and sets the area of ​​the subject to be photographed by the ILC 20 as the region of interest (ROI).

[0043] The UI display unit 303 displays the monitor image captured by the camera unit 130 on the display unit 11, and also presents a UI (auxiliary UI) to assist (guide) the image acquisition process by the ILC 20. The auxiliary UI may be presented as a GUI (Graphical User Interface) displayed on the display unit 11, or it may be presented by tactile feedback via vibration or audio output.

[0044] The imaging control unit 304 causes the image acquisition process for generating three-dimensional information to be executed, triggered by a user operation on the operation unit 140, the position and orientation of the smartphone 10, or the like. That is, the imaging control unit 304 outputs imaging control information for causing the ILC 20 to execute the image acquisition process for generating three-dimensional information.

[0045] The calibration processing unit 305 performs calibration processing for estimating the camera parameters of the camera unit 130 and the ILC 20 using the image captured by the camera unit 130 and the image captured by the ILC 20 (camera unit 210).

[0046] The three-dimensional model generation unit 306 simply generates a three-dimensional model based on the image captured by the camera unit 130 or the image captured by the ILC 20.

[0047] (Flow of auxiliary UI presentation processing) Referring to the flowchart of FIG. 5, the flow of the auxiliary UI presentation processing in the imaging navigation system 1 will be described. The processing in FIG. 5 is started, for example, triggered by the operation unit 140 being operated while the monitor image is captured by the camera unit 130 of the smartphone 10 and displayed on the display unit 11.

[0048] In step S1, the position and orientation calculation unit 301 calculates position and orientation information indicating the position and orientation of the camera unit 130 by SLAM (Simultaneous Localization and Mapping) using the sensor data from the IMU 110 and the camera unit 130.

[0049] In step S2, the object recognition unit 302 generates a mesh (three-dimensional shape information) indicating the three-dimensional shape of the object serving as the subject based on the calculated position and orientation information and the depth data (sensor data from the depth sensor 120) obtained by scanning the subject.

[0050] In step S3, the object recognition unit 302 sets a region of interest (ROI) based on the generated mesh. The shape and size of the ROI can be manually adjusted by the photographer.

[0051] Then, in step S4, the UI display unit 303 displays an auxiliary UI on the monitor image displayed on the display unit 11. The auxiliary UI displayed here is a UI that shows the shooting range of the ILC 20 projected onto the three-dimensional shape of the object, based on the ROI set by scanning the object. This makes it clear which part of the object to be 3D modeled should be photographed in the shooting navigation system 1 that combines the smartphone 10 and the ILC 20, and enables the shooting for 3D modeling to be successfully realized.

[0052] (Navigation screen) Referring to Figure 6, a basic example of the configuration of the navigation screen displayed on the display unit 11 of the smartphone 10 will be described. On the navigation screen, the monitor image captured by the camera unit 130 is displayed, and the auxiliary UI described above is presented.

[0053] Figure 6 shows an example of a navigation screen displayed on the display unit 11 of a smartphone 10 held horizontally. In the example in Figure 6, a monitor image including the subject SJ is displayed on the navigation screen. While the ILC 20 is performing image acquisition processing, the monitor image is superimposed with polygons (vertices, edges, faces) that constitute a mesh showing the three-dimensional shape of all objects included in the shooting range of the camera unit 130. However, in the example in Figure 6, for simplicity, polygons constituting the mesh are drawn only in a portion of the range including the subject SJ, and this will be omitted from the illustration thereafter.

[0054] The navigation screen (monitor image) displays an AF (Auto Focus) frame 401, which indicates the focus status of the ILC 20 (camera unit 210). The AF frame 401 is projected three-dimensionally onto the real space displayed on the monitor image. That is, the display position of the AF frame 401 on the monitor image changes according to the positional relationship between the ILC 20 and the subject. Furthermore, the display pattern of the AF frame 401 changes according to the distance and angle to the subject (angle in the normal direction of the mesh on the subject's surface).

[0055] Furthermore, various GUIs are arranged along the left, right, and top edges of the navigation screen. When the thumbnail 402 located in the middle of the right edge of the navigation screen is tapped, a preview screen is displayed for viewing images captured by the ILC20 so far. When the button 403 located in the lower right edge of the navigation screen is tapped, a confirmation screen is displayed for viewing a simplified three-dimensional model of the subject generated based on images captured by the ILC20 so far.

[0056] Furthermore, the UI on the navigation screen is not limited to the form shown in Figure 6, but can take various forms as will be described later.

[0057] <3. Shooting Flow for 3D Modeling> When shooting for 3D modeling using a shooting navigation system that combines a smartphone and ILC, it is necessary to generate a mesh of the subject to be 3D modeled, as explained with reference to the flowchart in Figure 5.

[0058] Figure 7 shows an example of the conventional 3D modeling workflow using a shooting navigation system that combines a smartphone and an ILC (International Linear Collider).

[0059] In conventional 3D modeling, a calibration process is performed to estimate the camera parameters of the smartphone and ILC, followed by the generation of a mesh of the subject to be 3D modeled. During mesh generation, shooting and mesh updates are repeatedly performed for each part of the subject. In terms of UI display, a score is calculated and displayed to evaluate the accuracy of the subject's 3D data as the mesh updates are repeated.

[0060] Thus, in conventional 3D modeling photography, the overall processing load becomes very high because a computationally intensive calibration process is repeatedly performed followed by a computationally intensive mesh update process.

[0061] Figure 8 shows an example of the shooting flow for 3D modeling according to this disclosure, which can be realized by a shooting navigation system that combines a smartphone and an ILC.

[0062] In the 3D modeling imaging described herein, first, a mesh is generated for the entire subject by scanning the entire subject to be 3D modeled (mesh scan), and then the ROI, which will be the imaging range of the ILC, is set (mesh crop).

[0063] Next, based on the set ROI, sparse shooting (shooting in the first mode) is performed by the ILC. This type of shooting is hereafter referred to as normal mode shooting. In normal mode shooting, calibration processing is performed by simultaneous shooting with the smartphone and the ILC. At this time, camera parameters are updated according to the parallax, and the shooting range of the ILC according to the parallax with the smartphone is visualized on the navigation screen.

[0064] After normal mode shooting is complete, detailed shooting (second mode shooting) is performed using the ILC. This type of shooting is hereafter referred to as detail mode shooting. In detail mode shooting, no calibration process is performed; only the ILC's shooting range, corresponding to the parallax with the smartphone, needs to be visualized on the navigation screen. In terms of UI display, a score for evaluating the accuracy of the subject's 3D data is calculated and displayed while normal mode shooting and detail mode shooting are taking place.

[0065] Thus, in the 3D modeling imaging process described herein, the processing load on subsequent stages can be significantly reduced by performing a high-processing-load mesh scan beforehand.

[0066] As explained with reference to Figure 8, the process of setting the ROI (Region of Interest) that constitutes the ILC's shooting range by scanning the entire subject for mesh generation, and then performing image acquisition by the ILC, is called "two-step shooting."

[0067] <4.2-Step Shooting> (Flow of 2-Step Shooting) Here, with reference to the flowchart in Figure 9, the flow of 2-step shooting performed by the shooting navigation system 1 to which the technology of this disclosure is applied will be explained. The process in Figure 9 is started, for example, when the operation unit 140 is operated while the navigation screen is displayed on the display unit 11 of the smartphone 10.

[0068] In step S101, the UI display unit 303 displays a size selection screen on the display unit 11 for selecting the size of the subject to be 3D modeled.

[0069] Figure 10 shows an example of a size selection screen.

[0070] In the example shown in Figure 10, the size selection screen 411 is superimposed on the navigation screen displayed on the display unit 11. On the size selection screen 411, you can select one of three sizes for the subject: "Small" (less than 1m square), "Medium" (approximately 4m square), or "Large / Area" (10m square or larger).

[0071] When the size of the subject is selected on the size selection screen, in step S102, the object recognition unit 302 performs a pre-scan to scan the entire subject in order to generate a mesh. The pre-scan is performed by the photographer pointing the smartphone 10 at the subject and taking a picture so that all surfaces of the subject are scanned. By performing the pre-scan, the object recognition unit 302 generates a mesh of the entire subject, as explained with reference to Figure 8.

[0072] Once a mesh of the entire subject is generated, in step S103, the object recognition unit 302 sets a region of interest (ROI) based on the generated mesh.

[0073] Figure 11 shows an example of a bounding box representing ROI.

[0074] In the example shown in Figure 11, a bounding box BX in the shape of a rectangular parallelepiped surrounding the subject SJ is projected three-dimensionally on the navigation screen (monitor image) displayed on the display unit 11. The photographer can manually adjust the shape and size of the bounding box BX.

[0075] Subsequently, in step S104, the shooting control unit 304 controls the ILC 20 to perform normal mode shooting, which involves sparse shooting using the ILC 20.

[0076] Figure 12 shows an example of the navigation screen when shooting in normal mode.

[0077] In the example shown in Figure 12, the navigation screen (monitor image) displayed on the display unit 11 shows that shooting is being performed by aligning the AF frame 401 with the subject SJ. In addition, a progress indicator 421 showing the progress of shooting in normal mode shooting is displayed in the upper center of the navigation screen. The number of shots in normal mode shooting is set according to, for example, the number of polygons that make up the mesh generated by pre-scan. In the example shown in Figure 12, the progress indicator 421 shows that 180 of the 351 shots performed in normal mode shooting have been completed. Note that each shot in normal mode shooting may be performed manually by the photographer, or it may be performed automatically according to the position and orientation of the ILC 20 relative to the polygons that make up the mesh.

[0078] For example, once all shooting in normal shooting mode is complete, in step S106, the shooting control unit 304 determines whether or not to switch the shooting mode to detail mode shooting, which performs detailed shooting using the ILC20, in response to an operation on the GUI displayed on the navigation screen.

[0079] If it is determined in step S106 to switch to detail mode shooting, the process proceeds to step S107, where the shooting control unit 304 controls the ILC 20 to perform detail mode shooting, which involves high-resolution shooting using the ILC 20.

[0080] On the other hand, if it is determined in step S106 that the system does not switch to detail mode shooting, step S107 is skipped and the process ends.

[0081] Figure 13 shows an example of the navigation screen when determining the shooting mode.

[0082] In the example shown in Figure 13, a pop-up 431 is displayed on the navigation screen (monitor image) shown on the display unit 11, which accepts a switch from normal mode shooting to detail mode shooting. The photographer can switch the shooting mode from normal mode shooting to detail mode shooting by operating the pop-up 431. Also in the example shown in Figure 13, a progress indicator 421 is displayed that shows the progress of shooting in detail mode shooting, indicating that 851 shots will be taken in detail mode shooting. Each shot in detail mode shooting may also be performed manually by the photographer, or it may be performed automatically according to the position and orientation of the ILC 20 relative to the polygons that make up the mesh.

[0083] Furthermore, the pop-up 431 can be displayed even if all shooting in normal shooting mode is not completed. In other words, even if all shooting in normal shooting mode is not completed, the shooting mode can be switched from normal shooting mode to detail shooting mode. Also, the pop-up 431 can be displayed even while detail shooting is in progress. In this case, the shooting mode can be switched from detail shooting mode to normal shooting mode.

[0084] As described above, in two-step imaging, performing a pre-scan, which has a high processing load, beforehand makes it possible to significantly reduce the processing load on the subsequent steps.

[0085] Furthermore, in two-step shooting, it is possible to switch from normal mode shooting to detail mode shooting. In other words, in shooting with the ILC20, in order to increase the success rate by taking advantage of the characteristics of SfM (Structure from Motion) processing used in photogrammetry and Novel View Synthesis, normal mode shooting allows the entire image to be connected and alignment and loop closing to be performed. This improves the stability of alignment. In detail mode shooting, where feature points tend to be lost due to close-ups, maintaining the matching with normal mode shooting ensures the success of alignment. If it is desired to finish with fewer shots, although the quality will be reduced, it is also possible to finish with only normal mode shooting.

[0086] (GUI in the navigation screen) Further forms of GUI in the navigation screen will be explained.

[0087] - Referring to the first example figure 14, we will explain the first example of the GUI in the navigation screen.

[0088] In the lower left corner of the navigation screen shown in Figure 14, there is a start button 441 for initiating the pre-scan, normal mode shooting, and detail mode shooting described above. When the start button 441 is tapped, its display changes to a stop button 441a for stopping these shooting modes.

[0089] In the upper right corner of the navigation screen shown in Figure 14, a thumbnail 442 is provided for viewing images taken in normal mode shooting or detail mode shooting. When the thumbnail 442 is tapped, a preview screen is displayed for viewing the latest image taken by the ILC20.

[0090] In the navigation screen shown in Figure 14, a mode indicator 443 is provided in the middle right corner, indicating the shooting mode being performed by the ILC 20. The display of the mode indicator 443 indicates that pre-scanning is being performed. The display changes to mode indicator 443a when automatic shooting is being performed in normal mode shooting or detail mode shooting, and to mode indicator 443b when manual shooting is being performed. Automatic shooting (automatic shooting) and manual shooting (manual shooting) can be switched by tapping mode indicator 443a or 443b.

[0091] In the lower right corner of the navigation screen shown in Figure 14, there is a confirmation button 444 for saving images taken in normal mode or detail mode. When the confirmation button 444 is tapped, its display changes to a progress display button 444a that shows the progress (progress rate) of normal mode shooting or detail mode shooting, and the captured image is saved. Also, when pre-scanning is being performed, the confirmation button 444 is replaced with an exit button 444b that ends the pre-scan without saving the data.

[0092] A second example of the GUI in the navigation screen will be explained with reference to Figures 15 and 16.

[0093] Figure 15 shows an example of the GUI on the navigation screen during pre-scan execution.

[0094] A start button 451 for initiating a pre-scan is provided at the bottom left of the navigation screen shown in Figure 15. When the start button 451 is tapped, its display changes to a stop button 451a for stopping the pre-scan.

[0095] A mode indicator 452 is provided in the upper right corner of the navigation screen shown in Figure 15 to indicate the scan mode. The display of the mode indicator 452 indicates that a pre-scan is being performed. When the mode indicator 452 is tapped, its display changes to mode indicator 452a, which indicates that a more detailed scan than the pre-scan is being performed, and a more detailed scan than the pre-scan is started.

[0096] Figure 16 shows an example of the GUI on the navigation screen when shooting in normal mode or detail mode.

[0097] In the navigation screen shown in Figure 16, a setting button 461 is provided at the bottom left to select whether to perform manual or automatic shooting. The display of the setting button 461 indicates that manual shooting is in progress. When the setting button 461 is tapped, its display changes to a setting button 461a indicating that automatic shooting is in progress, and automatic shooting begins.

[0098] A mode indicator 462 is provided in the middle right of the navigation screen shown in Figure 16 to indicate the shooting mode. The display of the mode indicator 462 indicates that normal mode shooting is being performed. When the mode indicator 462 is tapped, its display changes to mode indicator 462a, which indicates that detail mode shooting is being performed, and detail mode shooting begins.

[0099] In the shooting navigation system 1, various processes in two-step shooting become possible by operating the GUI of the navigation screen as described above.

[0100] (Shooting Quality) In normal mode shooting and detail mode shooting in two-step shooting, the quality of shooting by ILC20 in the area of ​​interest is displayed on the navigation screen. The quality of shooting by ILC20 may be determined based on a scoring result that evaluates the accuracy of 3D data that can be generated using images obtained from previously performed shooting, for example, as disclosed in Patent Document 1.

[0101] The quality of images captured by the ILC20 may be expressed, for example, by the difference in color added to the ROI (subject) area on the navigation screen (monitor image).

[0102] Furthermore, the image quality obtained by the ILC20 may be represented by a needle-shaped image as shown in Figure 17.

[0103] In the navigation screen shown in Figure 17, a needle-shaped image p1 is displayed at the intersection of the optical axis of the ILC20 lens and the mesh for each subject shown in the monitor image, indicating the optical axis direction of the ILC20 lens. In this case, the thickness of the needle, or the color and length of the needle may be changed, as shown in the needle-shaped image p2, so that the quality of the shooting can be checked, such as if there was camera shake during shooting. Alternatively, the needle-shaped image p1 may be displayed for all meshes regardless of whether shooting is complete or not, and the color of the needle-shaped image p1 may be changed to indicate whether shooting is complete or not.

[0104] (Overhead view) During shooting with ILC20, it may be possible to check the range of ROI (subject) that has been photographed so far from an overhead view.

[0105] Figure 18 shows an example of an overhead view displayed on the navigation screen.

[0106] In the navigation screen shown in Figure 18, a three-dimensional model (MD) of the subject is displayed in a virtual space viewed from an overhead (bird's-eye) perspective. In addition, thumbnail images (CP) representing images previously captured by the ILC20 are displayed in the virtual space shown on the navigation screen, at positions corresponding to the capture locations of those images.

[0107] Displaying an overhead view in such a virtual space can be achieved by simply generating and constructing a three-dimensional model of the environment, including the subject, based on images previously captured by ILC20, and by recording the position and orientation at the time of each capture.

[0108] Furthermore, in the virtual space viewed from an overhead perspective, as shown in Figure 18, the quality of the shots according to the scoring results described above, as well as whether or not there were any missed shots, may be represented by differences in color.

[0109] (Shooting progress) In normal mode shooting and detail mode shooting in two-step shooting, the progress of shooting by ILC20 in the area of ​​interest is displayed on the navigation screen. The shooting progress may be displayed in the form of "(number of completed shots) / (total number of shots)" as described above, or it may be shown as a progress rate.

[0110] The progress rate is calculated as the percentage of polygons that make up the entire mesh generated by pre-scanning and ROI settings, where the score representing the quality of the image assigned to each polygon has reached a certain level or higher, as shown in the following formula, for example.

[0111]

[0112] The progress rate is not limited to this; it may also be calculated by determining the projection area when images are taken at an appropriate distance in advance, and then determining the ratio of this area to the total area of ​​the generated mesh, based on the number of images taken.

[0113] (ROI setting) In two-step imaging, after pre-scanning (mesh generation), ROI setting is performed to allow the operator to more accurately define the scanning range (the range for generating three-dimensional information). This eliminates the need to manually specify the scanning range again when generating three-dimensional information.

[0114] The ROI (Route of Interest) can be set manually by editing the vertices, edges, and faces of the bounding box, or it can be set automatically.

[0115] One possible method for automatically setting the ROI is to set it based on the position and orientation of the ILC20 and the distance to the subject.

[0116] Figure 19 is a diagram illustrating the automatic setting of ROI. Figure 19A shows a method of setting the ROI by scanning the subject from the outside inward (Outside-In method). Figure 19B shows a method of setting the ROI by scanning the subject from the inside outward (Inside-Out method).

[0117] During imaging with the ILC20, the position and orientation are acquired by the IMU110 and camera unit130 for imaging navigation, and the distance to the subject is acquired by the depth sensor120. Therefore, the field of view is estimated from the information of the lens mounted on the ILC20, and an AABB (Axis-Aligned Bounding Box) is set to encompass that range. This algorithm may be applied only to the XY planes that constitute the imaging surface, and for the Z-axis direction (depth direction), where changes due to movement are relatively small, the AABB may be set based on the minimum / maximum values ​​of the generated mesh.

[0118] (Appropriateness of distance to subject) In normal mode shooting and detail mode shooting in two-step shooting, the navigation screen may display whether the distance to the subject in the area of ​​focus is appropriate for shooting with the ILC20. This will allow the subject to be photographed at the optimal distance, thereby improving the quality of the image.

[0119] Now, referring to Figure 20, we will explain the relationship between the distance to the subject and the resolution.

[0120] As shown in Figure 20, let p [pix] be the effective number of pixels (vertical) of the ILC20 image sensor, s [mm] be the image sensor size, f be the focal length, and d be the distance to the subject. In this case, the length L of the projection surface when projecting a subject of height L to fill the entire height of the screen is expressed by the following formula.

[0121] L = s * d / f

[0122] In this case, the resolution r of the captured image is expressed by the following formula.

[0123] r = p / L [pix / mm]

[0124] Based on the above relationship, once the desired resolution r is determined, it becomes possible to determine the distance d to the subject.

[0125] Furthermore, whether or not the distance d to the subject is a pre-set appropriate shooting distance is indicated, for example, by the display format of the AF frame 401.

[0126] Figure 21 shows examples of display modes for the AF frame 401. Figure 21 shows five examples of display modes for the AF frame 401 (AF frames 401a to 401e) according to the distance from the ILC 20 to the subject and the angle to the subject.

[0127] The AF frame 401 consists of hook-shaped frame sections positioned at the four corners of a rectangle, and rectangular areas drawn along these frame sections. The frame sections represent the focus state of the ILC 20. The rectangular areas represent the appropriateness of the distance to the subject and the angle to the subject. The size of the rectangular areas changes depending on whether the distance to the subject is an appropriate shooting distance. Furthermore, the shape of the rectangular areas changes depending on the angle between the normal direction of the mesh surface of the subject and the shooting direction of the ILC 20.

[0128] The display mode of the AF frame 401a indicates that the ILC20 is not in focus, but the distance to the subject and the angle to the subject are appropriate. When the distance to the subject and the angle to the subject are appropriate, the size of the rectangular area matches the frame, and the color of the rectangular area changes.

[0129] The display pattern of the AF frame 401b indicates that the ILC20 is in focus and that the distance to the subject and the angle to the subject are appropriate. When the ILC20 is in focus, the color of the frame changes.

[0130] Furthermore, as shown in AF frame 401a and AF frame 401b, when the distance to the subject and the angle to the subject are appropriate, not only may the color of the rectangular area change, but a circular image may also be added to its center. This can improve accessibility for the photographer.

[0131] The display pattern of AF frame 401c indicates that the ILC20 is not in focus and the angle to the subject is not appropriate. When the angle to the subject is not appropriate, the shape of the rectangular area changes as if it is tilted according to the angle to the subject.

[0132] The display pattern of AF frame 401d indicates that the ILC20 is not in focus and the subject is too far away. When the subject is too far away, the size of the rectangular area changes to be smaller than the frame area, depending on the distance to the subject.

[0133] The display pattern of AF frame 401e indicates that the ILC20 is not in focus and the distance to the subject is too close. When the distance to the subject is too close, the size of the rectangular area changes significantly compared to the frame area, depending on the distance to the subject.

[0134] The AF frame 401 indicates the focus position of the ILC20, and on the navigation screen, it is displayed corrected to the position as seen from the smartphone 10 according to the calibrated parallax. Therefore, the AF frame 401 is not generally displayed in the center of the navigation screen.

[0135] <5. UI Layout> (GUI layout on the navigation screen) Refer to Figure 22 to explain the GUI layout on the navigation screen.

[0136] The UI display unit 303 places GUIs in the display unit 11 in areas that do not overlap with the shooting range of the ILC 20 included in the monitor image. Specifically, as shown in Figure 22, in the navigation screen (monitor image) displayed on the display unit 11 of the smartphone 10, various GUIs are placed in the left edge area 511, the right edge area 512, and the upper edge area 513.

[0137] As explained with reference to Figures 1 and 2, when a smartphone 10 is attached to the top of the ILC 20, the shooting range of the ILC 20 is positioned towards the lower part of the navigation screen displayed on the display unit 11 of the smartphone 10. For example, the AF frame 401 mentioned above is displayed below the center of the navigation screen. Therefore, by not placing the GUI in the lower part of the navigation screen (monitor image), it is possible to prevent the AF frame 401 and the shooting range of the ILC 20 themselves from becoming difficult to see.

[0138] Furthermore, in the navigation screen (monitor image) displayed on the display unit 11, a GUI that accepts settings that are frequently changed during shooting with the ILC 20 is displayed in the area furthest from the gripping unit 22 of the ILC 20, i.e., the leftmost area 511. The GUI displayed in the leftmost area 511 is a GUI that accepts settings that the user wants to change immediately during shooting. This allows the photographer, who is holding the gripping unit 22 with their right hand, to operate on settings that they want to change immediately with their left hand, which is not holding the ILC 20.

[0139] On the other hand, in the navigation screen (monitor image) displayed on the display unit 11, the area closest to the gripping part 22 of the ILC 20, i.e., the rightmost area 512, displays a GUI that accepts settings that are not frequently changed during shooting with the ILC 20. The GUI displayed in the rightmost area 512 is a GUI that accepts settings that need to be carefully checked during shooting, such as a confirmation button. This makes it more difficult for the photographer, who is holding the gripping part 22 with their right hand, to operate settings that need to be carefully checked, thus naturally encouraging them to operate cautiously.

[0140] Furthermore, the upper edge area 513 of the navigation screen (monitor image) displayed on the display unit 11 should display various status information and other information that does not require the photographer to operate, with its position fixed regardless of screen transitions.

[0141] (Physical buttons on the ILC) As described above, the ILC 20 is provided with physical buttons such as a shutter button and custom buttons as an operation unit 23, which can be operated by the photographer while holding the grip unit 22. In the shooting navigation system 1 to which the technology of this disclosure is applied, the operation of the smartphone 10 can be controlled in response to the operation of the physical buttons on the ILC 20.

[0142] Figure 23 is a flowchart illustrating the shooting control process of the smartphone 10.

[0143] In step S211, the shooting control unit 304 determines whether or not a physical button of the ILC 20 (for example, a custom button or a shutter button) has been operated, based on information from the processing unit 230 of the ILC 20.

[0144] If it is determined that a physical button on the ILC 20 has been operated, in step S212, the shooting control unit 304 controls the shooting by the camera unit 130 of the smartphone 10. The shooting by the camera unit 130 may be the acquisition of a still image or the acquisition of three-dimensional information. If it is not determined that a physical button on the ILC 20 has been operated, step S212 is skipped.

[0145] With this type of processing, the photographer can operate the smartphone 10 while holding the ILC 20, or synchronize the operation of the ILC 20 with the operation of the smartphone 10.

[0146] Figure 24 is a flowchart explaining the process of indicating the timing of the photo shoot.

[0147] In step S221, the UI display unit 303 determines whether or not a physical button of the ILC 20 (for example, a shutter button) has been operated, based on information from the processing unit 230 of the ILC 20.

[0148] If it is determined that the physical button on the ILC20 has been operated, in step S222, the UI display unit 303 displays an effect on the navigation screen (monitor image) shown on the display unit 11 of the smartphone 10 indicating the timing of shooting by the ILC20. If it is not determined that the physical button on the ILC20 has been operated, step S222 is skipped.

[0149] This processing allows the photographer to easily recognize the timing of shooting with the ILC20 while checking the navigation screen.

[0150] <6. Simultaneous Shooting and Calibration> As mentioned above, in normal mode shooting, calibration processing is performed by simultaneous shooting with the smartphone 10 and the ILC20.

[0151] Here, referring to the flowchart in Figure 25, we will explain the simultaneous shooting process using the smartphone 10 and the ILC 20. The process in Figure 25 starts when automatic shooting is set to be performed in normal mode shooting.

[0152] In step S311, the shooting control unit 304 determines whether or not the change in the position and orientation of the ILC 20 has stopped, based on the position and orientation information calculated by the position and orientation calculation unit 301.

[0153] If it is determined that the change in the position and orientation of the ILC 20 has stopped, that is, when the ILC 20 is stationary, the process proceeds to step S312, where the shooting control unit 304 controls simultaneous shooting by the camera unit 130 of the smartphone 10 and the ILC 20 (camera unit 210). If it is not determined that the change in the position and orientation of the ILC 20 has stopped, step S312 is skipped.

[0154] This process makes it possible to achieve simultaneous shooting between the smartphone 10 and the ILC20 without any delay in shutter timing, and without camera shake caused by operating the shutter button.

[0155] Next, referring to the flowchart in Figure 26, we will explain the calibration process performed based on the simultaneous shooting process by the smartphone 10 and the ILC 20.

[0156] Steps S321 to S322 in the flowchart of Figure 26 are the same as steps S311 to S312 (simultaneous shooting process) in the flowchart of Figure 25, so their explanation will be omitted.

[0157] In other words, in step S323, the calibration processing unit 305 performs a calibration process to estimate the camera parameters of the camera unit 130 and the ILC 20 using the images captured by the simultaneous shooting process, namely the image captured by the camera unit 130 (first image) and the image captured by the ILC 20 (second image).

[0158] This type of processing allows for calibration using images without shutter timing discrepancies, thereby improving the estimation accuracy of intrinsic and extrinsic parameters.

[0159] The calibration process described above is performed during normal mode shooting. The images used for the calibration process may also be used to generate a simplified three-dimensional model.

[0160] Here, referring to the flowchart in Figure 27, we will explain the three-dimensional model generation process performed based on the calibration process.

[0161] Note that steps S331 to S333 in the flowchart of Figure 27 are the same as steps S321 to S323 (calibration process) in the flowchart of Figure 26, so their explanation will be omitted.

[0162] In other words, in step S334, the three-dimensional model generation unit 306 generates a three-dimensional model (three-dimensional information) of the subject based on one of the images used in the calibration process, namely the image captured by the camera unit 130 or the image captured by the ILC 20.

[0163] This process generates a three-dimensional model during normal mode shooting, making it possible to display the three-dimensional shape of the subject from an overhead perspective on the navigation screen, as explained with reference to Figure 18, for example.

[0164] The calibration process described above is performed each time the simultaneous shooting process is executed. The camera parameters may be updated each time the calibration process is performed.

[0165] Now, referring to the flowchart in Figure 28, we will explain the camera parameter update process that is executed each time the simultaneous shooting process is performed.

[0166] Steps S341 to S343 in the flowchart of Figure 28 are the same as steps S321 to S323 (calibration process) in the flowchart of Figure 26, so their explanation will be omitted.

[0167] In other words, in step S344, the calibration processing unit 305 updates the camera parameters by performing a calibration process for each simultaneous image capture.

[0168] With this process, each time a shot is taken, the camera parameters of the camera unit 130 and the ILC 20 are updated, and the positional relationship between the camera unit 130 and the ILC 20 is corrected. As a result, the shooting range of the ILC 20 can be displayed more accurately on the navigation screen.

[0169] As described above, by performing calibration shooting simultaneously with the smartphone 10 and ILC20 during normal mode shooting, calibration can be achieved without the need for a chart, thus minimizing the burden on the photographer.

[0170] Furthermore, since the calibration process is performed during normal mode shooting, it may be difficult for the photographer to grasp the status of the calibration process. Therefore, for example, as shown in Figure 29, status information 611 indicating the success or failure of the calibration process, or status information (not shown) indicating the progress, may be displayed in the upper area of ​​the navigation screen displayed on the display unit 11. Alternatively, a virtual frustum representing the shooting posture information applied to the calibration process may be projected three-dimensionally within the navigation screen (monitor image), changing color depending on the success or failure of the calibration process.

[0171] <7. Tactile Presentation> In the shooting navigation system 1, tactile presentation may be provided via the other smartphone 10 or ILC 20 by vibrating one of them.

[0172] Figure 30 is a flowchart illustrating the haptic feedback processing in the shooting navigation system 1. The processing in Figure 30 is performed when the navigation screen is displayed on the display unit 11 of the smartphone 10, such as during the pre-scan, normal mode shooting, or detail mode shooting described above.

[0173] In step S411, the UI display unit 303 determines whether the state of the smartphone 10 has changed. The state of the smartphone 10 here includes the display state of the navigation screen, the position and orientation of the smartphone 10, and the internal state of the smartphone 10.

[0174] If it is determined that the state of the smartphone 10 has changed, the process proceeds to step S412, where the UI display unit 303 vibrates the smartphone 10 to provide haptic feedback via the ILC 20. If it is not determined that the state of the smartphone 10 has changed, step S412 is skipped.

[0175] This processing allows the photographer holding the ILC20 to understand changes in the status of the smartphone 10 without looking at the navigation screen. For example, even if the photographer is performing shooting operations without looking at the navigation screen, they can know the timing of shooting by the ILC20 solely through touch, without needing to equip the ILC20 with a vibration element.

[0176] In the above description, haptic feedback is provided via the ILC 20 by vibrating the smartphone 10. However, haptic feedback may also be provided via the smartphone 10 by vibrating the ILC 20 in response to changes in the state of the ILC 20.

[0177] <8. Description of a computer to which the technology relating to this disclosure is applied> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, the computer includes computers that are built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0178] Figure 31 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program.

[0179] In a computer, the processing circuit 901, ROM (Read Only Memory) 902, and RAM (Random Access Memory) 903 are interconnected by a bus 904.

[0180] An input / output interface 905 is further connected to the bus 904. An input unit 906, an output unit 907, a storage unit 908, a communication unit 909, and a drive 910 are connected to the input / output interface 905.

[0181] The input unit 906 may include physical or virtual operating means that the user operates to input information, such as a keyboard, mouse, or touch panel, as well as means that the user inputs information through voice, eye gaze, etc. Furthermore, the input unit 906 may include sensors for inputting various physical quantities to the computer. For example, the input unit 906 may include sensors that acquire physical quantities such as light (including infrared light other than visible light) or sound, such as a camera or microphone. Also, for example, the input unit 906 may include sensors that acquire other physical quantities such as temperature, moisture content, acceleration, distance, etc. The output unit 907 may include means that present information to the user by stimulating the user's perception, such as a display, speaker, or haptic device. The storage unit 908 is composed of a hard disk, non-volatile or volatile memory, etc., and stores various types of information (including programs). The communication unit 909 is a network interface, etc., and performs wired or wireless communication with the outside. The drive 910 drives removable media 911 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0182] The processing circuit 901 includes a processor that executes programs such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The processing circuit 901 (its processor) performs the above-described series of processes by loading the program stored in the storage unit 908 into the RAM 903 via the input / output interface 905 and the bus 904 and executing it. The processing circuit 901 can output the processing results of the series of processes from the output unit 907 via the bus 904 and the input / output interface 905 as needed. The processing circuit 901 can also store the processing results in the storage unit 908 or transmit them from the communication unit 909.

[0183] The program executed by the computer (processing circuit 901) can be provided by recording it on a removable medium 911, such as a package medium. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0184] In a computer, a program can be installed in the storage unit 908 via the input / output interface 905 by inserting a removable media 911 into the drive 910. Alternatively, a program can be received by the communication unit 909 from another device, such as a server, via a wired or wireless transmission medium, and installed in the storage unit 908. Furthermore, programs can be pre-installed in the ROM 902 or the storage unit 908.

[0185] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0186] The processes that a computer performs according to a program do not necessarily have to follow the order described in the flowchart. In other words, the processes that a computer performs according to a program include processes that are executed in parallel or individually (e.g., parallel processing and object-based processing).

[0187] The program may be processed by a single computer (processor), or it may be processed in a distributed manner by multiple computers. Furthermore, the program may be transferred to a remote computer and executed there.

[0188] When the computer executes a program to perform the above-described series of processes, the input unit 906 functions as a sensor group including the camera unit 130, the processing circuit 901 functions as a processing unit 150 that realizes each functional block by executing the program, and the output unit 907 functions as a display unit 11.

[0189] In this specification, a system means one component or a collection of multiple components (devices, modules (parts), etc.). Therefore, one or more components of a computer, for example, only the processor, or a combination of the processor and memory (for example, only the processing circuit 901, or a combination of the processing circuit 901 to the bus 904, etc.), constitute a system. Regarding a collection of multiple components, it is not necessary whether all components reside in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, or a single device containing multiple modules within a single enclosure, are all systems. Furthermore, for example, the entire computer, or a combination of a computer and other devices such as a server (not shown), also constitute a system.

[0190] The components (blocks) of the apparatus illustrated in this specification are functional conceptual blocks, and the actual apparatus does not need to have the illustrated configuration. That is, the apparatus can have any configuration in which the functions of the illustrated components are divided and / or integrated into any unit, for example, a configuration having one block in which the functions of all components are integrated.

[0191] The embodiments of the technology relating to this disclosure are not limited to those described above, and various modifications are possible without departing from the gist of the technology relating to this disclosure. Furthermore, the embodiments of the technology relating to this disclosure may be a combination of at least one of the embodiments described above.

[0192] The effects described herein are merely illustrative and not limited to those described herein; other effects may also occur.

[0193] Furthermore, the technology relating to this disclosure can take the following configurations: (1) An information processing system comprising: an information processing terminal equipped with a sensor group including a camera unit and a display unit; a shooting device connected to the information processing terminal and performing image acquisition processing for generating three-dimensional information of a subject; and a processing circuit that displays on the display unit an image including the shooting range of the shooting device, which is a monitor image captured by the camera unit, and presents a UI for assisting the shooting as the image acquisition processing by the shooting device. (2) The information processing system according to (1), wherein the processing circuit presents the UI indicating the shooting range of the shooting device based on a region of interest set by scanning the subject. (3) The information processing system according to (2), wherein the processing circuit presents the quality of shooting by the shooting device in the region of interest. (4) The information processing system according to (2) or (3), wherein the processing circuit presents the progress of shooting by the shooting device in the region of interest. (5) The information processing system according to any one of (2) to (4), wherein the region of interest is used to determine the range of the generation of the three-dimensional information. (6) The information processing system according to any one of (2) to (5), wherein the processing circuit indicates whether the distance to the subject in the shooting by the shooting device is appropriate based on the area of ​​interest. (7) The information processing system according to any one of (2) to (6), wherein the processing circuit accepts a switch from a first mode of shooting in which the shooting device performs sparse shooting to a second mode of shooting in which the shooting device performs dense shooting as the image acquisition process. (8) The information processing system according to (1), wherein the processing circuit places a GUI in the display unit in an area that does not overlap with the shooting range of the shooting device included in the monitor image. (9) The information processing system according to (8), wherein the shooting device includes a gripping part that is held in one hand, and the processing circuit displays the GUI that accepts settings that are frequently changed during shooting by the shooting device in the area on the display unit that is farther from the gripping part.(10) The information processing system according to (8) or (9), wherein the imaging device comprises a gripping part that can be held with one hand, and the processing circuit displays the GUI that accepts settings that are not changed frequently during imaging by the imaging device in the area on the side of the display unit closest to the gripping part. (11) The information processing system according to any one of (8) to (10), wherein the imaging device comprises a gripping part that can be held with one hand and an operating part that can be operated while the gripping part is being held, and the processing circuit controls imaging by the camera unit in response to operations on the operating part. (12) The information processing system according to any one of (8) to (11), wherein the imaging device comprises a gripping part that can be held with one hand and an operating part that can be operated while the gripping part is being held, and the processing circuit displays an effect on the display unit indicating the timing of imaging by the imaging device in response to operations on the operating part. (13) The information processing system according to (1), wherein the processing circuit controls simultaneous shooting by the camera unit and the shooting device in response to the cessation of changes in the position and orientation of the shooting device. (14) The information processing system according to (13), wherein the processing circuit performs calibration processing to estimate the camera parameters of the camera unit and the shooting device using a first image captured by the camera unit and a second image captured by the shooting device. (15) The information processing system according to (14), wherein the processing circuit generates the three-dimensional information based on the first image or the second image. (16) The information processing system according to (14), wherein the processing circuit updates the camera parameters by performing the calibration processing each time the simultaneous shooting is performed. (17) The information processing system according to (1), wherein the processing circuit provides tactile feedback via the other of the information processing terminal and the shooting device by vibrating one of the information processing terminal and the shooting device. (18) The information processing system according to (17), wherein the processing circuit provides the tactile presentation via the other of the information processing terminal and the imaging device in response to a change in the state of one of the information processing terminal and the imaging device.(19) An information processing system comprising an information processing terminal having a sensor group including a camera unit and a display unit, and a shooting device connected to the information processing terminal and performing image acquisition processing for generating three-dimensional information of a subject, wherein the information processing system displays on the display unit an image including the shooting range of the shooting device, which is a monitor image captured by the camera unit, and presents a UI for assisting the shooting as the image acquisition processing by the shooting device. (20) An information processing system comprising an information processing terminal having a sensor group including a camera unit and a display unit, and a processing circuit that displays on the display unit an image including the shooting range of a shooting device connected to the information processing terminal and performing image acquisition processing for generating three-dimensional information of a subject, which is a monitor image captured by the camera unit, and presents a UI for assisting the shooting as the image acquisition processing by the shooting device.

[0194] 1 Shooting navigation system, 10 Smartphone, 11 Display unit, 20 ILC, 21 Display unit, 22 Gripping unit, 23 Operation unit, 110 IMU, 120 Depth sensor, 130 Camera unit, 140 Operation unit, 210 Camera unit, 301 Position and orientation information calculation unit, 302 Object recognition unit, 303 UI presentation unit, 304 Shooting control unit, 305 Calibration processing unit, 306 Three-dimensional model generation unit

Claims

An information processing terminal equipped with a sensor group including a camera unit and a display unit, A shooting device connected to the aforementioned information processing terminal, which performs image acquisition processing to generate three-dimensional information of the subject, A processing circuit that displays an image including the shooting range of the shooting device, which is captured by the camera unit, on the display unit, and also presents a UI to assist in the shooting process as the image acquisition process by the shooting device. An information processing system that includes this. 　 The processing circuit presents the UI indicating the shooting range of the shooting device based on the area of ​​interest set by scanning the subject. The information processing system according to claim 1. 　 The processing circuit presents the quality of the image taken by the imaging device in the region of interest. The information processing system according to claim 2. 　 The processing circuit displays the progress of imaging by the imaging device in the area of ​​interest. The information processing system according to claim 2. 　 The aforementioned region of interest is used to determine the range of the three-dimensional information generation. The information processing system according to claim 2. 　 The processing circuit indicates, based on the area of ​​interest, whether the distance to the subject in the imaging device is appropriate or not. The information processing system according to claim 2. 　 The processing circuit receives a switch from a first mode of shooting, in which the shooting device performs sparse shooting, to a second mode of shooting, in which the shooting device performs dense shooting, as part of the image acquisition process. The information processing system according to claim 2. 　 The processing circuit places the GUI in the display unit in an area that does not overlap with the shooting range of the shooting device included in the monitor image. The information processing system according to claim 1. 　 The aforementioned photographic device is equipped with a gripping part that can be held in one hand, The processing circuit displays the GUI that accepts settings that are frequently changed during shooting by the shooting device in the area of ​​the display unit furthest from the gripping unit. The information processing system according to claim 8. 　 The aforementioned photographic device is equipped with a gripping part that can be held in one hand, The processing circuit displays the GUI in the area of ​​the display unit closest to the gripping portion, which accepts settings that are not frequently changed during shooting by the shooting device. The information processing system according to claim 8. 　 The aforementioned photographic device comprises a gripping part that is held in one hand, and an operating part that can be operated while the gripping part is being held. The processing circuit controls the camera unit to take pictures in response to operations performed on the operation unit. The information processing system according to claim 8. 　 The aforementioned photographic device comprises a gripping part that is held in one hand, and an operating part that can be operated while the gripping part is being held. The processing circuit, in response to an operation on the operation unit, displays an effect on the display unit indicating the timing of the shooting by the shooting device. The information processing system according to claim 8. 　 The processing circuit controls simultaneous shooting by the camera unit and the shooting device in response to the cessation of changes in the position and orientation of the shooting device. The information processing system according to claim 1. 　 The processing circuit performs a calibration process to estimate the camera parameters of the camera unit and the imaging device using the first image captured by the camera unit and the second image captured by the imaging device. The information processing system according to claim 13. 　 The processing circuit generates the three-dimensional information based on the first image or the second image. The information processing system according to claim 14. 　 The processing circuit updates the camera parameters by performing the calibration process each time the simultaneous shooting is performed. The information processing system according to claim 14. 　 The processing circuit vibrates one of the information processing terminal and the imaging device, thereby providing tactile feedback via the other of the information processing terminal and the imaging device. The information processing system according to claim 1. 　 The processing circuit provides tactile feedback via the other information processing terminal or the imaging device in response to a change in the state of one of the two devices. The information processing system according to claim 17. 　 An information processing terminal equipped with a sensor group including a camera unit and a display unit, An information processing system including an imaging device connected to the aforementioned information processing terminal and which performs image acquisition processing to generate three-dimensional information of a subject, The display unit displays an image including the shooting range of the aforementioned shooting device, which is captured by the camera unit, and also presents a UI to assist in the shooting process as an image acquisition process by the shooting device. Information processing methods. An information processing terminal equipped with a sensor group including a camera unit and a display unit, A processing circuit that displays a monitor image captured by the camera unit on the display unit, and presents a UI to assist in the image acquisition process performed by the shooting device, which is connected to the information processing terminal and performs image acquisition processing to generate three-dimensional information of a subject. An information processing system that includes this.

Images