Information processing device, information processing method, and program

Abstract

An information processing device according to an embodiment of the present technology includes an image acquisition unit, a viewpoint image estimation unit, and a determination unit. The image acquisition unit acquires a first image, which is a captured image of a target object in a real space, and a second image, which is a captured image of the target object and has a higher quality than the first image. The viewpoint image estimation unit estimates a viewpoint image, which is an image of the target object as captured from the current viewpoint, on the basis of the acquired first image. The determination unit determines whether or not it is necessary to acquire the second image, on the basis of the difference between the currently acquired first image and the estimated viewpoint image.

Description

Information Processing Apparatus, Information Processing Method, and Program

[0001] The present technology relates to an information processing apparatus, an information processing method, and a program applicable to the generation of a 3D model.

[0002] Patent Document 1 discloses a learning apparatus used for generating a 3D model. In this learning apparatus, only a part obtained by removing an area located farther than a predetermined background point from each captured image is used for generating the 3D model. As a result, the process of removing the background from the captured image becomes unnecessary.

[0003] Japanese Patent No. 7360757

[0004] In such a field, a technology that enables efficient generation of a 3D model is required.

[0005] In view of the above circumstances, an object of the present technology is to provide an information processing apparatus, an information processing method, and a program that enable efficient generation of a 3D model.

[0006] To achieve the above object, an information processing apparatus according to one embodiment of the present technology includes an image acquisition unit, a viewpoint image estimation unit, and a determination unit. The image acquisition unit acquires a first image that is a captured image of an object in real space and a second image that is a captured image of the object and is of higher quality than the first image. The viewpoint image estimation unit estimates a viewpoint image that is an image of the object captured from the current viewpoint based on the acquired first image. The determination unit determines whether it is necessary to acquire the second image based on the difference between the currently acquired first image and the estimated viewpoint image.

[0007] In this information processing apparatus, a viewpoint image is estimated based on the first image, and it is determined whether it is necessary to acquire a second image that is of higher quality than the first image based on the difference between the first image and the viewpoint image. As a result, it becomes possible to efficiently generate a 3D model.

[0008] The viewpoint image estimation unit may estimate the viewpoint image using a NeRF (Neural Radiance Fields) model or a GS (Gaussian Splatting) model.

[0009] The viewpoint image estimation unit may perform machine learning on the NeRF model or the GS model using the acquired second image as ground truth data.

[0010] The viewpoint image estimation unit may estimate the current viewpoint based on the acquired first image.

[0011] The viewpoint image estimation unit may generate a 3D model of the object based on the acquired second image.

[0012] The viewpoint image estimation unit may estimate the viewpoint image based on the current viewpoint and the generated 3D model.

[0013] The information processing device may further include a guidance unit that guides the user taking a photograph of the object to manipulate the field of view of the first image so that the determination unit determines that it is necessary to acquire the second image.

[0014] The guidance unit may highlight areas on the first image where the difference between the first image and the viewpoint image is large.

[0015] The guidance unit may superimpose arrows on the first image to guide the operation of the field of view.

[0016] The determination unit may determine that it is necessary to acquire the second image if the difference is greater than a predetermined threshold, and may accept a setting regarding the threshold by the user taking the photograph of the object.

[0017] The information processing device may further include an imaging unit that performs imaging of the object. In this case, the image acquisition unit may acquire the image from the imaging unit.

[0018] If the determination unit determines that it is necessary to acquire the second image, it may transmit a signal to the imaging unit instructing it to acquire the second image.

[0019] An information processing method according to one embodiment of this technology includes acquiring a first image, which is an image of an object in real space. Based on the acquired first image, a viewpoint image is estimated, which is an image of the object taken from the current viewpoint. Based on the difference between the currently acquired first image and the estimated viewpoint image, it is determined whether or not it is necessary to acquire a second image, which is an image of the object and of higher quality than the first image.

[0020] A program according to one embodiment of the present invention causes a computer system to perform the following steps: acquiring a first image, which is an image of an object in real space; estimating a viewpoint image, which is an image of the object taken from the current viewpoint, based on the acquired first image; and determining whether it is necessary to acquire a second image, which is an image of the object of higher quality than the first image, based on the difference between the currently acquired first image and the estimated viewpoint image.

[0021] This is a schematic diagram showing an overview of this technology. This is a schematic diagram showing an example configuration of an information processing device related to this technology. This is a flowchart showing the processing content of this technology. This is a schematic diagram showing the content of viewpoint image generation. This is a schematic diagram showing an overview of difference images. This is a schematic diagram showing the content of retraining. This is a schematic diagram showing an example of guidance by the guidance unit. This is a schematic diagram showing the content of guidance to an uncaptured location. This is a schematic diagram showing an example configuration related to guidance to an uncaptured location. This is a schematic diagram showing an example configuration according to an embodiment using only edge devices. This is a schematic diagram showing an example configuration according to an embodiment in video production. This is a block diagram showing an example of a computer hardware configuration that can realize edge devices, cloud, and remote devices.

[0022] The embodiments of this technology will be described below with reference to the drawings. This technology is mainly used in the field of photogrammetry. Photogrammetry is a technology that uses images to measure objects, and by using photogrammetry, it is possible to generate a 3D model of an object that exists in real space.

[0023] Specifically, the user takes multiple images of the object, and a 3D model of the object is generated based on these images. When creating 3D models used in metaverses, etc., it is possible for the user to draw the 3D model themselves using specialized software, but this requires advanced skills and is often inefficient in terms of time and cost. On the other hand, photogrammetry makes it possible to generate 3D models using a relatively simple method of photographing objects that exist in real space.

[0024] This technology may be applied to fields other than photogrammetry. In other words, this technology may be applied to any field, not just photogrammetry.

[0025] Figure 1 is a schematic diagram illustrating the overview of this technology. As shown in the upper left of Figure 1, in this technology, user 1 wants to obtain a 3D model 29 of an object 2 in real space, and a camera 3 captures a still image of the object 2. In this example, an architectural building is schematically shown as the object 2, but it is not limited to this, and the object 2 can be any object such as a small object.

[0026] While camera 3 is running, images within the field of view are continuously captured at a predetermined frame rate, such as 30 fps. Hereinafter, these images will be referred to as live view images 4. Figure 1 schematically shows live view images 4, but the actual images will be images that include the object 2. Live view images 4 are output to a display unit provided on camera 3. This allows user 1 to view images within the field of view of camera 3 through the display unit. Camera 3 corresponds to one embodiment of the imaging unit according to this technology. Live view images 4 correspond to one embodiment of the first image, which is an image of the object according to this technology.

[0027] Furthermore, when user 1 presses the shutter button on camera 3, a higher quality image than the live view image 4 is captured. Hereafter, this image will be referred to as shutter image 5. Alternatively, shutter image 5 may be captured automatically when a signal is sent from the system to camera 3 instructing it to capture shutter image 5.

[0028] The shutter image 5 is, for example, an image with higher resolution than the live view image 4, but "high quality" does not mean only high resolution, and its specific indicators are not limited. Furthermore, "high quality" merely means that the quality of the shutter image 5 is relatively higher than the quality of the live view image 4, and does not limit the absolute quality of the live view image 4 and the shutter image 5. The shutter image 5 corresponds to one embodiment of the second image, which is an image of the object related to this technology.

[0029] In this technology, each time a shutter image 5 is captured, the 3D model 29 is retrained based on the shutter image 5 (step 101). The 3D model 29 includes either a NeRF (Neural Radiance Fields) model 6 or a GS (Gaussian Splatting) model 7. The 3D model 29 also includes positional information of three-dimensional points, but this is not shown in the diagram. Next, the shutter image 5 is stored separately in a memory unit or the like (step 102).

[0030] NeRF is a technology that generates viewpoint images 8 from any viewpoint in real space. Specifically, a NeRF model 6, which is a machine learning model such as a multilayer perceptron, is trained to learn the radiance and density at each location in real space. By inputting a viewpoint into the trained machine learning model, a viewpoint image 8 from that viewpoint is output by volume rendering.

[0031] GS, like NeRF, is a technique for generating viewpoint image 8. In GS, information about each point in real space (position, color, transparency, etc.) is represented using a Gaussian distribution. GS is one of the techniques that has attracted attention in recent years due to its extremely fast learning and viewpoint image 8 generation speeds.

[0032] Furthermore, the NeRF model 6 and GS model 7 generate a 3D model 29 of the object 2 based on the shutter image 5. This 3D model 29 is ultimately the output to the user 1.

[0033] When the live view image 4 is captured, the current position and orientation of the camera 3 and its camera parameters (the position, orientation, and camera parameters at the time the live view image 4 was captured) are calculated separately. The position and orientation are information represented by six values, such as the three-dimensional position (x, y, z) and orientation (Rx, Ry, Rz) in real space. Then, the current position and orientation are input to the NeRF model 6 or GS model 7, and the viewpoint image 8 is output.

[0034] Next, the difference between the live view image 4 and the viewpoint image 8 is calculated (step 103). The difference is an indicator of how different the live view image 4 and the viewpoint image 8 are, and is calculated according to a predetermined standard.

[0035] If the NeRF model 6 or GS model 7 has not been sufficiently retrained, there may be a significant difference between the output viewpoint image 8 and the live view image 4. In this case, the difference will be large (Yes in step 104). On the other hand, if the retraining has been sufficiently performed, the viewpoint image 8 and the live view image 4 will be roughly identical, and the difference will be small (No in step 104).

[0036] If the difference is large (Yes in step 104), it is determined that it is necessary to take a shutter image 5 at the current viewpoint, and a notification prompting user 1 to take the picture is issued. Then, user 1 takes a shutter image 5 (step 105). Alternatively, a signal is sent to camera 3, and the shutter image 5 is automatically captured.

[0037] If the difference is small (No. in step 104), it is determined that taking the shutter image 5 is unnecessary, and user 1 moves to the next position (step 106). This changes the viewpoint, and the same process is repeated under the new viewpoint.

[0038] In other words, this technology assumes that if the difference is large, sufficient shutter images 5 for training have not been captured, and therefore, sufficient shutter images 5 for generating the 3D model 29 have not been captured. By determining that shutter images 5 need to be captured only when the difference is large, it is possible to capture shutter images 5 at the appropriate timing.

[0039] Furthermore, this technology performs a process to guide user 1 in manipulating the field of view of the live view image 4. The guidance is performed so that the live view image 4 contains many areas with a large difference from the viewpoint image 8.

[0040] As shown in the upper right of Figure 1, in this example, the lower part of object 2 is captured in the live view image 4. The area on the right side of the live view image 4 is the area with the largest difference, and a frame 9 is displayed surrounding this area. User 1, upon seeing this display, either moves to the right themselves or shifts the field of view to the right. As a result, the area with the largest difference that was originally on the right side of the field of view is now included within the field of view, and the live view image 4 contains more areas with the largest difference.

[0041] [Information Processing Device] Figure 2 is a schematic diagram showing an example configuration of the information processing device 10 according to this technology. The information processing device 10 includes a camera 3 and an edge device 11, and in this embodiment, some functions are configured on the cloud 12. The camera 3 may be any type of camera and constantly captures images within the field of view as live view images 4.

[0042] In this embodiment, the shutter image 5 is described as being automatically captured in response to the reception of a signal. However, if the shutter image 5 is captured by user 1, a shutter button may be provided.

[0043] The edge device 11 is a device that user 1 carries with them when taking pictures, and for example, a smartphone may be used. In this case, the camera 3 may be a camera built into the smartphone. The specific type of edge device 11 is not limited, and a device dedicated to this technology may be used.

[0044] Edge device 11 has a display unit 13, an operation unit 14, a communication unit 15, a storage unit 16, and a controller 17. These are interconnected via a bus 18. Instead of the bus 18, each block may be connected using a communication network or a non-standardized proprietary communication method, etc.

[0045] The display unit 13 is a display device using, for example, liquid crystal, EL (Electro-Luminescence), etc., and displays a GUI (Graphical User Interface), etc. In this embodiment, the live view image 4 is displayed on the display unit of the camera 3, but the live view image 4 may also be displayed on the display unit 13 of the edge device 11. The operation unit 14 is a touch panel and is configured integrally with the display unit 13. When the edge device 11 is a device other than a smartphone, the operation unit 14 may be a keyboard or a pointing device.

[0046] The communication unit 15 is a communication module for communicating with other devices via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network). A wireless LAN module such as WiFi or a communication module for short-range wireless communication such as Bluetooth (registered trademark) may be provided. Also, communication devices such as a modem or a router may be used. In this embodiment, communication with the cloud 12 is performed via the communication unit 15. ,

[0047] The storage unit 16 is a storage device such as a non-volatile memory, and for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive), etc. is used. Other arbitrary non-transitory computer-readable storage media may be used. In the storage unit 16, for example, a control program for controlling the overall operation of the edge device 11 is stored. The method of installing the control program in the storage unit 16 is not limited. Also, the shutter image 5 and the generated 3D model 29 are stored in the storage unit 16.

[0048] The controller 17 controls the operations of each block of the edge device 11. The controller 17 has hardware circuits necessary for a computer, such as a CPU and a memory (RAM, ROM), for example. By the CPU executing the program according to the present technology stored in the storage unit 16 and the memory, various processes are executed. As the controller 17, for example, a PLD (Programmable Logic Device) such as an FPGA (Field Programmable Gate Array), or other devices such as an ASIC (Application Specific Integrated Circuit) may be used.

[0049] In the present embodiment, when the CPU of the controller 17 executes the program according to the present technology (for example, an application program), as functional blocks, a live view image acquisition unit 19, a model reception unit 20, a position and orientation estimation unit 21, a viewpoint image generation unit 22, a shooting determination unit 23, a setting acquisition unit 24, a signal transmission unit 25, a shutter image acquisition unit 26, an image etc. transmission unit 27, and a guidance unit 28 are realized. And by these functional blocks, the information processing method according to the present embodiment is executed. Note that dedicated hardware such as an IC (integrated circuit) may be appropriately used to realize each functional block.

[0050] The live view image acquisition unit 19 constantly acquires a live view image 4 from the camera 3. The acquisition of the live view image 4 may be performed via the communication unit 15, or may be performed by wire after the camera 3 and the edge device 11 are connected by a cable or the like. This is the same for the acquisition of the shutter image 5 by the shutter image acquisition unit 26 described later.

[0051] The model reception unit 20 acquires the 3D model 29 generated on the cloud 12 via the communication unit ********** The 3D model 29 acquired by the model reception unit 20 is schematically illustrated in Fig. 2. The acquired 3D model 29 is stored in the storage unit 16.

[0052] The position and orientation estimation unit 21 estimates the current position and orientation (x, y, z, Rx, Ry, Rz) of the camera 3 based on the live view image 4 and the 3D model 29. The position and orientation of the camera 3 may also be estimated in any other format. The viewpoint can be calculated, for example, by appropriately converting the position and orientation of the camera 3. The position and orientation estimation unit 21 also estimates the current position and orientation based on the shutter image 5 when a shutter image 5 is acquired.

[0053] The viewpoint image generation unit 22 calculates the current viewpoint of the camera 3 as five values ​​(x, y, z, θ, φ) based on the position and orientation estimated by the position and orientation estimation unit 21. Then, based on the calculated current viewpoint and the 3D model 29, it generates a viewpoint image 8. In other words, the estimated result is an image that is thought to be acquired when imaging is performed from the current viewpoint. This estimation is performed using the NeRF model 6 or the GS model 7. The viewpoint image estimation unit 22 also calculates the viewpoint when the position and orientation are estimated based on the shutter image 5.

[0054] The shooting determination unit 23 determines whether or not it is necessary to acquire the shutter image 5 based on the difference between the live view image 4 and the viewpoint image 8. Specifically, it determines that it is necessary to acquire the shutter image 5 if the difference is greater than a predetermined threshold. In this embodiment, the user 1 can set the threshold via the operation unit 14, and the setting acquisition unit 24 acquires the content of the setting.

[0055] When the shooting determination unit 23 determines that it is necessary to acquire the shutter image 5, the signal transmission unit 25 transmits a signal to the camera 3 instructing it to capture the shutter image 5. The camera 3 receives this signal and captures the shutter image 5.

[0056] In this embodiment, the camera 3 automatically captures the shutter image 5. However, if the user 1 manually captures the shutter image 5, a notification unit is configured, for example, to notify the user 1 of an image or sound prompting them to press the shutter button, instead of the signal transmission unit 25.

[0057] The shutter image acquisition unit 26 acquires the shutter image 5 from the camera 3 when the camera 3 captures the shutter image 5. The image transmission unit 27 transmits the shutter image 5 and the current viewpoint to the cloud 12 via the communication unit 15.

[0058] The guidance unit 28 guides the user 1 in manipulating the field of view of the live view image 4. Specifically, it transmits an image to the camera 3 prompting the user to change the field of view. As a result, the image is displayed on the display unit of the camera 3. Alternatively, the output of an audio prompting the user to change the field of view may be controlled. In this case, the edge device 11 is provided with a speaker.

[0059] For example, a server is used as the cloud 12. The cloud 12 has a communication unit 30, a storage unit 31, and a controller 32. These are interconnected via a bus 33. Furthermore, when a program is executed by the controller 17, a functional block is realized, consisting of an image receiving unit 34, a retraining unit 35, and a model transmission unit 36.

[0060] The image receiving unit 34 receives the shutter image 5 and viewpoint transmitted from the edge device 11 via the communication unit 30. The retraining unit 35 uses the shutter image 5 and viewpoint to retrain (incrementally learn) the 3D model 29 stored in the memory unit 31. Figure 2 schematically illustrates the 3D model 29, but the NeRF model 6, GS model 7, and position information of the 3D points included in the 3D model 29 are not shown. The model transmission unit 36 ​​transmits the 3D model 29 to the edge device 11 via the communication unit 30.

[0061] Furthermore, the specific configurations of the camera 3, edge device 11, and cloud 12 are not limited, and their configurations may be modified as appropriate within the scope of the feasibility of this technology. The live view image acquisition unit 19 and the shutter image acquisition unit 26 correspond to one embodiment of the image acquisition unit according to this technology. The position and orientation estimation unit 21, viewpoint image generation unit 22, and retraining unit 35 correspond to one embodiment of the viewpoint image estimation unit according to this technology. The shooting determination unit 23, setting acquisition unit 24, and signal transmission unit 25 correspond to one embodiment of the determination unit according to this technology.

[0062] [Processing Flow] Figure 3 is a flowchart showing the processing details of this technology. The series of processes shown in Figure 3 are executed at a predetermined frame rate, such as 30 fps. In Figure 3, processes executed on the edge device 11 are shown with dashed lines, and processes executed on the cloud 12 are shown with solid lines. Specifically, steps 201 to 209, 214, and 215 are executed on the edge device 11, and steps 210 to 213 are executed on the cloud 12.

[0063] The live view image 4 captured by the camera 3 is acquired by the live view image acquisition unit 19 (step 201). Next, the position and orientation estimation unit 21 estimates the current position and orientation based on the live view image 4 and the 3D model 29 currently stored in the edge device 11 (step 202).

[0064] Specifically, the position and orientation estimation unit 21 first estimates the current position and orientation of the camera 3 based on the live view images 4 acquired in the present and past. Position and orientation estimation is performed, for example, by SfM (Structure from Motion). In this case, the position information of three-dimensional points included in the 3D model 29 is used for estimation. Alternatively, estimation may be performed by any method, and the NeRF model 6 or GS model 7 may be used for estimation.

[0065] Figure 4 is a schematic diagram showing the process of generating the viewpoint image 8. The viewpoint image 8 is generated by the viewpoint image generation unit 22 (step 203 in Figure 3). First, as shown in Figure 4, the viewpoint on the light ray of each pixel is calculated for each pixel by appropriately converting the position and orientation of the camera 3 estimated by the position and orientation estimation unit 21 (step 203-1). Next, based on the viewpoint and the NeRF model 6 or GS model 7, the color and transmittance of all viewpoints on the light ray are added for each pixel (step 203-2). This generates the viewpoint image 8.

[0066] Figure 5 is a schematic diagram showing an overview of the difference image 39. The shooting determination unit 23 performs the calculation of the difference image 39 (step 204). Specifically, first, the difference between the live view image 4 and the viewpoint image 8 is calculated for each pixel. Here, the difference means, for example, the sum of the differences in pixel values, i.e., the sum of the differences in R values, G values, and B values. In addition, the difference may have any content such that the value increases as the live view image 4 and the viewpoint image 8 are more different.

[0067] Next, the calculated difference is binarized using a predetermined threshold (diff_th). That is, each pixel is classified into two groups: pixels where the difference > diff_th and pixels where the difference ≤ diff_th. Figure 5 shows an image of the difference image 39 generated by this classification. In the difference image 39, white pixels represent pixels where the difference > diff_th, and black pixels represent pixels where the difference ≤ diff_th. In this example, the difference is generally larger in the right-hand region of the difference image 39.

[0068] Furthermore, the shooting determination unit 23 determines whether the percentage of pixels with difference > diff_th among all pixels is N% or greater, which is a predetermined threshold (step 205). At this time, processing such as ignoring small areas in the difference image 39 may be performed. This makes it possible to reduce the effect of noise. If the percentage of pixels is N% or greater, it is determined that it is necessary to acquire the shutter image 5 (Yes in step 205). On the other hand, if it is less than N%, it is determined that it is not necessary to acquire the shutter image 5 (No in step 205).

[0069] In addition to the flow shown in Figure 3, the setting acquisition unit 24 constantly accepts threshold settings from User 1. For example, User 1 can set the shooting frequency parameter. The setting acquisition unit 24 acquires the set parameter and outputs the threshold setting value corresponding to that parameter to the shooting determination unit 23.

[0070] If the shooting frequency parameter is set to a high value, the setting acquisition unit 24 outputs a low value of N to the shooting determination unit 23. The shooting determination unit 23 then updates the value of N to the lower value. This increases the likelihood that step 205 will determine that it is necessary to acquire shutter images 5, requiring the capture of more shutter images 5, but the resulting 3D model 29 will also be of higher quality.

[0071] Conversely, if the parameter is set low, the value of N is updated to a high value. As a result, the 3D model 29 will be of relatively low quality, but the number of shutter images 5 will be taken will be reduced.

[0072] Furthermore, the value of diff_th may be updated to a higher value when the shooting frequency parameter is set to a high value, and to a lower value when the parameter is set to a low value. In addition, arbitrary setting details may be acquired by the setting acquisition unit 24. For example, setting details unrelated to the judgment made by the shooting judgment unit 23 may be acquired.

[0073] If the shooting determination unit 23 determines that it is unnecessary to acquire the shutter image 5 (No in step 205), the process ends. On the other hand, if it is determined that it is necessary to acquire the shutter image 5 (Yes in step 205), a shutter signal is transmitted from the signal transmission unit 25 to the camera 3 (step 206). The camera 3 then receives the shutter signal and captures the shutter image 5.

[0074] The shutter image acquisition unit 26 acquires the shutter image 5 from the camera 3 (step 207). Furthermore, the position and orientation estimation unit 21 estimates the current viewpoint (the viewpoint when the shutter image 5 was captured) based on the shutter image 5 (step 208). The method of this estimation may be the same as that described in step 202. Then, the image transmission unit 27 transmits the shutter image 5 and the viewpoint to the cloud 12 (step 209). The image receiving unit 34 of the cloud 12 receives the shutter image 5 and the viewpoint (step 210).

[0075] Figure 6 is a schematic diagram illustrating the content of the relearning process. The relearning unit 35 performs incremental GS / NeRF processing (relearning process) (step 211). Specifically, as shown in Figure 6, the relearning unit 35 inputs the shutter image 5 (ground truth data) and the current viewpoint as training data into the 3D model 29. This executes relearning and updates the 3D model 29. In other words, the relearning unit 35 generates the 3D model 29 based on the shutter image 5.

[0076] The new 3D model 29 is stored in the memory unit 31 (step 212). The model transmission unit 36 ​​then transmits the 3D model 29 to the edge device 11 (step 213). The model receiving unit 20 of the edge device 11 receives the 3D model 29 (step 214) and stores it in the memory unit 16 (step 215).

[0077] Note that when the information processing device 10 is started up, the processing of this flow is not executed until the first shutter image 5 is captured. Then, when the first shutter image 5 is captured at the user 1's discretion, the processing of steps 207 to 215 of this flow is executed, and thereafter a series of processes are executed at predetermined intervals.

[0078] [Guidance Unit] Figure 7 is a schematic diagram showing an example of the guidance provided by the guidance unit 28. In this embodiment, separate from the flow shown in Figure 3, the guidance unit 28 provides guidance to the user 1 on manipulating the field of view of the live view image 4. The guidance is performed so that the shooting determination unit 23 determines that it is necessary to acquire the shutter image 5. In other words, the guidance is performed so that the field of view of the live view image 4 includes many areas where the difference from the viewpoint image 8 is large.

[0079] In Figure 7A, areas with a large difference between the live view image 4 and the viewpoint image 8 are highlighted by being enclosed in a frame 9. For example, the display is controlled so that the same area as the white area in the difference image 39 is enclosed in a frame 9. This processing can be achieved by the guidance unit 28 acquiring the difference image from the shooting determination unit 23.

[0080] In Figure 7B, an arrow 42 is superimposed on the live view image 4 to guide the angle of view adjustment. This arrow 42 is displayed over the area with the largest difference. Alternatively, the arrow 42 may be displayed so that the area with the largest difference is located on the other side of the arrow 42. In this example, the arrow is displayed pointing to the right, but the arrow 42 may be displayed in any direction, including three-dimensional directions (foreground, background). In addition, in Figures 7A and 7B, text is displayed to guide the angle of view adjustment.

[0081] It is likely that user 1, upon seeing these frames 9, arrows 42, and text, will move themselves or adjust the field of view. As a result, a large area that was originally outside the field of view will be newly included within the field of view, increasing the likelihood that the shooting determination unit 23 will determine that it is necessary to acquire the shutter image 5.

[0082] Figure 8 is a schematic diagram showing the process of guiding the user to an uncaptured location. Figure 9 is a schematic diagram showing an example of the configuration related to guiding the user to an uncaptured location. In Figure 8A, the positions 43 (43a, 43b, 43c) where user 1 previously captured shutter images 5 are schematically shown as white triangles. The user 1's movement path is shown as a dashed arrow. The shaded triangle represents user 1's current position 44.

[0083] In Figure 8B, an arrow 42 is displayed within the live view image 4 at position 44. In this embodiment, the display of the arrow 42 is controlled by the guidance unit 28 to guide the user 1 to an uncaptured position (a position where shooting has not yet been performed). For example, in this example, since shooting has not yet been performed from the right side of the object 2 (the position of the star in Figure 8A), the arrow 42 is displayed to guide the user 1 to the right.

[0084] Such control is achieved by storing image position information 45 (the position when the shutter image 5 was captured) together with the 3D model 29, as shown in Figure 9. In this embodiment, as shown by the thick arrows in Figure 9, the guidance unit 28 acquires the image position information 45 stored in the storage unit 16 and calculates the uncaptured position based on the image position information 45. Then, by acquiring the current position and orientation of the camera 3 from the position and orientation estimation unit 21, it calculates the direction of the uncaptured position within the field of view of the live view image 4 and displays the arrow 42 so that it faces that direction.

[0085] In the information processing device 10 according to this embodiment, a viewpoint image 8 is estimated based on the live view image 4, and based on the difference between the live view image 4 and the viewpoint image 8, it is determined whether or not it is necessary to acquire a shutter image 5 of higher quality than the live view image 4. This makes it possible to efficiently generate a 3D model 29.

[0086] Generating high-quality 3D models using photogrammetry requires advanced expertise, as it necessitates proper adjustment of factors such as displacement, parallax, and overlap rate during photo capture. Therefore, it is often a high hurdle for users.

[0087] Therefore, one could either commission an expert with specialized knowledge to take the photos, or, if taking the photos oneself, take a vast number of redundant shots. However, either way, it would require cost and time. Another method is to make heuristic decisions based on overlap rates and movement amounts, but this cannot be universally applied to various environments and subjects. Consequently, knowing the right timing to take photos has been a challenge.

[0088] In this technology, the timing for taking the shutter image 5 is determined based on the difference between the live view image 4 and the viewpoint image 8. Therefore, user 1 can simply walk around pointing the camera 3 at the object 2 to know when to take the picture (or the shutter will be released automatically). As a result, photogrammetry can be performed without any special knowledge.

[0089] Furthermore, this technology generates viewpoint image 8 using NeRF Model 6 or GS Model 7, which are arbitrary viewpoint image synthesis techniques. This makes it possible to determine the timing of the image capture with even greater accuracy.

[0090] Furthermore, in this technology, machine learning is performed on the NeRF model 6 or GS model 7 using the shutter image 5 as ground truth data. This makes it possible to determine the appropriate shooting timing by utilizing the fact that if a sufficient number of shutter images 5 are not obtained for the generation of the 3D model 29, the NeRF model 6 or GS model 7 will not have been sufficiently trained, and the viewpoint image 8 will also be inaccurate.

[0091] Furthermore, this technology estimates the current viewpoint based on the live view image 4. This makes it possible to estimate the viewpoint with even greater accuracy.

[0092] Furthermore, this technology generates a 3D model 29 of the object 2 based on the shutter image 5. This makes it possible to generate the 3D model 29 with even greater accuracy.

[0093] Furthermore, this technology estimates the viewpoint image 8 based on the current viewpoint and the 3D model 29. This makes it possible to estimate the viewpoint image 8 with even greater accuracy, and also improves the accuracy of determining the timing of the image capture.

[0094] Furthermore, in this technology, the user 1 is guided to manipulate the field of view of the live view image 4 so that it is determined that it is necessary to acquire the shutter image 5. This reduces the time required to capture the shutter image 5 and improves the efficiency of shooting.

[0095] Furthermore, in this technology, areas with a large difference between the live view image 4 and the viewpoint image 8 are highlighted by a frame 9. Alternatively, arrows 42 guiding the user to adjust the field of view are superimposed on the live view image 4. This allows user 1 to visually understand how to adjust the field of view, thereby improving user convenience.

[0096] Furthermore, in this technology, if the difference between the live view image 4 and the viewpoint image 8 is greater than a predetermined threshold, it is determined that a shutter image 5 needs to be taken. This makes it possible to make the determination with even greater accuracy. In addition, the threshold setting by the user 1 is accepted. This allows the user 1 to select a setting appropriate to the situation, such as when a high-quality 3D model 29 is required even if it takes time to take the picture, or when the 3D model 29 can be of relatively lower quality but the user does not want to spend too much time taking the picture, thereby further improving convenience.

[0097] Furthermore, this technology includes a camera 3, and captured images are acquired by a live view image acquisition unit 19 and a shutter image acquisition unit 26. This makes it possible to acquire high-quality captured images.

[0098] Furthermore, in this technology, when the shooting determination unit 23 determines that it is necessary to acquire the shutter image 5, a signal is sent to the camera 3 instructing it to capture the shutter image 5. This makes it possible to capture images with even greater accuracy.

[0099] <Other Embodiments> This technology is not limited to the embodiments described above, and various other embodiments can be realized. [Embodiment with Edge Device Only] Figure 10 is a schematic diagram showing an example configuration according to an embodiment with only an edge device 11. In this example, the cloud 12 in Figure 2 does not exist, and the retraining unit 35 that was configured in the cloud 12 is included in the edge device 11. Also, since communication between the edge device 11 and the cloud 12 is not required, the image transmission unit 27, image reception unit 34, model transmission unit 36, and model reception unit 20 in Figure 2 are not configured.

[0100] In this embodiment, wireless communication with the cloud 12 is unnecessary, so even if the communication environment deteriorates, the retraining process (3D model 29 generation process) can be performed without any problems.

[0101] [Examples in Video Production] Figure 11 is a schematic diagram showing an example configuration according to an embodiment in video production. In this embodiment, when scouting locations in remote areas for video production, it is assumed that several people go to the site to photograph the subject 2 (the scenery that will be the setting), and then staff 48 check the 3D model 29 of the subject 2 in the studio.

[0102] In this example, in addition to the edge device 11 and cloud 12 shown in Figure 2, a remote device 49 is also configured. The edge device 11 is a device owned by user 1 who performs the shooting on-site. The remote device 49 is a device used by staff 48 in the studio, and for example, a PC (Personal Computer) is used.

[0103] The remote device 49 comprises a model receiving unit 50, a viewpoint image generation unit 51, and a destination transmission unit 52 as functional blocks. An operating device 53 is also provided for operation by the staff 48. The operating device 53 can be, for example, a display, VR (Virtual Reality) goggles, or a controller.

[0104] The model receiving unit 50 receives the 3D model 29 from the cloud 12. The viewpoint image generation unit 51 generates a viewpoint image 8 in the same manner as the viewpoint image generation unit 22 of the edge device 11. At this time, the staff 48 inputs a desired viewpoint via the operation device 53. The viewpoint image generation unit 22 generates a viewpoint image 8 for that viewpoint based on the viewpoint and the 3D model 29. This makes it possible for the staff 48 to view the viewpoint image 8 for any viewpoint.

[0105] In addition, if there is a viewpoint where the viewpoint image 8 is unclear, staff member 48 can instruct user 1 to take another shutter image 5 at that viewpoint. In this case, the destination (the viewpoint where the image should be taken) is specified using the operation device 53. The specified destination is transmitted to the edge device 11 by the destination transmission unit 52 and received by the destination receiving unit 54 of the edge device 11. The destination is then acquired by the guidance unit 28, and guidance to the destination is performed using a frame 9, arrows 42, etc.

[0106] When conducting location scouting in remote areas, it is often necessary for hundreds of staff members to travel to the site. In this embodiment, a small number of users 1 can travel to the site and photograph the target object 2, allowing staff 48 to verify a 3D model 29 that accurately reproduces the target object 2. Therefore, it is possible to reduce the budget for travel expenses to the site and to significantly shorten the production period.

[0107] [Modification] In generating the viewpoint image 8 and 3D model 29, other technologies besides NeRF and GS may be used, to the extent that this technology is feasible. For example, methods using polygon meshes may be used. However, using NeRF and GS has the advantage of generating photorealistic viewpoint images 8 and 3D models 29 with less noise and a natural appearance of reflective and translucent objects.

[0108] The edge device 11, the cloud 12, and the remote device 49 may be implemented by multiple computers or by a single computer.

[0109] Figure 12 is a block diagram showing an example hardware configuration of a computer 500 capable of realizing an edge device 11, a cloud 12, and a remote device 49. The computer 500 includes a CPU 501, ROM 502, RAM 503, an input / output interface 505, and a bus 504 connecting these to each other. A display unit 506, an input unit 507, a storage unit 508, a communication unit 509, and a drive unit 510 are connected to the input / output interface 505.

[0110] The display unit 506 is a display device using, for example, liquid crystal, EL, etc. The input unit 507 is, for example, a keyboard, pointing device, touch panel, or other operating device. If the input unit 507 includes a touch panel, the touch panel may be integrated with the display unit 506. The storage unit 508 is a non-volatile storage device, for example, an HDD, flash memory, or other solid memory. The drive unit 510 is a device capable of driving the removable recording medium 511, for example, an optical recording medium or magnetic recording tape. The communication unit 509 is a modem, router, or other communication device for communicating with other devices, which can be connected to a LAN, WAN, etc. The communication unit 509 may communicate using either wired or wireless methods. The communication unit 509 is often used separately from the computer 500.

[0111] Information processing by the computer 500 having the hardware configuration described above is realized through the cooperation of software stored in the memory unit 508 or ROM 502, etc., and the hardware resources of the computer 500. Specifically, the information processing method related to this technology is realized by loading the programs that constitute the software, stored in the ROM 502, etc., into the RAM 503 and executing them.

[0112] The program is installed on the computer 500, for example, via a removable recording medium 511. Alternatively, the program may be installed on the computer 500 via a global network or the like. In addition, any non-transient storage medium that the computer 500 can read may be used.

[0113] In this disclosure, "system" means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules in one enclosure, are both considered systems.

[0114] The execution of the information processing method related to this technology by a computer system includes both cases where the acquisition of live view images 4 and shutter images 5, viewpoint estimation, generation of viewpoint images 8, calculation of difference images 39, transmission of shutter signals, retraining processing, and generation of 3D models 29 are performed by a single computer, and cases where each process is performed by different computers. Furthermore, the execution of each process by a predetermined computer includes having another computer perform part or all of the process and obtaining the results. In other words, the information processing method related to this technology can also be applied to cloud computing configurations in which a single function is shared and processed jointly by multiple devices via a network.

[0115] The edge device 11, cloud 12, remote device 49, live view image 4, and processing flows described with reference to each drawing are merely one embodiment and can be arbitrarily modified without departing from the spirit of this technology. In other words, other arbitrary configurations and algorithms may be adopted to implement this technology.

[0116] Where the word "abbreviated" is used in this disclosure, it is used solely to facilitate understanding of the explanation, and there is no special meaning in the use or non-use of the word "abbreviated." In other words, in this disclosure, concepts that define shape, size, positional relationships, states, etc., such as "center," "central," "uniform," "equal," "same," "orthogonal," "parallel," "symmetric," "extending," "axial," "cylindrical," "cylindrical shape," "ring shape," and "annular shape," include concepts such as "substantially centered," "substantially central," "substantially uniform," "substantially equal," "substantially the same," "substantially orthogonal," "substantially parallel," "substantially symmetric," "substantially extending," "substantially axial," "substantially cylindrical," "substantially cylindrical shape," "substantially ring shape," and "substantially annular shape." For example, states that fall within a predetermined range (e.g., a range of ±10%) based on "perfectly centered," "perfectly central," "perfectly uniform," "perfectly equal," "perfectly the same," "perfectly orthogonal," "perfectly parallel," "perfectly symmetric," "perfectly extending," "perfectly axial," "perfectly cylindrical," "perfectly cylindrical shape," "perfectly ring shape," and "perfectly annular shape" are also included. Therefore, even if the word "abbreviated" is not added, the concept may still be included in what is typically expressed with "abbreviated" added. Conversely, the state expressed with "abbreviated" does not mean that the complete state is excluded.

[0117] In this disclosure, expressions using "greater than A" such as "greater than A" and "less than A" are expressions that comprehensively include both concepts that include cases where something is equivalent to A and concepts that do not include cases where something is equivalent to A. For example, "greater than A" is not limited to cases where something is not equivalent to A, but also includes "greater than or equal to A". Similarly, "less than A" is not limited to "less than A", but also includes "less than or equal to A". When implementing this technology, you may appropriately adopt specific settings from the concepts included in "greater than A" and "less than A" so that the effects described above are achieved.

[0118] It is also possible to combine at least two of the feature features of the present technology described above. In other words, the various feature features described in each embodiment may be combined arbitrarily, regardless of the specific embodiment. Furthermore, the various effects described above are merely examples and not limiting, and other effects may also be exhibited.

[0119] Furthermore, this technology can also be configured as follows: (1) An information processing device comprising: an image acquisition unit that acquires a first image which is an image of an object in real space and a second image which is an image of the object which is of higher quality than the first image; a viewpoint image estimation unit that estimates a viewpoint image which is an image of the object taken from the current viewpoint based on the acquired first image; and a determination unit that determines whether or not it is necessary to acquire the second image based on the difference between the currently acquired first image and the estimated viewpoint image. (2) An information processing device according to (1), wherein the viewpoint image estimation unit estimates the viewpoint image using a NeRF (Neural Radiance Fields) model or a GS (Gaussian Splatting) model. (3) An information processing device according to (2), wherein the viewpoint image estimation unit performs machine learning on the NeRF model or the GS model using the acquired second image as ground truth data. (4) An information processing device according to any one of (1) to (3), wherein the viewpoint image estimation unit estimates the current viewpoint based on the acquired first image. (5) An information processing device according to any one of (1) to (4), wherein the viewpoint image estimation unit generates a 3D model of the object based on the acquired second image. (6) An information processing device according to (5), wherein the viewpoint image estimation unit estimates the viewpoint image based on the current viewpoint and the generated 3D model. (7) An information processing device according to any one of (1) to (6), further comprising a guidance unit that guides the user performing the photography of the object to manipulate the field of view of the first image so that the determination unit determines that it is necessary to acquire the second image. (8) An information processing device according to (7), wherein the guidance unit highlights areas on the first image where the difference between the viewpoint image and the first image is large. An information processing device according to (9), (7), or (8), wherein the guidance unit superimposes and displays arrows on the first image that guide the operation of the field of view.(10) An information processing device according to any one of (1) to (9), wherein the determination unit determines that it is necessary to acquire the second image when the difference is greater than a predetermined threshold, and accepts a setting of the threshold by a user who is taking a photograph of the object. (11) An information processing device according to any one of (1) to (10), further comprising an imaging unit that performs imaging of the object, wherein the image acquisition unit acquires the imaged image from the imaging unit. (12) An information processing device according to (11), wherein the determination unit determines that it is necessary to acquire the second image, and transmits a signal to the imaging unit to instruct it to take the second image. (13) An information processing method that acquires a first image which is an image of an object in real space, estimates a viewpoint image which is an image of the object taken from the current viewpoint based on the acquired first image, and determines whether it is necessary to acquire a second image which is an image of the object and of higher quality than the first image based on the difference between the currently acquired first image and the estimated viewpoint image. (14) A program that causes a computer system to perform the following steps: acquire a first image which is an image of an object in real space; estimate a viewpoint image which is an image of the object taken from the current viewpoint based on the acquired first image; and determine whether it is necessary to acquire a second image which is an image of the object and of higher quality than the first image based on the difference between the currently acquired first image and the estimated viewpoint image.

[0120] 1...User 2...Object 3...Camera 4...Live View Image 5...Shutter Image 6...NeRF Model 7...GS Model 8...Viewpoint Image 9...Frame 10...Information Processing Device 11...Edge Device 12...Cloud 17...Controller 19...Live View Image Acquisition Unit 20...Model Reception Unit 21...Position and Orientation Estimation Unit 22, 51...Viewpoint Image Generation Unit 23...Shooting Determination Unit 24...Setting Acquisition Unit 25...Signal Transmission Unit 26...Shutter Image Acquisition Unit 28...Guidance Unit 29...3D Model 35...Retraining Unit 39...Difference Image 42...Arrow

Claims

1. An information processing device comprising: an image acquisition unit that acquires a first image, which is an image of an object in real space, and a second image, which is an image of the object of higher quality than the first image; a viewpoint image estimation unit that estimates a viewpoint image, which is an image of the object taken from the current viewpoint, based on the acquired first image; and a determination unit that determines whether or not it is necessary to acquire the second image based on the difference between the currently acquired first image and the estimated viewpoint image.

2. An information processing device according to claim 1, wherein the viewpoint image estimation unit estimates the viewpoint image using a NeRF (Neural Radiance Fields) model or a GS (Gaussian Splatting) model.

3. An information processing device according to claim 2, wherein the viewpoint image estimation unit performs machine learning on the NeRF model or the GS model using the acquired second image as ground truth data.

4. An information processing device according to claim 1, wherein the viewpoint image estimation unit estimates the current viewpoint based on the acquired first image.

5. An information processing apparatus according to claim 1, wherein the viewpoint image estimation unit generates a 3D model of the object based on the acquired second image.

6. An information processing device according to claim 5, wherein the viewpoint image estimation unit estimates the viewpoint image based on the current viewpoint and the generated 3D model.

7. An information processing device according to claim 1, further comprising a guidance unit that guides a user performing a photograph of the object to manipulate the field of view of the first image so that the determination unit determines that it is necessary to acquire the second image.

8. An information processing device according to claim 7, wherein the guidance unit highlights areas on the first image where the difference between the first image and the viewpoint image is large.

9. An information processing device according to claim 7, wherein the guidance unit superimposes and displays arrows on the first image to guide the operation of the field of view.

10. An information processing apparatus according to claim 1, wherein the determination unit determines that acquisition of the second image is necessary when the difference is greater than a predetermined threshold, and accepts a setting regarding the threshold by a user who is taking a photograph of the object.

11. An information processing apparatus according to claim 1, further comprising an imaging unit for performing imaging of the object, wherein the image acquisition unit acquires the imaged image from the imaging unit.

12. An information processing device according to claim 11, wherein the determination unit determines that it is necessary to acquire the second image, and transmits a signal to the imaging unit instructing it to acquire the second image.

13. An information processing method that acquires a first image which is an image of an object in real space, estimates a viewpoint image which is an image of the object taken from the current viewpoint based on the acquired first image, and determines whether it is necessary to acquire a second image which is an image of the object and of higher quality than the first image based on the difference between the currently acquired first image and the estimated viewpoint image.

14. A program that causes a computer system to perform the following steps: acquire a first image, which is an image of an object in real space; estimate a viewpoint image, which is an image of the object taken from the current viewpoint, based on the acquired first image; and determine whether it is necessary to acquire a second image, which is an image of the object and of higher quality than the first image, based on the difference between the currently acquired first image and the estimated viewpoint image.

Images