Imaging processing system and imaging processing method
The imaging processing system uses a second mobile body and triangulation with neighboring images to estimate external parameters when marker feature points are absent, ensuring accurate three-dimensional visual information reconstruction and free-viewpoint video generation.
Patent Information
- Application Number
- PCT/JP2025/026056
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-08
- Filing Date
- 2025-07-23
- Publication Date
- 2026-11-12
AI Technical Summary
Existing imaging processing systems struggle to accurately estimate the external parameters of an imaging device when the feature points necessary for marker recognition cannot be captured, which is crucial for generating free-viewpoint videos, especially in real-time applications like disaster sites.
An imaging processing system that utilizes a second mobile body with a second imaging unit to capture images including a subject and markers, employing an image processing device to estimate external parameters through triangulation using neighboring images when marker feature points are unavailable, and a three-dimensional spatial model generation unit to reconstruct three-dimensional visual information.
Accurately estimates the external parameters of the imaging unit even when marker feature points are not captured, enabling robust reconstruction of three-dimensional visual information and generation of high-quality free-viewpoint videos.
Smart Images

Figure JP2025026056_12112026_PF_FP_ABST
Abstract
Description
Imaging Processing System and Imaging Processing Method
[0001] The present disclosure relates to an imaging processing system and an imaging processing method.
[0002] Conventionally, there is known an imaging processing system that restores three-dimensional visual information of a subject from an entire image of the subject that includes feature points for recognizing a marker imaged by an imaging device in the image, and external parameters of the imaging device at the time of capturing the entire image, and generates a free-viewpoint video using the three-dimensional visual information. The external parameters of the imaging device are, for example, the position and orientation of the imaging device in a specific coordinate system. The external parameters of the imaging device are estimated using the two-dimensional coordinates of the marker captured in the entire image, the three-dimensional coordinates of the marker in a three-dimensional space, and the internal parameters and distortion parameters of the imaging device.
[0003] By the way, when a subject is imaged by an imaging device, it is not always possible to capture the entire image of the subject in which all the feature points of the marker are included in the image by capturing the marker, and the subject may be imaged in a state where the marker cannot be used because the feature points required for recognizing the marker cannot be fully captured. For example, in a case where a free-viewpoint video must be generated in real time and the on-site situation must be checked in real time, such as at a disaster site, a subject image obtained by imaging the subject in a state where the marker cannot be used may also be used. In this case, it is necessary to estimate the external parameters of the imaging device corresponding to the subject image. For example, Patent Document 1 discloses a tracking technique that tracks the position and change of a terminal device having an imaging unit using Visual-SLAM (Simultaneous Localization and Mapping). In tracking using Visual-SLAM, a map of the surrounding environment is created, and the movement amount and orientation change amount of the imaging unit in the user coordinate system are estimated based on the transition of the feature amount of the image captured by the imaging unit, so as to estimate the position and orientation of the imaging unit on the map of the surrounding environment. Thereby, even if no marker part exists in the imaging range of the imaging unit, tracking of the position and orientation of the imaging unit can be continuously performed.
[0004] Japanese Unexamined Patent Application Publication No. 2022-140891
[0005] However, the technology disclosed in Patent Document 1 uses Visual-SLAM to estimate the position and orientation of the imaging unit from the features in the image. Therefore, if sufficient features cannot be obtained from the image, there is a risk that the external parameters of the imaging unit may not be accurately estimated.
[0006] This disclosure is made in view of the above, and aims to provide an imaging processing system that can accurately estimate the external parameters of the imaging unit corresponding to a subject image taken when the subject is in a state where the feature points necessary for recognizing the marker cannot be captured and the marker cannot be used.
[0007] To solve the above-mentioned problems and achieve the objective, the imaging processing system according to this disclosure comprises at least one second mobile body having a second imaging unit that moves around a subject and captures an image including the subject by capturing a scene including the subject and at least one marker placed around the subject, and an image processing device that processes the image using the information obtained by the second mobile body. The image processing device comprises an external parameter information holding unit that holds the external parameters of the second imaging unit corresponding to the image captured by the second imaging unit in association with each image, a neighboring image search unit that searches for at least two neighboring images captured in the vicinity of an image whose external parameters are unknown from the external parameter information held by the external parameter information holding unit, and an external parameter estimation unit that estimates the external parameters of the second imaging unit corresponding to the image. If the image captured by the second imaging unit is a subject image in which the feature points necessary to recognize the marker cannot be captured and the marker cannot be used, the external parameter estimation unit uses at least two neighboring images searched by the neighboring image search unit and the external parameters corresponding to the neighboring images as a set and estimates the external parameters of the second imaging unit corresponding to the subject image by triangulation.
[0008] The imaging processing system described herein has the effect of accurately estimating the external parameters of the imaging unit corresponding to a subject image taken in a state where the feature points necessary for marker recognition cannot be captured and the marker cannot be used.
[0009] A schematic overall configuration diagram showing the imaging processing system according to Embodiment 1. A block diagram showing the imaging processing system according to Embodiment 1. An explanatory diagram showing an example of a search method of the nearby image search unit in the image processing device of the imaging processing system according to Embodiment 1. A flowchart showing the procedure for estimating the external parameters of the second imaging unit from the overall image in the external parameter estimation unit in the image processing device of the imaging processing system according to Embodiment 1. An explanatory diagram showing an example of the processing of the external parameter estimation unit in the image processing device of the imaging processing system according to Embodiment 1. A flowchart showing a different procedure for estimating the external parameters of the second imaging unit from the overall image in the external parameter estimation unit in the image processing device of the imaging processing system according to Embodiment 1. A flowchart showing the imaging processing method of the imaging processing system according to Embodiment 1. A block diagram showing a modified example 1 of the imaging processing system according to Embodiment 1. A block diagram showing a modified example 2 of the imaging processing system according to Embodiment 1. A block diagram showing the imaging processing system according to Embodiment 2. A flowchart showing the imaging processing method of the imaging processing system according to Embodiment 2.
[0010] The imaging processing system and imaging processing method according to the embodiments of this disclosure will be described in detail below with reference to the drawings.
[0011] Embodiment 1. Figure 1 is a schematic overall configuration diagram showing the shooting processing system according to Embodiment 1. The shooting processing system 100 according to Embodiment 1 reconstructs three-dimensional visual information of a subject 50 from multiple images of the subject 50 that have been photographed, and generates a free-viewpoint video using the reconstructed three-dimensional visual information, etc. Note that the three-dimensional visual information includes, for example, three-dimensional shape. As shown in Figure 1, the shooting processing system 100 according to Embodiment 1 comprises a first mobile body 1, a second mobile body 2, and an image processing device 3. As shown by the white arrows in Figure 1, in the shooting processing system 100, the information obtained by the first mobile body 1 and the second mobile body 2 is processed by the image processing device 3, and the free-viewpoint video generated by the image processing device 3 is displayed on the display device 4. In the shooting processing system 100 according to Embodiment 1, the first mobile body 1, the second mobile body 2, and the image processing device 3 are each configured as standalone devices.
[0012] As shown in Figure 1, the first mobile body 1 is, for example, a drone, and has the function of photographing the subject 50 while moving around the subject 50. However, the first mobile body 1 does not necessarily have to photograph the subject 50. The first mobile body 1 is provided with one or more markers 51 of different shapes on its outer surface. When multiple markers 51 are provided on the outer surface of the first mobile body 1, each marker 51 should have a different shape in order to distinguish them. The second mobile body 2 is, for example, a drone, and has the function of photographing the subject 50 while moving around the subject 50. Figure 1 shows a configuration having one first mobile body 1 and one second mobile body 2. The first mobile body 1 is not limited to the one shown, and may be two or more. Similarly, the second mobile body 2 is not limited to the one shown, and may be two or more. The movement of the first mobile body 1 and the second mobile body 2 may be operated manually or controlled automatically.
[0013] First, the configurations of the first mobile body 1 and the second mobile body 2 will be described. Figure 2 is a block diagram of the imaging processing system according to Embodiment 1. As shown in Figure 2, the first mobile body 1 has a first imaging unit 10, a first data communication unit 11, a first position information receiving unit 12, and a gyro sensor 13.
[0014] As shown in Figure 1, the first imaging unit 10 photographs the subject 50 and acquires a subject image 5B. The first data communication unit 11 transmits the information acquired by the first mobile body 1 to the image processing device 3 via an external network. The first position information receiving unit 12 receives three-dimensional position information from a three-dimensional position information transmitting device that transmits the three-dimensional position information of the mobile body, and stores the three-dimensional position information associated with a timestamp, which is the date and time information of the time of reception. The three-dimensional position information transmitting device is used to calculate the three-dimensional coordinates of the first mobile body 1 in world coordinates, and is, for example, a GNSS (Global Navigation Satellite System) satellite, or a base station in RTK (Real Time Kinematic) that improves the accuracy of signals from GNSS satellites. By receiving the three-dimensional position information with the first position information receiving unit 12, the first mobile body 1 can acquire its three-dimensional coordinate position in the world coordinate system. The gyro sensor 13 acquires the attitude of the first mobile body 1. The first mobile body 1 can acquire the three-dimensional coordinate position and orientation in the world coordinate system, which are external parameters of the first imaging unit 10, by using the three-dimensional coordinate position acquired by the first position information receiving unit 12 and the orientation acquired by the gyro sensor 13. The first mobile body 1 associates the subject image 5B captured by the first imaging unit 10, the three-dimensional coordinate position acquired by the first position information receiving unit 12, and the orientation of the first mobile body 1 acquired by the gyro sensor 13 with the ID (IDentifier) of the first mobile body 1 and transmits it to the image processing device 3. Note that the first mobile body 1 is not necessarily required to transmit the subject image 5B captured by the first imaging unit 10 to the image processing device 3, and the transmission of the subject image 5B may be omitted.
[0015] The marker 51 provided on the outer surface of the first mobile body 1 has a feature point portion and is used to calculate the position and orientation, which are external parameters of the second mobile body 2. As shown in Figure 1, for example, if a square-shaped marker 51 is used, the four corners of the outer circumference can be used as feature points. The marker 51 may be a designated marker such as an ArUco marker or a QR marker, or it may be a marker with a simple shape such as a linear, circular, star-shaped, or square. In addition, the corners of the marker 51 that serve as feature points may be colored or lights may be installed. Furthermore, if markers 51 are provided on multiple first mobile bodies 1, markers of the same type and shape may be provided, or markers of different types and shapes may be provided.
[0016] As shown in Figure 2, the second mobile body 2 has a second shooting unit 20, an image discrimination unit 21, and a second data communication unit 22. As shown in Figure 1, the second shooting unit 20 takes an image including the subject 50. Specifically, the second shooting unit 20 takes an image of the scene including the subject 50 and the marker 51 provided on the outer surface of the first mobile body 1 to obtain an overall image 5A, and stores the overall image 5A in association with the shooting time. The overall image 5A includes all of the feature points of the marker. In addition, the second shooting unit 20 may take an image of the subject 50 in a state where the feature points necessary to recognize the marker 51 cannot be captured and the marker 51 cannot be used, and obtain a subject image 5B, and stores the subject image 5B in association with the shooting time. The subject image 5B in a state where the marker 51 cannot be used means an image in which part of the marker 51 is captured to the extent that the marker 51 cannot be used, or an image in which the marker 51 is not captured at all.
[0017] The image discrimination unit 21 distinguishes between the overall image 5A and the subject image 5B by determining the presence or absence of the marker 51 in the image captured by the second shooting unit 20. If only a part of the marker 51 is visible in the image to the extent that the marker 51 cannot be used, it is determined to be the subject image 5B. The determination of the presence or absence of the marker 51 is performed by known methods such as pattern matching or image segmentation. The second data communication unit 22 transmits the overall image 5A and the subject image 5B, determined by the image discrimination unit 21, to the image processing device 3 via an external network, associating them with the ID of the second mobile device 2. The image discrimination unit 21 does not necessarily have to be located in the second mobile device 2; it may be located in the image processing device 3. In this case, the second data communication unit 22 transmits the image captured by the second shooting unit 20 to the image processing device 3 via an external network without distinguishing between the overall image 5A and the subject image 5B.
[0018] Next, the configuration of the image processing device 3 will be described. As shown in Figure 2, the image processing device 3 includes an information communication unit 30, a marker information holding unit 31, an external parameter information holding unit 32, a nearby image search unit 33, an external parameter estimation unit 34, and a three-dimensional spatial model generation unit 35.
[0019] The information and communication unit 30 communicates with the first data communication unit 11 via an external network and receives information acquired by the first mobile device 1. The information and communication unit 30 also communicates with the second data communication unit 22 via an external network and receives information acquired by the second mobile device 2. The information and communication unit 30 also transmits the free-viewpoint video generated by the three-dimensional spatial model generation unit 35 to the display device 4 via an external network.
[0020] The marker information holding unit 31 holds information about the marker 51. The information about the marker 51 includes, for example, the type and shape of the marker 51, and the difference in coordinate values between feature points in three-dimensional space held by the marker 51 in x, y, and z, which are defined as the three-dimensional axes of the world coordinate system. The type and shape of the marker 51 are, as described above, designated markers such as ArUco markers and QR markers, free markers, etc. This information is stored in the marker information holding unit 31 in advance as prior information about the marker 51.
[0021] The external parameter information holding unit 32 holds the external parameters of the second imaging unit 20 corresponding to the image captured by the second imaging unit 20, associating them with the image. Specifically, it holds the external parameters of the second imaging unit 20 corresponding to the overall image 5A estimated by the external parameter estimation unit 34, associating them with the overall image 5A. It also holds the external parameters of the second imaging unit 20 corresponding to the subject image 5B estimated by the external parameter estimation unit 34, associating them with the subject image 5B. Note that the external parameters of the first imaging unit 10 corresponding to the subject image 5B captured by the first imaging unit 10 may or may not be held as associated with the subject image 5B.
[0022] The neighboring image search unit 33 searches for at least two neighboring images taken in the vicinity of the subject image 5B, whose external parameters are unknown, from the external parameter information stored in the external parameter information storage unit 32. A neighboring image is an image in which a common feature point A is captured with the subject image 5B in which a certain feature point A is captured. Furthermore, a neighboring image is either the overall image 5A, whose external parameters are known and whose coordinate system is the world coordinate system, or the subject image 5B, whose external parameters are known. The number of feature points A common to the images may be one or multiple.
[0023] Figure 3 is an explanatory diagram showing an example of a search method of the nearby image search unit of the image processing device of the imaging processing system according to Embodiment 1. As shown in Figure 3, the nearby image search unit 33 searches for a nearby image of a subject image 5B in which a feature point A is captured, while retaining the common feature point A, and the overall image 5A located at a distance L from the subject image 5B as far away in the depth direction as possible. Note that the overall image 5A shown in Figure 3 is shown with the marker 51 omitted. In this way, the greater the parallax, that is, the more three-dimensional the image, the better the accuracy of depth estimation in the triangulation described later, and the better the accuracy of the estimation of external parameters.
[0024] The external parameter estimation unit 34 estimates the external parameters of the second imaging unit 20 in the world coordinate system that correspond to the overall image 5A captured by the second imaging unit 20. The external parameter estimation unit 34 also estimates the external parameters of the second imaging unit 20 in the world coordinate system that correspond to the subject image 5B captured by the second imaging unit 20. The external parameters estimated by the external parameter estimation unit 34 are associated with the overall image 5A or subject image 5B that corresponds to the external parameters and output to the external parameter information holding unit 32, where they are held as external parameter information.
[0025] First, a method for estimating the external parameters of the second imaging unit 20 corresponding to the overall image 5A captured by the second imaging unit 20 will be explained. If the overall image 5A captured by the second imaging unit 20 includes a marker 51, the external parameter estimation unit 34 estimates the external parameters of the second imaging unit 20 based on the two-dimensional coordinates of the feature points of the marker 51, the three-dimensional coordinate position and orientation of the marker 51 in the world coordinate system identified by the first position information receiving unit 12 and the gyro sensor 13, and the internal parameters and distortion parameters of the second imaging unit 20. Note that the overall image 5A captured by the second imaging unit 20 and the three-dimensional coordinate position and orientation of the marker 51 acquired by the first position information receiving unit 12 and the gyro sensor 13 must be acquired at the same time. Furthermore, the information of the marker 51 identified by the first position information receiving unit 12 and the gyro sensor 13 may consist of six parameters representing the three-dimensional coordinate values and orientation values of the marker 51, or it may consist of the three-dimensional coordinate values of four or more feature points on the marker 51.
[0026] Figure 4 is a flowchart showing the procedure for estimating the external parameters of the second imaging unit from an overall image in the external parameter estimation unit of the image processing device of the imaging processing system according to Embodiment 1. As shown in Figure 4, the external parameter estimation unit 34 acquires the overall image 5A captured by the second imaging unit 20 of the second mobile body 2 (step S11) and extracts the markers 51 included in the overall image 5A (step S12). If there are multiple second mobile bodies 2, the external parameter estimation unit 34 acquires the overall image 5A separately from each second mobile body 2 and performs parallel processing.
[0027] Next, the external parameter estimation unit 34 calculates the three-dimensional coordinate position of the marker 51 in the world coordinate system, which has been identified by the first position information receiving unit 12 and the gyro sensor 13 (step S13). Finally, the external parameter estimation unit 34 calculates the external parameters of the second imaging unit 20 based on the two-dimensional pixel coordinates of the extracted feature points of the marker 51 in the overall image 5A, the three-dimensional coordinate position of the marker 51 in the world coordinate system, and the internal parameters and distortion parameters of the second imaging unit 20 (step S14). Here, the external parameters are the three-dimensional coordinate position and orientation of the second imaging unit 20 in the world coordinate system.
[0028] Here, we will explain how the external parameter estimation unit 34 calculates the external parameters of the second imaging unit 20 in the world coordinate system. Figure 5 is an explanatory diagram showing an example of the processing of the external parameter estimation unit in the image processing device of the imaging processing system according to Embodiment 1. As shown in Figure 5, the external parameter estimation unit 34 estimates the external parameters of the second imaging unit 20 using the two-dimensional coordinates of the feature points of the marker 51 in the overall image 5A, the three-dimensional coordinates of the marker 51 in the three-dimensional space which is the real space, and the internal parameters and distortion parameters of the second imaging unit 20. The internal parameters of the second imaging unit 20 are, for example, the focal length or the coordinates of the image center, which are internal features of the second imaging unit 20. The distortion parameters of the second imaging unit 20 are, for example, lens distortion. Specifically, as shown in Figure 5, the external parameter estimation unit 34 uses the two-dimensional coordinates (u, v) of the feature point P1 of the marker 51 in the overall image 5A and the three-dimensional coordinates (X) of the feature point P2 of the marker 51 in the real space which corresponds to feature point P1. w , Y w , Z wThe external parameters of the second imaging unit 20 are estimated by solving the so-called PnP problem (Perspective-n-Point Problem) based on the correspondence with the marker 51. When solving the PnP problem, if the marker 51 is placed two-dimensionally on two axes of the reference world coordinate system, i.e., on a plane, at least four feature points are required that correspond to three-dimensional points in real space and two-dimensional points on the image. Also, when solving the PnP problem, if the marker 51 is placed three-dimensionally in the reference world coordinate system, at least six feature points are required that correspond to three-dimensional points in real space and two-dimensional points on the image. However, if the orientation of the marker 51 is known, the external parameters of the second imaging unit 20 can be solved by solving the PnP problem even with only four feature points.
[0029] The processing procedure of the external parameter estimation unit 34 in the image processing device 3 of the imaging processing system 100 according to Embodiment 1 is not limited to the above configuration. Figure 6 is a flowchart showing different procedures for estimating the external parameters of the second imaging unit from the overall image in the external parameter estimation unit of the image processing device of the imaging processing system according to Embodiment 1.
[0030] As shown in Figure 6, the external parameter estimation unit 34 acquires the overall image 5A captured by the second imaging unit 20 of the second mobile body 2 (step S21) and extracts the marker 51 included in the overall image 5A (step S22). Next, the external parameter estimation unit 34 estimates the external parameters of the second imaging unit 20 when one specific feature point of the marker 51 is set as the origin (step S23). Hereinafter, the coordinate system with one specific feature point of the marker 51 as the origin is called the marker coordinate system. The external parameters of the second imaging unit 20 in the marker coordinate system can be calculated based on the two-dimensional pixel coordinates of the extracted marker 51 feature points in the overall image 5A, the coordinate positions of the feature points of the marker 51 in the marker coordinate system, and the internal parameters and distortion parameters of the second imaging unit 20.
[0031] Next, the external parameter estimation unit 34 calculates a conversion formula to convert the marker coordinate system to the world coordinate system based on the first position information receiving unit 12 and the gyro sensor 13 (step S24). Finally, using the calculated conversion formula, the external parameters of the second imaging unit 20 in the marker coordinate system are converted to the external parameters of the second imaging unit 20 in the world coordinate system (step S25).
[0032] Next, a method for estimating the external parameters of the second imaging unit 20 corresponding to the subject image 5B captured by the second imaging unit 20 will be described. When the image captured by the second imaging unit 20 is the subject image 5B, the external parameter estimation unit 34 uses at least two neighboring images retrieved by the neighboring image search unit 33 and the external parameters of those neighboring images as a set to estimate the external parameters of the second imaging unit 20 corresponding to the subject image 5B by triangulation. A neighboring image is either a whole image 5A whose coordinate system within the image is the world coordinate system and whose external parameters are known, or a subject image 5B whose external parameters are known.
[0033] Figure 7 is a flowchart showing the shooting process method of the shooting processing system according to Embodiment 1. As shown in Figure 7, first, the second shooting unit 20 of the second mobile body 2 captures a scene including the subject 50 and the marker 51 provided on the outer surface of the first mobile body 1 to acquire an overall image 5A, and stores the overall image 5A in association with the shooting time. In addition, the second shooting unit 20 of the second mobile body 2 may capture the subject 50 in a state where the marker 51 cannot be used because it has not captured the feature points necessary to recognize the marker 51, and acquire a subject image 5B, and stores the subject image 5B in association with the shooting time (step S301).
[0034] Next, the image discrimination unit 21 determines whether or not there is a marker 51 in the image captured by the second shooting unit 20, thereby distinguishing between the overall image 5A and the subject image 5B (step S302).
[0035] Next, if the image captured by the second imaging unit 20 is a whole image 5A that includes all the feature points of the marker 51 (step S303: Yes), the external parameter estimation unit 34 estimates the external parameters of the second imaging unit 20 that correspond to the whole image 5A captured by the second imaging unit 20 (step S304). Then, the estimated external parameters are associated with the whole image 5A and stored in the external parameter information holding unit 32 (step S305).
[0036] On the other hand, if the subject image 5B captured by the second imaging unit 20 is in a state where the marker 51 cannot be used because it does not capture the feature points necessary to recognize the marker 51 (step S303: No), the neighboring image search unit 33 searches for at least two images that are in the vicinity of the subject image 5B from among the overall image 5A or subject image 5B held in the external parameter information holding unit 32 (step S306). The external parameter estimation unit 34 uses the at least two neighboring images found by the neighboring image search unit 33 and the external parameters of those neighboring images as a set to estimate the external parameters of the second imaging unit 20 corresponding to the subject image 5B by triangulation (step S307). Then, the estimated external parameters are associated with the subject image 5B and held in the external parameter information holding unit 32 (step S308).
[0037] Next, the three-dimensional spatial model generation unit 35 will be described. The three-dimensional spatial model generation unit 35 uses techniques such as NeRF (Neural Radiance Fields), volumetric video, or 3D Gaussian splatting to reconstruct the subject 50 contained in the subject image 5B into three-dimensional visual information, and generates a free-viewpoint video using the reconstructed three-dimensional visual information. The three-dimensional spatial model generation unit 35 acquires the overall image 5A captured by the second shooting unit 20 and the external parameters of the second mobile body 2 corresponding to the overall image 5A in association. The three-dimensional spatial model generation unit 35 also acquires the subject image 5B captured by the second shooting unit 20 and the external parameters of the second mobile body 2 corresponding to the subject image 5B in association. The three-dimensional spatial model generation unit 35 may or may not acquire the subject image 5B captured by the first shooting unit 10 and the external parameters of the first mobile body 1 corresponding to the subject image 5B in association. Then, the three-dimensional spatial model generation unit 35 reconstructs the subject 50 into three-dimensional visual information based on the acquired information, and generates a free-viewpoint video using the reconstructed three-dimensional visual information.
[0038] However, for example, when the second mobile body 2 photographs the marker 51 at a position far from the first mobile body 1, even if three-dimensional visual information is restored using the overall image 5A or the subject image 5B, the three-dimensional visual information may become slightly blurred. If the three-dimensional visual information is restored in a blurred state, the free-viewpoint video may break down. Therefore, the three-dimensional spatial model generation unit 35 determines whether there are any errors in the external parameters of the second shooting unit 20 corresponding to the overall image 5A or the subject image 5B, based on the restored three-dimensional visual information of the subject 50, in order to prevent the free-viewpoint video from breaking down. If it determines that there are errors, it corrects the external parameters of the second shooting unit 20 corresponding to the overall image 5A or the subject image 5B to restore the three-dimensional visual information. This makes it possible to restore the three-dimensional visual information with high accuracy and suppress the breakdown of the free-viewpoint video. The three-dimensional spatial model generation unit 35 then outputs the external parameters of the second imaging unit 20 corresponding to the corrected overall image 5A or subject image 5B to the external parameter information holding unit 32, and updates the external parameter information held in the external parameter information holding unit 32.
[0039] Furthermore, the three-dimensional spatial model generation unit 35 may use depth information obtained from captured images as a means of reconstructing three-dimensional visual information in order to more robustly estimate the surface position in three-dimensional space. Depth information can be obtained by using a stereo camera in the second imaging unit 20 or by using a depth estimation AI model.
[0040] Furthermore, the three-dimensional spatial model generation unit 35 may be configured as a separate device independent of the image processing device 3, and connected to the information and communication unit 30 via an external network.
[0041] The free-viewpoint video generated by the three-dimensional spatial model generation unit 35 is transmitted to the display device 4 via an external network upon request from the display device 4, and output by the display device 4. The display device 4 may be, for example, a smartphone, a tablet device, VR goggles with VR (Virtual Reality) image distribution functionality, AR (Augmented Reality) glasses, or a display.
[0042] FIG. 8 is a block diagram illustrating Modification Example 1 of the imaging processing system according to the first embodiment. The imaging processing system 100A shown in FIG. 8 is configured such that the first mobile object 1 receives and aggregates the information acquired by the second mobile object 2 via the first data communication unit 11. The first mobile object 1 collectively transmits the information acquired by the second mobile object 2 together with the information acquired by the first mobile object 1 to the image processing apparatus 3. The information acquired by the first mobile object 1 refers to the subject image 5B captured by the first imaging unit 10, the three-dimensional coordinate position of the first mobile object 1 in the world coordinate system acquired by the first position information receiving unit 12, the attitude of the first mobile object 1 acquired by the gyro sensor 13, and the ID of the first mobile object 1 associated therewith. The information acquired by the second mobile object 2 refers to the entire image 5A and the subject image 5B captured by the second imaging unit 20, and the ID of the second mobile object 2 associated therewith.
[0043] FIG. 9 is a block diagram illustrating Modification Example 2 of the imaging processing system according to the first embodiment. The imaging processing system 100B shown in FIG. 9 is configured such that the first mobile object 1 does not include the first imaging unit 10. That is, the first mobile object 1 is provided with a marker 51 on its outer surface, moves around the subject 50, and is used only for estimating the external parameters of the second imaging unit 20. Note that the imaging processing system 100A shown in FIG. 8 may also be configured such that the first mobile object 1 does not include the first imaging unit 10.
[0044] As described above, in the imaging processing system 100 according to Embodiment 1, if the image captured by the second imaging unit 20 is a subject image 5B in a state where the marker 51 cannot be used because it does not capture the feature points necessary for recognizing the marker 51, the system uses at least two neighboring images retrieved by the neighboring image search unit 33 and the corresponding external parameters of those neighboring images as a set to estimate the external parameters of the second imaging unit 20 corresponding to the subject image 5B by triangulation. Therefore, in the imaging processing system 100 according to Embodiment 1, even if the subject image 5B is a subject image 5B taken of the subject 50 in a state where the marker 51 cannot be used because it does not capture the feature points necessary for recognizing the marker 51, the external parameters of the second imaging unit 20 corresponding to the subject image 5B can be accurately estimated by triangulation using neighboring images for the subject image 5B, and the three-dimensional visual information of the subject 50 can be reconstructed using the subject image 5B. In addition, in the imaging processing system 100 according to Embodiment 1, it is also possible to create three-dimensional point cloud information using the reconstructed three-dimensional visual information.
[0045] Furthermore, in the first embodiment of the imaging processing system 100, based on the three-dimensional visual information of the restored subject 50, it is determined whether there are any errors in the external parameters of the second imaging unit 20 corresponding to the overall image 5A or the subject image 5B. If it is determined that there are errors, the system corrects the errors and updates the external parameter information held in the external parameter information holding unit 32. Therefore, in the first embodiment of the imaging processing system 100, the external parameters of the second imaging unit 20 corresponding to the subject image 5B, whose external parameters are unknown, can be estimated more accurately using a neighboring image of the subject image 5B.
[0046] Note that the marker 51 does not necessarily need to be provided on the outer surface of the first moving body 1, and may, for example, be arranged around the subject 50. In this case, the configuration of the first moving body 1 is omitted, and the system is configured only with the second moving body 2. Then, when the image captured by the second imaging unit 20 is the entire image 5A that includes all feature points of the marker 51 in the image, the external parameter estimation unit estimates the external parameters of the second imaging unit 20 based on the two-dimensional coordinates of the feature points of the marker 51 included in the entire image 5A, and the known three-dimensional coordinate position and orientation of the marker 51 in the world coordinate system.
[0047] Second Embodiment. Next, an imaging processing system 101 according to the second embodiment will be described. Figure 10 is a block diagram illustrating the imaging processing system according to the second embodiment. Note that constituent elements identical to those in the first embodiment are denoted by the same reference numerals, and descriptions thereof are omitted as appropriate. As shown in Figure 10, the imaging processing system 101 according to the second embodiment further includes a neighboring image generation unit 36 in addition to the configuration of the first embodiment.
[0048] When no neighboring image of the subject image 5B is stored in the external parameter information holding unit 32, or when the neighboring images stored in the external parameter information holding unit 32 are insufficient, the neighboring image search unit 33 cannot search for the neighboring image of the subject image 5B. In such a case, the neighboring image generation unit 36 generates, based on the three-dimensional visual information restored by the three-dimensional space model generation unit 35, an image of the subject 50 corresponding to a known external parameter from the known external parameter as a generated image. In the imaging processing system 101, while the entire image 5A or the subject image 5B is captured by the first moving body 1 and the second moving body 2, the three-dimensional visual information is restored by the three-dimensional space model generation unit 35 at the same time. Therefore, by inputting a known external parameter into the already restored three-dimensional visual information, an image of the subject 50 corresponding to the external parameter can be generated. The generated image generated by the neighboring image generation unit 36 is stored in the external parameter information holding unit 32 in association with the known external parameter.
[0049] In Embodiment 2, the neighboring image search unit 33 searches for at least two neighboring images taken near the subject image 5B, whose external parameters are unknown, from the external parameter information held in the external parameter information holding unit 32, within a preset threshold range. A neighboring image is either a whole image 5A, whose external parameters are known and whose coordinate system is the world coordinate system, or a subject image 5B, whose external parameters are known. The threshold is, for example, the distance traveled and the rotation angle of the second mobile body 2 that took the subject image 5B, whose external parameters are unknown, relative to the position and orientation of the second mobile body 2 that took the whole image 5A or subject image 5B, whose external parameters are known. If the neighboring image search unit 33 cannot find at least two neighboring images within the preset threshold range, the neighboring image generation unit 36 generates an image of the subject 50, which will be a neighboring image of the subject image 5B, as a generated image. The neighboring image search unit 33 then searches for at least two images, including the generated image generated by the neighboring image generation unit 36, as neighboring images. The neighboring images here may all be generated images, or they may be images that combine a generated image with an overall image 5A or subject image 5B with known external parameters.
[0050] If the neighboring image search unit 33 searches for at least two neighboring images within a threshold range, the external parameter estimation unit 34 uses the at least two searched neighboring images and the external parameters of those neighboring images as a set to estimate the external parameters of the second imaging unit 20 corresponding to the subject image 5B by triangulation.
[0051] On the other hand, if the neighboring image search unit 33 is unable to find at least two neighboring images within a threshold range, the external parameter estimation unit 34 uses at least two images, including the generated images generated by the neighboring image generation unit 36, and the external parameters of those images as a set to estimate the external parameters of the second imaging unit 20 corresponding to the subject image 5B by triangulation. Alternatively, the external parameter estimation unit 34 may combine the overall image 5A or subject image 5B, whose external parameters are known, with the generated images generated by the neighboring image generation unit 36 to create at least two images as neighboring images, and use these images to estimate the external parameters of the second imaging unit 20 corresponding to the subject image 5B.
[0052] Furthermore, when performing triangulation using the generated image produced by the neighboring image generation unit 36, the feature points used may be selected from the non-blurred image regions, excluding the blurred image regions, after estimating the blurred image regions within the image. Methods for estimating blurred image regions include, for example, analyzing an image with edges enhanced using a Laplacian filter, frequency domain analysis such as Fourier transform, or machine learning. Other methods may also be used to estimate blurred image regions. This prevents the selection of feature points from blurred image regions, which are areas where three-dimensional visual information has not been sufficiently restored, thus enabling accurate estimation of the external parameters of the second imaging unit 20 for the subject image 5B by triangulation.
[0053] Furthermore, when the external parameter estimation unit 34 estimates the external parameters of the second imaging unit 20 corresponding to the subject image 5B, the estimation accuracy is higher when using the overall image 5A than when using the generated image. Therefore, if the overall image 5A is acquired after the external parameters of the second imaging unit 20 corresponding to the subject image 5B have been estimated using the generated image, the external parameter estimation unit 34 will estimate the external parameters of the second imaging unit 20 corresponding to the subject image 5B using triangulation with the overall image 5A, and the external parameter information held in the external parameter information holding unit 32 will be updated.
[0054] Furthermore, in Embodiment 2, the three-dimensional spatial model generation unit 35 also determines whether there are any errors in the external parameters of the second shooting unit 20 corresponding to the overall image 5A or subject image 5B, based on the three-dimensional visual information of the restored subject 50, so that the free-viewpoint image does not break down. If it determines that there are errors, it corrects the external parameters of the second shooting unit 20 corresponding to the overall image 5A or subject image 5B to restore the three-dimensional visual information. The three-dimensional spatial model generation unit 35 then outputs the corrected external parameters of the second shooting unit 20 corresponding to the overall image 5A or subject image 5B to the external parameter information holding unit 32, and updates the external parameter information held in the external parameter information holding unit 32.
[0055] Figure 11 is a flowchart showing the imaging processing method of the imaging processing system according to Embodiment 2. Steps S401 to S405 shown in Figure 11 have the same configuration as steps S301 to S305 shown in Figure 7, so their explanation is omitted.
[0056] If the subject image 5B captured by the second imaging unit 20 is in a state where the marker 51 cannot be used because it does not capture the feature points necessary to recognize the marker 51 (Step S403: No), the neighboring image search unit 33 searches for images in the vicinity of the subject image 5B from among the overall image 5A or subject image 5B held in the external parameter information holding unit 32 (Step S406). The neighboring image search unit 33 then determines whether there are at least two neighboring images with known external parameters that were captured near the subject image 5B, within a preset threshold range (Step S407). If there are at least two neighboring images for the subject image 5B within the threshold range (Step S407: Yes), the external parameter estimation unit 34 uses the at least two neighboring images retrieved by the neighboring image search unit 33 and the external parameters of those neighboring images as a set, and estimates the external parameters of the second imaging unit 20 corresponding to the subject image 5B by triangulation (Step S408). Then, the estimated external parameters are associated with the subject image 5B and stored in the external parameter information holding unit 32 (step S409).
[0057] On the other hand, if there are no neighboring images for the subject image 5B within the threshold range (step S407: No), the neighboring image generation unit 36 generates an image of the subject 50 corresponding to the known external parameters from the three-dimensional visual information already restored by the three-dimensional spatial model generation unit 35 as a generated image (step S410). The generated image generated by the neighboring image generation unit 36 is held by the external parameter information holding unit 32. The neighboring image search unit 33 then searches for at least two images, including the generated image, as neighboring images from the external parameter information held in the external parameter information holding unit 32 (step S411). The external parameter estimation unit 34 uses the at least two images, including the generated image, found by the neighboring image search unit 33, and the external parameters of those images as a set, and estimates the external parameters of the second imaging unit 20 corresponding to the subject image 5B by triangulation (step S412). The estimated external parameters are then associated with the subject image 5B and held in the external parameter information holding unit 32 (step S413).
[0058] As described above, in the shooting processing system 101 according to Embodiment 2, if the nearby image search unit 33 is unable to search for a nearby image within a preset threshold range, the nearby image generation unit 36 generates an image of the subject 50 that will be a nearby image of the subject image 5B as a generated image. The external parameter estimation unit 34 then uses at least two images, including the generated image generated by the nearby image generation unit 36, and the external parameters corresponding to those images as a set, and estimates the external parameters of the second shooting unit 20 corresponding to the subject image 5B by triangulation. Therefore, even in the shooting processing system 101 according to Embodiment 2, even if the subject image 5B is taken in a state where the feature points necessary for marker 51 recognition cannot be captured and the marker 51 cannot be used, the external parameters of the second shooting unit 20 corresponding to the subject image 5B can be accurately estimated by triangulation using the nearby image or generated image for the subject image 5B, and the three-dimensional visual information of the subject 50 can be reconstructed using the subject image 5B.
[0059] Furthermore, in the shooting processing system 101 according to Embodiment 2, based on the three-dimensional visual information of the restored subject 50, it is determined whether there are any errors in the external parameters of the second shooting unit 20 corresponding to the overall image 5A or the subject image 5B. If it is determined that there are errors, the errors are corrected and the external parameter information held in the external parameter information holding unit 32 is updated. Therefore, in the shooting processing system 101 according to Embodiment 2, the external parameters of the second shooting unit 20 corresponding to the subject image 5B, whose external parameters are unknown, can be estimated more accurately using a neighboring image of the subject image 5B.
[0060] The configurations shown in the above embodiments are examples only, and it is possible to combine them with other known technologies, combine different embodiments, and omit or modify parts of the configuration without departing from the gist of the invention.
[0061] 1 First mobile body, 2 Second mobile body, 3 Image processing device, 4 Display device, 5A Overall image, 5B Subject image, 10 First shooting unit, 11 First data communication unit, 12 First position information receiving unit, 13 Gyro sensor, 20 Second shooting unit, 21 Image discrimination unit, 22 Second data communication unit, 30 Information communication unit, 31 Marker information holding unit, 32 External parameter information holding unit, 33 Nearby image search unit, 34 External parameter estimation unit, 35 Three-dimensional spatial model generation unit, 36 Nearby image generation unit, 50 Subject, 51 Marker, 100, 100A, 101 Shooting processing system.
Claims
1. The system comprises at least one second mobile body having a second shooting unit that moves around a subject and captures a scene including the subject and at least one marker placed around the subject to capture an image including the subject, and an image processing device that processes the image using information obtained by the second mobile body, wherein the image processing device has an external parameter information holding unit that holds the external parameters of the second shooting unit corresponding to the image captured by the second shooting unit in association with each image, a nearby image search unit that searches for at least two nearby images captured in the vicinity of the image for which the external parameters are unknown from the external parameter information held by the external parameter information holding unit, and an external parameter estimation unit that estimates the external parameters of the second shooting unit corresponding to the image, wherein the external parameter estimation unit is A shooting processing system characterized in that, if the image captured by the second shooting unit is a subject image in which the marker cannot be used because it does not capture the feature points necessary to recognize the marker, the system uses at least two of the neighboring images retrieved by the neighboring image search unit and the external parameters corresponding to the neighboring images as a set to estimate the external parameters of the second shooting unit corresponding to the subject image by triangulation.
2. The imaging processing system according to claim 1, further comprising a three-dimensional spatial model generation unit that reconstructs three-dimensional visual information of the subject using the image captured by the second imaging unit and the external parameters of the second imaging unit estimated by the external parameter estimation unit.
3. The device comprises: at least one first mobile body having one or a plurality of markers of different shapes on its outer surface and moving around a subject; at least one second mobile body having a second imaging unit that moves around the subject and captures an image including at least the subject; and an image processing device that processes the image using information obtained by the first mobile body and the second mobile body, wherein the image processing device includes: an external parameter information holding unit that holds the external parameters of the second imaging unit corresponding to the image captured by the second imaging unit in association with each image; a nearby image search unit that searches for at least two nearby images captured in the vicinity of an image whose external parameters are unknown from the external parameter information held by the external parameter information holding unit; an external parameter estimation unit that estimates the external parameters of the second imaging unit corresponding to the image; and a three-dimensional spatial model generation unit that reconstructs the three-dimensional visual information of the subject using the image captured by the second imaging unit and the external parameters of the second imaging unit estimated by the external parameter estimation unit. A shooting processing system comprising: a neighbor image generation unit that generates an image of the subject corresponding to known external parameters as a generated image based on the three-dimensional visual information already restored by the three-dimensional spatial model generation unit; a neighbor image search unit that searches for at least two neighbor images taken in the vicinity of the image for which the external parameters are unknown, within a preset threshold range, from the external parameter information held in the external parameter information holding unit, and if at least two neighbor images cannot be found within the threshold range, searches for at least two images including the generated image generated by the neighbor image generation unit as the neighbor images; and an external parameter estimation unit that, if the image taken by the second shooting unit is a subject image in which the feature points necessary for recognizing the marker cannot be captured and the marker cannot be used, uses at least two neighbor images found by the neighbor image search unit and the external parameters corresponding to the neighbor images as a set, and estimates the external parameters of the second shooting unit corresponding to the subject image by triangulation.
4. The imaging processing system according to claim 2 or 3, characterized in that the three-dimensional spatial model generation unit determines whether there is an error in the external parameters of the second imaging unit corresponding to the image based on the three-dimensional visual information of the restored subject, and if it determines that an error has occurred, corrects the error and updates the external parameter information held in the external parameter information holding unit.
5. The imaging processing system according to any one of claims 2 to 4, characterized in that the three-dimensional spatial model generation unit reconstructs three-dimensional visual information using depth information obtained from the captured image.
6. The shooting processing system according to any one of claims 1 to 5, characterized in that the nearby image search unit searches for nearby images as images that are located at a distance in the depth direction from the subject image while retaining the common feature points with respect to the subject image in which the feature points are captured.
7. The imaging processing system according to any one of claims 1 to 6, characterized in that, if the image captured by the second imaging unit is a whole image that includes all the feature points of the marker in the image, the external parameter estimation unit estimates the external parameters of the second imaging unit based on the two-dimensional coordinates of the feature points of the marker included in the whole image and the known three-dimensional coordinate position and orientation of the marker in the world coordinate system.
8. The imaging processing system according to any one of claims 1 to 7, further comprising at least one first moving body having one or a plurality of markers of different shapes provided on its outer surface, which moves around a subject.
9. The imaging processing system according to 8, wherein the first mobile body includes a first position information receiving unit that receives three-dimensional position information from a three-dimensional position information transmitting device that transmits three-dimensional position information of the mobile body, and a gyro sensor that acquires its own orientation, and the external parameter estimation unit estimates the external parameters of the second imaging unit based on the two-dimensional coordinates of the feature points of the marker included in the whole image and the three-dimensional coordinate position and orientation of the marker in the world coordinate system identified by the first position information receiving unit and the gyro sensor, when the image captured by the second imaging unit is a whole image that includes all the feature points of the marker in the whole image.
10. The imaging processing system according to claim 8 or 9, characterized in that the first mobile body further comprises a first imaging unit that photographs the subject and acquires a subject image.
11. The imaging processing system according to any one of claims 8 to 10, characterized in that the first mobile body further comprises a first data communication unit for transmitting acquired information to the image processing device, and the second mobile body further comprises a second data communication unit for transmitting acquired information to the image processing device.
12. The imaging processing system according to any one of claims 8 to 10, characterized in that the first mobile body further comprises a first data communication unit for transmitting acquired information to the image processing device, the second mobile body further comprises a second data communication unit for transmitting acquired information to the first mobile body, and the first mobile body receives and aggregates the information acquired by the second mobile body, and transmits the aggregated information to the image processing device via the first data communication unit.
13. A method for processing an image to reconstruct three-dimensional visual information of a subject, which involves processing an image using information obtained from at least one mobile body having a shooting unit that moves around a subject and captures a scene including the subject and at least one marker placed around the subject, the image being captured by the shooting unit is a whole image that includes all the feature points of the marker in the image, the method comprising: estimating the external parameters of the shooting unit based on the two-dimensional coordinates of the feature points of the marker included in the whole image and the three-dimensional coordinate position and orientation of the marker in the world coordinate system, and storing the estimated external parameters in an external parameter information holding unit; and, if the image being captured by the shooting unit is a subject image in which the marker cannot be used because it fails to capture the feature points necessary to recognize the marker, the method comprising: searching for at least two neighboring images with known external parameters, which were captured near the subject image, from the external parameter information held in the external parameter information holding unit. A method for photographing an object, comprising the steps of: using at least two of the aforementioned neighboring images and external parameters corresponding to the neighboring images as a set; estimating the external parameters of the imaging unit corresponding to the subject image by triangulation; and storing the estimated external parameters in an external parameter information holding unit.
14. A method for processing an image to reconstruct three-dimensional visual information of a subject, which involves processing an image using information obtained from at least one mobile body having a shooting unit that moves around a subject and captures a scene including the subject and at least one marker placed around the subject, the image being captured by the shooting unit is a whole image that includes all the feature points of the marker, the method comprising: estimating the external parameters of the shooting unit based on the two-dimensional coordinates of the feature points of the marker included in the whole image and the three-dimensional coordinate position and orientation of the marker in the world coordinate system, and storing the estimated external parameters in an external parameter information holding unit; and, if the image being captured by the shooting unit is a subject image in which the marker cannot be used because it does not capture the feature points necessary to recognize the marker, the method comprising: searching for at least two neighboring images with known external parameters, captured near the subject image, from the external parameter information held in the external parameter information holding unit, within a preset threshold range. A shooting processing method characterized by the following steps: when the image captured by the shooting unit is a subject image in which the marker cannot be used because it fails to capture the feature points necessary to recognize the marker, and at least two neighboring images cannot be found within the threshold range, the method generates an image of the subject corresponding to the external parameters from known external parameters based on the already restored three-dimensional visual information, and searches for at least two images including the generated image as neighboring images; and using at least two of the neighboring images and the external parameters corresponding to the neighboring images as a set, the method estimates the external parameters of the shooting unit corresponding to the subject image by triangulation, and stores the estimated external parameters in the external parameter information holding unit.
