Image processing system, image processing method, and image processing program
The image processing system addresses the challenge of low-quality blurred regions by estimating and masking these areas, allowing robust estimation and efficient generation of three-dimensional visual information and free-viewpoint videos.
Patent Information
- Application Number
- PCT/JP2025/026057
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-08
- Filing Date
- 2025-07-23
- Publication Date
- 2026-11-12
AI Technical Summary
Existing image processing systems struggle to accurately estimate the relative position and orientation of imaging devices when low-quality blurred regions are present in captured images, leading to reduced reliability and efficiency in generating three-dimensional visual information and free-viewpoint videos.
An image processing system that estimates blurred areas with low image quality, extracts feature points from non-blurred regions, applies masks to these areas, and reconstructs three-dimensional visual information using the relative position and orientation of the imaging device, enabling robust estimation and efficient generation of free-viewpoint videos.
The system effectively estimates the relative position and orientation of imaging devices even with blurry areas, ensuring high-quality image processing and efficient generation of three-dimensional visual information and free-viewpoint videos.
Smart Images

Figure JP2025026057_12112026_PF_FP_ABST
Abstract
Description
Image processing system, image processing method, and image processing program
[0001] The present disclosure relates to an image processing system, an image processing method, and an image processing program.
[0002] Conventionally, there is known an image processing system that uses feature points extracted from subject images captured by an imaging device to estimate the relative position and relative orientation of the imaging device that captured two subject images, restores three-dimensional visual information of the subject from the subject images and the relative position and relative orientation of the imaging device corresponding to the subject images, and generates a free-viewpoint video using the three-dimensional visual information. If a blurred region exists in a part of the subject image captured by the imaging device and the image quality of the image is low, the reliability of the extracted feature points may decrease. In this case, the estimation accuracy of the relative position and relative orientation of the imaging device that captured the two subject images deteriorates, the three-dimensional visual information of the subject cannot be sufficiently restored, and the free-viewpoint video may be corrupted. For example, Patent Document 1 discloses a method of using only high-image-quality images when generating a three-dimensional space.
[0003] Japanese Unexamined Patent Application Publication No. 2023-75860
[0004] However, the technique disclosed in Patent Document 1 uses only high-image-quality images without using low-image-quality images, which may lead to reduced efficiency in generating the three-dimensional space.
[0005] The present disclosure has been made in view of the above, and an object thereof is to obtain an image processing system that can robustly estimate the relative position and relative orientation of an imaging device that captured at least two subject images even when there is a low-image-quality blurred region in a part of the subject images captured by the imaging device, and can efficiently generate a free-viewpoint video.
[0006] To solve the above-mentioned problems and achieve the objective, the image processing system according to this disclosure is an image processing system for reconstructing three-dimensional visual information of a subject from an image of a subject captured by a camera, comprising: an image processing unit that estimates blurred areas with low image quality in the image of the subject; a feature point extraction unit that extracts a plurality of feature points from non-blurred areas excluding the blurred areas; a position and orientation estimation unit that estimates the relative position and orientation of a camera that captured another image of a subject having the same feature points to a camera that captured an image of a subject having the same feature points, using the plurality of feature points in the image; a mask processing unit that applies a mask to the blurred areas; and a three-dimensional spatial model generation unit that reconstructs three-dimensional visual information of a subject using the subject image from which the blurred areas have been masked and the relative position and orientation of the camera corresponding to the subject image.
[0007] The image processing system described herein has the effect of being able to robustly estimate the relative position and relative orientation of the camera that captured at least two subject images, even if there are blurry areas with low image quality in a portion of the subject image captured by the camera, and efficiently generate free-viewpoint video.
[0008] Overall configuration diagram schematically showing the image processing system according to Embodiment 1 Block diagram showing the image processing system according to Embodiment 1 Flowchart showing the processing procedure of the image processing system according to Embodiment 1 Flowchart showing the processing procedure of the image processing system according to Embodiment 1 Block diagram showing modified example 1 of the image processing system according to Embodiment 1 Block diagram showing modified example 2 of the image processing system according to Embodiment 1 An explanatory diagram showing depth estimation performed by the image processing unit in the image processing system according to Embodiment 2 Flowchart showing the processing procedure of the image processing system according to Embodiment 2 Flowchart showing the processing procedure of the image processing system according to Embodiment 2 An explanatory diagram showing an example of the configuration of a computer system that realizes the image processing system according to this embodiment
[0009] The image processing system, image processing method, and image processing program according to embodiments of this disclosure will be described in detail below with reference to the drawings.
[0010] Embodiment 1. Figure 1 is a schematic overall diagram showing the image processing system according to Embodiment 1. Figure 2 is a block diagram showing the image processing system according to Embodiment 1. As shown in Figures 1 and 2, the image processing system 100 according to Embodiment 1 reconstructs three-dimensional visual information of a subject 50 from a plurality of subject images 5 captured by the shooting device 1, and generates a free-viewpoint image using the three-dimensional visual information. Note that the three-dimensional visual information includes, for example, three-dimensional shape. The image processing system 100 according to Embodiment 1 comprises a shooting device 1, an image processing device 2, and a three-dimensional spatial reconstruction device 3. As shown by the white arrows in Figure 1, in the image processing system 100, processing is performed in the order of shooting device 1, image processing device 2, and three-dimensional spatial reconstruction device 3, and the free-viewpoint image generated by the three-dimensional spatial reconstruction device 3 is displayed on the display device 4. In the image processing system 100 according to Embodiment 1, the shooting device 1, image processing device 2, and three-dimensional spatial reconstruction device 3 are each configured as standalone devices.
[0011] The imaging device 1 is, for example, composed of a monocular camera. The imaging device 1 photographs the subject 50 to acquire a subject image 5 and stores the subject image 5 in association with the time of shooting. The imaging device 1 transmits the captured subject image 5 to the image processing device 2, for example, via an external network.
[0012] As shown in Figure 2, the image processing device 2 includes a first image acquisition unit 20, an image processing unit 21, a feature point extraction unit 22, a position and orientation estimation unit 23, a three-dimensional coordinate calculation unit 24, and a position information holding unit 25. The first image acquisition unit 20 has the function of acquiring and holding a subject image 5 from the imaging device 1, for example, via an external network. The image processing unit 21 estimates areas with low image quality and blurring within the subject image 5 held by the first image acquisition unit 20 as blurred areas, and divides the image into blurred areas and high-quality non-blurred areas excluding the blurred areas. The image processing unit 21 may estimate the blurred areas in the subject image 5 using methods such as analyzing an image with edges enhanced using a Laplacian filter, frequency domain analysis such as Fourier transform, or machine learning, or it may use other methods. If the image processing unit 21 cannot estimate blurred areas based on a threshold, it determines that there are no blurred areas in the subject image 5.
[0013] Furthermore, the image processing unit 21 may be configured to perform image quality enhancement processing on the estimated blurred area. Image quality enhancement processing may include, for example, sharpening processing or super-resolution technology. The image processing unit 21 then re-estimates the blurred area in the blurred area where image quality enhancement processing has been performed.
[0014] The feature point extraction unit 22 extracts multiple feature points from the estimated non-blurred regions in the subject image 5. Feature points are subject information captured in the image, such as corners or edges with large image brightness gradients. Methods for extracting feature points include, for example, SIFT (Scale-Invariant Feature Transform) or FAST (Features from Accelerated Segment Test).
[0015] The position and orientation estimation unit 23 uses a plurality of feature points in the image extracted by the feature point extraction unit 22 to estimate the relative position and orientation of the imaging device 1 that captured another subject image 5 having the same feature points as the imaging device 1 that captured the subject image 5 having the same feature points. Specifically, the position and orientation estimation unit 23 uses a specific subject image 5 captured by the imaging device 1 as a reference image, and arbitrarily sets the distance between the imaging device 1 that captured the reference image 5 and the imaging device 1 that captured the other subject image 5, for other subject images 5 captured in the vicinity of this reference image. It is assumed that these two subject images 5 are captured at a position where their feature points overlap. The position and orientation estimation unit 23 then calculates an essential matrix from pairs of common feature points between the reference image 5 and the other subject images 5. Then, by performing Singular Value Decomposition (SVD) from the essential matrix, the relative position and orientation of the imaging device 1 that captured the other subject image 5 with respect to the imaging device 1 that captured the reference subject image 5 are estimated. However, when calculating the essential matrix, the intrinsic parameters and distortion parameters of the imaging device 1 at the time the subject image 5 was captured are required.
[0016] The three-dimensional coordinate calculation unit 24 calculates the three-dimensional coordinates of feature points in an image by triangulation, based on the relative position and orientation of the shooting device 1 that captured the reference image 5 relative to the shooting device 1 that captured the other subject image 5. The calculated three-dimensional coordinates of the feature points are stored by the position information storage unit 25. Furthermore, the relative position and orientation of the shooting device 1 that captured the other subject image 5 relative to the shooting device 1 that captured the reference image 5, and the three-dimensional coordinates of common feature points contained in the subject image 5 may be further corrected by performing bundle adjustment using the relative position and orientation of the shooting device 1 that captured the reference image 5 relative to the shooting device 1, and the three-dimensional coordinates of common feature points contained in the subject image 5. The corrected three-dimensional coordinates of the feature points are stored by the position information storage unit 25. The position information storage unit 25 stores the three-dimensional coordinates of feature points in the image of the subject image 5 captured by the shooting device 1 as three-dimensional position information in real space.
[0017] Subsequently, if the feature points in the newly captured subject image 5 by the imaging device 1 match the feature points whose three-dimensional positional information is known and is held in the position information holding unit 25, the position and orientation estimation unit 23 can estimate the relative position and orientation of the imaging device 1 that captured the new subject image 5 relative to the imaging device 1 that captured the reference image 5 by calculating PnP (Perspective-n-Point) using each feature point. Furthermore, the relative position and orientation between the imaging devices 1 and the three-dimensional coordinates of the feature points may be further corrected by performing bundle adjustment using the calculated relative position and orientation and the three-dimensional coordinates of the feature points used when calculating the relative position and orientation with PnP. In this case, the three-dimensional coordinates of the feature points held in the position information holding unit 25 are updated to the corrected three-dimensional coordinates. The position and orientation estimation unit 23 uses a specific subject image 5 as a reference image and sequentially estimates the relative position and orientation of the shooting device 1 that took other subject images 5 with respect to the shooting device 1 that took the subject image 5 which serves as the reference image, as well as the three-dimensional coordinates of feature points.
[0018] The three-dimensional coordinates of feature points calculated by the three-dimensional coordinate calculation unit 24 are calculated using an arbitrary scale initially set for a specific subject image 5 that serves as the reference image in the position and orientation estimation unit 23. Therefore, the three-dimensional coordinates of feature points do not match the scale, which is the unit of distance in reality. To address this, PnP is calculated using images taken from at least three viewpoints of feature points such as markers whose three-dimensional distances in real space are known, and the internal parameters and distortion parameters of the imaging device 1 that captured the images. For example, at least three absolute positions and absolute orientations of the imaging device 1 that captured an image of a certain feature point on a marker are calculated, and the three-dimensional coordinates of all feature points held in the position information holding unit 25 are updated. The position and orientation estimation unit 23 can then estimate the absolute position and absolute orientation of the imaging device 1 that captured each subject image 5, with the feature points included in the marker as the origin. In addition, as a method for updating the scale of the three-dimensional coordinates of feature points, the three-dimensional coordinates of all feature points held in the position information holding unit 25 can also be updated by solving the closed circle problem when the imaging device 1 returns to a previously captured position.
[0019] The three-dimensional spatial reconstruction device 3 includes a second image acquisition unit 30, a mask processing unit 31, and a three-dimensional spatial model generation unit 32. The second image acquisition unit 30 has the function of acquiring and storing image data processed by the image processing device 2, for example, via an external network. The mask processing unit 31 performs a masking process on the blurred areas separated by the image processing unit 21 in the subject image 5 held by the second image acquisition unit 30. Masking is a process that, for example, fills the blurred areas with black or white. As a result, the subject image 5 consists only of non-blurred areas with high image quality.
[0020] The three-dimensional spatial model generation unit 32 uses techniques such as NeRF (Neural Radiance Fields), volumetric video, or 3D Gaussian splatting to reconstruct the subject 50 contained in the subject image 5 into three-dimensional visual information using the subject image 5 and the relative position and relative orientation of the shooting device 1 corresponding to the subject image 5 estimated by the position and orientation estimation unit 23, and generates a free-viewpoint video using the three-dimensional visual information. The subject image 5 includes images in which blurred areas have been masked by the mask processing unit 31, and images in which no blurred areas exist. The free-viewpoint video generated by the three-dimensional spatial model generation unit 32 is transmitted to the display device 4 via an external network in response to a request from the display device 4, and output by the display device 4. The display device 4 is, for example, a smartphone, a tablet terminal, VR goggles with VR (Virtual Reality) image distribution functionality, AR (Augmented Reality) glasses, or a display.
[0021] Next, an example of the processing performed by the image processing system 100 described above will be explained based on a flowchart. Figures 3 and 4 are flowcharts showing the processing procedure of the image processing system according to Embodiment 1. As shown in Figure 3, first, the first image acquisition unit 20 acquires a subject image 5 captured by the shooting device 1 (step S101). Next, the image processing unit 21 estimates the blurred areas with low image quality in the subject image 5 acquired by the first image acquisition unit 20 (step S102). If the image processing unit 21 determines that there are blurred areas in the image (step S103: Yes), it divides the image into blurred areas and non-blurred areas with high image quality (step S104). Next, the feature point extraction unit 22 extracts a plurality of feature points from the non-blurred areas in the image (step S105). If the number of feature points extracted by the feature point extraction unit 22 is N or more (step S106: Yes), the position and orientation estimation unit 23 uses the multiple feature points to set a specific subject image 5 containing the feature points as a reference image, and estimates the relative position and orientation of the shooting device 1 that captured the subject image 5 relative to the shooting device 1 that captured the reference image 5 (step S107). If the feature points in the image of the acquired subject image 5 match feature points for which the three-dimensional position information held in the position information holding unit 25 is known, the position and orientation estimation unit 23 may use each feature point to calculate PnP and estimate the relative position and orientation of the shooting device 1 that captured the subject image 5 relative to the shooting device 1 that captured the reference image 5. Then, the mask processing unit 31 applies a mask to the blurred area (step S108). The three-dimensional spatial model generation unit 32 uses the subject image 5, which has been masked in the blurred area, and the relative position and relative orientation of the shooting device 1 corresponding to the subject image 5, to reconstruct the three-dimensional visual information of the subject 50 contained in the subject image 5, and generates a free-viewpoint video using the reconstructed three-dimensional visual information (step S109).
[0022] On the other hand, in step S103, if the image processing unit 21 determines that there are no blurry areas with low image quality in the image (step S103: No), the feature point extraction unit 22 extracts multiple feature points from the entire image (step S110). If the number of feature points extracted by the feature point extraction unit 22 is N or more (step S111: Yes), the position and orientation estimation unit 23 uses the multiple feature points to set a specific subject image 5 having the feature points as a reference image, and estimates the relative position and orientation of the shooting device 1 that took the acquired subject image 5 relative to the shooting device 1 that took the subject image 5 (step S112). If the feature points in the acquired subject image 5 match feature points for which the three-dimensional position information held in the position information holding unit 25 is known, the position and orientation estimation unit 23 may use each feature point to calculate PnP and estimate the relative position and orientation of the shooting device 1 that took the acquired subject image 5 relative to the shooting device 1 that took the subject image 5. Then, the three-dimensional spatial model generation unit 32 uses the subject image 5 and the relative position and relative orientation of the shooting device 1 corresponding to the subject image 5 to reconstruct the three-dimensional visual information of the subject 50 contained in the subject image 5, and generates a free-viewpoint video using the reconstructed three-dimensional visual information (step S109). In step S111, if the number of feature points extracted by the feature point extraction unit 22 is less than N (step S111: No), the image processing unit 21 decides not to use the subject image 5 for generating the free-viewpoint video (step S113).
[0023] Furthermore, in step S106, if the number of feature points extracted by the feature point extraction unit 22 is less than N (step S106: No), the image processing unit 21 determines whether the subject image 5 has already undergone image enhancement processing (step S114), as shown in Figure 4. If the image processing unit 21 determines that the image has already undergone image enhancement processing (step S114: Yes), it decides not to use the subject image 5 for generating the free-viewpoint video (step S116). On the other hand, if the image processing unit 21 determines that the image has not yet undergone image enhancement processing (step S114: No), it applies image enhancement processing to the blurred areas (step S115) and then estimates the blurred areas from the subject image 5 again (step S102).
[0024] In step S106, if the number of feature points extracted by the feature point extraction unit 22 is less than N (step S106: No), it may be decided not to use the subject image 5 for generating the free-viewpoint video without determining whether or not the subject image 5 has already undergone high-resolution processing.
[0025] Figure 5 is a block diagram showing a modified example 1 of the image processing system according to Embodiment 1. As shown in the image processing system 101 of Modified Example 1 in Figure 5, the image processing device 2 may be configured to have a mask processing unit 31. In this case, after the blurred area is masked by the mask processing unit 31, the data processed by the image processing device 2 is sent to the three-dimensional spatial reconstruction device 3, for example, via an external network.
[0026] Figure 6 is a block diagram showing a modified example 2 of the image processing system according to Embodiment 1. As shown in the modified example 2 image processing system 102 in Figure 6, the image processing device 2 may be configured to include a mask processing unit 31 and a three-dimensional spatial model generation unit 32. In other words, the image processing device 2 and the three-dimensional spatial reconstruction device 3 shown in Figures 2 and 5 may be configured as a single device. The free-viewpoint image generated by the three-dimensional spatial model generation unit 32 is transmitted to the display device 4 and output by the display device 4.
[0027] Furthermore, the imaging device 1 is not limited to a monocular camera; it may be a stereo camera or a camera with equivalent functionality to a stereo camera. When the imaging device 1 is a stereo camera, the three-dimensional distance between the two cameras constituting the stereo camera is known. Therefore, the three-dimensional coordinate calculation unit 24 can calculate the three-dimensional coordinates of the feature points of the subject image 5 captured by the two pairs of imaging devices 1 constituting the stereo camera using triangulation. These three-dimensional coordinates of the feature points are calculated in units similar to those of real space, such as meters. The calculated three-dimensional coordinates of the feature points are stored by the position information holding unit 25.
[0028] The position and orientation estimation unit 23 calculates PnP using the feature points in the newly captured subject image 5 taken by the imaging device 1, when the feature points in the image match the feature points in the three-dimensional position information held in the position information holding unit 25 that are known. This allows the position and orientation estimation unit 23 to estimate the relative position and orientation of the imaging device 1 that took the reference image 5 with respect to the imaging device 1 that took the reference image 5. The relative position and orientation between the imaging devices 1 and the three-dimensional coordinates of the feature points may be further corrected by performing bundle adjustment using the calculated relative position and orientation and the three-dimensional position information of the feature points used when calculating the relative position and orientation with PnP. In this case, the three-dimensional coordinates of the feature points held in the position information holding unit 25 are updated to the corrected three-dimensional coordinates. The position and orientation estimation unit 23 uses a specific subject image 5 as a reference image and sequentially estimates the relative position and orientation of the imaging device 1 that took other subject images 5 with respect to the imaging device 1 that took the reference image 5, as well as the three-dimensional coordinates of the feature points.
[0029] As described above, the image processing system 100 according to Embodiment 1 includes an image processing unit 21 that estimates blurry areas with low image quality in the image of the subject 5, a feature point extraction unit 22 that extracts a plurality of feature points from non-blurry areas excluding the blurry areas, a position and orientation estimation unit 23 that estimates the relative position and orientation of a shooting device 1 that has taken Therefore, the image processing system 100 according to Embodiment 1 can mask blurred areas with low image quality in part of the subject image 5, resulting in an image that only has high-quality, non-blurred areas. This allows for robust estimation of the relative position and relative orientation between at least two subject images 5, and enables efficient generation of free-viewpoint video using the subject images 5.
[0030] Embodiment 2. Next, an image processing system 100 according to Embodiment 2 will be described. Figure 7 is an explanatory diagram showing depth estimation performed by the image processing unit in the image processing system according to Embodiment 2. As shown in Figure 7, the image processing unit 21 in Embodiment 2 is configured to perform depth estimation of the subject image 5 and to estimate the blurred area in stages according to the depth of the subject image 5. If the shooting device 1 is a stereo camera, depth estimation using the stereo camera is used. If the shooting device 1 is a monocular camera, monocular depth estimation is used. For depth estimation, for example, an AI model such as a deep learning-based model can be used. The image processing unit 21 converts the subject image 5 into a depth-estimated image 6 and divides the depth region above a threshold into a foreground region 60 and the depth region below a threshold into a background region 61. Then, it estimates the blurred area for the foreground region 60 and the background region 61. Furthermore, when estimating the depth of subject image 5 using monocular depth estimation, the depth information may be calibrated for each corresponding region across multiple images to correct for errors in depth information between images and then estimate the depth. This makes it possible to match the depth estimation information for each image and prevent discrepancies in the depth region for each image.
[0031] Next, an example of the processing performed by the image processing system 100 described above will be explained based on a flowchart. Figures 8 and 9 are flowcharts showing the processing procedure of the image processing system according to Embodiment 2. As shown in Figure 8, first, the first image acquisition unit 20 acquires a subject image 5 captured by the shooting device 1 (step S201). Next, the image processing unit 21 estimates the depth from the subject image 5 acquired by the first image acquisition unit 20 (step S202). The image processing unit 21 divides the image into foreground regions 60 and background regions 61, where the depth region is above a threshold (step S203). Then, the feature point extraction unit 22 extracts a plurality of feature points from the foreground region 60 and background region 61 (step S204).
[0032] In step S204, if the number of feature points extracted from the foreground region 60 and the background region 61 is less than N, the image processing may be performed on the foreground region 60 and the background region 61, and feature points may be extracted from the foreground region 60 and the background region 61 again. If, even after performing the image processing on the foreground region 60 and the background region 61, the number of extracted feature points is still less than N, the image processing unit 21 may decide not to use the subject image 5 for generating a free-viewpoint video.
[0033] Next, the image processing unit 21 determines whether the parts containing feature points are blurred in the foreground region 60 and the background region 61 (step S205). As shown in Figure 9, if it is determined that the parts containing feature points are not blurred in both the foreground region 60 and the background region 61 (step S206: Yes), the position and orientation estimation unit 23 uses the multiple feature points extracted in the foreground region 60 and the background region 61 to estimate the relative position and orientation of the shooting device 1 that took the acquired subject image 5 relative to the shooting device 1 that took the subject image 5 which will be the reference image, using the specific subject image 5 containing the feature points as the reference image (step S207). If the feature points in the acquired subject image 5 match the feature points of known three-dimensional position information held in the position information holding unit 25, the position and orientation estimation unit 23 may use each feature point to calculate PnP and estimate the relative position and orientation of the shooting device 1 that took the acquired subject image 5 relative to the shooting device 1 that took the subject image 5 which will be the reference image. Then, the three-dimensional spatial model generation unit 32 uses the subject image 5 and the relative position and relative orientation of the shooting device 1 corresponding to the subject image 5 to reconstruct the three-dimensional visual information of the subject 50 contained in the subject image 5, and generates a free-viewpoint video using the reconstructed three-dimensional visual information (step S208).
[0034] On the other hand, in step S206, if the image processing unit 21 determines that at least one of the parts of the foreground region 60 and the background region 61 having feature points is blurred (step S206: No), it determines whether the part of the foreground region 60 having feature points is not blurred (step S209). If it is determined that the part of the foreground region 60 having feature points is not blurred (step S209: Yes), the position and orientation estimation unit 23 uses a plurality of feature points extracted from the foreground region 60 to estimate the relative position and orientation of the shooting device 1 that took the acquired subject image 5 relative to the shooting device 1 that took the subject image 5 which is the reference image (step S210). Furthermore, if the feature points in the acquired subject image 5 match the feature points of known three-dimensional position information held in the position information holding unit 25, the position and orientation estimation unit 23 may use each feature point to calculate PnP and estimate the relative position and orientation of the shooting device 1 that took the acquired subject image 5 relative to the shooting device 1 that took the subject image 5 which serves as the reference image. Then, the mask processing unit 31 applies a mask to the background region 61 (step S211). The three-dimensional spatial model generation unit 32 uses the subject image 5 with the background region 61 masked and the relative position and orientation of the shooting device 1 corresponding to the subject image 5 to reconstruct the three-dimensional visual information of the subject 50 contained in the subject image 5, and generates a free-viewpoint video using the reconstructed three-dimensional visual information (step S208).
[0035] Furthermore, in step S209, if the image processing unit 21 determines that the foreground area 60 is blurred (step S209: No), it determines whether the background area 61 is blurred (step S212). If it is determined that the background area 61 is not blurred (step S212: Yes), the position and orientation estimation unit 23 uses a plurality of feature points extracted from the background area 61 to estimate the relative position and orientation of the shooting device 1 that took the acquired subject image 5 relative to the shooting device 1 that took the subject image 5, using a specific subject image 5 having the feature points as the reference image (step S213). Note that if the feature points in the acquired subject image 5 match feature points for which the three-dimensional position information held in the position information holding unit 25 is known, the position and orientation estimation unit 23 may use each feature point to calculate PnP and estimate the relative position and orientation of the shooting device 1 that took the acquired subject image 5 relative to the shooting device 1 that took the subject image 5. Then, the mask processing unit 31 applies a mask to the foreground region 60 (step S214). The three-dimensional spatial model generation unit 32 uses the subject image 5 with the foreground region 60 masked and the relative position and relative orientation of the shooting device 1 corresponding to the subject image 5 to reconstruct the three-dimensional visual information of the subject 50 contained in the subject image 5, and generates a free-viewpoint image using the reconstructed three-dimensional visual information (step S208).
[0036] Furthermore, in step S212, if the image processing unit 21 determines that the background area 61 is blurred (step S212: No), it decides not to use the subject image 5 for generating the free-viewpoint video (step S215).
[0037] Figure 10 is an explanatory diagram showing an example configuration of a computer system that implements the image processing system according to this embodiment. In this embodiment, the image processing systems 100, 101, and 102 function as image processing systems 100, 101, and 102 when a computer program describing the processing in the image processing systems 100, 101, and 102 is executed on the computer system. As shown in Figure 10, this computer system comprises a processor 200, a memory 201, a storage 202, and a communication device 203, which are connected via a system bus 204.
[0038] The processor 200, memory 201, storage 202, and communication device 203 can send and receive information from each other via the system bus 204. The processor 200 is, for example, an example of a processing circuit and includes one or more of the following: CPU (Central Processing Unit), DSP (Digital Signal Processor), and system LSI (Large Scale Integration). The memory 201 includes one or more of the following: RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), and EEPROM® (Electrically Erasable Programmable Read Only Memory). The memory 201 also includes a recording medium on which a computer-readable program is recorded. Such a recording medium includes one or more of the following: non-volatile or volatile semiconductor memory, magnetic disk, flexible memory, optical disk, compact disk, and DVD (Digital Versatile Disc). Memory 201 stores the program that the processor 200 should execute, necessary data obtained during processing, and so on. Memory 201 is also used as a temporary storage area for the program. The communication device 203 consists of a receiver and transmitter that perform communication processing. Note that the computer system is not limited to the configuration shown in Figure 10 and may include other components.
[0039] Here, an example of the operation of the computer system until the program of this embodiment becomes executable will be described. In a computer system with the above configuration, for example, a computer program is installed in storage 202 from media such as a CD-ROM or DVD-ROM set in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). When the program is executed, the program read from storage 202 is stored in the main memory area of memory 201. In this state, the processor 200 executes processing as the image processing system 100, 101, 102 according to this embodiment, in accordance with the program stored in memory 201.
[0040] In the above description, a program describing the processing in the image processing systems 100, 101, and 102 is provided using a CD-ROM or DVD-ROM as the recording medium. However, the system is not limited to this, and depending on the configuration of the computer system, the capacity of the program to be provided, a program provided via a transmission medium such as the internet through the communication device 203 may also be used.
[0041] The program in this embodiment causes a computer system to perform the following steps: estimate blurry areas with low image quality in the subject image 5; extract multiple feature points from the non-blurry areas excluding the blurry areas; estimate the relative position and orientation of the shooting device 1 that took a picture of another subject image 5 having the same feature points as the shooting device 1 that took a picture of the subject image 5 having the same feature points, using the multiple feature points in the image; apply a mask to the blurry areas; and reconstruct the three-dimensional visual information of the subject 50 using the subject image 5 with the mask applied to the blurry areas and the relative position and orientation of the shooting device 1 corresponding to the subject image 5.
[0042] The configurations shown in the above embodiments are examples only, and it is possible to combine them with other known technologies, combine different embodiments, and omit or modify parts of the configuration without departing from the gist of the invention.
[0043] 1. Imaging device, 2. Image processing device, 3. Three-dimensional spatial reconstruction device, 4. Display device, 5. Subject image, 6. Depth estimation image, 20. First image acquisition unit, 21. Image processing unit, 22. Feature point extraction unit, 23. Position and orientation estimation unit, 24. Three-dimensional coordinate calculation unit, 25. Position information holding unit, 30. Second image acquisition unit, 31. Mask processing unit, 32. Three-dimensional spatial model generation unit, 50. Subject, 60. Foreground area, 61. Background area, 100, 101, 102. Image processing system, 200. Processor, 201. Memory, 202. Storage, 203. Communication device, 204. System bus.
Claims
1. An image processing system for reconstructing three-dimensional visual information of a subject from an image of the subject captured by a camera, comprising: an image processing unit that estimates blurred areas with low image quality within the image of the subject; a feature point extraction unit that extracts a plurality of feature points from non-blurred areas excluding the blurred areas; a position and orientation estimation unit that estimates the relative position and orientation of the camera that captured another image of the subject having the same feature points, using the plurality of feature points in the image; a mask processing unit that applies a mask to the blurred areas; and a three-dimensional spatial model generation unit that reconstructs three-dimensional visual information of the subject using the subject image from which the blurred areas have been masked, and the relative position and orientation of the camera corresponding to the subject image.
2. The image processing system according to claim 1, further comprising: a three-dimensional coordinate calculation unit that calculates the three-dimensional coordinates of feature points in an image by triangulation based on the relative position and relative orientation of the shooting device that took the other subject image estimated by the position and orientation estimation unit; and a position information holding unit that holds the three-dimensional coordinates of the feature points calculated by the three-dimensional coordinate calculation unit, wherein the position and orientation estimation unit estimates the relative position and relative orientation of the newly taken subject image with respect to the reference image by calculating PnP using each feature point when the feature points in the image of the subject image newly taken by the shooting device match feature points whose three-dimensional coordinates are known and held in the position information holding unit.
3. The image processing system according to claim 1 or 2, characterized in that the image processing unit divides the subject image into a foreground region and a background region using depth estimation, and estimates blurred regions in the foreground region and the background region, respectively.
4. An image processing method for reconstructing three-dimensional visual information of a subject from an image of the subject captured by a camera, comprising: a step of estimating a blurred area with low image quality in the image of the subject; a step of extracting a plurality of feature points from the non-blurred area excluding the blurred area; a step of using the plurality of feature points in the image to estimate the relative position and relative orientation of the camera that captured another image of the subject having the same feature points as the camera that captured the subject image having the same feature points; a step of applying a mask to the blurred area; and a step of reconstructing three-dimensional visual information of the subject using the subject image from which the blurred area has been masked, and the relative position and relative orientation of the camera corresponding to the subject image.
5. An image processing program for reconstructing three-dimensional visual information of a subject from an image of the subject captured by a camera, characterized in that it causes a computer to perform the following steps: estimate blurred areas with low image quality in the image of the subject; extract a plurality of feature points from non-blurred areas excluding the blurred areas; estimate the relative position and orientation of the camera that captured another image of a subject having the same feature points, relative to the camera that captured the image of the subject having the same feature points, using the plurality of feature points in the image; apply a mask to the blurred areas; and reconstruct the three-dimensional visual information of the subject using the image of the subject with the mask applied to the blurred areas and the relative position and orientation of the camera corresponding to the image of the subject.
