Light Field Image Processing Using Depth-Based Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image processing techniques for light field photography face challenges in achieving smooth blur for subjects out of focus, often resulting in artifacts and increased processing burdens, especially when estimating subject shapes and tracing light beams.
Innovation Solution
An image processing apparatus and method that uses a filtering unit to calculate filter coefficients based on the position of a virtual sensor, distance to subjects, and image capturing device characteristics, applying filters to multi-viewpoint image data to achieve smooth blur with reduced computational load and robustness to shape estimation errors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If light beam tracing is performed to reproduce smooth blur, then blur smoothness is improved, but processing burden increases significantly
Solution Approach 1:
The patent replaces the mechanical light beam tracing process with a mathematical filtering approach. Instead of simulating light propagation through complex geometric calculations, the invention applies convolution filters to the captured images, substituting a computationally intensive geometric optics simulation with efficient signal processing operations that achieve the same blur effect.
Solution Approach 2:
The patent changes the approach from spatial domain light tracing to frequency domain filtering. By transforming the problem into applying filters with specific kernel parameters in the spatial domain (or FFT-based convolution in frequency domain), the method achieves smooth blur effects with significantly reduced computational complexity compared to ray tracing algorithms.
2Manufacturing precision
If subject shape estimation is performed for light beam tracing, then blur accuracy is improved, but processing complexity increases
Solution Approach 1:
The patent extracts only the essential information needed for blur rendering - the depth information - without requiring complete subject shape estimation. By separating the depth mapping function from the blur rendering function, the method achieves accurate depth-dependent blur without the complexity of full 3D shape reconstruction and light beam tracing.
Solution Approach 2:
The patent creates a simplified 2.5D representation (depth map) that copies only the necessary depth information from the scene, rather than performing complete 3D shape estimation. This depth map serves as a surrogate for full geometric reconstruction, enabling efficient filter-based blur rendering without the computational burden of accurate shape modeling.
3Productivity
If projective transformation and simple averaging is used to combine images, then processing speed is improved, but artifact occurrence increases
Solution Approach 1:
The patent applies different filtering operations to different regions of the image based on local depth information. Instead of uniform averaging, the method adjusts filter kernels and convolution parameters locally according to the depth map, preserving image quality in focus regions while applying appropriate blur to out-of-focus regions, thereby eliminating artifacts.
Solution Approach 2:
The patent performs preliminary depth estimation and filter parameter calculation before the actual image combination process. By pre-computing the depth map and determining appropriate filter kernels for each region beforehand, the method prepares all necessary parameters in advance, enabling high-speed filtering without sacrificing image quality during the combination phase.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In the conventional method, the burden of the processing is very heavy because tracking of light beams is performed and an erroneous estimation of a shape causes a factor to deteriorate image quality. An image processing apparatus that generates composite image data using multi-viewpoint image data obtained by capturing images from a plurality of viewpoints is characterized by including a filter processing unit configured to perform filter processing on the multi-viewpoint image data based on distance information indicative of a distance to a subject and a generation unit configured to generate composite image data by combining the multi-viewpoint image data on which the filter processing has been performed.