Synthetic Shallow Depth of Field Rendering via Sparse Sampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Small form-factor cameras, such as those in mobile devices, struggle to achieve a shallow depth of field effect due to smaller apertures, leading to images with a large depth of field, making it difficult to synthetically create a natural-looking out-of-focus blur without high computational resources and requiring meticulous user intervention.
Innovation Solution
The method involves capturing an input image and corresponding depth information, converting it into a desired color space, and performing sparse sampling with a blur map to adjust pixel values, weighting neighborhood pixels, and applying anti-aliasing to generate a computationally-efficient synthetic shallow depth of field effect, while preserving scene highlights and edges.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If a naïve approach is used to simulate SDOF effects by spreading every pixel's light intensity onto every other pixel within its blurring radius, then the out-of-focus blur effect can be achieved, but the computational resources required become excessively large and processing time increases significantly
Solution Approach 1:
The patent segments the image processing task by separating foreground and background regions using depth information. Only background pixels are subjected to the computationally intensive SDOF blur processing, while foreground pixels remain sharp. This segmentation drastically reduces the number of pixel interactions needed compared to processing the entire image uniformly.
Solution Approach 2:
The patent applies partial action by performing the full SDOF simulation only on background regions rather than the entire image. The blur processing is selectively applied based on depth map classification, performing the computationally expensive operation only where needed to achieve the desired aesthetic effect while minimizing computational overhead.
2Manufacturing precision
If meticulous user intervention and fine tuning are applied to adjust the synthetic SDOF effect, then physically accurate rendering with minimal occlusions and mischaracterizations can be achieved, but the ease of operation decreases and user burden increases
Solution Approach 1:
The system performs self-service by automatically classifying foreground and background pixels using depth information and autonomously adjusting the blur parameters. The algorithm independently determines which pixels to blur and to what extent based on depth map analysis, eliminating the need for manual user intervention and fine-tuning while maintaining physical accuracy.
Solution Approach 2:
The patent uses depth information as feedback to automatically control the blur application. The depth map provides real-time feedback about scene geometry, allowing the system to自适应ly adjust blur parameters for each pixel based on its depth relative to the focus plane, achieving accurate SDOF rendering without manual intervention.
3Volume of moving object
If a small form factor camera with smaller aperture is used, then the device size and portability are improved, but the ability to achieve shallow depth of field effect deteriorates
Solution Approach 1:
The patent replaces the mechanical/optical system (large aperture lens) with a computational system. Instead of relying on physical aperture size to achieve SDOF, the system uses image processing algorithms that analyze depth information and synthetically apply blur to background regions, substituting computational methods for optical hardware limitations.
Solution Approach 2:
The patent introduces depth information (depth map) as an intermediary between the captured image and the final SDOF effect. This intermediate representation of scene geometry enables the system to accurately determine which regions should be blurred, bridging the gap between the LDOF optical capture and the desired SDOF aesthetic output.
4Manufacturing precision
If background light sources are rendered with large blurred colored discs in SDOF images, then the aesthetic quality and natural look are improved, but the measurement precision of light source position and the accuracy of boundary transitions deteriorate
Solution Approach 1:
The patent applies local quality by using depth information to determine the blur radius for each pixel individually. Background light sources naturally form larger blurred discs because their depth values result in larger blur radii, while pixels near foreground objects have smaller or zero blur radii. This local variation in blur quality creates natural-looking bokeh while maintaining accurate boundary transitions through the depth-based parameter adjustment.
Data Source
AI summary
This disclosure relates to techniques for synthesizing out of focus effects in digital images. Digital single-lens reflex (DSLR) cameras and other cameras having wide aperture lenses typically capture images with a shallow depth of field (SDOF). SDOF photography is often used in portrait photography, since it emphasizes the subject, while deemphasizing the background via blurring. Simulating this kind of blurring using a large depth of field (LDOF) camera may require a large amount of computational resources, i.e., in order to simulate the physical effects of using a wide aperture lens while constructing a synthetic SDOF image. However, cameras having smaller lens apertures, such as mobile phones, may not have the processing power to simulate the spreading of all background light sources in a reasonable amount of time. Thus, described herein are techniques to synthesize out-of-focus background blurring effects in a computationally-efficient manner for images captured by LDOF cameras.