Bokeh Image Refinement Using Confidence Maps and CNNs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional mobile device cameras struggle to produce high-quality Bokeh images due to limitations in lens and sensor size, relying on computational methods that result in artifacts from imperfect depth estimation, separating depth estimation and depth-of-field rendering into sequential and independent modules without machine learning techniques.
Innovation Solution
A Bokeh network architecture that employs convolutional neural networks (CNNs) to generate and refine Bokeh images by processing reference images to create confidence and disparity maps, and then using these maps to generate and refine Bokeh images, integrating machine learning for improved image quality and user experience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If computational methods are used to achieve Bokeh effect on mobile devices, then Bokeh effect can be produced, but image quality deteriorates due to artifacts from imperfect depth estimation
Solution Approach 1:
The patent replaces traditional mechanical optical systems (lens and sensor size limitations) with a computational approach using deep learning models. The CNN-based depth estimation and Bokeh rendering system substitutes the physical limitations of mobile device optics with algorithmic processing, achieving DSLR-quality Bokeh effects through neural network inference rather than relying on hardware aperture size.
Solution Approach 2:
The patent changes the parameters of depth estimation by using trained CNN models that output both disparity maps and confidence maps. The confidence map parameter allows the system to weight different depth estimation regions differently, improving overall image quality by focusing computational resources on high-confidence regions and reducing artifacts in low-confidence areas.
2Device complexity
If depth estimation and depth-of-field rendering are separated into sequential modules, then processing can be modularized, but image quality deteriorates due to lack of integration and refinement
Solution Approach 1:
The patent implements a feedback mechanism where the output of the depth estimation module (disparity map and confidence map) is fed back into the Bokeh rendering module. The confidence map serves as feedback information that allows the rendering module to adjust its processing, creating an iterative refinement loop that improves final image quality while maintaining modular architecture.
Solution Approach 2:
The patent merges the depth estimation and depth-of-field rendering modules into an integrated pipeline where both processes operate simultaneously with shared computational resources. The CNN model processes both depth estimation and Bokeh rendering tasks through unified neural network layers, eliminating the sequential bottleneck and allowing for joint optimization of both functions.
3Device complexity
If conventional camera lenses are used on mobile devices, then hardware simplicity is maintained, but Bokeh quality deteriorates due to small aperture size
Solution Approach 1:
The patent substitutes the mechanical aperture system with a computational Bokeh generation system. Instead of relying on a large physical aperture to create natural Bokeh, the system uses CNN-based depth estimation to compute depth information and then applies algorithmic rendering to create the Bokeh effect, achieving high-quality results without requiring large-aperture hardware.
Solution Approach 2:
The patent creates a computational copy of the optical Bokeh effect. Rather than capturing actual optical Bokeh with a large aperture lens, the system captures the scene with a standard mobile lens and then generates a synthetic Bokeh effect that replicates the appearance of optical Bokeh through image processing and deep learning models trained on reference images.
Data Source
AI summary
A method includes processing, using at least one processor of an electronic device, multiple reference images of a scene using a first convolutional neural network (CNN) to generate a confidence map and a disparity map. The method also includes generating, using the at least one processor, an initial Bokeh image based on the disparity map and the reference images using a depth-of-field (DoF) renderer. The method further includes refining, using the at least one processor, the initial Bokeh image using a second CNN to generate a refined Bokeh image, where the second CNN uses the confidence map, the disparity map, and the reference images to generate the refined Bokeh image.