Synthetic Depth-of-Field Rendering for Mobile Video
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for generating Bokeh effects in videos, especially on mobile devices, face challenges such as inaccurate depth estimation, lack of blur transition in regions with gradient depth, and inability to identify objects at the same depth level, leading to poor aesthetic quality and inability to achieve special blurring effects like re-focusing.
Innovation Solution
A system and method for synthetic depth-of-field effect rendering that involves obtaining multiple video and depth frames, generating blur kernel maps, reducing depth errors through spatial refinement, performing temporal smoothing to suppress artifacts, and generating blur effects using these maps to create visually plausible Bokeh effects in real-time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional computational Bokeh methods are used on mobile devices, then processing speed is improved, but depth estimation accuracy deteriorates
Solution Approach 1:
The system performs preliminary depth refinement by identifying motion boundary regions and applying bilateral filtering to correct depth errors before generating blur kernel maps. This preliminary action improves depth estimation accuracy without significantly increasing processing time during video playback.
Solution Approach 2:
The processing is segmented into distinct stages: motion boundary detection, depth error correction in specific regions, blur kernel map generation, and temporal smoothing. By segmenting the process, the system applies computationally intensive operations only where needed rather than uniformly across the entire video stream.
2Device complexity
If simple blur application is used, then device complexity is reduced, but blur transition quality in gradient depth regions deteriorates
Solution Approach 1:
The system applies different processing quality to different regions of the video frame. Motion boundary regions receive enhanced depth correction and blur transition processing, while other regions use standard blur application. This local quality approach ensures high blur transition quality where needed without uniformly increasing overall processing complexity.
3Loss of time
If depth maps are processed without temporal smoothing, then processing time is reduced, but temporal artifacts between frames increase
Solution Approach 1:
The system applies temporal smoothing selectively rather than to the entire depth map. By applying smoothing only to specific regions and using adaptive filtering, the system achieves sufficient temporal consistency without the excessive processing time that would result from comprehensive temporal smoothing of all pixels.
Data Source
AI summary
A method includes obtaining, using at least one processor of an electronic device, multiple video frames of a video stream and multiple depth frames corresponding to the multiple video frames. The method also includes generating, using the at least one processor, multiple blur kernel maps based on the multiple depth frames. The method further includes reducing, using the at least one processor, depth errors in each of the multiple blur kernel maps. The method also includes performing, using the at least one processor, temporal smoothing on the multiple blur kernel maps to suppress temporal artifacts between different ones of the multiple blur kernel maps. In addition, the method includes generating, using the at least one processor, blur effects in the video stream using the multiple blur kernel maps.