Scene-Adaptive Denoise Scheduling for Multi-Exposure Deghosting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional image processing pipelines for mobile devices are scene-specific, leading to unsatisfactory results when applied to different types of scenes, such as over-filtering or computational bottlenecks, and lack flexibility in adapting to varying lighting conditions.
Innovation Solution
A unified image processing pipeline that adapts to different scenes by generating alignment, shadow, and motion maps to determine the need for machine learning-based denoising on specific frames, optimizing denoising operations for improved picture quality and reduced processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If machine learning-based denoising is performed on all image frames, then image quality is improved, but processing time and computational resources increase significantly
Solution Approach 1:
The patent applies machine learning-based denoising selectively only to specific image frames that meet certain criteria (e.g., frames with high noise levels, frames containing important visual information), rather than processing all frames uniformly. This local quality approach ensures that denoising is applied where most needed while avoiding unnecessary processing on other frames, thus improving image quality where necessary without incurring the full computational cost of processing every frame.
Solution Approach 2:
The patent performs denoising on a subset of image frames rather than all frames. By selecting only certain frames for denoising based on scene analysis, motion detection, or noise level assessment, the system achieves partial action that balances quality improvement with computational efficiency, avoiding the excessive processing time that would result from denoising every frame.
2Manufacturing precision
If different image processing pipelines are used for different scene types, then image quality for specific scenes is improved, but system complexity increases
Solution Approach 1:
The patent employs a single unified image processing pipeline that can adapt to different scene types through scene classification and conditional processing logic. Rather than maintaining separate dedicated pipelines for night scenes, day scenes, indoor, and outdoor environments, the system uses one versatile pipeline that automatically adjusts its processing steps based on the detected scene type, thereby reducing system complexity while maintaining high image quality across all scenarios.
Solution Approach 2:
The patent implements a dynamic processing pipeline where the processing steps and parameters are adjusted in real-time based on scene characteristics. The system dynamically determines which processing operations to apply (such as whether to perform denoising, tone mapping, or color adjustment) based on the current scene type, making the pipeline flexible and adaptive rather than static and scene-specific, thus reducing overall system complexity.
3Manufacturing precision
If denoising is performed on multiple exposure frames, then noise reduction is improved, but motion artifacts and ghosting increase
Solution Approach 1:
The patent performs motion detection and scene analysis before committing to multi-frame denoising processing. By preliminarily assessing the scene for motion content and determining the appropriate processing strategy, the system can decide whether to proceed with multi-frame denoising or use alternative approaches, thereby preventing motion artifacts from occurring in the first place rather than trying to correct them afterward.
Solution Approach 2:
The patent incorporates feedback mechanisms where the results of preliminary scene analysis and motion detection inform the subsequent denoising processing decisions. The system uses feedback from motion estimation and scene classification to adjust denoising parameters and select appropriate frames for processing, ensuring that denoising is applied in a way that minimizes motion artifacts while maintaining noise reduction effectiveness.
Data Source
AI summary
A method includes generating alignment maps for a first image frame having a first exposure level and a second image frame having a second exposure level different than the first exposure level. The method also includes generating, for the second image frame and a third image frame having a third exposure level different than the first and second exposure levels, shadow maps, saturation maps, and multi-exposure (ME) motion maps based on the alignment maps. The method further includes determining, based on the shadow maps, saturation maps, and ME motion maps, whether to perform machine learning-based denoising and, if so, on which image frame(s) to perform the machine learning-based denoising. In addition, the method includes updating at least one saturation map and at least one ME motion map for at least one of the second and third image frames depending on the image frame(s) on which the denoising is to be performed.