Temporal Low Light Filtering with Global and Local Motion Compensation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing temporal filtering methods in mixed-reality systems face challenges such as ghosting effects and reduced image quality due to global camera movements and local object movements within the scene.
Innovation Solution
The method involves acquiring a history frame and a current frame, performing global motion compensation to align the camera poses, identifying and shifting pixels corresponding to locally moving objects using optical flow vectors, and then temporally filtering the current frame with the adjusted history frame to generate a high-quality image.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If temporal filtering is performed by combining multiple consecutively captured images, then image quality is improved, but ghosting effects occur when objects or camera move during capture
Solution Approach 1:
The patent applies preliminary motion compensation by estimating camera motion and object motion before performing temporal filtering. By pre-aligning the images through motion compensation, the patent prevents ghosting effects from occurring during the filtering process, thereby maintaining both high image quality and eliminating motion-related artifacts.
Solution Approach 2:
The patent segments the motion compensation process into two distinct components: camera motion compensation and object motion compensation. This segmentation allows the system to handle different types of motion separately, applying appropriate compensation techniques to each, which effectively eliminates ghosting while preserving image quality.
2Manufacturing precision
If multiple images are combined to improve low light image quality, then image data quality is improved, but processing complexity increases
Solution Approach 1:
The patent performs preliminary motion estimation and compensation before the actual temporal filtering process. By pre-processing the images to correct for motion, the patent simplifies the subsequent filtering operation, reducing the overall processing complexity while still achieving high image data quality in low light conditions.
Solution Approach 2:
The system uses the image data itself to estimate motion parameters through optical flow analysis. This self-service approach allows the patent to derive motion information directly from the input images without requiring additional sensors or complex external processing, thereby reducing processing complexity while maintaining image quality improvement.
3Object-generated harmful factors
If motion compensation is applied to eliminate ghosting, then ghosting effects are reduced, but processing time increases
Solution Approach 1:
The patent segments motion compensation into two efficient stages: camera motion compensation using global motion models, and object motion compensation using optical flow. This segmentation allows each stage to be optimized independently, reducing overall processing time while effectively eliminating ghosting effects from both camera and object motion.
Solution Approach 2:
The patent replaces complex mechanical motion tracking with computational methods. By using optical flow algorithms to estimate object motion and reprojection techniques for camera motion, the patent achieves effective ghosting reduction through software-based solutions rather than mechanical systems, thereby minimizing processing time.
Data Source
AI summary
Techniques for generating a temporally filtered image designed to compensate for global motions of a camera and to compensate for local motions of an object are disclosed. A history frame and a current frame are acquired. A global motion compensation operation is performed on the history frame to reproject a pose of the history frame to match the pose of the current frame. The history frame is compared against the current frame to identify pixels that represent moving objects. For each of those pixels, an optical flow vector is computed. The optical flow vectors are then applied to those pixels to shift those pixels to new locations. These new positions, which are in the history frame, correspond to positions that were identified in the current frame. Afterwards, the current frame is temporally filtered with the history frame.