XR Rolling Shutter Image Correction Using Motion Matrix Projection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AR systems face challenges with slow computation speed and robustness issues in correcting rolling shutter (RS) camera images, which affect the user experience and downstream tasks.
Innovation Solution
A method and apparatus that utilize motion matrices and neural networks to project and correct RS camera images onto global shutter (GS) images, determining the need for motion matrix calculation based on camera velocity and optical flow thresholds, and employing convolutional and U-network neural networks for image enhancement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If motion inversion estimation using multiple frames is used for depth estimation or correction, then correction accuracy is improved, but computation speed becomes too slow for fast AR scenes
Solution Approach 1:
The patent extracts only the essential motion information (motion matrix) from the multiple frames rather than performing full motion inversion estimation. By taking out only the critical motion parameters needed for correction, the system achieves adequate correction accuracy without the computational burden of complete motion inversion, thus resolving the contradiction between correction accuracy and computation speed.
Solution Approach 2:
The patent applies partial action by using a simplified correction approach that processes only the necessary motion compensation rather than complete multi-frame motion inversion. This partial processing provides sufficient correction for AR scenes while maintaining real-time computation speed, avoiding the excessive computational requirements of full motion inversion estimation.
2Device complexity
If RS camera image correction is assumed to be effective, then processing simplicity is improved, but robustness of downstream tasks deteriorates
Solution Approach 1:
The patent implements dynamic correction by adaptively adjusting the correction strength and method based on detected motion magnitude. When motion is small, simpler correction is applied; when motion is large, more robust correction methods are activated. This dynamic approach maintains processing simplicity for normal cases while ensuring downstream task robustness when needed, resolving the contradiction between simplicity and reliability.
Solution Approach 2:
The patent changes correction parameters (such as correction intensity, method selection) based on motion conditions detected from the motion matrix. By dynamically adjusting these parameters, the system maintains simple processing under normal conditions while enhancing robustness when motion artifacts significantly impact downstream tasks, thus resolving the contradiction between processing simplicity and task robustness.
Data Source
AI summary
A processor-implemented method includes obtaining a first motion matrix corresponding to an extended reality (XR) system and a second motion matrix based on a conversion coefficient from an XR system coordinate system into a rolling shutter (RS) camera coordinate system, and projecting an RS color image of a current frame onto a global shutter (GS) color image coordinate system based on the second motion matrix and generating a GS color image of the current frame, wherein the second motion matrix is a motion matrix of a timestamp of a depth image captured by a GS camera corresponding to a timestamp of a first scanline of an RS color image captured by the GS camera.