Rolling Shutter Correction Using Row-Wise Pose and 3D Depth Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mixed-reality systems face challenges in accurately presenting images due to parallax errors and rolling shutter artifacts, particularly when using stereo cameras with a vertical offset from the user's eyes, leading to incorrect perception of real-world objects.
Innovation Solution
A system for compensating rolling shutter artifacts involves identifying readout timepoints and poses for sensor rows, generating a 3D representation of the environment, and projecting it using separate projection poses to correct for rolling shutter artifacts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a rolling shutter sensor is used to capture images in mixed-reality systems, then the device complexity is reduced and manufacturing cost is lowered, but rolling shutter artifacts are introduced that cause incorrect perception of real-world objects
Solution Approach 1:
The system performs preliminary actions by capturing pose information at multiple readout timepoints during the rolling shutter exposure period. By proactively collecting temporal data points before the image is fully processed, the system establishes a foundation for later artifact correction through depth map generation and parallax compensation calculations.
Solution Approach 2:
The system changes parameters by introducing temporal variation into the depth map generation process. Instead of using a single static depth map, the system generates multiple depth maps corresponding to different readout timepoints and uses these to compute parallax corrections that account for camera movement during the exposure period.
2Measurement precision
If the stereo cameras are physically separated from the user's eyes to enable depth detection, then depth detection capability is improved, but parallax errors are introduced that cause vertical offset and incorrect depth perception
Solution Approach 1:
The system introduces an intermediary computational process that bridges the gap between the camera's physical position and the user's eye position. By generating depth maps and computing parallax corrections as intermediate steps, the system translates the camera's perspective into the user's perspective, compensating for the physical separation between components.
Solution Approach 2:
The system adds a temporal dimension to the traditional spatial depth mapping approach. By capturing pose information at multiple timepoints during the rolling shutter readout and generating corresponding depth maps, the system creates a four-dimensional representation that accounts for camera movement through time, enabling more accurate parallax correction.
3Ease of manufacture
If rolling shutter sensors with sequential row readout are used, then the manufacturing cost is reduced, but temporal offsets between different sensor rows cause rolling shutter artifacts
Solution Approach 1:
The system segments the image capture process into multiple temporal components by treating each sensor row's readout timepoint as a distinct measurement. By dividing the rolling shutter readout into discrete time segments and generating separate depth maps for each segment, the system can process and correct artifacts for individual rows before combining them into a final corrected image.
Solution Approach 2:
The system implements feedback by using the captured pose information at each readout timepoint to inform the depth map generation and parallax correction process. The pose data serves as feedback that allows the system to adjust and compensate for camera movement, improving the accuracy of the final image despite the sequential readout nature of the rolling shutter sensor.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems having rolling shutter sensors with a plurality of sensor rows are configured for compensating for rolling shutter artifacts that result from different sensor rows in the plurality of sensor rows outputting sensor data at different times. The systems compensate for the rolling shutter artifacts by identifying readout timepoints for the plurality of sensor rows of the rolling shutter sensor while the rolling shutter sensor captures an image of an environment and identifying readout poses each readout timepoint, as well as obtaining a depth map based on the image. The depth map includes a plurality of different rows of depth data that correspond to the different sensor rows. The system further compensates for the rolling shutter artifacts by generating a 3D representation of the environment while unprojecting the rows of depth data into 3D space using the readout poses.