Bundle Adjustment System for Rolling Shutter Camera Distortion Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Aerial survey systems using rolling shutter image sensors face significant challenges with image distortion and residual distortions due to camera motion, leading to insufficient bundle adjustment solutions and slow convergence, especially under rapid and non-linear motion.
Innovation Solution
The implementation of pose and position time domain polynomials for each scanline, modeled using 3rd order polynomials with adjustable parameters, and a bundle adjustment process that minimizes the distance error between rays projected from tie point features in image space, reducing the number of parameters to be adjusted and improving convergence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If rolling shutter sensors are used to reduce cost and improve performance, then device complexity and cost are reduced, but image distortion occurs due to sequential scanline exposure during motion
Solution Approach 1:
The patent transforms the camera model parameters by introducing time-dependent pose parameters (position and orientation at each scanline exposure time) and distortion parameters specific to rolling shutter geometry. This allows the system to accurately model and correct the sequential exposure distortion while maintaining the simplicity and low cost of rolling shutter sensors.
Solution Approach 2:
The patent segments the image formation process into individual scanline exposures, each with its own position and pose parameters. By treating each scanline as a separate exposure event with distinct temporal characteristics, the system can accurately model the distortion introduced by camera motion during the rolling shutter sequence and apply针对性的 corrections.
2Measurement precision
If the number of images per ground point is increased to improve photogrammetric solution, then measurement precision is improved, but productivity decreases due to reduced ground area captured per hour
Solution Approach 1:
The patent changes the approach from increasing image quantity to improving parameter utilization. By introducing time-dependent pose parameters and rolling shutter distortion parameters, the system extracts more information from existing images, achieving improved photogrammetric accuracy without requiring additional images or reduced flight speed.
3Productivity
If flying faster or higher is done to increase productivity, then ground area per hour is increased, but image resolution deteriorates due to image blur
Solution Approach 1:
The patent applies dynamics by introducing time-dependent pose parameters that capture the instantaneous position and orientation of the camera at each scanline exposure moment. This dynamic modeling allows the system to account for and correct motion-induced blur and distortion, enabling high-speed flight without sacrificing image resolution or measurement precision.
4Speed
If traditional bundle adjustment is applied to rolling shutter images, then processing speed is maintained, but measurement precision deteriorates due to insufficient modeling of scanline-specific position and pose
Solution Approach 1:
The patent extends the traditional bundle adjustment parameter set by adding time-dependent pose parameters (position and orientation as functions of scanline exposure time) and rolling shutter distortion parameters. This enhanced parameterization improves measurement precision while maintaining computational efficiency through the use of polynomial representations and efficient optimization algorithms.
Data Source
Figure 1~2
Figure 3~4b
Figure 5
AI summary
A bundle adjustment system is disclosed for determining a solution for orientation information associated with a plurality of image frames captured by at least one rolling shutter camera, wherein the at least one rolling shutter camera is part of a rotating camera lens type imaging system having a camera lens that moves in an oscillating manner as image frames are captured. The system is arranged to define a plurality of pose time domain polynomials for the rolling shutter camera, the pose time domain polynomials together defining pose information for the rolling shutter camera for scanlines of an image frame captured by the rolling shutter camera, and each pose time domain polynomial including a plurality of parameters to be adjusted in a bundle adjustment process. The system is also arranged to carry out a bundle adjustment process using the pose time domain polynomials to produce a bundle adjustment solution wherein the pose time domain polynomial parameters are adjusted to reduce error. The bundle adjustment process is arranged to use a bundle adjustment cost function wherein a cost measure of the cost function is a minimum distance error between 2 rays projected from 2 different locations of a tie point feature.