Rolling Shutter Image Projection Algorithm for Autonomous Vehicles
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in accurately mapping points from LIDAR sensors to locations within images captured by cameras operating in rolling shutter mode, especially in environments where the camera is in motion, due to the complexity of determining the camera's pose at different exposure times.
Innovation Solution
A method is introduced that estimates the time when light from a point in the environment was sensed by the camera, using an iterative process that evaluates a cost function to update the estimated time, allowing for the projection of points into rolling-shutter images with reduced computational resources and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a rolling shutter camera is used to capture images, then the camera can operate in motion with continuous exposure of different rows, but accurately mapping LIDAR points to image locations becomes complex due to varying exposure times
Solution Approach 1:
The system pre-determines exposure time intervals for different rows of the rolling shutter camera and stores this timing information. When mapping LIDAR points to image pixels, the system uses these pre-established exposure time relationships to efficiently determine which pixel corresponds to each LIDAR point without complex real-time calculations, thus resolving the mapping complexity while maintaining continuous capture capability
Solution Approach 2:
The patent introduces a time dimension to the mapping process by associating each row of pixels with specific exposure time intervals. This temporal dimension allows the system to handle the rolling shutter's sequential exposure by matching LIDAR point timestamps with corresponding pixel exposure windows, transforming a spatial mapping problem into a spatio-temporal mapping problem that resolves the complexity
2Measurement precision
If iterative cost function evaluation is performed to update estimated time, then mapping accuracy improves, but computational resources and processing time increase
Solution Approach 1:
The system performs a limited number of iterative updates (N iterations) rather than exhaustive optimization. This partial action approach achieves sufficient accuracy for autonomous vehicle applications while constraining computational resource usage, balancing precision requirements with energy and processing constraints
Solution Approach 2:
The iterative process uses feedback from cost function evaluation to progressively refine the estimated time. Each iteration uses the previous estimate to guide the next refinement, allowing the system to converge to an accurate solution efficiently. The feedback mechanism enables the algorithm to stop when sufficient accuracy is achieved, preventing excessive computational resource consumption
Data Source
AI summary
An improved, efficient method for mapping world points from an environment (e.g., points generated by a LIDAR sensor of an autonomous vehicle) to locations (e.g., pixels) within rolling-shutter images taken of the environment is provided. This improved method allows for accurate localization of the world point in a rolling-shutter image via an iterative process that converges in very few iterations. The method poses the localization process as an iterative process for determining the time, within the rolling-shutter exposure period of the image, at which the world point was imaged by the camera. The method reduces the number of times the world point is projected into the normalized space of the camera image, often converging in three or fewer iterations.