Dynamic Seam Placement for Saliency-Aware Surround View Stitching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing vehicle Surround View Systems suffer from noticeable seams and artifacts in stitched images due to noise, white balance configurations, and limitations in 3D visualization, particularly ignoring important objects outside the ultrasonic sensing range and failing to accurately represent the area under the vehicle, leading to reduced visual quality and obscured useful information.
Innovation Solution
Implementing dynamic seam placement based on object saliency and ego-object state, an adaptive 3D bowl model that changes shape based on distance and direction to detected objects, and reconstructing the area under the vehicle using cached sensor data and ego-motion, while streaming visualizations remotely for enhanced visualization and safety.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Area of stationary object
If conventional image stitching techniques are used to create surround view visualization, then the system can provide a 360-degree view of the surrounding environment, but noticeable seams and stitching artifacts appear in the stitched images
Solution Approach 1:
The patent implements dynamic seam placement that adapts to the driving state and detected objects. The seam position is not fixed but changes dynamically based on ultrasonic sensor detections and ego-vehicle state (parking, reversing, etc.), allowing the system to avoid placing seams over important objects while maintaining 360-degree coverage
Solution Approach 2:
The patent applies different processing to different regions of the stitched image. By identifying salient objects through ultrasonic sensors and analyzing their position in the stitched image, the system selectively places seams in less important regions while preserving quality in critical areas, creating local quality variations optimized for safety
2Measurement precision
If ultrasonic sensors are used to detect objects for seam placement, then seams can be avoided on close objects, but objects outside the ultrasonic sensing range are ignored
Solution Approach 1:
The patent merges ultrasonic sensor data with camera image data to create a comprehensive object detection system. The ultrasonic sensors detect close objects with high precision, while the camera captures the full 360-degree environment including distant objects. The system combines these data sources to make seam placement decisions that consider both close and distant objects
Solution Approach 2:
The camera acts as an intermediary that extends the detection range beyond ultrasonic sensors. While ultrasonic sensors provide precise detection for very close objects, the camera captures visual information about distant objects that would otherwise be undetected, bridging the gap in sensing range
3Area of stationary object
If fisheye cameras are used to capture surrounding views, then 360-degree coverage is achieved, but geometric distortions and misalignments occur in the stitched image
Solution Approach 1:
The patent applies geometric transformation parameters to correct fisheye distortion. By using calibration parameters and transformation models, the system converts the distorted fisheye images into geometrically accurate representations that can be properly aligned and stitched together, maintaining both wide coverage and geometric precision
4Device complexity
If the area under the vehicle is not visualized, then the system complexity is reduced, but useful visual information about the under-vehicle environment is lost
Solution Approach 1:
The patent creates a virtual copy of the under-vehicle area using image processing techniques. By analyzing images from surrounding cameras and using the vehicle's motion data, the system synthesizes a representation of the area under the vehicle that appears in the surround view display, providing visual information without adding physical sensors under the vehicle
Data Source
AI summary
In various examples, dynamic seam placement is used to position seams in regions of overlapping image data to avoid crossing salient objects or regions. Objects may be detected from image frames representing overlapping views of an environment surrounding an ego-object such as a vehicle. The images may be aligned to create an aligned composite image or surface (e.g., a panorama, a 360° image, bowl shaped surface) with regions of overlapping image data, and a representation of the detected objects and/or salient regions (e.g., a saliency mask) may be generated and projected onto the aligned composite image or surface. Seams may be positioned in the overlapping regions to avoid or minimize crossing salient pixels represented in the projected masks, and the image data may be blended at the seams to create a stitched image or surface (e.g., a stitched panorama, stitched 360° image, stitched textured surface).