Multi-View Camera Processing for Efficient Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing object detection systems using multiple cameras struggle with efficient computation, error-prone 3D reconstruction, limited information access, and high computational requirements, especially when dealing with overlapping camera views and embedded hardware constraints.
Innovation Solution
A method that groups camera images to minimize overlap, stitches them into composite images, and processes these combined images using a neural network for robust object detection and association, reducing computation power and time, and enabling direct 3D reconstruction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If images from multiple cameras are processed independently using deep neural networks, then object detection can be performed in each image, but the backbone computation is carried out multiple times even when cameras capture overlapping redundant information
Solution Approach 1:
The patent merges images from multiple cameras into a composite image, allowing the deep neural network to process all camera views simultaneously in a single backbone computation rather than independently processing each image separately. This eliminates redundant computations while maintaining detection accuracy across all views.
Solution Approach 2:
The patent segments the processing pipeline into two distinct stages: first merging the images from multiple cameras into a composite image, then performing deep neural network processing on the merged result. This segmentation allows the computationally intensive backbone to run once on combined data rather than multiple times on individual images.
2Reliability
If images from multiple cameras are processed independently, then object detection can be performed, but the three-dimensional reconstruction process may contain errors resulting in further errors in the subsequent association/merging step
Solution Approach 1:
The patent performs image merging as a preliminary action before deep neural network processing and 3D reconstruction. By combining images first, the system establishes a unified reference frame that eliminates errors in subsequent 3D reconstruction and object association steps, rather than attempting to merge results after independent processing.
3Reliability
If a transformer network is used to compute correspondence between vehicle frame and camera images, then object detection can be performed, but the computational requirements increase making it unsuitable for embedded hardware
Solution Approach 1:
The patent extracts and eliminates the computationally intensive transformer network component from the processing pipeline. Instead of using transformer networks to compute correspondences, the system achieves the same goal through simpler image merging techniques that are computationally efficient and suitable for embedded hardware deployment.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
The technical problem solved by the teaching of the present application is to efficiently detect one or more objects surrounding or in the environment of an observing object (e.g., a vehicle, a robot or the like) from multiple camera images captured from multiple cameras mounted on the observing object. In particular, the present invention addresses the problem of the association between objects detected in different camera images from different cameras (wherein these different cameras have fields of view that intersect), while reducing the required computation power and time. It teaches a multi-view object detection method, system, computer program and computer-readable storage medium comprising instructions for detecting objects in the surroundings of an observing object, wherein the method comprises the step of: capturing images from at least two cameras mounted on the observing object such that their fields of view intersect at least partially; determining two or more channel groups of cameras, wherein the channel groups are groups of one or more cameras which are determined such that an overlap of the image of the one or more cameras in the group is minimized; stitching, for each determined channel group of cameras, the captured images of the cameras of the determined channel group together to generate a composite image; processing, using a neural network, the composite image of each channel group to detect one or more surrounding objects in the composite images; assigning the one or more detected surrounding objects to different cameras and defining at least one association between at least two objects assigned to at least two different cameras, wherein the at least one association defines that the at least two objects assigned to at least two different cameras are the same surrounding object.