Camera-LiDAR Fusion for Object Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current object tracking systems face challenges with blur phenomena and low tracking performance due to fast movement or low Frame Per Second (FPS) when using a single camera, and increasing costs with multi-camera or LiDAR solutions, especially in on-device environments.
Innovation Solution
An electronic apparatus combining a camera and a LiDAR sensor with a processor that controls the camera's photographing direction based on scanning information to track objects, even if they are not identified in images, by matching object locations between camera images and LiDAR data and adjusting the camera's view accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If a single camera is used for object tracking, then the manufacturing cost is low, but tracking performance deteriorates due to blur phenomena and low FPS when objects move fast
Solution Approach 1:
The patent introduces LiDAR as an intermediary device to assist the camera in object tracking. The LiDAR provides depth information and object location data that complements the camera's visual information, enabling reliable tracking even when the camera alone cannot capture fast-moving objects clearly. This mediator approach allows the system to maintain low manufacturing cost while improving tracking performance through sensor fusion.
Solution Approach 2:
The patent combines camera and LiDAR sensors into an integrated tracking system. The camera captures visual information while the LiDAR provides depth and spatial data, and their results are merged through data fusion algorithms. This combination allows the system to overcome the limitations of a single camera, achieving reliable tracking performance without requiring multiple expensive camera systems.
2Reliability
If multiple cameras are used to track objects outside the field of view, then tracking performance improves, but manufacturing cost increases
Solution Approach 1:
Instead of adding more cameras to expand the field of view, the patent uses LiDAR as a mediator to detect objects outside the camera's field of view. The LiDAR's scanning capability provides spatial information about objects in directions not covered by the camera, allowing the system to track objects broadly without requiring multiple camera units, thus avoiding increased manufacturing costs.
Solution Approach 2:
The patent replaces the mechanical approach of using multiple cameras (which would require additional mounting structures, synchronization mechanisms, and complex calibration) with an optical sensing approach using LiDAR. The LiDAR's ability to rapidly scan and detect objects in three-dimensional space provides the same functional benefit as multiple cameras but with simpler system architecture and lower cost.
3Area of stationary object
If multi-channel 360° LiDAR is used for object tracking, then tracking coverage improves, but manufacturing cost increases and recognition performance is lower than camera
Solution Approach 1:
Instead of implementing a full 360° multi-channel LiDAR system, the patent uses a single LiDAR sensor that provides sufficient tracking coverage for the application. The LiDAR performs scanning in the relevant directions to detect objects of interest, applying partial action rather than complete omnidirectional coverage. This approach achieves adequate tracking coverage without the high cost and complexity of a full multi-channel 360° LiDAR system.
Solution Approach 2:
The patent makes the single LiDAR sensor perform multiple functions: it detects object presence, determines object location, provides depth information, and guides camera pointing. This multi-functional use of a single LiDAR replaces the need for multiple specialized sensors, achieving comprehensive tracking coverage while minimizing manufacturing cost.
4Reliability
If camera FPS is increased to reduce blur, then tracking performance improves, but processing speed and system complexity increase
Solution Approach 1:
The LiDAR acts as an intermediary that provides object location information independently of camera frame rate. By using LiDAR to detect object positions and then guiding the camera to those positions, the system can maintain reliable tracking without requiring the camera to capture high-FPS images. This mediator approach decouples tracking performance from camera processing speed.
Solution Approach 2:
The LiDAR performs preliminary detection of object locations before the camera captures images. By pre-identifying object positions using LiDAR's rapid scanning capability, the system can then direct the camera to those specific locations, reducing the need for high-FPS full-field imaging. This preliminary action by LiDAR enables the camera to operate at lower FPS while maintaining tracking performance.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables continuous object tracking and re-acquisition of images by changing the camera's direction based on LiDAR information, reducing manufacturing costs and improving processing speed without the need for multiple cameras or high-cost LiDAR systems.
Implementation Method 1
a Light Detection And Ranging (LiDAR) sensor... based on scanning information obtained by the LiDAR sensor
Data Source
AI summary
An electronic apparatus is provided. The electronic apparatus includes a camera, a Light Detection And Ranging (LiDAR) sensor, and a processor configured to track an object based on a plurality of images photographed by the camera sequentially, wherein, to track the object based on the plurality of images, the processor is configured to, based on the object being identified in a first image from among a plurality of images and subsequently, not being identified in a second image after the first image, control a photographing direction of the camera based on scanning information obtained by the LiDAR sensor.