Mobile Robot Human 3D Positioning Using 2D LiDAR-Camera Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing 2D LiDAR systems face issues with sparse signals, limited computation, and the need for manual annotation of large labels when detecting the position of a human being using a mobile robot.
Innovation Solution
A system utilizing a 2D range sensor, camera, and transformer neural network for accurate 3D position estimation of a person, trained through self-supervised learning with knowledge distillation, and using a segmentation network and clustering algorithm for efficient data labeling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If 2D LiDAR is used for human position detection, then the system is simpler and uses less energy, but the signals are sparse and computation is limited
Solution Approach 1:
The patent introduces a transformer neural network as an intermediary that processes sparse 2D LiDAR points and camera images to accurately determine human position. The transformer acts as a mediator that combines information from both sensors, resolving the contradiction by enabling precise position detection despite the sparsity of LiDAR signals through sophisticated data fusion and contextual understanding.
Solution Approach 2:
The system combines multiple sensors (2D LiDAR and camera) and multiple processing functions (point classification, bounding box detection, depth estimation) into a unified framework. This multi-functional approach allows the system to achieve both simplicity in hardware selection and high precision in position detection by leveraging the complementary strengths of different sensors and processing methods.
2Productivity
If traditional methods are used for position estimation, then manual labeling is required, but this increases time consumption and labor requirements
Solution Approach 1:
The patent implements self-supervised learning where the system automatically generates its own training data through a segmentation network that labels points based on semantic information from camera images. This self-service mechanism eliminates the need for manual annotation by having the system teach itself the relationship between visual features and spatial positions, dramatically reducing time consumption and labor requirements.
Solution Approach 2:
The system performs preliminary processing by pre-segmenting the image into semantic regions and pre-identifying potential human points before final position estimation. This preliminary action prepares the data in advance, reducing the need for time-consuming manual labeling while maintaining high accuracy in position detection.
3Measurement precision
If more computational resources are allocated, then position estimation accuracy improves, but the computation load increases
Solution Approach 1:
The patent applies partial action by processing only the necessary portions of data - specifically, only the points within the bounding box detected by the transformer network are subjected to detailed analysis. This selective processing approach maintains high position estimation accuracy for the region of interest while significantly reducing overall computational load compared to processing the entire point cloud.
Solution Approach 2:
The system applies different processing qualities to different regions: high-quality detailed analysis is applied locally to the bounding box region where human presence is detected, while coarser processing is applied to other areas. This local quality differentiation enables accurate position estimation where needed without proportionally increasing overall computation load.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables accurate 3D position estimation of a person with reduced computational load and without manual labeling, allowing the robot to perform tasks like transportation effectively.
Implementation Method 1
a 2D (two-dimensional) range sensor configured to detect a distance to a nearby point
Implementation Method 2
a camera configured to take an image of an area around the mobile robot
Data Source
AI summary
A system according to the present disclosure includes: a robot including a 2D range sensor configured to detect a distance to a nearby point; a camera configured to take an image of an area around the robot; a detection unit configured to detect a bounding box surrounding a person included in the image; a determination unit configured to determine, for each point included in the bounding box detected by the range sensor, whether or not the detected point corresponds to the person; and an estimation unit configured to estimate a 3D position of the person based on a distance to a detected point determined to correspond to the person.