Obstacle avoidance using fused depth and intensity for NNT training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cleaning robots face challenges in accurately detecting obstacles due to their low profile and unique floor-level viewpoint, leading to misidentification when using neural networks trained with images from different perspectives.
Innovation Solution
The method involves capturing images with image sensors on cleaning robots, transmitting data to a remote neural network for machine learning, and training the network using fused intensity and depth data from PMD sensors to classify potential hazards, allowing the robot to adjust its operation accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If neural networks are trained with images from different perspectives (online libraries), then the neural network can recognize a wide variety of objects, but cleaning robots with low profile and floor-level viewpoint experience misidentification due to perspective differences
Solution Approach 1:
The patent applies local quality by creating a specialized training dataset that matches the specific local conditions of cleaning robots - floor-level perspectives, low profile viewpoints, and cleaning robot-specific camera angles. This localized training data ensures the neural network is adapted to the specific operational context rather than using generic high-perspective images, thereby resolving the mismatch between general object recognition capabilities and specific obstacle detection accuracy.
2Ease of manufacture
If only intensity data from image sensors is used for training, then the training process is simpler and faster, but the neural network lacks depth information needed for accurate obstacle detection and differentiation
Solution Approach 1:
The patent merges multiple data types - combining intensity data from image sensors with depth data from time-of-flight sensors - to create a fused multi-modal training dataset. This combination allows the neural network to receive both the simplicity of intensity-based images and the spatial depth information simultaneously, enabling accurate object differentiation and obstacle detection without sacrificing training feasibility.
3Productivity
If cleaning robots use generic object recognition models trained on high-perspective images, then implementation is easier and faster, but the robots cannot accurately identify objects from their unique floor-level viewpoint
Solution Approach 1:
The patent applies preliminary action by pre-processing and curating training images specifically for floor-level perspectives before deploying the model to cleaning robots. This includes capturing images at robot operational height, annotating them with obstacle labels, and preparing the dataset in advance. This preliminary preparation ensures that when the model is deployed, it immediately achieves high reliability for floor-level obstacle identification without requiring complex runtime adjustments.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach enhances obstacle detection accuracy by providing a high-fidelity model trained on depth and intensity images, enabling the robot to differentiate and measure objects more effectively, thereby improving navigation and avoiding potential hazards.
Implementation Method 1
a non-moving laser emitter and detector can be used to give a 360 degree view, such as the PMD Technologies 3D sensor for TOF
Data Source
Figure 1~2
Figure 3
Figure 4~5
AI summary
Embodiments provide a method for obstacle avoidance in a mobile robot. Images of objects are captured with mobile robot image sensors with a floor level perspective. Data corresponding to the images is used to train a neural network. The identified objects are classified by indicating whether the objects are a potential hazard. In one embodiment, an image sensor on a robot provides intensity and depth data for each of a plurality of pixels of the images. The intensity and depth data are fused to produce fused data. The fused data is then used to train the neural network.