Multimodal Spiking Neural Navigation for Dynamic Obstacle Avoidance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing robot obstacle avoidance methods struggle to effectively perceive and navigate around dynamic obstacles, such as moving people or objects, due to limitations in traditional laser radar strategies and the lack of efficient processing methods for dynamic environments.
Innovation Solution
A robot dynamic obstacle avoidance method based on a multimodal spiking neural network that fuses laser radar data and processed event camera data, using a hybrid spiking variational autoencoder module, population coding module, and a middle fusion decision module with a learnable threshold to guide the robot's movement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional laser radar strategy is used for obstacle avoidance, then the system is simple and easy to implement, but it cannot effectively perceive and process dynamic obstacles moving suddenly in the environment
Solution Approach 1:
The patent combines laser radar and event camera into a fused perception system. The laser radar provides accurate range measurements while the event camera captures dynamic changes with high temporal resolution. The fusion module integrates both data sources to achieve reliable dynamic obstacle perception that neither sensor could achieve alone.
Solution Approach 2:
The event camera serves multiple functions: it detects dynamic obstacles, provides high-temporal-resolution motion information, and operates with low power consumption. This multi-functional sensor replaces the need for multiple specialized sensors, achieving reliable dynamic perception without proportionally increasing system complexity.
2Measurement precision
If event camera is used to perceive dynamic obstacles, then the temporal resolution is very high (up to 1 MHz) and power consumption is very low, but the data stream output format is completely different from traditional camera frame output and cannot be simply used directly
Solution Approach 1:
The patent introduces a data fusion module as an intermediary that translates and integrates event camera data with laser radar data. This fusion module converts the asynchronous event stream into a format compatible with traditional processing pipelines, enabling high temporal resolution measurement without overwhelming data processing complexity.
Solution Approach 2:
The system transforms the event camera's high-frequency asynchronous event stream into a processed data format with adjusted temporal parameters. By changing the data representation parameters through fusion processing, the system maintains the high temporal resolution advantage while making the data suitable for integration with other sensors and control systems.
3Extent of automation
If deep reinforcement learning is used for obstacle avoidance learning, then the system can learn independently without manual collection of annotated data sets, but the perception of dynamic obstacles that move rapidly is not complete enough to implement efficient obstacle avoidance strategies
Solution Approach 1:
The patent segments the perception task into two complementary parts: laser radar handles static environment mapping and structured spatial information, while the event camera specializes in detecting dynamic changes and motion. This segmentation allows the reinforcement learning system to receive comprehensive perceptual input without requiring complete manual annotation of all obstacle types.
Solution Approach 2:
The system creates a composite perception input by combining data from laser radar and event camera, analogous to composite materials. This fused data structure provides both the geometric precision of laser radar and the dynamic sensitivity of event camera, enabling complete dynamic obstacle perception that supports effective autonomous learning and decision-making.
Data Source
AI summary
The present invention provides a robot dynamic obstacle avoidance method based on a multimodal spiking neural network. The present invention realizes a robot obstacle avoidance method in a dynamic environment by fusing laser radar data and processed event camera data and combining with the intrinsic learnable threshold of the spiking neural network for a scenario comprising dynamic obstacles. It solves the difficulty of failure of obstacle avoidance due to the difficulty in perceiving the dynamic obstacles in the obstacle avoidance task of a robot. The present invention helps the robot to fully perceive the static information and the dynamic information of the environment, uses the learnable threshold mechanism of the spiking neural network for efficient reinforcement learning training and decision making, and realizes autonomous navigation and obstacle avoidance in the dynamic environment. An event data enhanced model is combined to better adapt to the dynamic environment for obstacle avoidance.