Visual-Tactile Spiking Sensing for Low-Latency Robot Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing tactile sensors for robots are difficult to scale and integrate with robot platforms due to high latency and complex wiring, and event-driven perception systems lack effective training procedures for spiking neural networks (SNNs) to efficiently combine multiple sensory modalities.
Innovation Solution
A neuromorphic tactile sensor (NeuTouch) with asynchronous data transmission and a visual-tactile spiking neural network (VT-SNN) that encodes and merges event-based outputs from vision and tactile sensors using spiking neural networks (SNNs) for efficient classification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional tactile sensors are used for robot platforms, then tactile sensing capability is provided, but the system suffers from high latency and complex wiring that make scaling and integration difficult
Solution Approach 1:
The patent replaces conventional mechanical/electrical tactile sensor systems with a neuromorphic tactile sensor that uses biological-inspired neural processing. The sensor integrates sensing elements directly connected to a neuromorphic processing chip, eliminating complex external wiring by embedding the processing functionality within the sensor itself, thus reducing wiring complexity while maintaining tactile sensing capability
Solution Approach 2:
The patent introduces a neuromorphic processing layer as an intermediary between the tactile sensing elements and the robot control system. This intermediary processes tactile information using biological-inspired neural networks, reducing the need for complex wiring and high-latency digital processing while maintaining reliable tactile sensing
2Measurement precision
If conventional deep learning methods are used for processing visual and tactile data, then classification accuracy can be achieved, but energy consumption is high
Solution Approach 1:
The patent changes the fundamental processing parameters from conventional deep learning to neuromorphic processing. The neuromorphic system uses sparse, event-driven processing that mimics biological neural networks, dramatically reducing energy consumption while maintaining classification accuracy through efficient temporal and spatial processing of sensory data
Solution Approach 2:
The patent implements event-driven processing where neural networks are activated only when changes occur in the sensory input, rather than continuous processing. This periodic, event-triggered operation reduces energy consumption significantly while maintaining high classification accuracy by focusing computational resources only when needed
3Loss of time
If event-based sensors are used for vision and tactile data, then low-latency data capture is achieved, but effective training procedures for spiking neural networks are lacking
Solution Approach 1:
The patent develops and implements preliminary training procedures for spiking neural networks before deployment. These training procedures are specifically designed for event-based data and are integrated into the system development process, eliminating the barrier of lacking training methods while preserving the low-latency benefits of event-based sensing
Data Source
AI summary
A classifying sensing system, a classifying method performed using a sensing system, a tactile sensor, and a method of fabricating a tactile sensor. The classifying sensing system comprises a first spiking neural network, SNN, encoder configured for encoding an event-based output of a vision sensor into individual vision modality spiking representations with a first output size; a second SNN encoder configured for encoding an event-based output of a tactile sensor into individual tactile modality spiking representations with a second output size; a combination layer configured for merging the vision modality spiking representations and the tactile modality spiking representations; and a task SNN configured to receive the merged vision modality spiking representations and tactile modality spiking representations and output vision-tactile modality spiking representations with a third output size for classification.