Visual-Haptic Robotic Grasping Under Complex Illumination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing mobile mechanical arm technologies face challenges in accurately identifying and positioning target objects under complex illumination conditions, and effectively utilizing haptic information for precise grabbing tasks.
Innovation Solution
An autonomous mobile grabbing method based on visual-haptic fusion, utilizing a multi-sensor module comprising a communication module, image acquisition module, and force sensing module, which includes depth cameras, 6-dimensional force sensors, and 24-dimensional capacitive haptic sensors to fuse visual and haptic information for accurate object positioning and grabbing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If visual information is used for target object identification and positioning, then the mechanical arm can perform autonomous navigation and operation, but the positioning accuracy deteriorates under complex illumination conditions
Solution Approach 1:
The patent combines visual sensors (RGB-D cameras) with haptic sensors (6-dimensional force sensors, 24-dimensional capacitive haptic sensors) to create a fused sensing system. This merging of different sensing modalities allows the system to maintain accurate target object positioning under complex illumination conditions by compensating for visual limitations with haptic feedback information.
Solution Approach 2:
The haptic sensor information acts as an intermediary that bridges the gap between visual detection and precise positioning. When visual information becomes unreliable under complex illumination, the haptic sensors provide alternative contact information about the target object's position, stiffness, shape, and weight, enabling accurate positioning despite illumination challenges.
2Device complexity
If only visual sensors are used for grabbing tasks, then the system structure remains simple, but the grabbing precision and object interaction capability deteriorate
Solution Approach 1:
The patent merges multiple sensor types (visual RGB-D cameras with haptic 6-dimensional force sensors and 24-dimensional capacitive haptic sensors) into an integrated sensing system. This combination enables precise grabbing operations by fusing visual information about object location with haptic information about object contact, stiffness, shape, and weight, thereby improving grabbing precision beyond what either sensor type could achieve alone.
3Measurement precision
If haptic sensors are introduced to improve grabbing precision, then the grabbing accuracy improves, but the device complexity increases
Solution Approach 1:
The haptic sensor system is designed to perform multiple functions simultaneously: detecting contact force, determining object stiffness, identifying object shape, and measuring object weight. This multi-functionality allows a single sensor module to provide diverse information needed for precise grabbing operations, reducing the need for separate specialized sensors and thereby managing device complexity while improving grabbing accuracy.
Solution Approach 2:
The patent implements feedback loops where haptic sensor measurements of contact force and object properties are continuously fed back to adjust the grabbing mechanism in real-time. This feedback enables dynamic adjustment of grabbing force and positioning based on actual contact conditions, improving grabbing accuracy while the integrated control system manages the complexity of coordinating multiple sensors.
Data Source
AI summary
The present disclosure discloses an autonomous mobile grabbing method for a mechanical arm based on visual-haptic fusion under a complex illumination condition, which mainly includes approaching control over a target position and feedback control over environment information.According to the method, under the complex illumination condition, weighted fusion is conducted on visible light and depth images of a preselected region, identification and positioning of a target object are completed based on a deep neural network, and a mobile mechanical arm is driven to continuously approach the target object; in addition, the pose of the mechanical arm is adjusted according to contact force information of a sensor module, the external environment and the target object; and meanwhile, visual information and haptic information of the target object are fused, and the optimal grabbing pose and the appropriate grabbing force of the target object are selected.By adopting the method, the object positioning precision and the grabbing accuracy are improved, the collision damage and instability of the mechanical arm are effectively prevented, and the harmful deformation of the grabbed object is reduced.