Visual Servoing Path Planning for Camera Occlusion in Robotic Arms
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Visual servoing systems in robotic arms face challenges with occlusions, where the target object becomes hidden from the camera's field of view, leading to difficulties in maintaining control and tracking during tasks, especially in dynamic and uncontrolled environments.
Innovation Solution
The implementation of multiple cameras positioned on different links of a redundant manipulator robotic arm, allowing for continuous tracking of a target by using extra degrees of freedom, and employing a visibility-aware path planning algorithm to optimize trajectories and maintain line of sight, even when occlusions occur.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single camera is mounted on the end effector for visual servoing, then the system structure is simple, but the target becomes occluded when the end effector approaches the target, causing loss of visual tracking
Solution Approach 1:
The single camera system is segmented into multiple cameras positioned at different locations on the robotic arm (end effector, intermediate links, base). Each camera provides a different viewpoint, and the system segments the visual monitoring task across multiple sensors to eliminate blind spots and occlusions that a single camera cannot avoid.
Solution Approach 2:
Intermediate cameras mounted on link segments of the robotic arm serve as mediators between the base camera and the end effector camera. These intermediary cameras provide alternative viewing angles when the end effector blocks the target from the base camera's view, acting as visual relays to maintain continuous tracking.
2Manufacturing precision
If the end effector moves close to the target for precise manipulation, then the manipulation precision is improved, but the target occlusion increases making visual servoing difficult
Solution Approach 1:
The system transitions from a single-viewpoint (2D image plane) to multi-viewpoint (3D spatial distribution of cameras) visual monitoring. By distributing cameras across different spatial dimensions on the robotic arm structure, the system captures the target from multiple angles simultaneously, ensuring that at least one camera maintains an unobstructed view even when the end effector is in close proximity to the target.
Solution Approach 2:
The system performs preliminary visual assessment using multiple cameras before the end effector executes close-proximity manipulation. The base and intermediate cameras detect potential occlusions in advance, allowing the control system to select optimal camera views and plan manipulation trajectories that maintain visual contact, preventing information loss before it occurs.
3Reliability
If multiple cameras are positioned on different links of the robotic arm, then the target visibility is maintained through extra degrees of freedom, but the system complexity and computational requirements increase
Solution Approach 1:
The robotic arm's redundant degrees of freedom, originally designed for manipulation tasks, are made multi-functional by simultaneously using them for camera positioning. The same joint movements that position the end effector for manipulation also reposition the mounted cameras, allowing a single actuator set to serve both manipulation and visual monitoring functions, thereby reducing overall system complexity despite multiple cameras.
Solution Approach 2:
The camera positioning system is made dynamic and adaptive rather than static. Cameras are mounted on movable links that can change position during task execution, allowing the system to dynamically adjust camera viewpoints based on the current task phase and occlusion conditions. This dynamic repositioning enables a relatively simple mechanical mounting structure to achieve complex visual monitoring capabilities.
4Reliability
If visibility-aware path planning is used to maintain line of sight, then the visual servoing reliability is improved, but the path planning computation time increases
Solution Approach 1:
The system performs preliminary computation of visibility-aware paths before actual task execution. By pre-calculating trajectories that maintain line-of-sight for all cameras and storing these planned paths, the system avoids real-time computation during execution, thereby reducing operational delays while maintaining visual reliability through pre-optimized visibility-constrained trajectories.
Solution Approach 2:
The system implements feedback mechanisms where the actual camera views and occlusion states are continuously monitored during task execution. This feedback allows the system to verify whether pre-planned visibility-aware paths are being maintained and to make minor real-time adjustments without requiring complete re-planning, thus balancing reliability with computational efficiency through iterative refinement rather than full re-computation.
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
Various aspects of techniques, systems, and use cases may be used for camera and end-effector planning for visual servoing for example in redundant manipulators. A technique may include generating a set of paths of an end effector of a robotic arm, and ranking the set of paths using an objective function that results in an improvement to a distance between the end effector and a target while maintaining a collision mitigation path for the end effector and minimizing occlusion of a camera affixed to a joint of the robotic arm. The technique may include converting a path of the set of paths into a trajectory based on the ranking, and outputting the trajectory for controlling the robotic arm to move the end effector.