Remote Working Machine Control Using Visual Tags for Depth Perception
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Remote operation of working machines, such as excavators and loaders, faces challenges in providing depth perception, making it difficult for operators to accurately control tools like buckets with movable loose material, impacting productivity.
Innovation Solution
A system that uses visual tags on load carriers, detected by cameras, to determine the distance between the tool and the load carrier, providing operators with essential information for safe and efficient operation, and potentially incorporating sensor data for autonomous or semi-autonomous assistance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard 2D image/video feeds from cameras are used for remote operation, then the system complexity is low, but depth perception is lost making it difficult to determine tool position relative to load carrier
Solution Approach 1:
Visual tags are introduced as intermediary objects attached to the load carrier. These tags serve as mediators between the camera system and the load carrier, enabling the remote operator to perceive depth and distance information without requiring complex 3D imaging systems. The tags provide reference points that transform the 2D image into a depth-aware visualization.
Solution Approach 2:
The system changes the parameter of visual information by overlaying computed distance measurements and depth indicators onto the standard 2D camera feed. This transforms the plain 2D image into an enhanced visualization that conveys depth perception through additional visual parameters such as distance markers, depth overlays, and positional indicators.
2Measurement precision
If visual tags and distance determination systems are added to provide depth perception, then depth perception improves, but the device complexity increases
Solution Approach 1:
The visual tag serves multiple functions: it acts as a visual reference for depth perception, a target for computer vision algorithms to track position, and a marker for distance calculation. This multi-functionality reduces the need for separate systems for each function, thereby limiting the increase in overall system complexity.
Solution Approach 2:
Instead of using complex 3D cameras or LIDAR systems, the patent creates a virtual copy of depth information by using 2D camera images combined with visual tags and computational algorithms. This copying approach reconstructs depth perception software-based rather than hardware-based, avoiding the complexity of expensive 3D sensing equipment.
3Productivity
If remote operation is implemented without depth perception, then setup complexity is reduced, but productivity decreases due to difficulty in controlling tool positioning
Solution Approach 1:
The system implements feedback by continuously displaying computed distance information and tool position relative to the load carrier on the remote operator's screen. This real-time feedback loop allows the operator to make precise positioning adjustments, improving productivity without requiring complex manual coordination.
Solution Approach 2:
The patent replaces the need for complex mechanical 3D visualization systems with a software-based solution using computer vision and image processing. By substituting mechanical/optical complexity with computational algorithms, the system achieves depth perception capability while keeping hardware complexity manageable.
Data Source
AI summary
A system and a method for remote operation of a working machine comprising a tool is provided. The system includes an on-board controller configured to receive signals from a remote control station remotely controlling the operation of the working machine, and to obtain and send camera images to an off-board controller. The system also includes an off-board controller configured to receive the camera images from the on-board controller. The off-board controller identifies at least one visual tag in the camera images located on a load carrier, determines at least one distance between the tool and the load carrier based on the identified at least one visual tag, and provides, to the on-board controller and/or to an operator of the working machine, information based on the determined at least one distance between the tool and the load carrier in order to support the remote operation of the working machine.