Remote Operation Gaze Sharing for Working Machine Action Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Less skilled operators face challenges in grasping remotely operated actions of a working machine from a skilled operator's viewpoint, necessitating improved information provision for effective remote operation.
Innovation Solution
A remote operation system comprising a working machine with an imaging device, a first remote operation apparatus with a sight line detector, and a second remote operation apparatus that displays a highlighted image region corresponding to the detected sight line, using wireless communication to transmit and display environmental images, allowing the second operator to visualize the first operator's actions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a skilled operator performs remote operation of a working machine, then the operation efficiency is improved, but a less skilled operator cannot grasp the operated action from the skilled operator's viewpoint
Solution Approach 1:
The system creates a visual copy of the skilled operator's viewpoint by detecting the sight line and displaying it as a highlighted region in the environmental image. This allows the less skilled operator to see exactly what the skilled operator is looking at, effectively copying the visual perspective without requiring physical presence or complex communication
Solution Approach 2:
The patent uses color changes to highlight the designated image region corresponding to the sight line. By displaying this region in a different color or with different visual properties than surrounding areas, the system draws attention to critical areas and helps the less skilled operator understand where the skilled operator is focusing attention
2Ease of operation
If the second operator views the environmental image without sight line information, then the image display is simple, but the second operator cannot recognize the first operator's actions
Solution Approach 1:
The sight line information acts as an intermediary element that bridges the gap between the first operator's actions and the second operator's understanding. By introducing this intermediate visual indicator, the system enables action recognition without complicating the basic image display functionality
Solution Approach 2:
The environmental image is segmented into a designated image region (highlighted) and other regions. This segmentation allows the second operator to quickly identify the area of interest without being overwhelmed by the entire image, maintaining simplicity while providing critical information
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Provided is a system which can supply appropriate information to one operator from a viewpoint that the one operator grasps a remotely operated action of a working machine by another operator. In a first remote operation apparatus 10, a sight line detector 112 detects a sight line of a first operator, and transmits sight line detection data corresponding to the sight line of the first operator. In a second remote operation apparatus 20, a second image output device 221 displays a designated image region spreading with reference to the sight line of the first operator corresponding to the sight line detection data in an environmental image corresponding to captured image data acquired by an imaging device 401 of a working machine 40, in a form different from a form of an image region around the designated image region.