Teleoperator Visualization for Remote Driving Network Latency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Latency in the teleoperation control loop due to wireless communication network delays impairs the ability of teleoperators to safely and correctly remotely drive vehicles, especially in dynamic urban environments.

Innovation Solution

A teleoperator station that processes live vehicle environment data to generate visualizations considering network latency, including graphical indicators representing current and anticipated vehicle pathways, to enhance the teleoperator's awareness and control accuracy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If a teleoperator uses a conventional teleoperator station to remotely drive a vehicle via wireless communication, then the teleoperator can control the vehicle from a distance, but the transmission latency severely impairs the teleoperator's ability to correctly and safely drive the vehicle in dynamic environments

Engineering Contradiction:

Improveremote driving capabilityVSAvoiddriving accuracy

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The system performs preliminary actions by predicting the vehicle's future position and environment state before the teleoperator receives the delayed live video stream. The prediction processing unit calculates where the vehicle will be and what the environment will look like after the latency period, allowing the teleoperator to make decisions based on anticipated rather than past information.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system introduces feedback by continuously comparing the predicted future state with the actual delayed state received from the vehicle. This feedback loop allows the prediction model to be continuously refined and adjusted, improving the accuracy of the anticipated environment presentation over time.

Inventive Principle:

Principle #23Feedback

2Adaptability or versatility

If the system transmits live video stream and control commands via wireless network, then remote operation is enabled, but the control loop latency causes the vehicle to be in a different situation than what the teleoperator perceives

Engineering Contradiction:

Improveremote operation flexibilityVSAvoidenvironmental state information

Core Design Contradiction:

Adaptability or versatilityVSLoss of information

Solution Approach 1:

The prediction processing unit performs preliminary calculations to determine the vehicle's future position, orientation, and environmental context before the teleoperator needs this information. This advance computation compensates for the information loss caused by transmission latency.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The prediction processing unit acts as an intermediary between the delayed live video stream and the teleoperator. It processes the received data and generates predicted future state information, serving as a mediator that bridges the gap caused by network latency.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Ease of operation

If the teleoperator reacts to past environment information due to latency, then wireless remote control is achieved, but the drive control commands are executed only after significant time delay

Engineering Contradiction:

Improveremote control capabilityVSAvoidcontrol loop delay

Core Design Contradiction:

Ease of operationVSLoss of time

Solution Approach 1:

The system performs preliminary prediction of the vehicle's state and environment before the teleoperator formulates control commands. By anticipating future conditions, the teleoperator can issue commands that are timely and appropriate for the actual current state rather than reacting to outdated information.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Instead of waiting for the delayed live video stream to process current state information, the system inverts the approach by predicting future state information in advance. This inversion allows the teleoperator to work with anticipated current state rather than past state, effectively compensating for the time delay.

Inventive Principle:

Principle #13The other way round (Inversion)

Data Source

PatentEP3702864B1Accounting for latency in teleoperated remote driving

Publication Date: 2021.10.27 VAY TECH GMBH

Figure 1
Figure 2(a)~2(c)
Figure 3

AI summary

The present invention relates to a teleoperator station (40) for use by a teleoperator to remotely drive a vehicle (20) via a wireless communication network (30). The teleoperator station comprises a communication unit (41) for receiving a live representation of the vehicle's environment from the vehicle and for sending drive control commands for controlling the driving of the vehicle to the vehicle. The teleoperator station further comprises a processing unit (42) for processing the live representation to generate a visualization of the vehicle's environment that considers a latency of the wireless communication network, a display unit (43) for displaying the visualization to the teleoperator, and a control interface (44) for use by the teleoperator to generate the drive control commands in response to the visualization.