Remote Driving Video Overlay for Latency-Aware Vehicle Positioning

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Teleoperated remote driving systems experience latency issues due to video and control latency, leading to a 'past' view of the vehicle's environment and delayed command execution, which compromises safety and reliability.

Innovation Solution

Modify the video stream to include a visualization of the vehicle's estimated position, adjusting colors, highlighting, sharpness, focus, and resolution based on video latency and uncertainty, to intuitively aid the teleoperator in understanding the vehicle's actual position and safely control it.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of information

If a live video stream is transmitted from the vehicle to the teleoperator station, then the teleoperator can see the vehicle's environment, but the video latency causes the teleoperator to see a 'past' view instead of real-time conditions

Engineering Contradiction:

Improvereal-time environmental informationVSAvoidvideo transmission latency

Core Design Contradiction:

Loss of informationVSLoss of time

Solution Approach 1:

The system performs preliminary actions by predicting the vehicle's future position based on current motion state (velocity, acceleration, steering angle) and displays this predicted position to the teleoperator. This allows the teleoperator to see where the vehicle will be rather than where it was when the video frame was captured, effectively compensating for video latency without requiring faster transmission.

Inventive Principle:

Principle #10Preliminary action

2Measurement precision

If the video stream is modified to include visualization of the vehicle's estimated position, then the teleoperator can better understand the vehicle's actual position, but the computational intensity increases

Engineering Contradiction:

Improvevehicle position accuracy perceptionVSAvoidcomputational energy at teleoperator station

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The system changes parameters by selectively applying different levels of visualization based on video latency thresholds. When video latency is below a threshold, no additional visualization is added. When latency exceeds the threshold, the system adds predicted position indicators with varying levels of detail (such as uncertainty regions or confidence levels) based on the magnitude of latency, thus optimizing computational energy usage while maintaining position accuracy perception.

Inventive Principle:

Principle #35Parameter changes

3Reliability

If the system provides detailed visualization of vehicle position and uncertainty, then the teleoperator can make safer control decisions, but the device complexity increases

Engineering Contradiction:

Improvesafe control decision-makingVSAvoidvideo stream processing complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system segments the visualization information into distinct components: the actual vehicle position, the predicted future position, and the uncertainty region. Each component is rendered separately and overlaid on the video stream. This segmentation allows the teleoperator to clearly distinguish between different types of information and makes the processing pipeline more manageable and modular.

Inventive Principle:

Principle #1Segmentation

4Loss of time

If the video latency is reduced through faster transmission, then the teleoperator sees more real-time information, but the system cost and infrastructure requirements increase

Engineering Contradiction:

Improvevideo transmission timeVSAvoidtransmission energy and infrastructure cost

Core Design Contradiction:

Loss of timeVSLoss of energy

Solution Approach 1:

The system converts the harmful effect of video latency into a beneficial feature by using the latency time to compute and display predicted future vehicle positions. Instead of merely showing where the vehicle was when the frame was captured, the system leverages the time delay to provide forward-looking information that helps the teleoperator anticipate the vehicle's trajectory and make better control decisions, effectively turning the latency from a disadvantage into an advantage.

Inventive Principle:

Principle #22Blessing in disguise (Convert harm into benefit)

Data Source

PatentUS12537921B2Systems and methods to account for latency associated with remote driving applications

Publication Date: 2026.01.27 VAY TECH GMBH

AI summary

Systems and methods to account for latency associated with remote driving applications may include a vehicle having an imaging device and a teleoperator station in communication with each other via a network. Imaging data that is captured by the imaging device may be transmitted to the teleoperator station for presentation to a teleoperator. In order to account for latency in the transmission, receipt, processing, and presentation of the imaging data, one or more visualizations of the vehicle, with various visual characteristics, may be rendered within or overlaid onto the imaging data, in order to facilitate safe and reliable remote operation of the vehicle by the teleoperator at the teleoperator station.