Remote Robotic Control Using Predicted Future Video Frames

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Latency in data transmission between a robotic device and its operator leads to control inputs being based on past contexts, resulting in inaccurate or unsuccessful task performance due to changes in the operating environment.

Innovation Solution

Systems and methods predict estimated future frames using machine learning models to compensate for communication latency, allowing control inputs to be based on future contexts, thereby improving accuracy and simplifying operations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If real-time video streaming is used for remote control, then the operator can see the current environment, but the control input arrives too late due to transmission latency

Engineering Contradiction:

Improvecontrol accuracyVSAvoidcontrol latency

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by predicting future environment frames before the control input is executed. The machine learning model generates predicted future frames that represent the environment state at the time the control input will take effect, allowing the operator to make informed control decisions based on anticipated future states rather than reacting to past states.

Inventive Principle:

Principle #10Preliminary action

2Loss of time

If control is based on past video frames, then transmission latency is acceptable, but the control input becomes inaccurate for the current environment

Engineering Contradiction:

Improvecontrol latencyVSAvoidenvironment context accuracy

Core Design Contradiction:

Loss of timeVSMeasurement precision

Solution Approach 1:

The system transforms the temporal parameter of video frames by using machine learning to predict future frames. Instead of displaying past frames or current frames, the system generates predicted future frames that represent the environment state at a future timestamp, effectively changing the temporal parameter from past/present to future to compensate for control latency.

Inventive Principle:

Principle #35Parameter changes

3Reliability

If machine learning models predict future frames, then control accuracy improves, but computational resources increase

Engineering Contradiction:

Improvecontrol accuracyVSAvoidcomputational energy consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The system introduces a machine learning model as an intermediary between the video stream and the control interface. This intermediary predicts future environment frames based on the video input, providing the operator with anticipated future states without requiring complex real-time processing at the robotic device end, thus balancing accuracy improvement with manageable computational resources.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS12436530B2Latency management systems and methods for remote control of a robotic device

Publication Date: 2025.10.07 VERIZON PATENT & LICENSING INC

AI summary

An illustrative latency management system receives a sequence of captured frames in a video stream of an operating environment associated with a robotic device. The latency management system monitors a communication latency of a communication channel between the robotic device and a robotic device operator. The latency management system predicts one or more estimated future frames associated with the video stream based on the sequence of captured frames and the communication latency. The latency management system provides the one or more estimated future frames to the robotic device operator. Corresponding methods and systems are also disclosed.