Remote Vehicle Video Frame Synthesis for Latency and Frame Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Remote control of autonomous or semi-autonomous vehicles is limited by poor adaptation to video feed perspectives, latency, and dropped or interrupted video frames, leading to potential collisions with surrounding objects or pedestrians.
Innovation Solution
A remote vehicle control system that includes a vehicle-mounted sensor system with a video camera, data handling for frame synthesis using convolutional neural networks for interpolation and extrapolation, and a data handling system with forward error correction and prioritization to improve video stream reliability and frame rate.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If video frames are transmitted over a network for remote control, then the teleoperator can view the vehicle environment, but latency and dropped frames occur leading to poor control performance
Solution Approach 1:
The system performs preliminary actions by capturing and buffering video frames at the vehicle end before transmission is needed. The circular buffer stores multiple frames in advance, so when network transmission experiences latency or drops, the teleoperator can still receive and display previously captured frames without interruption, maintaining continuous video feed reliability.
2Reliability
If video frames are transmitted over a network for remote control, then the teleoperator can view the vehicle environment, but dropped frames occur leading to poor control performance
Solution Approach 1:
The system implements beforehand cushioning by maintaining a circular buffer that stores multiple video frames in advance at the vehicle end. This buffer acts as a cushion against network failures - when frames are dropped during transmission, the system can retrieve and resend previous frames from the buffer, ensuring video feed continuity and preventing gaps in the teleoperator's view.
3Adaptability or versatility
If multiple video streams are transmitted for different vehicle views, then comprehensive monitoring is improved, but network bandwidth consumption increases
Solution Approach 1:
The system applies local quality by prioritizing video streams based on their importance to vehicle control. Forward-facing camera streams that provide critical driving information are transmitted with higher priority and more frequent updates, while rearward or less critical views are transmitted with lower priority. This selective approach maintains comprehensive monitoring coverage while optimizing network bandwidth usage according to the specific control needs at each moment.
Data Source
AI summary
A remote vehicle control system includes a vehicle mounted sensor system including a video camera system for producing video data. A data handling system is connected to a network to transmit data to and receive data from a remote teleoperation site. A virtual control system is configured to receive the video, provide a user with a live video stream supported by machine intelligence directed frame synthesis, and transmit control instructions to the remote vehicle over the network. The frame synthesis is supported by a convolutional neural network. The frame synthesis may be used to interpolate frames to increase effective frames per second. The frame synthesis may be used to extrapolate frames to replace missing or damaged video frames.