Head-Mounted Display Video Alignment for Low-Latency Remote Operation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Remote operation of machinery, particularly with head-mounted displays, faces challenges due to latency issues that can be dangerous and disorienting, especially in hazardous environments where precise control is required.
Innovation Solution
The system reduces latency by using a remote capture device and a graphics processing unit to process video from cameras mounted on controllable machines, synchronizing the video with the head-mounted display's pose, and rendering images based on a rotation matrix to align the camera and display movements, minimizing processing latency and mechanical latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If video processing is performed remotely to provide the same viewing perspective, then the user gains proper viewing perspective for remote operation, but latency is introduced which is dangerous and disorienting
Solution Approach 1:
The patent extracts the video processing function from the remote location and relocates it to the head-mounted display device. By processing video locally on the HMD rather than remotely, the system eliminates transmission and remote processing delays, providing both accurate viewing perspective and real-time feedback without latency
Solution Approach 2:
The patent introduces a graphics processing unit (GPU) as an intermediary component within the HMD system. This GPU performs real-time video processing, rotation matrix calculations, and image rendering locally, acting as a mediator between the camera input and display output to minimize processing time and eliminate latency
2Loss of information
If video processing is done remotely, then the user receives processed video information, but mechanical latency occurs between camera position and display position
Solution Approach 1:
The patent implements preliminary action by pre-calculating rotation matrices and preparing rendering pipelines in advance. The system maintains ready-state processing capabilities so that when video frames arrive, they can be immediately processed and displayed without mechanical delays, ensuring both information quality and real-time performance
3Loss of time
If all video processing is done on the graphics processing unit, then processing latency is minimized, but device complexity increases
Solution Approach 1:
The patent applies multi-functionality by designing the graphics processing unit to handle multiple tasks simultaneously: video frame processing, rotation matrix calculations, image rendering, and synchronization. This consolidation of functions into a single versatile component minimizes latency while avoiding the need for multiple separate processing devices
Data Source
AI summary
Media, systems, and methods for reducing latency in a head-mounted display for the remote operation of machinery. A remote camera sends raw video to a graphics processing unit which processes the video and transmits the updated video to a head-mounted display with limited latency such that the control of remote devices may be improved. The image may be spherically rendered and projected onto the head-mounted display with a rectangular view. The graphics processing unit may determine a rotation matrix based on a pose of the remote camera and a pose of the head-mounted display, and the image may be projected based on the rotation matrix.