Frame Reprojection Using Motion Vectors for VR Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional reality technology systems, such as VR, face challenges in maintaining a realistic experience due to frame latency and image judder during head movements, especially in dynamic scenes and linear movements, as they lack the processing power to perform complex image processing for space warping and motion vector generation on low-powered client devices.
Innovation Solution
The system extracts and utilizes motion vector information from the encoded video stream generated at the server device to perform space warping and time warping on low-powered client devices, enabling accurate synthesis of frames that compensate for linear head movements and dynamic scenes by shifting and scaling images based on motion vectors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If complex image processing for space warping and motion vector generation is performed on client devices, then image quality and realism are improved, but device complexity and processing requirements increase
Solution Approach 1:
The patent extracts motion vector information from the encoded video stream at the client device, separating the complex space warping computation from the client device. The server generates and transmits motion vectors, which the client uses to perform simplified space warping, thereby improving image quality without significantly increasing client device complexity.
Solution Approach 2:
The server device performs preliminary computation of motion vectors during video encoding and transmits them to the client device. This preliminary action allows the client device to use pre-computed motion vectors for space warping, reducing the computational burden on the client device while maintaining high image quality.
2Loss of time
If frame rate is increased to reduce latency, then realism is improved, but bandwidth requirements and processing power increase
Solution Approach 1:
The patent uses motion vectors to create interpolated frames that copy and transform information from previous and next key frames. This allows the system to achieve higher effective frame rates for reducing latency without requiring the server to render and transmit every single frame at the target frame rate, thereby reducing processing power requirements.
Solution Approach 2:
The system performs space warping and motion compensation selectively based on detected scene changes. When scene changes are detected, the system uses motion vectors from surrounding frames; when no scene changes occur, simpler processing is used. This partial application of complex processing reduces overall processing power requirements while maintaining low latency during critical moments.
3Measurement precision
If motion vectors are extracted and used for space warping, then image accuracy during linear movement is improved, but processing complexity at the client device increases
Solution Approach 1:
The patent introduces motion vectors as an intermediary data structure that mediates between the encoded video stream and the space warping process. The client device extracts these pre-computed motion vectors and uses them to guide the space warping operation, achieving accurate image alignment during linear movement without requiring the client device to perform complex motion estimation algorithms.
Data Source
AI summary
A client device for use in processing reality technology content is provided. The client device comprises memory configured to store data and a processor in communication with the memory. The processor is configured to receive, from a server device via network communication channel, a video stream comprising encoded video frames and motion vector information and extract, at the client device, the motion vector information. The processor is also configured to decode the video frames, determine whether to at least one of time warp the video frames and space warp the video frames using the extracted motion vector information. The processor is also configured to display the warped video frame data.