Mixed Reality Video Compression Using Precomputed Motion Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing mixed reality systems face challenges in maintaining high frame rates and minimizing latency due to the time-consuming process of estimating motion vectors for encoding frames, which limits the user's immersive experience.
Innovation Solution
The system uses pre-determined motion vectors from head motion and rendering application data to encode frames, reducing the time required for motion compensation and embedding this information in the data stream for frame synthesis on the HMD, allowing for efficient encoding and display even without real-time frame reception.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional motion vector estimation is used to encode frames, then encoding accuracy is improved, but encoding time increases and frame rate decreases
Solution Approach 1:
The system performs motion vector determination in advance using sensor data from the HMD and rendering application, storing these pre-determined motion vectors for subsequent encoding operations. This preliminary action eliminates the need for time-consuming motion vector estimation during real-time frame encoding, thereby resolving the contradiction between encoding accuracy and frame rate.
2Measurement precision
If conventional motion vector estimation is used to encode frames, then encoding accuracy is improved, but processing time increases and latency increases
Solution Approach 1:
Motion vectors are determined and stored in advance based on sensor data and rendering application information. When encoding frames, these pre-determined motion vectors are directly utilized, eliminating the time required for real-time motion vector estimation and thereby reducing overall system latency while maintaining encoding accuracy.
3Productivity
If pre-determined motion vectors are used for encoding, then encoding speed is improved and frame rate is maintained, but encoding complexity increases
Solution Approach 1:
The system determines motion vectors in advance using sensor data from the HMD and rendering application, storing them for later use. This shifts the computational complexity from real-time encoding to a preliminary stage, allowing the encoding process itself to be simpler and faster, thereby resolving the contradiction between encoding speed and process complexity.
Data Source
AI summary
A mixed reality system including a head-mounted display (HMD) and a base station. Information collected by HMD sensors may be transmitted to the base via a wired or wireless connection. On the base, a rendering engine renders frames including virtual content based in part on the sensor information, and an encoder compresses the frames according to an encoding protocol before sending the frames to the HMD over the connection. Instead of using a previous frame to estimate motion vectors in the encoder, motion vectors from the HMD and the rendering engine are input to the encoder and used in compressing the frame. The motion vectors may be embedded in the data stream along with the encoded frame data and transmitted to the HMD over the connection. If a frame is not received at the HMD, the HMD may synthesize a frame from a previous frame using the motion vectors.