MR Video Compression Using Precomputed Motion Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing virtual reality and mixed reality systems face challenges in providing high-quality, immersive experiences due to limitations in bandwidth, latency, and frame rendering times, particularly when using conventional encoding methods that rely on previous frames for motion vector estimation.
Innovation Solution
The system employs motion vectors determined from sensors on the HMD and rendering application to encode frames, reducing the time required for encoding by using pre-determined head and virtual content motion vectors, and embeds this motion information in the data stream for frame synthesis on the HMD.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional encoding methods using previous frames for motion vector estimation are used, then encoding accuracy is maintained, but encoding time increases and latency worsens
Solution Approach 1:
The system performs motion vector estimation in advance by using sensor data from the HMD to predict head motion and pre-calculating motion vectors before encoding. This preliminary action allows the encoder to skip time-consuming motion estimation during the actual encoding process, thereby reducing encoding time while maintaining accuracy through pre-computed motion compensation parameters.
2Manufacturing precision
If high-resolution frames are transmitted to maintain image quality, then display quality improves, but bandwidth requirements increase
Solution Approach 1:
The system changes the parameter representation by transmitting motion vectors and sensor data instead of complete high-resolution frames. By encoding only the essential motion parameters and allowing the HMD to perform motion compensation locally, the system maintains high display quality while dramatically reducing the bandwidth required for transmission.
3Speed
If real-time rendering is performed to maintain immersive experience, then frame rate is improved, but processing complexity increases
Solution Approach 1:
The system segments the processing tasks by separating motion vector estimation from frame encoding. The HMD's sensors handle motion detection and the base station pre-calculates motion vectors, while the encoder only needs to apply motion compensation. This segmentation reduces processing complexity at each stage while maintaining high frame rates for real-time rendering.
Data Source
AI summary
A mixed reality system including a head-mounted display (HMD) and a base station. Information collected by HMD sensors may be transmitted to the base via a wired or wireless connection. On the base, a rendering engine renders frames including virtual content based in part on the sensor information, and an encoder compresses the frames according to an encoding protocol before sending the frames to the HMD over the connection. Instead of using a previous frame to estimate motion vectors in the encoder, motion vectors from the HMD and the rendering engine are input to the encoder and used in compressing the frame. The motion vectors may be embedded in the data stream along with the encoded frame data and transmitted to the HMD over the connection. If a frame is not received at the HMD, the HMD may synthesize a frame from a previous frame using the motion vectors.