Mixed Reality Rendering via Virtual Coordinate Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mixed Reality (MR) systems experience display distortion and jitter due to the MTP delay caused by remote server computation and decoding processes, leading to a mismatch between virtual content and user head pose, resulting in an unsatisfactory user experience.
Innovation Solution
A mixed reality rendering system that includes a display device and a server, where the server converts physical coordinates into virtual coordinates, generates a virtual streaming camera, and renders objects based on these coordinates, transmitting them back to the display device for seamless display, reducing latency and jitter.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If remote server computation and decoding processes are used, then computational load on HMD is reduced, but MTP delay increases causing display distortion and jitter
Solution Approach 1:
The system performs preliminary coordinate conversion by detecting the physical coordinate of the display device, converting it to virtual coordinate space, and pre-calculating the virtual streaming camera position and orientation before rendering. This preliminary action in coordinate space reduces the computational burden during the rendering pipeline while minimizing delays in the overall MTP process.
2Device complexity
If remote server computation is used, then processing power requirements are reduced, but synchronization between video content and head pose deteriorates
Solution Approach 1:
The system continuously receives head pose information from the display device, converts physical coordinates to virtual coordinates, and uses this feedback to dynamically adjust the virtual streaming camera position and rendering parameters. This closed-loop feedback mechanism ensures synchronization between video content and head pose while distributing computational load to the remote server.
Solution Approach 2:
The patent introduces coordinate conversion as an intermediary process between the display device and the rendering pipeline. By converting physical coordinates to virtual coordinates and establishing a virtual streaming camera, the system creates an intermediate representation that bridges the gap between real-world head movements and virtual content rendering, improving synchronization while reducing processing requirements.
Data Source
AI summary
A mixed reality rendering system includes a display device and a server, wherein the display device is at a user end, and the server is at a remote end. A native object is built in the server, wherein the native object has a virtual coordinate. The remote server converts a physical coordinate of the display device into a virtual coordinate. The server generates a virtual straight line based on the virtual coordinate of the display device and the virtual coordinate of the native object. A virtual streaming camera is generated on the virtual straight line based on a predetermined distance. The server renders a rendered object based on the virtual streaming camera.