View-Optimized RGBD Mesh Rendering for Resource-Limited Headsets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Resource-constrained devices, such as VR, AR, and MR headsets, struggle to render high-quality objects due to limited local power and resources, and server rendering at slower frame rates than desired.
Innovation Solution
A cloud-computing system renders scenes using a powerful GPU, generates simplified 3D meshes with depth maps, and streams compressed data to the client device, which reprojects images using efficient vertex shaders.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If local rendering is used on resource-constrained devices, then device independence is maintained, but rendering quality and performance are limited
Solution Approach 1:
A cloud rendering server acts as an intermediary between the 3D content source and the resource-constrained head-mounted device. The server performs computationally intensive rendering operations using powerful GPUs, while the device only receives compressed video streams and simplified depth meshes, eliminating the need for high-end local hardware while maintaining high rendering quality
Solution Approach 2:
Instead of transmitting complete 3D scene data and models to the device for local rendering, the system generates and transmits rendered image frames and simplified depth information as copies of the visual output. This allows the device to display high-quality rendered content without needing the computational resources to generate it locally
2Manufacturing precision
If server rendering is used, then rendering quality improves, but frame rate decreases due to transmission latency
Solution Approach 1:
The rendering pipeline is segmented into distinct stages: 3D model processing and rendering on the server, video compression, depth mesh generation, and data transmission to the device. This segmentation allows each component to be optimized independently, with the server focusing on high-quality rendering and the device focusing on low-latency display, thereby maintaining both quality and frame rate
Solution Approach 2:
The system performs preliminary rendering and compression actions on the server before data transmission. By pre-rendering scenes and pre-compressing video streams, the system reduces the computational burden on the device during actual display, allowing the device to maintain high frame rates by simply decoding and displaying pre-prepared data rather than performing real-time rendering
3Measurement precision
If complete 3D mesh data is transmitted, then rendering accuracy is maintained, but data bandwidth increases
Solution Approach 1:
The system extracts only the essential depth information from complete 3D mesh data by generating simplified depth meshes that capture depth contours and spatial relationships without transmitting full geometric detail. This extraction maintains sufficient depth accuracy for proper rendering while dramatically reducing the quantity of data that needs to be transmitted over the network
Data Source
AI summary
In one embodiment, a method includes receiving a first viewpoint associated with a head-mounted device from the head-mounted device, accessing a 3D mesh of a virtual scene, selecting a portion of the 3D mesh based on the first viewpoint, generating an image and a corresponding depth map of the virtual scene based on the selected portion of the 3D mesh, generating a simplified 3D mesh based on the depth map, wherein the simplified 3D mesh has fewer primitives than the selected portion of 3D mesh of the virtual scene, generating a texture for the simplified 3D mesh based on the image, and sending the simplified 3D mesh and the texture to the head-mount device, wherein the simplified 3D mesh and the texture are configured to be used for rendering the virtual scene from one or more viewpoints different from the first viewpoint.