Mixed-Reality Occlusion Using Server-Side 3D Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing mixed-reality (MR) technologies struggle with inaccurate occlusion between real-world and virtual objects due to reliance on depth sensors, leading to unnatural MR images and high computational intensity.
Innovation Solution
A system and method utilizing 3D reconstruction at a server side to determine and send only the non-occluded portion of virtual objects to the client device, reducing processing burden and ensuring accurate MR occlusion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If depth sensors (LIDAR, ToF, stereoscopic) are used to detect optical depths for occlusion, then occlusion can be performed, but the accuracy is limited by sensor distance and resolution, and computational load is high
Solution Approach 1:
The patent introduces an intermediary server that performs 3D reconstruction of the real-world environment and calculates occlusion relationships between virtual and real objects. This mediator handles the computationally intensive tasks, allowing client devices to achieve accurate occlusion without requiring powerful local processing or high-resolution depth sensors.
Solution Approach 2:
The system creates a 3D copy or digital twin of the real-world environment through 3D reconstruction. This virtual representation is then used to determine occlusion relationships, eliminating the need for complex real-time depth sensing and processing on client devices while maintaining high accuracy.
2Reliability
If depth sensors are used for occlusion detection, then occlusion can be achieved, but the realism and immersiveness of MR environment is compromised due to improper occlusion
Solution Approach 1:
By creating a accurate 3D reconstruction of the real environment and using it to determine occlusion relationships, the system achieves high measurement precision for occlusion. This virtual copy allows for realistic occlusion effects that enhance the immersiveness of the MR environment.
Solution Approach 2:
The system performs 3D reconstruction and occlusion calculation in advance on the server before rendering the final MR image. This preliminary action ensures that occlusion relationships are accurately determined before composition, improving the realism of the final output.
3Productivity
If MR device performs occlusion entirely locally, then occlusion can be applied, but processing is computationally-intensive and time-consuming
Solution Approach 1:
The server acts as an intermediary that handles the computationally intensive 3D reconstruction and occlusion calculation tasks. The client device only needs to send rendering requests and receive processed results, dramatically reducing local computational load while maintaining high processing speed for occlusion operations.
Solution Approach 2:
The system segments the occlusion processing workflow into server-side tasks (3D reconstruction, occlusion calculation) and client-side tasks (rendering composition). This division allows complex processing to occur on powerful servers while keeping client devices lightweight and responsive.
Data Source
AI summary
Disclosed is a system with server(s) communicably coupled to client device(s). The server(s) is configured to obtain a 3D model of a real-world environment; receive, from client device(s), viewpoint information indicative of a viewpoint from a perspective of which a mixed-reality (MR) image is to be generated; for virtual object(s) to be embedded in MR image, determine portion(s) of the virtual object(s) being occluded by real object(s) present in the real-world environment, based on optical depths determined from 3D model corresponding to a viewpoint, a position at which the virtual object(s) is to be embedded with respect to viewpoint, and at least one of: size of the virtual object(s), shape of the virtual object(s), orientation of the virtual object(s) with respect to viewpoint; and send a remaining portion of the virtual object(s) that is not being occluded to client device(s).