Semantic-Aware Occlusion in Augmented Reality via Server-Side 3D Meshes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current artificial-reality systems face challenges in providing persistent semantic augmented reality experiences without requiring client devices to store digital maps of the environment, and they struggle with accurate occlusion of artificial-reality objects behind physical objects.
Innovation Solution
A server maintains 3D representations of the real-world environment, including sparse 3D point features, dense triangular mesh layers, and semantic layers, allowing client devices to obtain re-localization and semantic information through message exchange, and enables accurate occlusion by determining intersection points and metadata associated with 3D meshes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If client devices store digital maps of the environment, then semantic augmented reality experiences can be provided, but device complexity and memory requirements increase
Solution Approach 1:
The patent extracts the digital map storage function from the client device and relocates it to a server. The server maintains the comprehensive 3D representations and semantic information, while client devices only store minimal anchor data and exchange messages with the server to obtain re-localization and semantic information. This extraction resolves the contradiction by maintaining persistent semantic augmented reality experiences through server-based storage while keeping client devices lightweight.
Solution Approach 2:
The server acts as an intermediary between the digital map data and client devices. Instead of client devices directly storing and processing complete digital maps, they communicate with the server which provides re-localization and semantic information on demand. This intermediary approach allows the system to maintain persistent semantic augmented reality experiences while avoiding the need for complex client-side storage.
2Productivity
If artificial-reality objects are rendered without accurate occlusion data, then rendering speed improves, but visual realism and measurement precision deteriorate
Solution Approach 1:
The server pre-computes 3D representations including sparse point features, dense triangular meshes, and identifies occlusion relationships before client devices need them. Anchor vertices are pre-established with transform data connecting key frames to mixed-reality elements. This preliminary action allows client devices to perform efficient rendering by simply applying pre-computed occlusion data rather than calculating it in real-time, thus maintaining both rendering efficiency and occlusion accuracy.
3Measurement precision
If detailed 3D representations are maintained on client devices, then occlusion accuracy improves, but memory usage and processing requirements increase
Solution Approach 1:
The patent segments the 3D representation data into two parts: comprehensive 3D models are stored on the server, while client devices only store lightweight anchor vertices and transform data. The server maintains detailed sparse point features and dense triangular meshes, and provides this data to clients only when needed for rendering specific views. This segmentation allows high-precision occlusion to be achieved through server-based storage while keeping client memory usage minimal.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In one embodiment, a method includes accessing a digital map of a real-world region, where the digital map includes one or more three-dimensional meshes corresponding to one or more three-dimensional objects within the real-world region, receiving an object query including an identifier for an anchor in the digital map, positional information relative to the anchor, and information associated with a directional vector, determining a position within the digital map based on the identifier for the anchor and the positional information relative to the anchor, determining a three-dimensional mesh in the digital map that intersects with a projection of the directional vector from the determined position within the digital, map, identifying metadata associated with the three-dimensional mesh, and sending the metadata to the second computing device.