AR Surface Reprojection for Misaligned Capture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current augmented reality (AR) devices face challenges in sharing AR experiences with multiple users due to limited power budgets and misalignment of virtual content with real-world images, leading to inefficient content re-rendering and incorrect overlay of virtual objects.
Innovation Solution
The system employs a late-latching reprojection engine to reproject AR surfaces based on a stabilized camera pose, disabling display corrections during capture and encoding AR content into a 2D rectilinear frame buffer for efficient sharing, and records metadata for 3D playback, allowing for accurate alignment and interaction with AR surfaces.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If AR surfaces are re-rendered for capture, then virtual content can be displayed, but system power consumption increases and processing efficiency decreases
Solution Approach 1:
The patent captures AR surfaces by copying their visual representation from the display buffer rather than re-rendering them. The reprojection engine captures the rendered surface data directly from memory, avoiding the computationally expensive re-rendering process while preserving visual fidelity. This copying approach significantly reduces power consumption and processing requirements.
Solution Approach 2:
The system performs preliminary rendering of AR surfaces for display, and then captures these pre-rendered surfaces for sharing. By capturing surfaces that have already been rendered for the user's view, the system eliminates the need for separate re-rendering operations, thereby reducing overall processing load and power consumption.
2Manufacturing precision
If display corrections are applied during AR capture, then virtual content alignment is improved, but processing complexity and time increase
Solution Approach 1:
The patent introduces a reprojection engine as an intermediary component that handles the complex alignment and projection transformations. This engine receives surface data, applies necessary projections based on camera pose and display geometry, and outputs correctly aligned capture data. By centralizing this complex processing in a dedicated intermediary component, the system achieves accurate alignment while managing processing complexity effectively.
3Adaptability or versatility
If AR content is captured in 3D format, then immersive experience is enhanced, but data processing and storage requirements increase
Solution Approach 1:
The patent segments AR content into distinct surface elements with associated metadata including 3D position, orientation, and size information. By representing virtual content as discrete surfaces with structured data attributes rather than continuous 3D volumetric data, the system maintains immersive 3D capabilities while significantly reducing data processing and storage requirements.
Solution Approach 2:
The system represents 3D AR surfaces using 2D image plane coordinates combined with depth and orientation metadata. This dimensional transformation allows the capture and sharing of 3D virtual content by encoding it in a more compact format that preserves spatial relationships while reducing overall data volume compared to full 3D volumetric representation.
Data Source
AI summary
In one embodiment, a method includes accessing, from a client system associated with a first user, multiple surfaces in an artificial reality scene, wherein the multiple surfaces are projected for display to the first user based on an eye pose of the first user; capturing a frame of the real-world environment while displaying the artificial reality scene to the first user, wherein the captured frame of the real-world environment is associated with a camera pose; reprojecting the multiple surfaces based on the camera pose; generating an aligned artificial reality scene based on the reprojected multiple surfaces and the captured frame of the real-world environment; and sending the aligned artificial reality scene to a second user for display.