World-Space Segmentation for Mixed Reality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional semantic segmentation neural networks are limited to screen-space, leading to temporal instability and high computational demands, making them unsuitable for real-time mixed reality applications, especially on low-end devices.
Innovation Solution
A method for generating a persistent world-space ground (or floor) segmentation map using a surface tracker and a temporal filter to project screen-space segmentation into world-space, reducing computational burden and enhancing stability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional semantic segmentation neural networks are used in screen-space, then segmentation can be achieved, but temporal instability occurs and computational demands become excessively high
Solution Approach 1:
The patent transforms the segmentation problem from screen-space (2D image coordinates) to world-space (3D physical coordinates). By projecting segmentation results from multiple frames into a persistent world-space representation, the system achieves temporal stability while maintaining segmentation accuracy. This dimensional transformation allows the segmentation to be invariant to camera motion and frame-to-frame variations.
Solution Approach 2:
The system pre-processes multiple image frames to establish a persistent world-space map before generating final segmentation results. By accumulating and integrating segmentation information across multiple frames in world-space, the system builds a stable reference framework that reduces temporal instability in the final output.
2Measurement precision
If conventional semantic segmentation neural networks are applied to each frame, then segmentation is achieved, but computational power requirements increase significantly
Solution Approach 1:
The patent divides the computational workload into two parts: (1) running the neural network on a subset of frames or at lower resolution to obtain initial segmentation, and (2) projecting and integrating these results into a persistent world-space map. This segmentation of the processing task reduces the computational burden per frame while maintaining overall segmentation quality.
Solution Approach 2:
Instead of running the neural network on every frame, the system uses a copy-projection approach where segmentation results from selected frames are copied and transformed into world-space coordinates. This persistent world-space representation is then reused across multiple frames, eliminating the need for repeated full neural network executions.
3Productivity
If screen-space segmentation is used, then segmentation can be performed, but the results are limited to the camera angle and require re-application when camera moves
Solution Approach 1:
The patent transforms segmentation from screen-space (camera-dependent 2D) to world-space (camera-independent 3D). By expressing segmentation results in world coordinates that represent the actual physical environment, the system achieves adaptability to camera movements. The persistent world-space map remains valid across different camera angles and positions, eliminating the need to re-run segmentation when the camera moves.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and devices for generating a persistent world-space ground (or floor) segmentation map (or "texture") for use in augmented or virtual reality 3D experiences.