World-Space Floor Segmentation With Temporal Filtering for Mixed Reality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional semantic segmentation neural networks are limited to screen-space and face challenges with temporal instability, computational power requirements, and inflexibility, making them difficult to use in real-time mixed reality applications, especially on low-end devices.
Innovation Solution
A method for generating a persistent world-space ground (or floor) segmentation map using a surface tracker and a temporal filter to project screen-space segmentation into world-space, reducing the need for continuous frame-by-frame processing and optimizing computational demands.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional semantic segmentation neural networks are used for screen-space segmentation, then segmentation can be achieved, but temporal instability and computational power requirements increase
Solution Approach 1:
The patent transforms the segmentation problem from 2D screen-space to 3D world-space by projecting segmentation results through depth information. This dimensional transformation allows the system to leverage spatial relationships and depth cues to reduce computational complexity while maintaining or improving segmentation accuracy, particularly for ground/floor detection in mixed reality environments.
Solution Approach 2:
The system performs preliminary depth estimation and world-space map construction before final segmentation. By pre-processing the depth information and establishing a world-space reference frame in advance, the computational burden is reduced during the actual segmentation process, enabling real-time performance on devices with limited computational resources.
2Reliability
If frame-by-frame segmentation is performed continuously, then up-to-date segmentation is maintained, but computational burden increases
Solution Approach 1:
Instead of continuous frame-by-frame segmentation, the system employs periodic updates combined with temporal filtering. The world-space segmentation map is updated at strategic intervals and refined using temporal filters that leverage information from previous frames. This periodic approach maintains temporal stability while significantly reducing the computational load compared to continuous processing.
Solution Approach 2:
The patent introduces a world-space map and temporal filter as intermediary structures between raw frame segmentation and final output. These intermediaries accumulate and smooth segmentation information over time, providing temporal stability without requiring every frame to be fully processed, thus improving computational efficiency while maintaining reliability.
3Device complexity
If screen-space segmentation is used, then processing is simpler, but flexibility and adaptability to different viewing conditions are reduced
Solution Approach 1:
The system extends segmentation from 2D screen-space to 3D world-space by incorporating depth information and camera pose data. This transformation enables the segmentation to be view-independent and adaptable to different viewing angles and distances, while the underlying processing remains relatively simple by building upon the screen-space segmentation results through geometric projection.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and devices for generating a persistent world-space ground (or floor) segmentation map (or "texture") for use in augmented or virtual reality 3D experiences.