Semantic Segmentation for Plane Detection in Mixed Reality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In mixed reality environments, placing computer-generated reality (CGR) objects in unmapped or dynamic scenes without suitable virtual substrates can result in poor user experiences, as these objects may float, occlude, or collide with real-world objects, due to lack of anchoring to real-world surfaces.
Innovation Solution
The method involves using semantic segmentation to classify pixels in an image of the scene, generating a point cloud, and then determining plane hypotheses to provide virtual substrates for CGR object placement, thereby anchoring objects to real-world surfaces.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If plane detection is performed without semantic segmentation, then the process is simpler, but the accuracy of surface mapping and object anchoring deteriorates
Solution Approach 1:
The patent applies segmentation by dividing the image processing task into two distinct stages: first, semantic segmentation classifies pixels into different object categories (e.g., floor, wall, table); second, plane detection operates on these segmented regions. This segmentation allows the system to focus computational resources on relevant regions, improving surface mapping accuracy while managing complexity through modular processing steps.
2Productivity
If computational complexity is reduced for real-time processing, then processing speed improves, but the accuracy of plane detection and virtual substrate generation deteriorates
Solution Approach 1:
The patent implements preliminary action by performing semantic segmentation before plane detection. This pre-processing step categorizes pixels into semantic regions (floor, wall, table, etc.), which provides structured information that guides subsequent plane detection. By preparing the data in advance with semantic labels, the system achieves both real-time processing speeds and high detection accuracy, as the plane detection algorithm can focus on predefined regions of interest rather than processing the entire image indiscriminately.
Data Source
AI summary
In one implementation, a method of generating a plane hypothesis is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels. The method includes obtaining a plurality of points of a point cloud based on the image of the scene. The method includes obtaining an object classification set based on the image of the scene. Each element of the object classification set includes a plurality of pixels respectively associated with a corresponding object in the scene. The method includes detecting a plane within the scene by identifying a subset of the plurality of points of the point cloud that correspond to a particular element of the object classification set.