Semantic Segmentation With Gravity-Normal Regularization for Tilted Views
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing 3D semantic reconstruction methods in extended reality devices provide inaccurate semantic labels, particularly in cases of tilted camera views, leading to misunderstandings of the scene and potential collisions, and existing solutions to improve accuracy require longer processing times or lack sufficient 3D information.
Innovation Solution
Implementing a gravity-normal regularization technique that utilizes a gravity vector and normal predictions to enhance the robustness of semantic segmentation, incorporating 3D information to improve the accuracy of semantic labeling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 3D semantic reconstruction is used to improve scene understanding, then the accuracy of semantic labeling is improved, but the processing time increases
Solution Approach 1:
The patent combines multiple tasks (semantic segmentation and normal estimation) into a single multi-task model that processes images simultaneously. This merging of tasks allows the system to achieve accurate 3D semantic reconstruction without requiring separate processing steps, thereby reducing overall processing time while maintaining high accuracy in semantic labeling even under tilted camera views
2Measurement precision
If existing 3D semantic reconstruction methods are used, then semantic labeling is provided, but the accuracy deteriorates in tilted camera views
Solution Approach 1:
The patent incorporates gravity vector information as a feedback mechanism that continuously guides the normal estimation process. By comparing the estimated surface normals with the gravity vector direction, the system can detect and correct inaccuracies caused by camera tilt, ensuring accurate semantic labeling even when the camera is not in a horizontal position
3Reliability
If multi-task model with gravity-normal regularization is implemented, then the robustness of semantic segmentation is improved, but the device complexity increases
Solution Approach 1:
The patent designs a multi-task model that simultaneously performs semantic segmentation and normal estimation using a single unified architecture. This universal model handles multiple functions (semantic labeling and geometric understanding) without requiring separate specialized systems, thereby improving robustness while keeping the overall device complexity manageable through shared computational resources
Data Source
AI summary
Disclosed are systems and techniques for image processing. For example, a computing device can generate, using a multi-task model, a segmentation output and a normal output based on image(s) of a scene and a gravity vector for the scene. The computing device can learn semantic prediction(s) based on comparing the segmentation output to at least one ground truth semantic segmentation map. The computing device can also learn normal prediction(s) based on comparing the normal output to at least one ground truth normal map. The computing device can extract a semantics normal from the semantic prediction(s) and the normal prediction(s). The computing device can optimize a regularization loss based on the semantics normal and the gravity vector for the scene by learning gravity-normal regularization(s) for the scene. The computing device can determine final semantic labels for regions of the scene based on the gravity-normal regularization(s).