Point-Cloud 3D Semantic Scene Graphs With Transformer-GNN Reasoning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current 3D scene graph generation technologies have limited accuracy in predicting object relationships, with existing models achieving only about 50% prediction accuracy, and there is a need for improved methods to enhance scene understanding in 3D environments.
Innovation Solution
A 3DSSG generation method utilizing a Transformer-based feature extractor, a multi-layer perceptron (MLP), and a graph neural network (NN)-based relationship reasoning module to determine a 3D semantic scene graph, incorporating object auxiliary information such as bounding box size, object length, and spatial distribution to improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If current 3D scene graph generation models are used, then the system can generate 3DSSG from point cloud data, but the prediction accuracy for object relationships is limited to about 50%
Solution Approach 1:
The model is segmented into distinct functional modules: Transformer-based feature extractor for global context, graph neural network for relationship reasoning, and MLP for prediction. This modular segmentation allows each component to specialize in specific tasks, improving overall prediction accuracy while maintaining manageable complexity through clear division of labor.
Solution Approach 2:
The patent transitions from traditional 2D image-based scene graphs to 3D point cloud-based scene graphs, adding a spatial dimension. This dimensional change enables more accurate representation of object spatial relationships and semantic information in three-dimensional space, directly addressing the limitation of 50% prediction accuracy.
2Measurement precision
If more object auxiliary information is incorporated, then the 3DSSG prediction accuracy improves, but the data processing complexity increases
Solution Approach 1:
The graph neural network module serves multiple functions simultaneously: it processes various types of object auxiliary information (bounding box size, object length, volume, spatial distribution SD), performs relationship reasoning between objects, and generates predictions. This multi-functionality reduces the need for separate processing pipelines for each type of information, managing complexity while comprehensively utilizing auxiliary data.
Solution Approach 2:
The model incorporates multiple object auxiliary information parameters (bounding box size, object length, volume, spatial distribution standard deviation) to enhance prediction accuracy. By changing and expanding the set of input parameters beyond basic point cloud data, the system achieves more accurate 3DSSG generation despite the increased data processing requirements.
Data Source
AI summary
The present disclosure provides a three-dimensional semantic scene graph (3DSSG) generation method and system, and an electronic device, and relates to the field of three-dimensional (3D) scene graph generation. The method includes: obtaining a point cloud set and an object segmentation result of a target scene; determining a point cloud subset of each object according to the point cloud set and the object segmentation result; and determining a 3DSSG of the target scene according to a point cloud subset of any object and object auxiliary information by using a 3DSSG prediction model, where the 3DSSG prediction model is obtained by training a 3DSSG initial prediction model by using a training dataset; and the 3DSSG prediction model includes a Transformer-based feature extractor, a first multi-layer perceptron (MLP), a graph neural network (NN)-based relationship reasoning module, and a scene graph generation module. The present disclosure improves accuracy of generating a 3DSSG.