Spatial semantic occupancy prediction method and apparatus, computer device, readable storage medium and program product
By converting point cloud data into multi-view feature planes and performing feature fusion, and utilizing visual generative and 3D feature classification models, the problem of excessive computational burden in existing technologies is solved, and efficient spatial semantic occupancy prediction is achieved.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- CHINA TELECOM CLOUD TECH CO LTD
- Filing Date
- 2025-11-17
- Publication Date
- 2026-06-04
AI Technical Summary
Existing spatial semantic occupancy prediction methods have an excessive computational burden when processing 3D point cloud data, resulting in low efficiency.
By converting raw point cloud data into multi-view feature planes, feature extraction and fusion are performed. Using a visual generative model and a 3D feature classification model, spatial semantic occupancy prediction results are output, reducing information loss and improving modeling efficiency.
It effectively reduces the computational and storage burden of the model, improves the efficiency of spatial semantic occupancy prediction, and is able to better model complex 3D scenes.
Smart Images
Figure CN2025135384_04062026_PF_FP_ABST
Abstract
Description
Spatial semantic occupancy prediction methods, apparatus, computer equipment, readable storage media, and program products
[0001] Related applications
[0002] This application claims priority to Chinese patent application No. 2024117046720, filed on November 26, 2024, entitled “Spatial Semantic Occupation Prediction Method, Apparatus, Computer Equipment, Readable Storage Medium and Program Product”, the entire contents of which are incorporated herein by reference. Technical Field
[0003] This application relates to the field of artificial intelligence technology, and in particular to a spatial semantic occupancy prediction method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology
[0004] The spatial semantic occupancy prediction task aims to build a more accurate model of a 3D spatial scene. This construction process relies on assigning a clear semantic label to each voxel in the 3D spatial scene.
[0005] However, point cloud voxel representations are highly dense in 3D spatial scenes, which means that spatial semantic occupancy prediction methods in related technologies often require a series of operations on a large amount of 3D point cloud data, resulting in a huge computational burden and hindering the improvement of the efficiency of spatial semantic occupancy prediction.
[0006] Therefore, spatial semantic occupancy prediction in related technologies suffers from low efficiency. Summary of the Invention
[0007] Therefore, it is necessary to provide a spatial semantic occupancy prediction method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the efficiency of spatial semantic occupancy prediction in response to the above-mentioned technical problems.
[0008] Firstly, this application provides a spatial semantic occupancy prediction method, the method comprising:
[0009] Acquire raw point cloud data from the target spatial scene;
[0010] The original point cloud data is converted into a multi-view feature plane, and features are extracted from the multi-view feature plane to obtain multi-view point cloud features.
[0011] Multi-view point cloud features are input into a pre-trained visual generative model to obtain a two-dimensional semantic image. Feature extraction is performed on the two-dimensional semantic image to obtain multi-view image features.
[0012] Multi-view fused features are obtained by fusing multi-view point cloud features and multi-view image features;
[0013] For any query point located in the target space scene, output the spatial semantic occupancy prediction result based on the multi-view fusion features corresponding to the query point.
[0014] In one embodiment, based on the multi-view fusion features corresponding to the query point, a spatial semantic occupancy prediction result is output, including:
[0015] Obtain the fusion features corresponding to each perspective in the multi-view fusion features corresponding to the query point;
[0016] By fusing the corresponding fusion features from various perspectives, the three-dimensional features corresponding to the query point are obtained.
[0017] The three-dimensional features of the query point are input into a pre-trained three-dimensional feature classification model to obtain the spatial semantic occupancy prediction result.
[0018] In one embodiment, converting raw point cloud data into multi-view feature planes includes:
[0019] The original point cloud data is divided into voxels in cylindrical coordinates to obtain voxelized point cloud data.
[0020] Pooling operations are performed on the voxelized point cloud data along the three axes of the cylindrical coordinate system to obtain planar features from three perspectives in the cylindrical coordinate system.
[0021] The planar features obtained from three perspectives are used to create a multi-view feature plane.
[0022] In one embodiment, feature extraction is performed on the multi-view feature plane to obtain multi-view point cloud features, including:
[0023] The encoder in the point cloud feature extraction model encodes the multi-view feature plane to obtain the feature plane encoding result.
[0024] The decoder in the point cloud feature extraction model decodes the feature plane encoding results to obtain multi-view point cloud features at the target resolution.
[0025] In one embodiment, feature extraction is performed on the two-dimensional semantic image to obtain multi-view image features, including:
[0026] The two-dimensional semantic image is encoded by the encoder in the image feature extraction model to obtain the semantic image encoding result;
[0027] The semantic image encoding result is decoded by the decoder in the image feature extraction model to obtain multi-view image features at the target resolution.
[0028] In one embodiment, acquiring raw point cloud data in the target space scene includes:
[0029] According to the preset sampling configuration information, the point cloud data in the target space scene is sampled to obtain the sampled data;
[0030] The sampled data is the raw point cloud data.
[0031] Secondly, this application also provides a spatial semantic occupancy prediction device, the device comprising:
[0032] The acquisition module is used to acquire raw point cloud data from the target spatial scene;
[0033] The conversion module is used to convert the raw point cloud data into a multi-view feature plane, extract features from the multi-view feature plane, and obtain multi-view point cloud features.
[0034] The generation module is used to input multi-view point cloud features into a pre-trained visual generative model to obtain a two-dimensional semantic image, and to extract features from the two-dimensional semantic image to obtain multi-view image features.
[0035] The fusion module is used to fuse multi-view point cloud features and multi-view image features to obtain multi-view fused features.
[0036] The output module is used to output the spatial semantic occupancy prediction result for any query point located in the target space scene, based on the multi-view fusion features corresponding to the query point.
[0037] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.
[0038] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.
[0039] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.
[0040] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description
[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the disclosed drawings without creative effort.
[0042] Figure 1 is an application environment diagram of a spatial semantic occupancy prediction method according to one or more embodiments;
[0043] Figure 2 is a flowchart illustrating a spatial semantic occupancy prediction method according to one or more embodiments;
[0044] Figure 3 is a network structure diagram of a spatial semantic occupancy prediction method according to one or more embodiments;
[0045] Figure 4 is a schematic diagram of a three-view feature encoding according to one or more embodiments;
[0046] Figure 5 illustrates a spatial semantic occupancy prediction system according to one or more embodiments;
[0047] Figure 6 is a flowchart illustrating a spatial semantic occupancy prediction method according to one or more other embodiments;
[0048] Figure 7 is a structural block diagram of a spatial semantic occupancy prediction device according to one or more embodiments;
[0049] Figure 8 is an internal structural diagram of a computer device according to one or more embodiments. Detailed Implementation
[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0051] The spatial semantic occupancy prediction method provided in this application embodiment can be applied to the application environment shown in Figure 1. The point cloud data acquisition device 102 communicates with the computer device 104. In practical applications, the point cloud data acquisition device 102 can capture the three-dimensional geometric information of various objects or environmental surfaces in the target spatial scene, obtaining point cloud data in the target spatial scene. The point cloud data acquisition device 102 can transmit the point cloud data in the target spatial scene to the computer device 104. The computer device 104 acquires the original point cloud data in the target spatial scene; the computer device 104 converts the original point cloud data into a multi-view feature plane, extracts features from the multi-view feature plane, and obtains multi-view point cloud features; the computer device 104 inputs the multi-view point cloud features into a pre-trained visual generative model to obtain a two-dimensional semantic image, extracts features from the two-dimensional semantic image, and obtains multi-view image features; the computer device 104 fuses the multi-view point cloud features and multi-view image features to obtain multi-view fused features; for any query point located in the target spatial scene, the computer device 104 outputs the spatial semantic occupancy prediction result based on the multi-view fused features corresponding to the query point. The point cloud acquisition device 102 may include, but is not limited to, various laser scanners, photogrammetry equipment, 3D scanners, drones equipped with lidar or high-definition cameras, mobile devices equipped with depth sensors, and structured light scanners. The computer equipment 104 may include, but is not limited to, various personal computers, laptops, smartphones, tablets, and servers. The server may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.
[0052] In one exemplary embodiment, as shown in FIG2, a spatial semantic occupancy prediction method is provided. Taking the application of the method to the computer device 104 in FIG1 as an example, the method includes the following steps S202 to S208.
[0053] in:
[0054] Step S202: Obtain the original point cloud data in the target space scene.
[0055] The target spatial scene can refer to the three-dimensional spatial scene for which spatial semantic occupancy prediction is to be performed.
[0056] For example, after acquiring point cloud data transmitted by the point cloud data acquisition device, the computer device can sample the point cloud data in the target spatial scene to obtain the aforementioned raw point cloud data.
[0057] Step S204: Convert the original point cloud data into a multi-view feature plane, extract features from the multi-view feature plane, and obtain multi-view point cloud features.
[0058] For example, after acquiring raw point cloud data from a target spatial scene, the computer device can convert the raw point cloud data into multi-view feature planes. Specifically, the computer device can convert the raw point cloud data into two-dimensional features in three different perspectives within a cylindrical coordinate system, resulting in three feature planes at different perspectives. The computer device then uses these three feature planes as the three-view feature plane, i.e., the aforementioned multi-view feature plane.
[0059] After acquiring multi-view feature planes, the computer device can extract features from these planes to obtain multi-view point cloud features. In practical applications, the computer device can use a two-dimensional neural network Conv... cloud This is used to encode the three-view feature plane and decode the high-resolution three-view point cloud features; where the three-view point cloud features can be represented as... Where hw, wd, and dh represent the three coordinate axes of the cylindrical coordinate system.
[0060] Step S206: Input the multi-view point cloud features into the pre-trained visual generative model to obtain a two-dimensional semantic image, and extract features from the two-dimensional semantic image to obtain multi-view image features.
[0061] Among them, the visual generative model can be a large model used to generate new visual content.
[0062] For example, after a computer device acquires multi-view point cloud features, it can input the multi-view point cloud features into a pre-trained visual generative model, and then process the multi-view point cloud features through the pre-trained visual generative model to output a two-dimensional semantic image.
[0063] Then, the computer device can extract features from the two-dimensional semantic image to obtain multi-view image features. In practical applications, the computer device can use a two-dimensional neural network Conv... image This is used to encode two-dimensional semantic images and decode high-resolution multi-view image features, such as three-view image features. The three-view image features can be represented as... Where hw, wd, and dh represent the three coordinate axes of the cylindrical coordinate system.
[0064] Step S208: Fuse multi-view point cloud features and multi-view image features to obtain multi-view fused features.
[0065] For example, a computer device can fuse multi-view point cloud features and multi-view image features to obtain multi-view fused features; wherein, the computer device performs a stitching operation on the multi-view point cloud features and multi-view image features to obtain multi-view fused features, such as three-view fused features. Continuing the previous example, the three-view fused feature can be represented as... The fusion features corresponding to the three-view fusion features under the three different views can be represented as follows:
[0066] Concat represents the concatenation operation.
[0067] Step S210: For any query point located in the target space scene, output the spatial semantic occupancy prediction result based on the multi-view fusion features corresponding to the query point.
[0068] Understandably, the computer device can model the target spatial scene using two-dimensional features through the aforementioned steps S202, S204 to S208. Then, the computer device can perform spatial semantic occupancy prediction. When performing spatial semantic occupancy prediction, for any query point located in the target spatial scene, the computer device can obtain the multi-view fusion features corresponding to that query point and output the spatial semantic occupancy prediction result.
[0069] Specifically, the computer device can determine the 3D features of the query point based on the multi-view fusion features corresponding to the query point. Then, based on the 3D features of the query point, the computer device can output the spatial semantic occupancy prediction result of the query point. For example, the semantic label associated with the query point in the target space acquisition. It should be noted that the following text will further explain how the computer device outputs the spatial semantic occupancy prediction result based on the multi-view fusion features corresponding to the query point.
[0070] In the aforementioned spatial semantic occupancy prediction method, the original point cloud data in the target spatial scene is acquired and converted into a multi-view feature plane. Feature extraction is performed on the multi-view feature plane to obtain multi-view point cloud features. The multi-view point cloud features are then input into a pre-trained visual generative model to obtain a two-dimensional semantic image. Feature extraction is performed on the two-dimensional semantic image to obtain multi-view image features. The multi-view point cloud features and multi-view image features are then fused to obtain multi-view fused features. For any query point in the target spatial scene, the spatial semantic occupancy prediction result is output based on the multi-view fused features corresponding to the query point. In this way, the two-dimensional image features obtained by encoding and decoding the two-dimensional point cloud features from multiple perspectives and the two-dimensional semantic image output by the visual generative model can be used to reduce information loss during the projection process. This can further improve the model's ability to model complex three-dimensional scenes while maintaining high efficiency. At the same time, by fusing multi-view point cloud features and multi-view image features in parallel, the model can effectively use two-dimensional features to model complex three-dimensional scenes, reducing the computational and storage burden of the model and achieving comprehensive two-dimensional scene modeling and efficient spatial semantic occupancy prediction.
[0071] In one exemplary embodiment, the spatial semantic occupancy prediction result is output based on the multi-view fusion features corresponding to the query point, including: obtaining the fusion features corresponding to each view in the multi-view fusion features corresponding to the query point; fusing the fusion features corresponding to each view to obtain the three-dimensional features corresponding to the query point; and inputting the three-dimensional features of the query point into a pre-trained three-dimensional feature classification model to obtain the spatial semantic occupancy prediction result.
[0072] For example, in the process of outputting spatial semantic occupancy prediction results based on the multi-view fusion features corresponding to a query point, the computer device can obtain the fusion features corresponding to each view in the multi-view fusion features corresponding to the query point. The multi-view fusion features are used as three-view fusion features. For example, computer devices can obtain the fused features of the query point from three perspectives. and Then, the computer device can aggregate the corresponding fusion features from the three perspectives using an additive approach. and Obtain the three-dimensional feature F corresponding to the query point sem , is represented as:
[0073] Then, the computer device can input the three-dimensional features corresponding to the query point into a pre-trained three-dimensional feature classification model, such as a lightweight classification network; the three-dimensional features corresponding to the query point are processed by the pre-trained three-dimensional feature classification model to obtain the classification result of the query point; the classification result of the query point can represent the semantic label associated with the query point in the target space acquisition, thereby realizing spatial semantic occupancy prediction.
[0074] To facilitate understanding by those skilled in the art, Figure 3 exemplarily provides a network structure diagram of a spatial semantic occupancy prediction method. The method involves inputting point cloud data from a target spatial scene into a computer device, which samples the point cloud data to obtain raw point cloud data. The computer device then performs three-view transformation on the raw point cloud data to obtain three-view feature planes. Next, the computer device performs three-view feature encoding, extracting features from the three-view feature planes to obtain three-view point cloud features. These three-view point cloud features are then input into a pre-trained visual generative model to obtain a two-dimensional semantic image. Features are extracted from the two-dimensional semantic image to obtain three-view image features. The three-view point cloud features and the three-view image features are then fused to obtain three-view fused features. The computer device then performs query point feature encoding, acquiring the fused features corresponding to each viewpoint within the three-view fused features for the query point. These fused features are then fused to obtain the three-dimensional features corresponding to the query point. Finally, the computer device inputs the three-dimensional features of the query point into a pre-trained three-dimensional feature classification model and outputs the prediction result.
[0075] The technical solution of this embodiment obtains the three-dimensional features corresponding to the query point by fusing the fusion features corresponding to each view in the multi-view fusion features corresponding to the query point, and inputs the three-dimensional features of the query point into a pre-trained three-dimensional feature classification model to obtain the spatial semantic occupancy prediction result. This enables the aggregation of the features of the query point on the multi-view plane by using the addition method, effectively solving the problem that two-dimensional feature methods are difficult to depict the fine structure in complex three-dimensional scenes, and achieving efficient spatial semantic occupancy prediction.
[0076] In one exemplary embodiment, converting the original point cloud data into a multi-view feature plane includes: dividing the original point cloud data into voxels in a cylindrical coordinate system to obtain voxelized point cloud data; performing pooling operations on the voxelized point cloud data in three coordinate axes of the cylindrical coordinate system to obtain planar features in three views in the cylindrical coordinate system; and obtaining the planar features in the three views as a multi-view feature plane.
[0077] For example, in the process of converting raw point cloud data into multi-view feature planes, the computer device can perform voxel partitioning on the raw point cloud data in cylindrical coordinates to obtain voxelized point cloud data; then, the computer device can perform pooling operations on the voxelized point cloud data in three coordinate axes of cylindrical coordinates to obtain planar features in three views in cylindrical coordinates, which serve as multi-view feature planes.
[0078] The technical solution of this embodiment obtains voxelized point cloud data by dividing the original point cloud data into voxels in a cylindrical coordinate system; and obtains planar features from three perspectives in the cylindrical coordinate system by pooling the voxelized point cloud data under the three coordinate axes of the cylindrical coordinate system, which can achieve efficient three-view transformation of the original point cloud data.
[0079] In one exemplary embodiment, feature extraction is performed on the multi-view feature plane to obtain multi-view point cloud features, including: encoding the multi-view feature plane through the encoder in the point cloud feature extraction model to obtain the feature plane encoding result; and decoding the feature plane encoding result through the decoder in the point cloud feature extraction model to obtain the multi-view point cloud features at the target resolution.
[0080] For example, in the process of extracting features from a multi-view feature plane to obtain multi-view point cloud features, the computer device can extract features from the multi-view feature plane using a pre-trained point cloud feature extraction model: the computer device can encode the multi-view feature plane using the encoder in the point cloud feature extraction model to obtain the feature plane encoding result; then, the computer device can decode the feature plane encoding result using the decoder in the point cloud feature extraction model to obtain the multi-view point cloud features at the target resolution.
[0081] The technical solution of this embodiment encodes the multi-view feature plane through the encoder in the point cloud feature extraction model and decodes the feature plane encoding result through the decoder in the point cloud feature extraction model. This can effectively capture the data features in the multi-view feature plane by using a two-dimensional neural network to perform encoding and decoding operations, thereby compressing and reconstructing the key information in the multi-view feature plane and extracting representative features from the multi-view feature plane as multi-view point cloud features.
[0082] In one exemplary embodiment, feature extraction of a two-dimensional semantic image to obtain multi-view image features includes: encoding the two-dimensional semantic image through an encoder in an image feature extraction model to obtain a semantic image encoding result; and decoding the semantic image encoding result through a decoder in an image feature extraction model to obtain multi-view image features at the target resolution.
[0083] For example, in the process of extracting features from a two-dimensional semantic image to obtain multi-view image features, the computer device can extract features from the two-dimensional semantic image through a pre-trained image feature extraction model. Specifically, the computer device can encode the two-dimensional semantic image through the encoder in the image feature extraction model to obtain the semantic image encoding result. Then, the computer device can decode the semantic image encoding result through the decoder in the image feature extraction model to obtain multi-view image features at the target resolution.
[0084] The technical solution of this embodiment encodes a two-dimensional semantic image through an encoder in the image feature extraction model and decodes the semantic image encoding result through a decoder in the image feature extraction model. This can effectively capture the data features in the two-dimensional semantic image by using a two-dimensional neural network to perform encoding and decoding operations. It can also effectively compress and reconstruct the key information in the two-dimensional semantic image, thereby extracting representative features from the two-dimensional semantic image as multi-view image features.
[0085] To facilitate understanding by those skilled in the art, Figure 4 provides an exemplary schematic diagram of three-view feature encoding. The computer device can input the three-view feature plane 402 into a two-dimensional neural network A, which encodes the three-view feature plane 402 and decodes the three-view point cloud features 404. Then, the computer device inputs the three-view point cloud features 404 into a pre-trained generative model to obtain a two-dimensional semantic image 406. Finally, the computer device inputs the two-dimensional semantic image 406 into a two-dimensional neural network B, which encodes the two-dimensional semantic image 406 and decodes the three-view image features 408.
[0086] In one exemplary embodiment, obtaining raw point cloud data in a target spatial scene includes: sampling point cloud data in the target spatial scene according to preset sampling configuration information to obtain sampled data; obtaining the sampled data as raw point cloud data.
[0087] Among them, sampling configuration information can refer to the configuration information for setting sampling parameters.
[0088] The sampling parameters may include sampling density, sampling method, etc. Sampling methods include uniform sampling, random sampling, importance sampling, etc.
[0089] For example, during the process of acquiring raw point cloud data in a target spatial scene, the computer device can obtain sampling configuration information pre-configured by the user; then, the computer device can sample the point cloud data in the target spatial scene according to the sampling method, sampling density, and other sampling parameters indicated by the sampling configuration information to obtain sampled data. The computer device uses the obtained sampled data as the raw point cloud data.
[0090] The technical solution of this embodiment, in the process of acquiring the original point cloud data in the target spatial scene, samples the point cloud data in the target spatial scene according to the preset sampling configuration information, and uses the sampled data as the original point cloud data. This can effectively reduce the data processing volume of computer equipment while minimizing the impact on the accuracy of spatial semantic occupancy prediction and improving the efficiency of spatial semantic occupancy prediction.
[0091] In one exemplary embodiment, as shown in FIG5, a spatial semantic occupancy prediction system is provided, including: a point cloud sampling unit, a three-view transformation unit, a three-view feature encoding unit, a query point feature encoding unit, and a three-dimensional semantic occupancy prediction output unit.
[0092] Among them, the point cloud sampling unit is used to sample the point cloud data in the target space scene to obtain the original point cloud data of the target space scene;
[0093] The three-view transformation unit is used to transform the original point cloud data in the target space scene into three-view features to obtain the three-view feature plane.
[0094] The three-view feature encoding unit is used to extract features from the three-view feature plane to obtain three-view point cloud features; the three-view point cloud features are input into a pre-trained visual generative model to obtain a two-dimensional semantic image; features are extracted from the two-dimensional semantic image to obtain three-view image features; and the three-view point cloud features and the three-view image features are fused to obtain three-view fused features.
[0095] The query point feature encoding unit is used to obtain the fusion features corresponding to each view in the three-view fusion features corresponding to the query point; and to fuse the fusion features corresponding to each view to obtain the three-dimensional features corresponding to the query point.
[0096] The 3D semantic occupancy prediction output unit is used to input the 3D features of the query point into a pre-trained 3D feature classification model and output the prediction result.
[0097] It should be noted that the specific limitations of the above steps can be found in the specific limitations of a spatial semantic occupancy prediction method described above.
[0098] The aforementioned spatial semantic occupancy prediction system acquires raw point cloud data from the target spatial scene, converts the raw point cloud data into a multi-view feature plane, extracts features from the multi-view feature plane to obtain multi-view point cloud features, inputs the multi-view point cloud features into a pre-trained visual generative model to obtain a two-dimensional semantic image, extracts features from the two-dimensional semantic image to obtain multi-view image features, and then fuses the multi-view point cloud features and multi-view image features to obtain multi-view fused features. For any query point located in the target spatial scene, the system outputs the spatial semantic occupancy prediction result based on the multi-view fused features corresponding to the query point. Thus, it can achieve the prediction of multiple spatial occupancy predictions. By encoding and decoding 2D point cloud features from different perspectives and 2D semantic images output by a visual generative model to obtain 2D image features, information loss during projection is reduced. This approach can further enhance the model's ability to model complex 3D scenes while maintaining high efficiency. Furthermore, by fusing multi-view point cloud features and multi-view image features in parallel and aggregating the features of query points on the three-view plane using an additive approach, the problem of 2D feature methods being unable to depict the fine structure in complex 3D scenes can be effectively solved. This approach effectively enables the modeling of complex 3D scenes using 2D features, reduces the computational and storage burden of the model, and thus improves the efficiency of spatial semantic occupancy prediction based on the model.
[0099] In another embodiment, as shown in Figure 6, a spatial semantic occupancy prediction method is provided. Taking the application of this method to the terminal in Figure 1 as an example, the method includes the following steps:
[0100] Step S602: Sample to obtain the original input point cloud. For example, sample the point cloud data in the target space scene to obtain the original point cloud data in the target space scene.
[0101] Step S604, Three-view transformation. For example, the original point cloud data is divided into voxels in cylindrical coordinate system to obtain voxelized point cloud data; pooling operation is performed on the voxelized point cloud data in three coordinate axes of cylindrical coordinate system to obtain planar features in three views in cylindrical coordinate system.
[0102] Step S606, three-view feature encoding. For example, feature extraction is performed on the three-view feature plane to obtain three-view point cloud features; the three-view point cloud features are input into a pre-trained visual generative model to obtain a two-dimensional semantic image; feature extraction is performed on the two-dimensional semantic image to obtain three-view image features; the three-view point cloud features and the three-view image features are fused to obtain three-view fused features.
[0103] Step S608, query point feature encoding. For example, obtain the fusion features corresponding to each viewpoint in the three-view fusion features corresponding to the query point; fuse the fusion features corresponding to each viewpoint to obtain the three-dimensional features corresponding to the query point. Input the three-dimensional features of the query point into a pre-trained three-dimensional feature classification model and output the prediction result.
[0104] It should be noted that the specific limitations of the above steps can be found in the specific limitations of a spatial semantic occupancy prediction method described above, and will not be repeated here.
[0105] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0106] Based on the same inventive concept, this application also provides a spatial semantic occupancy prediction device for implementing the spatial semantic occupancy prediction method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more spatial semantic occupancy prediction device embodiments provided below can be found in the limitations of the spatial semantic occupancy prediction method described above, and will not be repeated here.
[0107] In one exemplary embodiment, as shown in FIG7, a spatial semantic occupancy prediction device is provided, comprising:
[0108] The acquisition module 710 is used to acquire raw point cloud data in the target space scene;
[0109] The conversion module 720 is used to convert the original point cloud data into a multi-view feature plane, extract features from the multi-view feature plane, and obtain multi-view point cloud features.
[0110] The generation module 730 is used to input the multi-view point cloud features into a pre-trained visual generative model to obtain a two-dimensional semantic image, and to extract features from the two-dimensional semantic image to obtain multi-view image features.
[0111] The fusion module 740 is used to fuse the multi-view point cloud features and the multi-view image features to obtain multi-view fused features.
[0112] The output module 750 is used to output a spatial semantic occupancy prediction result for any query point located in the target spatial scene, based on the multi-view fusion features corresponding to the query point.
[0113] In one embodiment, the output module 750 is used to obtain the fusion features corresponding to each view in the multi-view fusion features corresponding to the query point; fuse the fusion features corresponding to each view to obtain the three-dimensional features corresponding to the query point; input the three-dimensional features of the query point into a pre-trained three-dimensional feature classification model to obtain the spatial semantic occupancy prediction result.
[0114] In one embodiment, the conversion module 720 is used to divide the original point cloud data into voxels in a cylindrical coordinate system to obtain voxelized point cloud data; to perform pooling operations on the voxelized point cloud data in the three coordinate axes of the cylindrical coordinate system to obtain planar features in three perspectives in the cylindrical coordinate system; and to obtain the planar features in the three perspectives as the multi-view feature plane.
[0115] In one embodiment, the conversion module 720 is used to encode the multi-view feature plane using an encoder in the point cloud feature extraction model to obtain a feature plane encoding result; and to decode the feature plane encoding result using a decoder in the point cloud feature extraction model to obtain the multi-view point cloud features at the target resolution.
[0116] In one embodiment, the generation module 730 is used to encode the two-dimensional semantic image through the encoder in the image feature extraction model to obtain a semantic image encoding result; and to decode the semantic image encoding result through the decoder in the image feature extraction model to obtain the multi-view image features at the target resolution.
[0117] In one embodiment, the acquisition module 710 is used to sample point cloud data in the target space scene according to preset sampling configuration information to obtain sampled data; the sampled data is obtained as the original point cloud data.
[0118] Each module in the aforementioned spatial semantic occupancy prediction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.
[0119] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram is shown in Figure 8. The computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a spatial semantic occupancy prediction method. The display unit of the computer device is used to form a visually visible image and may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.
[0120] Those skilled in the art will understand that the structure shown in Figure 8 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.
[0121] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the spatial semantic occupancy prediction method described above. The steps of the spatial semantic occupancy prediction method described here can be steps from one of the spatial semantic occupancy prediction methods in the various embodiments described above.
[0122] In one embodiment, a computer-readable storage medium is provided, storing a computer program that, when executed by a processor, causes the processor to perform the steps of the spatial semantic occupancy prediction method described above. The steps of the spatial semantic occupancy prediction method described here can be steps from one of the spatial semantic occupancy prediction methods in the various embodiments described above.
[0123] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, causes the processor to perform the steps of the spatial semantic occupancy prediction method described above. The steps of the spatial semantic occupancy prediction method described here may be steps from one of the spatial semantic occupancy prediction methods in the various embodiments described above.
[0124] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0125] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic resistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0126] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0127] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A spatial semantic occupancy prediction method, characterized in that, The method includes: Acquire raw point cloud data from the target spatial scene; The original point cloud data is converted into a multi-view feature plane, and features are extracted from the multi-view feature plane to obtain multi-view point cloud features. The multi-view point cloud features are input into a pre-trained visual generative model to obtain a two-dimensional semantic image. Feature extraction is performed on the two-dimensional semantic image to obtain multi-view image features. By fusing the multi-view point cloud features and the multi-view image features, a multi-view fused feature is obtained; For any query point located in the target spatial scene, the spatial semantic occupancy prediction result is output based on the multi-view fusion features corresponding to the query point.
2. The method according to claim 1, characterized in that, The step of outputting a spatial semantic occupancy prediction result based on the multi-view fusion features corresponding to the query point includes: Obtain the fusion features corresponding to each viewpoint in the multi-view fusion features corresponding to the query point; By fusing the fusion features corresponding to each viewpoint, the three-dimensional features corresponding to the query point are obtained; The three-dimensional features of the query point are input into a pre-trained three-dimensional feature classification model to obtain the spatial semantic occupancy prediction result.
3. The method according to claim 1, characterized in that, The step of converting the original point cloud data into a multi-view feature plane includes: The original point cloud data is divided into voxels in cylindrical coordinates to obtain voxelized point cloud data. The voxelized point cloud data is subjected to pooling operations under the three coordinate axes of the cylindrical coordinate system to obtain planar features from three perspectives in the cylindrical coordinate system. The planar features obtained from the three perspectives are the multi-view feature plane.
4. The method according to claim 1, characterized in that, The step of extracting features from the multi-view feature plane to obtain multi-view point cloud features includes: The multi-view feature plane is encoded using the encoder in the point cloud feature extraction model to obtain the feature plane encoding result; The feature plane encoding result is decoded by the decoder in the point cloud feature extraction model to obtain the multi-view point cloud features at the target resolution.
5. The method according to claim 4, characterized in that, The step of extracting features from the two-dimensional semantic image to obtain multi-view image features includes: The two-dimensional semantic image is encoded using the encoder in the image feature extraction model to obtain the semantic image encoding result; The semantic image encoding result is decoded by the decoder in the image feature extraction model to obtain the multi-view image features at the target resolution.
6. The method according to any one of claims 1 to 5, characterized in that, The acquisition of raw point cloud data in the target space scene includes: According to the preset sampling configuration information, the point cloud data in the target space scene is sampled to obtain the sampled data; The sampled data is obtained as the original point cloud data.
7. A spatial semantic occupancy prediction device, characterized in that, The device includes: The acquisition module is used to acquire raw point cloud data from the target spatial scene; The conversion module is used to convert the original point cloud data into a multi-view feature plane, and to extract features from the multi-view feature plane to obtain multi-view point cloud features. The generation module is used to input the multi-view point cloud features into a pre-trained visual generative model to obtain a two-dimensional semantic image, and to extract features from the two-dimensional semantic image to obtain multi-view image features. A fusion module is used to fuse the multi-view point cloud features and the multi-view image features to obtain multi-view fused features; The output module is used to output a spatial semantic occupancy prediction result for any query point located in the target spatial scene, based on the multi-view fusion features corresponding to the query point.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.