Data processing method and apparatus

Abstract

The present application relates to the technical field of artificial intelligence. Provided are a data processing method and apparatus. The method comprises: acquiring first point cloud data, wherein the first point cloud data comprises N points and N input features corresponding to the N points, and N is greater than 0; acquiring second point cloud data, wherein the second point cloud data comprises M points and M input features corresponding to the M points, and M is greater than 0; and on the basis of a difference between the first point cloud data and the second point cloud data, performing a sparse convolution operation on the second point cloud data. On the basis of the method, the computational overhead caused by table construction during a three-dimensional sparse convolution operation performed on point cloud data can be reduced, which is conducive to improving the operation performance of a processor in three-dimensional sparse convolution.

Description

Data processing methods and apparatus Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to methods and apparatus for data processing. Background Technology

[0002] With the continuous development of 3D sensors and artificial intelligence technologies, 3D point cloud processing technology plays a crucial role in perception, map creation, and localization modules in the field of autonomous driving. Due to the sparsity of point cloud data, after voxelization, the proportion of invalid voxels in 3D space is relatively high. Using the traditional sliding window method for convolution calculations results in a large amount of redundant computation on these invalid voxels. To reduce the waste of memory and computing resources, improve feature extraction capabilities, and enhance the efficiency of point cloud processing algorithms, 3D sparse convolution has been proposed and has become a commonly used algorithm in point cloud object detection neural networks.

[0003] In existing algorithms for 3D point cloud detection based on 3D sparse convolution, when performing sparse convolution calculations on the input point cloud, it is necessary to construct corresponding mapping tables, such as rulebooks, gather tables, and scatter tables. The process of constructing these mapping tables involves a large number of non-contiguous memory accesses, resulting in significant overhead during the entire 3D sparse convolution calculation process. This limits the computational performance of hardware platforms for 3D sparse convolution of continuous frame point clouds.

[0004] Summary of the Invention

[0005] This application provides a data processing method and apparatus that can reduce the overhead of establishing rule tables, gather tables and scatter tables during the three-dimensional sparse convolution operation on point cloud data, thereby helping to improve the computational performance of the processor in three-dimensional sparse convolution.

[0006] In a first aspect, a data processing method is provided, the method comprising: acquiring first point cloud data, the first point cloud data including N points and N input features corresponding to the N points, where N is greater than 0; acquiring second point cloud data, the second point cloud data including M points and M input features corresponding to the M points, where M is greater than 0; and performing sparse convolution operation on the second point cloud data based on the difference between the first point cloud data and the second point cloud data.

[0007] For example, the application scenarios corresponding to the above data processing methods can be autonomous driving solutions or functional scenarios involving three-dimensional sparse convolution in intelligent robot pathfinding, such as point cloud object detection. Accordingly, the methods proposed in this application embodiment can be deployed on edge devices with computing capabilities, such as intelligent driving computing platforms, intelligent robots, etc.

[0008] For example, the above sparse convolution operation can be a three-dimensional sparse convolution operation, and in some special application scenarios, the above sparse convolution operation can also be a two-dimensional sparse convolution operation.

[0009] For example, the first point cloud data and the second point cloud data mentioned above may be two adjacent frames of point cloud data in terms of time sequence, or there may be at least one frame of other point cloud data between the first point cloud data and the second point cloud data.

[0010] For example, the difference between the first and second point cloud data mentioned above refers to the addition and / or invalidation of points in the second point cloud data compared to the first point cloud data. Furthermore, utilizing the differences between these two point cloud frames, a relevant parameter mapping table can be constructed for performing sparse convolution operations on the second point cloud data; for example, a reference table (which could be a rulebook), a gather table, and a scatter table.

[0011] Based on the above technical solution, the difference information between the point cloud data of the later frame and the point cloud data of the earlier frame—that is, reusing historical data from the tables corresponding to the point cloud data of the earlier frame—can be used to maintain a parameter mapping table for sparse convolution operations on the second point cloud data. This enables the differential table building operation, and subsequently, sparse convolution operations can be performed on the point cloud data of the later frame based on the updated gather and scatter tables. This saves a significant amount of computational overhead caused by table building operations and helps improve the sparse convolution performance of the processor.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the positions of the aforementioned N points are represented by the coordinates of N voxels. The first point cloud data is used to construct a first relation table and a second relation table, wherein the first relation table is used to represent the correspondence between N voxels and N input features, and the second relation table is used to represent the correspondence between N voxels and N output features. The first relation table and the second relation table are updated according to the difference between the first point cloud data and the second point cloud data. Based on the updated first relation table and the second relation table, sparse convolution operation is performed on the second point cloud data.

[0013] For example, the first relation table could be the gather table mentioned above, and the second relation table could be the scatter table mentioned above.

[0014] For example, the correspondence between N voxels and N input features in the first relation table above can be the correspondence between the coordinates of N voxels and the index numbers of the N input features; similarly, the correspondence between N voxels and N output features in the second relation table above can be the correspondence between the coordinates of N voxels and the index numbers of the N output features.

[0015] For example, when there are multiple convolution kernels participating in the convolution operation on N input features, the first relation table and the second relation table mentioned above can also be used to indicate the convolution kernels corresponding to the N voxels respectively. The convolution kernels are used to perform the above convolution operation on the input features corresponding to the voxels to determine the corresponding output features. Further, the first relation table mentioned above can be used to indicate the convolution kernels corresponding to the N input features respectively, and correspondingly, the second relation table mentioned above can be used to indicate the convolution kernels corresponding to the N output features respectively.

[0016] For example, when there is only one convolution kernel participating in the convolution operation on N input features, the correspondence between the N input features and the convolution kernel can be reflected in the first relation table, and the correspondence between the N output features and the convolution kernel can be reflected in the second relation table.

[0017] For example, when multiple convolutional kernels participate in the sparse convolution operation on N input features, the relational table involved in this method may further include a third relational table. This third relational table is used to indicate at least one voxel coordinate corresponding to each convolutional kernel, where the at least one voxel coordinate belongs to the aforementioned N voxels, and all voxels corresponding to the multiple convolutional kernels include the aforementioned N voxels. Based on the first relational table, the second relational table, and the third relational table, the correspondence between N voxels and the multiple convolutional kernels can be determined. Furthermore, the correspondence between N input features or N output features and the multiple convolutional kernels can also be determined.

[0018] Based on the above technical solution, the first relation table and the second relation table used for sparse convolution operation on the second point cloud data can be maintained by using the difference information between the point cloud data of the later frame and the point cloud data of the previous frame, that is, by reusing the historical data in each table corresponding to the point cloud data of the previous frame. This enables the operation of differential table building, thereby saving a lot of computational overhead caused by table building operations and helping to improve the sparse convolution operation performance of the processor.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the number of entries in the first relation table and the number of entries in the second relation table are pre-configured.

[0020] For example, the number of entries in the first relation table and the number of entries in the second relation table can be determined based on historical data, which is used to indicate the number of entries in the first relation table and the second relation table that enable the three-dimensional sparse convolution operation to proceed smoothly in scenarios with different point cloud distributions.

[0021] Based on the above technical solution, by pre-allocating a certain number of table entries to the first relation table and the second relation table, it is ensured that there are corresponding empty table entries for adding input feature index numbers or output feature index numbers corresponding to the newly added voxels during the subsequent maintenance of the first relation table and the second relation table.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, the number of convolution kernels used for sparse convolution operations is Y. When Y is greater than 1, the first relation table includes Y first relation sub-tables, which correspond to Y convolution kernels. The second relation table includes Y second relation sub-tables, which correspond to Y convolution kernels. In this table, the N entries of the Y first relation sub-tables contain the index numbers of N input features, and the N entries of the Y second relation sub-tables contain the index numbers of N output features.

[0023] For example, when the number of convolution kernels used for sparse convolution operations is Y, the number of entries in the first relation table and the number of entries in the second relation table are greater than N×Y.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, the positions of the above M points are represented by the coordinates of M voxels. The first relation table and the second relation table are updated according to the first newly added voxel, and / or the first relation table is updated according to the first failed voxel. The first newly added voxel is a voxel among the M voxels that does not belong to the N voxels, and the first failed voxel is a voxel among the N voxels that does not belong to the M voxels.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, the first newly added voxel corresponds to the first newly added input feature and the first newly added output feature respectively, the index number related to the first newly added input feature is added to the first relation table, and / or, the content of the table entry related to the first invalid voxel in the first relation table is set to invalid; the index number related to the first newly added output feature is added to the second relation table.

[0026] For example, when the difference between two frames of point cloud data only contains faulty voxels, the first relation table can be updated based on the faulty voxels, and the second relation table does not need to be updated; when the difference between two frames of point cloud data only contains newly added voxels, the first and second relation tables can be updated only based on the newly added voxels; when the second point cloud data is completely consistent with the first point cloud data, the first and second relation tables corresponding to the first point cloud data can be directly reused.

[0027] Based on the above technical solutions, the first relation table is updated by reusing the first relation table corresponding to the previous frame point cloud data and supplementing the corresponding positions of the voxels of the newly added points in the subsequent frame point cloud data with the corresponding positions in the first relation table, and / or deleting the corresponding input features of the voxels of the reduced points from the first relation table. Similarly, the second relation table is updated by reusing the second relation table corresponding to the previous frame point cloud data and adding the corresponding positions of the voxels of the newly added points in the subsequent frame point cloud data with the corresponding positions in the second relation table. This update method is more cost-effective than rebuilding the relation table, helps improve the construction efficiency of the relation table, and helps improve the processor's performance in sparse convolution of continuous frame point cloud data.

[0028] In conjunction with the first aspect, in some implementations of the first aspect, a first relation table and a second relation table are constructed based on the reference table. The reference table includes a first column and a second column. The L entries in the first column correspond to the coordinates of L voxels, and N voxels belong to L voxels. The N entries in the second column correspond to the index numbers of N input features, and the N entries correspond to N voxels. The remaining (LN) entries in the second column contain invalid values.

[0029] For example, the invalid values ​​mentioned above can be negative numbers, null, or values ​​greater than the number of items in the second list.

[0030] For example, the reference table above can consist of L key-value pairs, where the key in a key-value pair is the voxel coordinate and the value is the index number of the input feature corresponding to the voxel coordinate; and the key and value in a key-value pair are located in the same row of the reference table.

[0031] For example, after establishing the above reference table, it is also necessary to establish the first relation table and the second relation table based on the reference table.

[0032] Based on the above technical solution, the necessary prerequisites for the differential table building operation in this data processing method can be provided to ensure the smooth execution of the differential table building operation, and it also helps to reduce the complexity of updating the first and second relation tables in the subsequent differential table building operation. Theoretically, when updating the first and second relation tables based on the above reference table, the computational complexity is O(1), and when updating the first and second relation tables without updating them based on the above reference table, the computational complexity is O(N), where N is greater than 1.

[0033] In conjunction with the first aspect, in some implementations of the first aspect, the reference table is updated based on the first failed voxel and / or the first newly added voxel. In the updated reference table, the entries corresponding to the coordinates and input features of the first failed voxel are invalid values, and / or, in the updated reference table, one entry in the first column contains the coordinates of the first newly added voxel, and the entry in the second column corresponding to the coordinates of the first newly added voxel contains the index number of the first newly added input feature.

[0034] For example, during the process of updating the reference table, the coordinates of the first failed voxel and the table entry containing the input feature in the reference table can be set to invalid values, the coordinates of the first newly added voxel can be written in the first column, and the index number of the first newly added input feature can be written in the table entry corresponding to the coordinates of the first newly added voxel in the second column, so as to update the reference table.

[0035] Based on the above technical solution, the reference table is updated through differential table building. During the update process, the reference table constructed based on the previous frame point cloud data is reused, so that before performing sparse convolution operation on the subsequent frame point cloud data, the reference table corresponding to the subsequent frame point cloud data can be obtained with a small computational overhead for use in subsequent sparse convolution operation.

[0036] In conjunction with the first aspect, in some implementations of the first aspect, when Y is greater than 1, according to the updated reference table, the index numbers of the input features corresponding to the Y first target voxels are written to the (N+1)th row of the Y first relation sub-tables, where the Y first target voxels are voxels within the receptive field of the convolutional kernel centered on the first newly added voxel; when the index number of the input feature corresponding to the second target voxel in the updated reference table is a valid value, the row number corresponding to the second target voxel in the updated reference table is determined to be the first target row number, where the second target voxel corresponds to the center of the first convolutional kernel within the receptive field of the Y convolutional kernels, including the first newly added voxel; and a valid value is written into the table entry corresponding to the first target row number of the first relation sub-table corresponding to the first convolutional kernel.

[0037] Based on the above technical solution, by reusing the first relation table corresponding to the previous frame point cloud data, and supplementing the corresponding position of the voxel where the newly added point in the subsequent frame point cloud data is located into the first relation table, the first relation table is updated. This update method is more cost-effective than rebuilding the first relation table, and helps to improve the construction efficiency of the first relation table, and also helps to improve the sparse convolution operation performance of the processor.

[0038] In conjunction with the first aspect, in some implementations of the first aspect, according to the table entry position of the index number of the input feature corresponding to the Y first target voxels written in the first relation table, the index number of the output feature corresponding to the Y first target voxels is written to the (N+1)th row of the Y second relation sub-tables.

[0039] Based on the above technical solution, by reusing the second relation table corresponding to the previous frame point cloud data, and supplementing the input features corresponding to the voxels of the newly added points in the subsequent frame point cloud data to the corresponding positions in the second relation table, the second relation table is updated. This update method is more cost-effective than rebuilding the second relation table, and helps to improve the construction efficiency of the second relation table, as well as the sparse convolution operation performance of the processor.

[0040] In conjunction with the first aspect, in some implementations of the first aspect, based on the updated reference table, the row number corresponding to the first failure voxel in the updated reference table is determined to be the second target row number; invalid values ​​are written into the entries corresponding to the second target row number of the Y first relational sub-tables; when the index number of the input feature corresponding to the third target voxel in the updated reference table is a valid value, the row number corresponding to the third target voxel in the updated reference table is determined to be the third target row number, and the third target voxel corresponds to the center of the second convolutional kernel within the receptive field of the Y convolutional kernels, including the first failure voxel; invalid values ​​are written into the entries corresponding to the third target row number of the first relational sub-table corresponding to the second convolutional kernel.

[0041] Based on the above technical solution, by reusing the first relation table corresponding to the previous frame point cloud data, and supplementing the corresponding positions of the voxels where the points of the subsequent frame point cloud data are invalid compared to the previous frame point cloud data with the input features of the voxels, the first relation table is updated. This update method is more cost-effective than rebuilding the first relation table, and helps to improve the construction efficiency of the first relation table, and also helps to improve the sparse convolution operation performance of the processor.

[0042] In conjunction with the first aspect, in some implementations of the first aspect, before performing sparse convolution operations on the second point cloud data based on the difference between the first point cloud data and the second point cloud data, it is determined that the number of newly added and / or invalid points in the second point cloud data compared to the first point cloud data is less than or equal to a first threshold.

[0043] It should be understood that after voxelizing the first point cloud data and the second point cloud data, the points added and / or invalidated in the second point cloud data compared to the first point cloud data can be understood as the first newly added voxels and the first invalidated voxels proposed in the aforementioned embodiments.

[0044] For example, the first threshold mentioned above can be pre-configured.

[0045] Based on the above technical solution, it is possible to enter the data processing method flow proposed in this application under appropriate circumstances, so as to ensure that when updating the first relation table and the second relation table based on the difference between the point cloud data of the later frame and the point cloud data of the previous frame, the previously determined first relation table and the second relation table can be reused more, so that the computational overhead of table building is kept at a low level.

[0046] In conjunction with the first aspect, in some implementations of the first aspect, the first point cloud data and the second point cloud data are acquired through sensors mounted on the vehicle. Before performing sparse convolution operation on the second point cloud data based on the difference between the first point cloud data and the second point cloud data, the vehicle speed is acquired, and the vehicle speed is less than or equal to a second threshold.

[0047] For example, a vehicle's speed can be obtained through onboard sensors.

[0048] For example, the second threshold mentioned above can be pre-configured.

[0049] For example, in the application scenario of intelligent vehicles, the decision to enter the data processing method proposed in this application can be determined by judging the current driving status of the intelligent vehicle and the activated intelligent driving functions. For instance, if the vehicle is not parked, and the traffic jam assistant (TJA) function in the vehicle's intelligent driving functions is activated, then the data processing method proposed in this application can be entered; or, if the vehicle is parked, and the auto parking assist (APA), auto valet parking (AVP), or valet parking driver (VPD) functions in the vehicle's intelligent driving functions are activated, then the data processing method proposed in this application can be entered.

[0050] Based on the above technical solution, under certain preconditions (such as vehicle speed, vehicle driving status, or surrounding traffic conditions), the first and second relation tables of the previous frame point cloud data are adaptively activated and updated based on the differences between the two frames of point cloud data. This avoids comparing every two frames of point cloud data, thus minimizing the computational resource consumption of comparison operations when processor resources are insufficient. This meets the algorithm requirements under different hardware scenarios and increases the robustness of the solution.

[0051] In a second aspect, a data processing apparatus is provided, comprising: an acquisition unit for acquiring first point cloud data, the first point cloud data including N points and N input features corresponding to the N points, where N is greater than 0, and for acquiring second point cloud data, the second point cloud data including M points and M input features corresponding to the M points, where M is greater than 0; and a processing unit for performing sparse convolution operations on the second point cloud data based on the differences between the first point cloud data and the second point cloud data.

[0052] In conjunction with the second aspect, in some implementations of the second aspect, the positions of the aforementioned N points are represented by the coordinates of N voxels. The first point cloud data is used to construct a first relation table and a second relation table, wherein the first relation table is used to represent the correspondence between N voxels and N input features, and the second relation table is used to represent the correspondence between N voxels and N output features. Based on this, the aforementioned operation unit 1120 is specifically used to: update the first relation table and the second relation table according to the difference between the first point cloud data and the second point cloud data; and perform sparse convolution operation on the second point cloud data according to the updated first relation table and the second relation table.

[0053] In conjunction with the second aspect, in some implementations of the second aspect, the number of entries in the first relation table and the number of entries in the second relation table are pre-configured.

[0054] In conjunction with the second aspect, in some implementations of the second aspect, the number of convolution kernels used for sparse convolution operations is Y. When Y is greater than 1, the first relation table includes Y first relation sub-tables, and the Y first relation sub-tables correspond to the Y convolution kernels. The second relation table includes Y second relation sub-tables, and the Y second relation sub-tables correspond to the Y convolution kernels. The N entries of the Y first relation sub-tables contain the index numbers of the N input features, and the N entries of the Y second relation sub-tables contain the index numbers of the N output features.

[0055] In conjunction with the second aspect, in some implementations of the second aspect, the positions of the above M points are represented by the coordinates of M voxels. The above-mentioned operation unit is specifically used to: update the first relation table and the second relation table according to the first newly added voxel, and / or update the first relation table according to the first failed voxel. The first newly added voxel is a voxel among the M voxels that does not belong to the N voxels, and the first failed voxel is a voxel among the N voxels that does not belong to the M voxels.

[0056] In conjunction with the second aspect, in some implementations of the second aspect, the aforementioned first newly added voxel corresponds to the first newly added input feature and the first newly added output feature, respectively. The aforementioned arithmetic unit is specifically used to: add the index number related to the first newly added input feature to the first relation table, and / or, invalidate the content of the table entry related to the first invalid voxel in the first relation table; and add the index number related to the first newly added output feature to the second relation table.

[0057] In conjunction with the second aspect, in some implementations of the second aspect, the above-mentioned operation unit is also used to: construct a first relation table and a second relation table according to the reference table, the reference table including a first column and a second column, the L entries in the first column correspond to the coordinates of L voxels, N voxels belong to L voxels, the N entries in the second column correspond to the index numbers of N input features, the N entries correspond to N voxels, and the remaining (LN) entries in the second column contain invalid values.

[0058] In conjunction with the second aspect, in some implementations of the second aspect, the aforementioned arithmetic unit is further configured to: update a reference table based on the first failed voxel and / or the first newly added voxel, wherein in the updated reference table, the entries containing the coordinates and input features corresponding to the first failed voxel are invalid values, and / or, in the updated reference table, an entry in the first column contains the coordinates of the first newly added voxel, and an entry in the second column corresponding to the coordinates of the first newly added voxel contains the index number of the first newly added input feature.

[0059] In conjunction with the second aspect, in some implementations of the second aspect, when Y is greater than 1, the above-mentioned operation unit is specifically used to: write the index numbers of the input features corresponding to the Y first target voxels into the (N+1)th row of the Y first relation sub-tables according to the updated reference table, where the Y first target voxels are voxels within the receptive field of the convolution kernel centered on the first newly added voxel; when the index number of the input feature corresponding to the second target voxel in the updated reference table is a valid value, determine that the row number corresponding to the second target voxel in the updated reference table is the first target row number, where the second target voxel corresponds to the center of the first convolution kernel within the receptive field of the Y convolution kernels, including the first newly added voxel; and write a valid value into the table entry corresponding to the first target row number of the first relation sub-table corresponding to the first convolution kernel.

[0060] In conjunction with the second aspect, in some implementations of the second aspect, the above-mentioned operation unit is specifically used to: write the index numbers of the output features corresponding to the Y first target voxels into the (N+1)th row of the Y second relation sub-tables according to the table entry positions of the table entries in the first relation table that contain the index numbers of the input features corresponding to the Y first target voxels respectively.

[0061] In conjunction with the second aspect, in some implementations of the second aspect, the aforementioned computational unit is specifically used to: determine, based on the updated reference table, the row number corresponding to the first failure voxel in the updated reference table as the second target row number; write invalid values ​​into the entries corresponding to the second target row number of the Y first relational sub-tables; when the index number of the input feature corresponding to the third target voxel in the updated reference table is a valid value, determine, the row number corresponding to the third target voxel in the updated reference table as the third target row number, wherein the third target voxel corresponds to the center of the second convolutional kernel within the receptive field of the Y convolutional kernels, including the first failure voxel; and write invalid values ​​into the entries corresponding to the third target row number of the first relational sub-table corresponding to the second convolutional kernel.

[0062] In conjunction with the second aspect, in some implementations of the second aspect, the above-mentioned apparatus may further include a determining unit, which determines that the number of newly added and / or invalid points in the second point cloud data compared to the first point cloud data is less than or equal to a first threshold before the computing unit performs sparse convolution operation on the second point cloud data based on the difference between the first point cloud data and the second point cloud data.

[0063] In conjunction with the second aspect, in some implementations of the second aspect, the first point cloud data and the second point cloud data are acquired by sensors mounted on the vehicle. Before the computing unit performs sparse convolution operation on the second point cloud data based on the difference between the first point cloud data and the second point cloud data, the determining unit is also used to determine that the speed of the vehicle is less than or equal to the second threshold.

[0064] Thirdly, a data processing apparatus is provided, including a processor and a memory, wherein the processor and the memory are connected together, wherein the memory is used to store program code, and the processor is used to call the program code to execute a method in any possible implementation of the method design of the first aspect described above.

[0065] Fourthly, a vehicle is provided, including the means as described in any possible implementation of the means design of the second or third aspect above.

[0066] Fifthly, a chip system is provided, which is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuits and processors are interconnected by lines; the interface circuits are used to receive signals from the memory of the electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes any possible implementation of the method design of the first aspect above.

[0067] In a sixth aspect, a computer-readable storage medium is provided storing a computer program or instructions for implementing the method in any possible implementation of the method design of the first aspect.

[0068] In a seventh aspect, a computer program product is provided, wherein when the computer program code or instructions are executed on a computer, the computer performs the method in any possible implementation of the method design of the first aspect described above. Attached Figure Description

[0069] Figure 1 is a functional block diagram of a vehicle 100 provided in an embodiment of this application;

[0070] Figure 2 is a flowchart of a three-dimensional sparse convolution algorithm 200;

[0071] Figure 3 is a flowchart illustrating a data processing method 300 proposed in an embodiment of this application;

[0072] Figure 4 is a flowchart illustrating another data processing method 400 proposed in an embodiment of this application;

[0073] Figure 5 is a flowchart illustrating another data processing method 500 proposed in an embodiment of this application;

[0074] Figure 6 is a flowchart illustrating another data processing method 600 proposed in an embodiment of this application;

[0075] Figure 7 is a schematic diagram of an initialized reference table, a first relation table, and a second relation table according to an embodiment of this application;

[0076] Figure 8 is a schematic diagram of a differential update reference table proposed in an embodiment of this application;

[0077] Figure 9 is a schematic diagram of a differential update of the first relation table proposed in an embodiment of this application;

[0078] Figure 10 is a schematic diagram of a differential update of the second relation table proposed in an embodiment of this application;

[0079] Figure 11 is a schematic block diagram of a data processing apparatus 1100 according to an embodiment of this application. Detailed Implementation

[0080] It should be noted that, in the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in this article is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0081] In the embodiments of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more, and "at least one" and "one or more" refer to one, two, or more than two. The singular expressions "a," "an," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context explicitly indicates otherwise.

[0082] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0083] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0084] Figure 1 is a functional block diagram of a vehicle 100 provided in an embodiment of this application.

[0085] Vehicle 100 may include a sensing system 110, a computing platform 120, and a display device 130. The sensing system 110 may include one or more sensors for sensing information about the environment surrounding the vehicle 100. For example, the sensing system 110 may include a positioning system, which may be a Global Positioning System (GPS), a BeiDou Navigation Satellite System, or another positioning system. As another example, the sensing system 110 may include one or more of an inertial measurement unit (IMU), an accelerometer, a lidar, millimeter-wave radar, ultrasonic radar, and a camera device. For instance, the accelerometer may include a sensor for detecting acceleration signals from the air suspension system, or it may include a sensor for detecting ESC acceleration signals.

[0086] Some or all of the functions of vehicle 100 can be controlled by computing platform 120. Computing platform 120 may include one or more processors, such as processors 121 to 12n (n being a positive integer). A processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction read and execute capabilities, such as a central processing unit (CPU), microprocessor, graphics processing unit (GPU) (which can be understood as a type of microprocessor), or digital signal processor (DSP). In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as a field-programmable gate array (FPGA). In reconfigurable hardware circuits, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, the processor can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), deep learning processing unit (DPU), etc. In addition, the computing platform 120 may also include a memory for storing instructions. Some or all of the processors 121 to 12n can call the instructions in the memory to implement the corresponding functions.

[0087] The in-cabin display devices 130 are mainly divided into two categories: the first is the in-vehicle display screen; the second is the projection display screen, such as the head-up display (HUD). An in-vehicle display screen is a physical display screen and an important component of the in-vehicle infotainment system. Multiple displays can be installed in the cabin, such as the digital instrument cluster display, the central control screen, the display screen in front of the front passenger (also known as the front-seat passenger), the display screen in front of the left rear passenger, the display screen in front of the right rear passenger, and even the car window can be used as a display screen. A head-up display, also known as a head-up display system, is mainly used to display driving information such as speed and navigation on a display device in front of the driver (such as the windshield). This reduces the driver's eye-shift time, avoids pupil changes caused by eye-shifting, and improves driving safety and comfort. Examples of HUDs include combiner-HUD (C-HUD) systems, windshield-HUD (W-HUD) systems, and augmented reality HUD (AR-HUD) systems. It should be understood that HUDs can also evolve into other types of systems as technology progresses, and this application does not limit them.

[0088] The above description of the display device 130 uses an in-vehicle display screen and a projection display screen as examples, but the embodiments of this application are not limited thereto. For example, the display device 130 can also be a light display screen or a projection screen.

[0089] The vehicles involved in this application can include road vehicles, water vehicles, air vehicles, industrial equipment, agricultural equipment, or entertainment equipment. For example, vehicles can include driverless vehicles. The term "vehicle" is used broadly and can refer to various types of vehicles, such as transportation vehicles (e.g., commercial vehicles, passenger cars, motorcycles, flying cars, trains), industrial vehicles (e.g., forklifts, trailers, tractors), engineering vehicles (e.g., excavators, bulldozers, cranes), agricultural equipment (e.g., lawnmowers, harvesters), amusement equipment, and toy vehicles. This application does not specifically limit the type of vehicle. For ease of description, this application uses an autonomous vehicle as an example for detailed explanation.

[0090] 3D point cloud processing technology plays a crucial role in enabling autonomous vehicles to perform functions such as perception, map creation, and localization. However, during actual vehicle operation, the point cloud data collected by vehicle sensors is typically sparse. In 3D point cloud processing, each frame of point cloud data usually needs to be voxelized before feature extraction is performed on the voxels corresponding to each point in 3D space. However, for sparse point cloud data, after voxelization, the proportion of invalid voxels (i.e., voxels not corresponding to points in the point cloud data) is relatively high. When using traditional sliding window algorithms for convolution calculations, a large number of invalid voxels participate in the convolution calculation, resulting in numerous redundant computational operations. To reduce the waste of memory and computing resources, improve the feature extraction capability of point cloud data, and increase the efficiency of point cloud processing algorithms, those skilled in the art have proposed a computational method for processing point cloud data based on 3D sparse convolution, which has become a commonly used algorithm in point cloud object detection neural networks.

[0091] For ease of understanding, the following is an introduction to the 3D sparse convolution technique.

[0092] The principle of 3D sparse convolution is mainly based on data structures such as hash tables. By recording the positions of non-zero elements (i.e., activated input points) and performing convolution calculations only on these elements and their neighborhoods, unnecessary computation is greatly reduced, thus improving computational efficiency. The algorithm flow is as follows:

[0093] Figure 2 is a flowchart of a three-dimensional sparse convolution algorithm 200.

[0094] It should be understood that the algorithm begins after voxelizing the entire three-dimensional space.

[0095] S210: Calculate the rule table based on the coordinates of the input valid voxels.

[0096] The rule table records the voxel location information (e.g., voxel coordinates) corresponding to the features extracted from the valid voxels of the input (referred to as input features), which is the content of the gather table. The rule table also records the voxel location information (e.g., voxel coordinates) corresponding to the output features obtained by processing the input features based on the convolution kernel and used to write back the output results, which is the content of the scatter table.

[0097] It should be understood that the purpose of establishing the above rule table is to enable quick access to the rule table to obtain information from the gather and scatter tables.

[0098] S220: Based on the position information k_i of multiple convolutional kernels and the rule table, read the input feature F_k corresponding to the position information k_i from the gather table.

[0099] The location information k_i can include the voxel coordinates of the kernel center point and the size of the kernel. It should be understood that for each effective voxel, the region covered by the kernel (also called a filter or receptive field) needs to be determined. This region is typically a three-dimensional cube centered on the effective voxel, the size of which is determined by the size of the kernel.

[0100] S230: Perform a convolution operation between the input feature f_k and the corresponding convolution kernel to determine the calculation result r_k.

[0101] S240: Based on the calculation result r_k and the rule table, determine the position of the calculation result r_k in the scatter table, and then write the calculation result r_k back to the output feature map.

[0102] In this table, the position of the calculated result r_k in the scatter table corresponds to the position of the input feature f_k in the gather table. For example, if the input feature f_k is located in the second row of the gather table, then the calculated result r_k will also be located in the second row of the scatter table. It should be understood that the same row in both the gather and scatter tables corresponds to the same effective voxel coordinates.

[0103] For table creation operations in S210, the method for building rule tables using GPU is as follows:

[0104] For each valid input voxel, taking the first valid voxel as an example, when traversing to the first valid voxel (in the rule table, the coordinates of the first valid voxel correspond to an index number, which is also called the index number of the input feature), it is necessary to determine the region of the convolution kernel containing this first valid voxel, thereby determining a set of valid output voxel coordinate index numbers (which are also called the index numbers of the output feature), and obtaining a series of coordinate offset values ​​of the convolution kernel participating in the convolution calculation with the first valid voxel (the voxel coordinates corresponding to the convolution kernel after offset). Then, the gather entry in the rule table corresponding to the coordinate offset value is set to the index number corresponding to the first valid voxel coordinate, and the scatter entry in the rule table corresponding to the coordinate offset value is set to the index number of the output voxel coordinate.

[0105] It should be understood that during the process of establishing the rule table, when calculating the output coordinates of effective voxels by traversing the convolution kernel positions, it is necessary to query whether the center position of the convolution kernel is an effective voxel. If there is no effective voxel, the query result is the effective output position, so as to establish the following correspondence: the correspondence between the information of effective voxel coordinates in the gather table, the position information corresponding to the convolution kernel, and the effective output coordinates in the scatter table.

[0106] So, if the number of effective voxels is N and the volume of the convolution kernel is K (for example, if the length, width, and height of the convolution kernel are k, then K = k...), then... 3 In the case of a sparse convolution algorithm, the process of building the rule table requires N×K table lookup operations. This large number of lookup operations results in significant overhead during the table building process, leading to a heavy processor load and limiting the processor's computational performance for sparse convolution on consecutive frame point clouds. Especially when the processor is a single-instruction multiple-data (SIMD) architecture, the computing platform limits the data read per operation to at least 32 bytes. Using this instruction to access memory and look up tables will result in a large amount of invalid data being read, accompanied by a significant amount of redundant computation.

[0107] In view of this, embodiments of this application propose a data processing method and apparatus, which determines whether to use a differential method to create a rule table by comparing the similarity of point cloud data in two adjacent frames. That is, the rule table corresponding to the point cloud data in the later frame can reuse the rule table corresponding to the point cloud data in the previous frame, and add voxel-related information corresponding to the point cloud, such as coordinates, input features and output features, on the basis of the rule table corresponding to the point cloud data in the previous frame, or delete voxel-related information corresponding to the point cloud that has become invalid in the point cloud data in the later frame.

[0108] In some possible embodiments, the data processing method proposed in this application can be applied to autonomous driving solutions or functional scenarios involving 3D sparse convolution in intelligent robot pathfinding, such as point cloud object detection. Accordingly, the method proposed in this application can be deployed on edge devices with computing capabilities, such as intelligent driving computing platforms and intelligent robots.

[0109] As can be seen from the foregoing description, three-dimensional sparse convolution mainly involves the steps of establishing a rule table (related to the gather table and scatter table), reading data based on the gather table, performing convolution operations, and accumulating and writing data based on the scatter table. The embodiments of this application mainly improve the process of establishing the rule table and the subsequent update mechanism of the gather table and scatter table.

[0110] Figure 3 is a flowchart illustrating a data processing method 300 proposed in an embodiment of this application.

[0111] S310: Obtain the first point cloud data, which includes N points and N input features corresponding to the N points, where N is greater than 0.

[0112] In some possible embodiments, the positions of the aforementioned N points can be represented by the coordinates of N voxels. The aforementioned first point cloud data is used to construct a first relation table and a second relation table. The first relation table represents the correspondence between N voxels and N input features; the second relation table represents the correspondence between N voxels and N output features. The N output features can be obtained by performing sparse convolution operations on the N input features, where N is greater than 0.

[0113] It should be noted that the action of "constructing the first relation table and the second relation table" proposed in this application embodiment can refer to the fact that the first relation table and the second relation table do not exist before the action is performed, or that the first relation table and the second relation table have been deleted. Therefore, for the first relation table and the second relation table, the process of performing this action can refer to the process of creating these two relation tables from scratch. Of course, the above action can also refer to the fact that the first relation table and the second relation table exist based on the previous frame point cloud data before the action is performed, and the meaning of "construction" can be the meaning of "update", that is, the process of performing this action can refer to the process of updating or maintaining these two relation tables.

[0114] In some possible embodiments, when multiple convolution kernels participate in the convolution operation on N input features, the aforementioned first and second relation tables can also be used to indicate the convolution kernels corresponding to the N voxels, which are used to perform the aforementioned convolution operation on the input features corresponding to the voxels to determine the corresponding output features. Further, the aforementioned first relation table can be used to indicate the convolution kernels corresponding to the N input features, and correspondingly, the aforementioned second relation table can be used to indicate the convolution kernels corresponding to the N output features. Of course, even when only one convolution kernel participates in the convolution operation on the N input features, the first relation table can reflect the correspondence between the N input features and the convolution kernel, and the second relation table can reflect the correspondence between the N output features and the convolution kernel.

[0115] In some possible embodiments, when multiple convolutional kernels participate in the sparse convolution operation on N input features, the relational table involved in method 300 may further include a third relational table. This third relational table is used to indicate at least one voxel coordinate corresponding to each convolutional kernel, where the at least one voxel coordinate belongs to the aforementioned N voxels, and all voxels corresponding to the multiple convolutional kernels include the aforementioned N voxels. Based on the first relational table, the second relational table, and the third relational table, the correspondence between N voxels and the multiple convolutional kernels can be determined. Furthermore, the correspondence between N input features or N output features and the multiple convolutional kernels can also be determined.

[0116] In some possible embodiments, the first relation table can be the gather table mentioned above, and the second relation table can be the scatter table mentioned above. Therefore, the first and second relation tables are the basis for sparse convolution of the second point cloud data.

[0117] In some possible embodiments, the correspondence between N voxels and N input features in the first relation table can be the correspondence between the coordinates of N voxels and the index numbers of the N input features; similarly, the correspondence between N voxels and N output features in the second relation table can be the correspondence between the coordinates of N voxels and the index numbers of the N output features.

[0118] S320: Obtain the second point cloud data, which includes M points and M input features corresponding to the M points, where M is greater than 0.

[0119] Similarly, the positions of the above M points are represented by the coordinates of the M voxels.

[0120] In some possible embodiments, the first point cloud data and the second point cloud data may be two adjacent frames of point cloud data in terms of time sequence, or there may be at least one frame of other point cloud data between the first point cloud data and the second point cloud data.

[0121] S330: Perform sparse convolution operation on the second point cloud data based on the difference between the first point cloud data and the second point cloud data.

[0122] In some possible embodiments, the above-mentioned sparse convolution operation can be a three-dimensional sparse convolution operation. However, with technological iteration, the above-mentioned sparse convolution operation can also include operations in other dimensions. For example, in the process of processing point cloud data, spatial information can be planarized, and the point cloud data can be processed in two-dimensional space. In this case, the above-mentioned sparse convolution operation can be a two-dimensional sparse convolution operation.

[0123] In some possible embodiments, the difference between the first point cloud data and the second point cloud data refers to the points added and / or the points that are invalidated in the second point cloud data compared to the first point cloud data.

[0124] It should be understood that in existing 3D sparse convolution algorithms, whenever a frame of point cloud data is input, the rule table needs to be reconstructed, and then the gather table and scatter table need to be reconstructed, which consumes a lot of computing resources.

[0125] Based on the above technical solution, the first relation table (which could be the gather table) and the second relation table (which could be the scatter table) can be maintained by using the difference information between the point cloud data of the later frame and the point cloud data of the earlier frame, i.e., by reusing historical data from the tables corresponding to the point cloud data of the earlier frame. This enables the differential table building operation. Subsequently, sparse convolution operations can be performed on the point cloud data of the later frame based on the updated gather and scatter tables. This will save a significant amount of computational overhead caused by table building operations and help improve the sparse convolution operation performance of the processor.

[0126] In some possible embodiments, the above-mentioned S330 can be implemented as follows: update the first relation table and the second relation table according to the difference between the first point cloud data and the second point cloud data; perform sparse convolution operation on the second point cloud data according to the updated first relation table and the second relation table.

[0127] In some possible embodiments, the number of entries in the first relation table and the number of entries in the second relation table are pre-configured.

[0128] In some possible embodiments, the term "table entry" in this application refers to a cell or entry in a table (grid) used to write specific information or data.

[0129] In some possible embodiments, updating the first relation table and the second relation table may include modifying the content of the first relation table and / or the second relation table, or it may include not modifying the content of the first relation table and / or the second relation table. In other words, if the content of the first relation table and / or the second relation table does not change before and after the update, it can also be considered that the first relation table and / or the second relation table has been updated, except that the updated content is empty.

[0130] In some possible embodiments, the number of convolution kernels used to perform the above-described sparse convolution operation is Y. When Y is greater than 1, the first relation table may include Y first relation sub-tables, which correspond to Y convolution kernels; correspondingly, the second relation table may include Y second relation sub-tables, which correspond to Y convolution kernels. The Y first relation sub-tables each contain N entries with N index numbers of input features, and the Y second relation sub-tables each contain N entries with N index numbers of output features.

[0131] In some possible embodiments, the first and second relation tables can be updated using the following methods: updating the first and second relation tables based on the first newly added voxels, and / or updating the first relation table based on the first failed voxels. The first newly added voxel is a voxel among M voxels that does not belong to N voxels; in other words, it is a voxel added compared to the first point cloud data. The first failed voxel is a voxel among N voxels that does not belong to M voxels; in other words, it is a voxel reduced (or failed) compared to the first point cloud data. It should be noted that the second point cloud data can have multiple newly added points compared to the first point cloud data. Correspondingly, the voxelized second point cloud data can also have multiple newly added voxels compared to the first point cloud data, such as the aforementioned first newly added voxel and second newly added voxel. For ease of description, the following explanation will focus on the first newly added voxel as an example to elaborate on the updating of the first and second relation tables; the same applies to the first failed voxel.

[0132] In some possible embodiments, when the difference between two frames of point cloud data contains only failed voxels, the first relation table can be updated based on the failed voxels, and the second relation table does not need to be updated; when the difference between two frames of point cloud data contains only newly added voxels, the first and second relation tables can be updated only based on the newly added voxels; when the second point cloud data is completely consistent with the first point cloud data, the first and second relation tables corresponding to the first point cloud data can be directly reused.

[0133] In some possible embodiments, the operation of updating the first relation table can be further implemented by adding the index number associated with the first newly added input feature to the first relation table, and / or invalidating the contents of the table entries associated with the first invalid voxel in the first relation table.

[0134] Alternatively, the operation of updating the second relation table can be further implemented by adding the index number related to the first newly added output feature to the second relation table.

[0135] Based on the above technical solutions, the first relation table is updated by reusing the first relation table corresponding to the previous frame point cloud data and supplementing the corresponding positions of the voxels where the points in the subsequent frame point cloud data are newly added compared to the previous frame point cloud data with the corresponding input features in the first relation table, and / or deleting the corresponding input features of the voxels where the points are reduced from the first relation table. Similarly, the second relation table is updated by reusing the second relation table corresponding to the previous frame point cloud data and adding the corresponding positions of the voxels where the points in the subsequent frame point cloud data are newly added compared to the previous frame point cloud data with the corresponding output features in the second relation table. This update method is more cost-effective than rebuilding the relation table, helps improve the efficiency of relation table building, and helps improve the processor's performance in sparse convolution operations on consecutive frame point cloud data.

[0136] In some possible embodiments, the triggering of the above method 300 requires the fulfillment of specific preconditions, and the determination method is as follows:

[0137] If the number of newly added and / or invalid points in the second point cloud data is less than or equal to the first point cloud data, S330 of the above method 300 can be triggered. That is, the above judgment operation needs to be completed before S330. Therefore, it can be seen that the judgment method is completed during the execution of method 300.

[0138] It should be understood that after voxelizing the first point cloud data and the second point cloud data, the points added and / or invalidated in the second point cloud data compared to the first point cloud data can be understood as the first newly added voxels and the first invalidated voxels proposed in the aforementioned embodiments.

[0139] Furthermore, given the limited processor performance, the above-described method 300 and the above-described judgment method may not be able to achieve parallel processing. Therefore, this application embodiment proposes another judgment method that can be performed before executing method 300.

[0140] Figure 4 is a flowchart illustrating another data processing method 400 proposed in an embodiment of this application. This method 400 is used to determine whether to trigger the execution of the above-described method 300, and includes the following steps:

[0141] S410: Obtain the third point cloud data, which includes J voxels.

[0142] S420: Obtain the fourth point cloud data, which includes K point clouds.

[0143] S430: Based on the third point cloud data and the fourth point cloud data, determine the second newly added voxel and / or the second failed voxel.

[0144] Among them, the second newly added voxel is a voxel in voxel K that does not belong to voxel J, and the second ineffective voxel is a voxel in voxel J that does not belong to voxel K.

[0145] S440: Based on the size relationship between the second newly added voxel and / or the second failed voxel and the first threshold, determine whether to trigger the construction of the first relation table and the second relation table by differential table building (i.e., trigger the above method 300).

[0146] In some possible embodiments, the above method 300 is triggered when the number of the second newly added voxels is determined to be less than or equal to the first threshold, or when the number of the second failed voxels is determined to be less than or equal to the first threshold, or when the sum of the number of the second newly added voxels and the number of the second failed voxels is determined to be less than or equal to the first threshold.

[0147] In some possible embodiments, x frames of point cloud data can be continuously acquired, and the number of changes in effective voxels between two adjacent frames of point cloud data in the x frames of point cloud data can be counted, m_i. The maximum value max(m_1,…,m_x) of the number of changes in effective voxels (including newly added voxels and invalid voxels) between each frame of point cloud data can be determined. If max(m_1,…,m_x) is less than or equal to the first threshold mentioned above, the execution of the above method 300 is triggered, that is, differential table building is adopted; otherwise, the above method 300 is not adopted for table building for subsequent y frames of point cloud data.

[0148] In some possible embodiments, the first threshold mentioned above may be pre-configured.

[0149] Based on the above technical solution, it is possible to enter the data processing method flow proposed in this application under appropriate circumstances, so as to ensure that when updating the first relation table and the second relation table based on the difference between the point cloud data of the later frame and the point cloud data of the previous frame, the previously determined first relation table and the second relation table can be reused more, so that the computational overhead of table building is kept at a low level.

[0150] Considering that comparing two frames of point cloud data also requires corresponding computing resources, when the processor's computing resources are limited, it is possible to first determine whether certain preconditions are met before proceeding to the judgment process used to trigger method 300.

[0151] In some possible embodiments, assuming that the first point cloud data and the second point cloud data are acquired by sensors mounted on the vehicle, then before the above S410, it can also be determined whether the vehicle speed is less than or equal to the second threshold.

[0152] When the vehicle speed is less than or equal to the second threshold, the judgment process for triggering method 300 described above can be entered; otherwise, the judgment process for triggering method 300 described above will not be entered, and the first relation table and the second relation table corresponding to the point cloud data acquired at this time will be reconstructed.

[0153] In some possible embodiments, the vehicle's speed can also be obtained through onboard sensors.

[0154] In some possible embodiments, the second threshold described above may be pre-configured.

[0155] In some possible embodiments, it is assumed that the data processing method proposed in this application is applied to other fields, such as intelligent robots, and the motion speed of the mounted object can be obtained through the corresponding speed sensor, thereby determining whether to enter the judgment process for triggering method 300.

[0156] In some possible embodiments, the data processing method proposed in this application is applied to a smart car scenario. It can determine whether to enter the judgment process for triggering method 300 by judging the current driving state of the smart car and the activated intelligent driving functions. For example, if the vehicle is in a non-parked state and the TJA function in the vehicle's intelligent driving functions is activated, the judgment process for triggering method 300 can be entered; or, if the vehicle is in a parked state and the Intelligent Parking Assist (APA), Valet Parking Assist (AVP), or Valet Parking Driver (VPD) functions in the vehicle's intelligent driving functions are activated, the judgment process for triggering method 300 can be entered.

[0157] It should be understood that the above-mentioned judgment process for determining whether to enter the above-mentioned triggering method 300 is based on vehicle speed, whether the vehicle is parked, and the vehicle's intelligent driving function. Essentially, it is based on the vehicle's driving status or the traffic conditions around the vehicle to determine whether to enter the above-mentioned triggering method 300. When the vehicle speed is low, the traffic conditions around the vehicle are relatively simple, and the environmental factors change slowly, it means that the above-mentioned triggering method 300 can be entered.

[0158] In some possible embodiments, the execution of the above method 300 can also be directly triggered when the vehicle speed is determined to be less than or equal to the second threshold, or when the vehicle activates the TJA function, APA function, AVP function, or VPD function.

[0159] Based on the above technical solution, under certain preconditions (such as vehicle speed, vehicle driving status, or surrounding traffic conditions), the first and second relation tables of the previous frame point cloud data are adaptively activated and reused. The first and second relation tables are then updated based on the differences between the two frames of point cloud data. This avoids comparing every two acquired frames of point cloud data, thus reducing the computational burden on processors with limited resources. This approach meets the algorithm requirements of different hardware scenarios and increases the robustness of the solution.

[0160] As can be seen from the foregoing embodiments, both the gather table and the scatter table have a mapping relationship with the rule table. Therefore, in this embodiment, it is also necessary to pre-build a reference table (which can be understood as the rule table mentioned above) and initialize the reference table for adding corresponding information later to establish a mapping relationship with the first relation table and the second relation table (which can be the gather table and the scatter table mentioned above) to ensure the normal execution of sparse convolution operations.

[0161] Figure 5 is a flowchart illustrating another data processing method 500 proposed in an embodiment of this application. This method 500 is used to initialize a reference table and write relevant information into the reference table. The method 500 includes the following steps:

[0162] S510: Create a reference table.

[0163] The reference table includes a first column and a second column. The first column contains the coordinates of L voxels that constitute the voxel space, and the second column is used to write the input features corresponding to the L voxels, where L is greater than or equal to M and N.

[0164] In some possible embodiments, the reference table described above can be created based on all valid voxels in a multi-frame point cloud, that is, L voxels in the reference table correspond to all valid voxels appearing in the multi-frame point cloud.

[0165] S520: Set the entries in the second column to invalid values.

[0166] In some possible embodiments, the invalid value mentioned above can be a negative number, or null, or a value greater than the number of items in the second list.

[0167] S530: Based on the first point cloud data, write the index numbers of the N input features into the N entries in the second column that correspond to the coordinates of the N voxels.

[0168] It should be understood that, after the above processing, the L entries in the first column of the above reference table correspond to the coordinates of L voxels, and N voxels belong to L voxels. The N entries in the second column of the above reference table correspond to the index numbers of N input features, and the N entries correspond to N voxels. The remaining (LN) entries in the second column contain invalid values.

[0169] In some possible embodiments, the reference table described above may consist of L key-value pairs, wherein the key in a key-value pair is a voxel coordinate, and the value is the index number of the input feature corresponding to the voxel coordinate; and the key and value in a key-value pair are located in the same row of the reference table.

[0170] It should be understood that the writing positions of the index numbers of the N input features in the first relation table correspond to the writing positions of the coordinates of the N voxels in the reference table, and the writing positions of the index numbers of the N output features in the second relation table correspond to the writing positions of the coordinates of the N voxels in the reference table.

[0171] Based on the above technical solution, the necessary prerequisites for the differential table building operation in method 300 can be provided to ensure the smooth execution of the differential table building operation, and it also helps to reduce the complexity of updating the first relation table and the second relation table in the subsequent differential table building operation. Theoretically, when updating the first relation table and the second relation table based on the above reference table, the computational complexity is O(1), and when updating the first relation table and the second relation table without updating based on the above reference table, the computational complexity is O(N), where N is greater than 1.

[0172] After establishing the above reference table, it is also necessary to establish the first relationship table and the second relationship table based on the reference table.

[0173] Figure 6 is a flowchart illustrating another data processing method 600 proposed in an embodiment of this application. Method 600 is used to initialize a first relation table and a second relation table, and to write relevant information into these two tables. Method 600 includes the following steps:

[0174] S610: Create the first relation table and the second relation table.

[0175] For details on the basic structure of the first and second relation tables, please refer to the corresponding embodiments described above, which will not be repeated here.

[0176] In some possible embodiments, the number of entries in the first relation table and the number of entries in the second relation table are both greater than the product of N and Y. The first relation table includes Y first relation sub-tables, and the Y first relation sub-tables correspond to Y convolution kernels. The second relation table includes Y second relation sub-tables, and the Y second relation sub-tables correspond to Y convolution kernels. The Y convolution kernels are used to perform convolution operations on the input features corresponding to the effective voxels to determine the corresponding output features.

[0177] For example, assuming the convolution kernel is a 3×3×3 cube, then 27 convolution kernels are needed in this 3D sparse convolution operation, corresponding to 27 of the first and second relation sub-tables mentioned above. When the convolution kernel has other geometric configurations, the number of convolution kernels, as well as the number of the first and second relation sub-tables, will be adjusted adaptively.

[0178] In some possible embodiments, the number of entries in the first relation table and the number of entries in the second relation table can be pre-configured to be greater than the product of N and Y, or determined based on historical data, which is used to indicate the number of entries that enable sparse convolution operations to proceed smoothly in scenarios with different point cloud distributions.

[0179] S620: Set the entries of the first relation table and the second relation table to invalid values.

[0180] It should be understood that although the first and second relation tables mentioned above can consist of only single columns, each column in the table has a mapping relationship with the corresponding voxel coordinates in the reference table. Therefore, when a certain entry in the first or second relation table is obtained, not only can the content within the entry be obtained, but also the voxel coordinates in the reference table that have a mapping relationship (or correspondence) with that entry can be obtained.

[0181] S630: According to the reference table, write N index numbers of input features into the entries of Y first relation sub-tables respectively, and write N index numbers of output features into the entries of Y second relation sub-tables respectively.

[0182] For example, assuming the voxel coordinates corresponding to the first voxel in N voxels are V1 = (x1, y1, z1), then when V1 is the center point of the convolution kernel (which is used to perform convolution operations on the input features corresponding to V1), the voxel coordinates within Y receptive fields can be obtained. At this time, the Y voxel coordinates can be searched in the reference table, and the search result is the index number of the Y input features corresponding to the Y voxel coordinates. Then, the obtained index numbers of the Y input features are written into the first row of the Y first relation sub-tables of the first relation table, and the output value position of the Y voxel coordinates all correspond to the aforementioned first voxel. Then, the index numbers of the Y input features are written into the first row of the Y second relation sub-tables of the second relation table.

[0183] Based on the above technical solution, the necessary prerequisites for the differential table building operation in method 300 can be provided to ensure the smooth execution of the differential table building operation.

[0184] In some possible embodiments, after determining the difference between the first point cloud data and the second point cloud data, the reference table can be updated by the following operations: updating the reference table according to the first failed voxel and / or the first newly added voxel, wherein in the updated reference table, the table entries corresponding to the coordinates and input features of the first failed voxel are invalid values, and / or, in the updated reference table, one table entry in the first column contains the coordinates of the first newly added voxel, and the table entry in the second column corresponding to the coordinates of the first newly added voxel contains the index number of the first newly added input feature.

[0185] In some possible embodiments, during the process of updating the reference table, the coordinates corresponding to the first failed voxel and the entry containing the input feature in the reference table can be set to invalid values, the coordinates of the first newly added voxel can be written in the first column, and the index number of the first newly added input feature can be written in the entry corresponding to the coordinates of the first newly added voxel in the second column, so as to update the reference table.

[0186] In some possible embodiments, setting the coordinates corresponding to the first failed voxel in the reference table to invalid values ​​can be done by deleting the coordinates from the reference table, setting them to null or a value greater than the number of entries, or adding a marker to the entry containing the coordinates to indicate that the voxel corresponding to those coordinates has failed. The same principle applies to setting the entry containing the input feature corresponding to the first failed voxel in the reference table to invalid values.

[0187] Based on the above technical solution, the reference table is updated through differential table building. During the update process, the reference table constructed based on the previous frame point cloud data is reused, so that before performing sparse convolution operation on the subsequent frame point cloud data, the reference table corresponding to the subsequent frame point cloud data can be obtained with a small computational overhead for use in subsequent sparse convolution operation.

[0188] In some possible embodiments, after updating the above reference tables, the first relation table and the second relation table can be updated by the following operations:

[0189] For updating the first relation table:

[0190] In the case where the difference between the second point cloud data and the first point cloud data is determined to include the first newly added voxel:

[0191] According to the updated reference table, the index numbers of the input features corresponding to the Y first target voxels are written into the (N+1)th row of the Y first relation sub-tables. The Y first target voxels are voxels within the receptive field of the convolution kernel centered on the first newly added voxel.

[0192] When the index number of the input feature corresponding to the second target voxel in the updated reference table is a valid value, the row number of the second target voxel in the updated reference table is determined to be the row number of the first target voxel. The second target voxel corresponds to the center of the first convolutional kernel within the receptive field of Y convolutional kernels, including the first newly added voxel.

[0193] Write a valid value into the table entry corresponding to the first target row number of the first relation sub-table corresponding to the first convolution kernel.

[0194] For example, taking the aforementioned Y first target voxels as an example, assuming that each first relation sub-table contains N rows of content, then the index numbers of the input features corresponding to the Y first target voxels can be filled into the (N+1)th row of the Y first relation sub-tables one by one. Then, determine the voxel coordinates of the center points of the (Y-1) convolutional kernels within the receptive field that include the aforementioned newly added voxel; select I voxel coordinates from the (Y-1) center point voxel coordinates, and then write the valid index numbers of the input features corresponding to the I center point voxel coordinates into at least one first relation sub-table associated with the aforementioned newly added voxel, wherein the newly added voxel does not belong to the voxel coordinates of the (Y-1) center points, and I is less than or equal to (Y-1).

[0195] Taking the center point V2 (i.e., voxel V2) among the above (Y-1) center points as an example, if the index number of the input feature corresponding to voxel V2 can be found as a valid value in the updated reference table, assuming that voxel V2 corresponds to the Kth row in the Y first relation sub-tables, then the index number of the input feature corresponding to voxel V2 is written in the Kth row of at least one first relation sub-table associated with the above-mentioned first newly added voxel; where, if the Kth row already contains relevant content, the original content can be overwritten.

[0196] In the case where the difference between the second point cloud data and the first point cloud data is determined to include the first failed voxel:

[0197] Based on the updated reference table, the row number corresponding to the first failure voxel in the updated reference table is determined to be the row number of the second target.

[0198] Write invalid values ​​into the table entries corresponding to the second target row numbers of the Y first relation sub-tables;

[0199] When the index number of the input feature corresponding to the third target voxel in the updated reference table is a valid value, the row number of the third target voxel in the updated reference table is determined to be the row number of the third target voxel. The third target voxel corresponds to the center of the second convolutional kernel within the receptive field of Y convolutional kernels, including the first failed voxel.

[0200] Write an invalid value into the table entry corresponding to the third target row number of the first relational subtable corresponding to the second convolution kernel.

[0201] For example, taking the first invalid voxel corresponding to the Qth row in the Y first relation sub-tables as an example, the content of the Qth row in the Y first relation sub-tables can be set to invalid values. Then, determine the voxel coordinates of the center points of the (Y-1) convolutional kernels that include the first invalid voxel within the receptive field; filter the voxel coordinates of H center points from the voxel coordinates of the (Y-1) center points, and the corresponding input features of the voxel coordinates of the H center points in the updated reference table are the valid index numbers, where H is less than or equal to (Y-1), and set the valid index numbers corresponding to the voxel coordinates of the H center points in the H first relation sub-tables to invalid values.

[0202] Taking the center point V3 (i.e., voxel V3) among the above (Y-1) center points as an example, if the index number of the input feature corresponding to voxel V3 can be found as a valid value in the updated reference table, assuming that voxel V3 corresponds to the Qth row in the Y first relation sub-tables, then an invalid value is written in the Qth row of at least one first relation sub-table associated with the first invalid voxel; where, if the Kth row already contains relevant content, the original content can be overwritten.

[0203] After updating the first relation table, proceed to update the second relation table as follows:

[0204] In the case where the difference between the second point cloud data and the first point cloud data is determined to include the first newly added voxel:

[0205] Based on the table entries in the first relation table that contain the index numbers of the input features corresponding to the Y first target voxels, the index numbers of the output features corresponding to the Y first target voxels are written into the (N+1)th row of the Y second relation sub-tables.

[0206] For example, since the first relation table has been updated before updating the second relation table, such as filling the index numbers of the input features corresponding to the Y first target voxels into the (N+1)th row of the Y first relation sub-tables one by one, then for updating the second relation table, the (N+1)th row of the Y second relation sub-tables can be set to a valid value.

[0207] Based on the above technical solutions, by reusing the first relation table corresponding to the previous frame point cloud data and supplementing the input features corresponding to the newly added voxels in the voxelized subsequent frame point cloud data to the corresponding positions in the first relation table, and / or deleting the input features corresponding to the reduced voxels from the first relation table, the first relation table is updated. Similarly, by reusing the second relation table corresponding to the previous frame point cloud data and adding the output features corresponding to the voxels of the points added in the subsequent frame point cloud data to the corresponding positions in the second relation table, the second relation table is updated. This update method is more cost-effective than rebuilding the relation table, helps improve the construction efficiency of the relation table, and helps improve the sparse convolution operation performance of the processor.

[0208] To facilitate understanding, the following example will be used to provide a detailed description of the data processing method proposed in this application, involving operations from table creation to table updating.

[0209] The differential table building method based on three-dimensional sparse convolution in a multi-frame point cloud continuous monitoring scenario proposed in this application embodiment can include the following three steps: 1. Reference table initialization; 2. Adaptive start of differential algorithm; 3. Differential update of relevant tables, including the reference table, based on differential algorithm.

[0210] Step 1 above can include the following process:

[0211] Figure 7 is a schematic diagram of an initialized reference table, a first relation table, and a second relation table proposed in an embodiment of this application.

[0212] S101: Pre-allocation reference table.

[0213] In some possible embodiments, the length of the reference table can be determined as x_max × y_max × z_max based on the size of the maximum voxel space (denoted as x_max, y_max, z_max) pre-configured by the user. Alternatively, the length of the reference table can be directly configured by the user, or the length of the reference table can be adaptively adjusted according to the current processing power of the processor.

[0214] Referring to Figure 7, the reference table may include two columns, and the number of rows corresponding to each column can be x_max×y_max×z_max or other configured lengths.

[0215] In some possible embodiments, each row of the reference table may correspond to two entries. Accordingly, each row of the reference table may correspond to a key-value pair, where the key may be used to represent the voxel coordinate (corresponding to the first column of the reference table). If the voxel coordinate corresponds to a point in the point cloud, then the value of the key-value pair may be used to represent the index number of the input feature corresponding to the point (corresponding to the second column of the reference table). The index number of the input feature may also be used to represent the index position of the voxel in the input feature.

[0216] S103: Write the coordinates of all voxels in the voxel space to the first column of the reference table, and set the second column of the reference table to an invalid value.

[0217] Among them, the above invalid values ​​can be negative numbers, null, or values ​​greater than x_max×y_max×z_max.

[0218] S105: Pre-allocate the first relation table and the second relation table.

[0219] In some possible implementations, the lengths of the first and second relation tables can be determined as e_max+1 based on the user-preconfigured maximum number of entries e_max. The value of e_max can also be determined through pre-experimentation to find reasonable values ​​for different driving scenarios or vehicle speeds.

[0220] In some possible embodiments, as shown in Figure 7, the first relation table and the second relation table may include Y sub-tables, and each sub-table may correspond to a convolution kernel. For example, if 27 convolution kernels are needed to participate in the three-dimensional sparse convolution operation, then the first relation table and the second relation table may each include 27 sub-tables.

[0221] It should be understood that different driving scenarios or different vehicle speeds will affect the magnitude of changes in the collected point cloud data between frames. Therefore, different driving scenarios can be constructed for the vehicle in the pre-experiment, and experiments can be conducted at different vehicle speeds in the same scenario to generate historical data. This historical data includes the mapping relationship between driving scenarios, vehicle speeds, and the value of e_max.

[0222] However, regardless of the driving scenario or speed of the vehicle, it is necessary to ensure that e_max > N×Y, where N is the number of voxels corresponding to the points in the point cloud data, and Y is the number of convolution kernels used in the 3D sparse convolution operation.

[0223] Referring to Figure 7, both the first and second relational tables include a list of entries. Entries in the first relational table are used to write the index number of the input feature corresponding to a valid voxel (representing a point in the point cloud data). After the index number of the input feature is written into an entry in the first relational table, the entry should correspond to a key-value pair in the reference table, and thus correspond to the voxel coordinates in that key-value pair. Entries in the second relational table are used to write the index number of the output feature obtained after convolving the input feature of the valid voxel with the convolution kernel. After the index number of the output feature is written into an entry in the second relational table, the entry should correspond to the index number (or the entry in the first relational table) of the corresponding input feature, and thus correspond to the voxel coordinates (in the reference table) corresponding to that input feature's index number. The index number of the output feature can also be used to represent the index position of the voxel in the output feature.

[0224] S107: Set the entries of the first relation table and the second relation table to invalid values.

[0225] Among them, the above invalid values ​​can be negative numbers, null, or values ​​greater than x_max×y_max×z_max.

[0226] After initializing the reference table, the first relation table, and the second relation table, the process of updating the reference table, the first relation table, and the second relation table begins. Before starting the differential algorithm, the contents of the reference table, the first relation table, and the second relation table can be updated as a whole based on the point cloud data of each frame obtained.

[0227] For example, when updating the reference table, upon acquiring point cloud data 1, the voxel coordinates corresponding to each point in point cloud data 1 and the index number of the input feature corresponding to each point are determined to identify multiple key-value pairs 1, and these multiple key-value pairs 1 are written to the corresponding entries in the reference table. Upon acquiring point cloud data 2, the data in the current reference table is cleared (or invalidated), the voxel coordinates corresponding to each point in point cloud data 2 and the index number of the input feature corresponding to each point are determined to identify multiple key-value pairs 2, and these multiple key-value pairs 2 are written to the corresponding entries in the reference table.

[0228] The update method for the first and second relation tables is similar. When the point cloud data of the next frame is obtained, the contents written into the first and second relation tables based on the point cloud data of the previous frame need to be cleared (or invalidated). Then, the index number of the input feature or output feature corresponding to each point in the point cloud data of the next frame is written into the corresponding table entry of the first or second relation table.

[0229] Before starting the differential algorithm, it is also necessary to simultaneously determine whether the differential algorithm needs to be started, which is step 2 above.

[0230] Taking the vehicle's corresponding sensors as an example to obtain point cloud data, step 2 above can include the following process:

[0231] S201: Determine whether the current vehicle speed is less than or equal to the speed threshold. If yes, proceed to S203. If no, use a non-differential algorithm to update the reference table, the first relation table, and the second relation table based on the currently acquired point cloud data.

[0232] S203: Determine whether the number of voxel changes between the two subsequently acquired point cloud data frames is less than or equal to the number threshold. If yes, determine to start the differential algorithm and update the reference table, the first relation table, and the second relation table. If no, return to S201 after the time for acquiring y frames of point cloud data has elapsed.

[0233] In some possible embodiments, x frames of point cloud data can be continuously acquired, and the number of changes in effective voxels between two adjacent frames of point cloud data in the x frames of point cloud data can be counted, m_i. The maximum value max(m_1,…,m_x) of the number of changes in effective voxels (including newly added voxels and invalid voxels) between each frame of point cloud data can be determined. If max(m_1,…,m_x) is less than or equal to the first threshold mentioned above, the execution of the above method 300 is triggered, that is, differential table building is adopted; otherwise, the above differential algorithm is not adopted to update the reference table, the first relation table and the second relation table for subsequent y frames of point cloud data.

[0234] Here, y is a positive integer. For y frames of point cloud data, a non-differential algorithm can be used to update the reference table, the first relation table, and the second relation table. The value of y can be adaptively adjusted according to the vehicle's driving environment or driving speed. For example, the smaller the change in the driving environment, the smaller y, or the lower the driving speed, the smaller y, etc.

[0235] After determining to start the differential algorithm, we can proceed to step 3 above, which may include the following process:

[0236] S301: Determine the difference between the valid voxels corresponding to the point cloud data of the current frame and the valid voxels corresponding to the point cloud data of the previous frame, and determine the set of invalid voxels a and the set of newly added voxels b.

[0237] S303: Determine the number of entries e_cur in the first or second relation table required for the point cloud data in this frame. If e_cur ≤ e_max, proceed to S305. If e_cur > e_max, use a non-differential algorithm to update the reference table, the first relation table, and the second relation table.

[0238] Here, e_cur equals the sum of the number of entries in the first or second relation table corresponding to the previous frame's point cloud data, e_pre, and the newly added entry, e_b. The newly added entry, e_b, can be determined by the number of voxels in the newly added voxel set b.

[0239] S305: Update the reference table based on the failed voxel set a and the newly added voxel set b.

[0240] Figure 8 is a schematic diagram of a differential update reference table proposed in an embodiment of this application.

[0241] For failed voxels in failed voxel set a, the coordinates of the failed voxels in the reference table and the index number of the input feature are set to invalid values. For newly added voxels in newly added voxel set b, the coordinates of the newly added voxel and the index number of the input feature corresponding to the newly added voxel are written to the invalidated entries in the reference table. It should be understood that the coordinates of the newly added voxel and the index number of the input feature corresponding to the newly added voxel should be written to two entries in the same row of the reference table. For the remaining entries, the current content of the reference table (i.e., the content of the reference table determined based on the previous frame point cloud data) is reused to realize the operation of updating the reference table based on the difference algorithm.

[0242] Referring to Figure 8, taking voxel (59,31,19) as an example, there is no point corresponding to voxel (59,31,19) in the current frame's point cloud data, while there is a corresponding point in the previous frame's point cloud data. Therefore, voxel (59,31,19) becomes an invalid voxel in the current frame's point cloud data. In the reference table, the coordinate entry corresponding to voxel (59,31,19) is marked as invalid, and the index number of the corresponding input feature is set to an invalid value of -1.

[0243] Taking voxel (198,102,28) as an example, there is no point corresponding to voxel (198,102,28) in the previous frame's point cloud data, but there is a corresponding point in the current frame's point cloud data. Therefore, voxel (198,102,28) is an invalid voxel in the current frame's point cloud data. In the reference table, the coordinates of voxel (198,102,28) and the corresponding input feature index N+1 are written into the invalid or blank entry.

[0244] S307: Update the first relation table and the second relation table based on the receptive field of the convolution kernel, the set of invalid voxels a, and the set of newly added voxels b.

[0245] Figure 9 is a schematic diagram of a differential update of the first relation table proposed in an embodiment of this application.

[0246] Figure 10 is a schematic diagram of a differential update of the second relation table proposed in an embodiment of this application.

[0247] In this embodiment, it is assumed that the volume of the convolution kernel is i×i×i, and the number of convolution kernels is Y, where Y = i×i×i. Accordingly, the first relation table and the second relation table correspond to Y sub-tables, and each sub-table corresponds to one convolution kernel.

[0248] For each failed voxel in the failed voxel set a, determine the Y voxel within the receptive field centered on that failed voxel. Then, look up the updated reference table. In the updated reference table, if the coordinates or index number of the input feature corresponding to the n voxels in the Y voxel set are valid values, where 0 ≤ n ≤ Y, then set the corresponding entry for the n voxels in the first relation table to invalid. The update of the second relation table is not affected by the failed voxel set a.

[0249] For each newly added voxel in the newly added voxel set b, determine k voxels within the receptive field centered on each newly added voxel. Then, look up the updated reference table. In the reference table, if the coordinates or index numbers of the input features corresponding to m voxels in the k voxels are valid values, where 0≤m≤k, then the corresponding entry for the m voxels in the first relation table is set to a valid value, and the corresponding entry for the m voxels in the second relation table is also set to a valid value.

[0250] Referring to Figures 9 and 10, for easier viewing, a lookup table can be added during the process of updating the first relation table and the second relation table. This lookup table also includes Y sub-tables, and each sub-table corresponds to a convolution kernel. The table entries are used to write the voxel coordinates corresponding to the points processed by the convolution kernel, so as to establish a mapping relationship with the corresponding entries of the reference table.

[0251] Referring to Figure 9, taking voxel (59,31,19) as an example, based on the aforementioned explanation, voxel (59,31,19) is an invalid voxel. Therefore, in the reference table, the coordinate entry corresponding to voxel (59,31,19) has been marked as invalid, and the index number of the corresponding input feature has been set to an invalid value of -1. Assuming the size of the convolution kernel is 3×3×3, then 3×3×3 is the receptive field corresponding to this convolution kernel. Then, the coordinates of other voxels within the 3×3×3 receptive field centered on voxel (59,31,19) are determined, for example, (60,31,19), (59,32,19), (59,31,20), (60,32,19), (58,32,18), etc., and so on, totaling 27 voxel coordinates including voxel (59,31,19). Then, look up the updated reference table. Based on the updated reference table, we know that (58,32,19) is located in the 4th row of the updated reference table. The 4th row of the above 27 first relation sub-tables corresponds to (58,32,19). At this time, we need to set the 4th row of each of the 27 first relation sub-tables to invalid values.

[0252] Furthermore, assuming the entry corresponding to voxel (58,32,18) in the reference table is a valid value, and (58,32,18) is the center point of the 27th convolutional kernel in the aforementioned receptive field, and based on the updated reference table, (58,32,18) is located in the 5th row of the updated reference table, then the 5th row of the 27th first relational sub-table needs to be set to an invalid value. This process continues for other voxels; the judgment procedure will not be repeated here.

[0253] Referring to Figure 9, taking voxel (198,102,28) as an example, based on the aforementioned description, voxel (198,102,28) is a newly added voxel. In the reference table, the coordinates of voxel (198,102,28) have been filled into the corresponding table entry, and the index number of the input feature corresponding to this voxel has been set to the valid value N+1. Assuming the receptive field is 3×3×3, the coordinates of other points within the 3×3×3 receptive field centered on voxel (198,102,28) are determined, such as (197,103,27), (198,101,28), etc., and so on. Including voxel (197,103,27), there are a total of 27 voxel coordinates. Each of these 27 voxel coordinates corresponds to one convolutional kernel, that is, each corresponds to one first relational sub-table. Then, look up the updated reference table and determine that (198,102,28) is located in the N+1th row of the reference table and corresponds to the index number N+1. Then, write the index number N+1 in the N+1th row of the above 27 first relation sub-tables.

[0254] Additionally, among the 27 voxel coordinates, at least one voxel coordinate whose corresponding input feature index number is valid in the updated reference table is identified. The first relation sub-table corresponding to each voxel coordinate is determined, as well as the row number of each voxel coordinate in the updated reference table. Then, based on the row number of each voxel coordinate in the updated reference table, index number N+1 is written to the corresponding row of the first relation sub-table corresponding to each voxel coordinate. For example, assuming that the entry corresponding to (197, 103, 27) in the updated reference table is invalid or blank, then an invalid value is written to the N+1th row of the first relation sub-table corresponding to (197, 103, 27).

[0255] Assuming that the entry corresponding to (197,103,27) is a valid value in the updated reference table, then the row number of (197,103,27) in the reference table is determined to be the Mth row. Then, the index number N+1 is written into the Mth row of the first relation sub-table corresponding to (197,103,27). This process continues in the same manner. The determination process for other voxels will not be repeated here.

[0256] Referring to Figure 10, taking voxel (59,31,19) as an example, based on the aforementioned explanation, voxel (59,31,19) is a failed voxel. Since the update of the second relation table is not affected by the set of failed voxels a, the entries in the second relation table corresponding to voxel (59,31,19) and other voxels within the receptive field centered on voxel (59,31,19) are not modified.

[0257] Referring to Figure 10, taking voxel (198, 102, 28) as an example, based on the aforementioned explanation, voxel (198, 102, 28) is a newly added voxel. Based on the aforementioned update of the first relation table, (198, 102, 28) is located in the (N+1)th row of the 27 first relation sub-tables. Therefore, in the second relation table, the index number N+1 is simply written in the (N+1)th row of the 27 second relation sub-tables.

[0258] This completes the update of the reference table, the first relation table, and the second relation table based on the difference algorithm.

[0259] It should be noted that in the examples shown in Figures 8 to 10 above, the index numbers corresponding to the voxel coordinate values, input features, and output features are all examples. In other application scenarios, the values ​​of these parameters can be adjusted accordingly. This application embodiment does not limit these values.

[0260] Furthermore, embodiments of this application also provide an apparatus for implementing any of the above methods. For example, an apparatus for data processing is provided, which includes units (or means) for implementing any of the above data processing methods.

[0261] Figure 11 is a schematic block diagram of a data processing apparatus 1100 according to an embodiment of this application.

[0262] Referring to Figure 11, the device 1100 includes:

[0263] The acquisition unit 1110 is used to acquire first point cloud data, which includes N points and N input features corresponding to the N points, where N is greater than 0; it is also used to acquire second point cloud data, which includes M points and M input features corresponding to the M points, where M is greater than 0.

[0264] The computation unit 1120 is used to perform sparse convolution operation on the second point cloud data based on the difference between the first point cloud data and the second point cloud data.

[0265] In some possible embodiments, the positions of the above N points are represented by the coordinates of N voxels. The first point cloud data is used to construct a first relation table and a second relation table, wherein the first relation table is used to represent the correspondence between N voxels and N input features, and the second relation table is used to represent the correspondence between N voxels and N output features. Based on this, the above-mentioned operation unit 1120 is specifically used to: update the first relation table and the second relation table according to the difference between the first point cloud data and the second point cloud data; and perform sparse convolution operation on the second point cloud data according to the updated first relation table and the second relation table.

[0266] In some possible embodiments, the number of entries in the first relation table and the number of entries in the second relation table are pre-configured.

[0267] In some possible embodiments, the number of convolution kernels used for sparse convolution operations is Y. When Y is greater than 1, the first relation table includes Y first relation sub-tables, and the Y first relation sub-tables correspond to the Y convolution kernels. The second relation table includes Y second relation sub-tables, and the Y second relation sub-tables correspond to the Y convolution kernels. The N entries of the Y first relation sub-tables contain the index numbers of the N input features, and the N entries of the Y second relation sub-tables contain the index numbers of the N output features.

[0268] In some possible embodiments, the positions of the above M points are represented by the coordinates of M voxels. Based on this, the above-mentioned operation unit 1120 is specifically used to: update the first relation table and the second relation table according to the first newly added voxel, and / or update the first relation table according to the first failed voxel. The first newly added voxel is a voxel among the M voxels that does not belong to the N voxels, and the first failed voxel is a voxel among the N voxels that does not belong to the M voxels.

[0269] In some possible embodiments, the first newly added voxel corresponds to the first newly added input feature and the first newly added output feature, respectively. Based on this, the above-mentioned operation unit 1120 is specifically used to: add the index number related to the first newly added input feature to the first relation table, and / or, invalidate the content of the table entry related to the first invalid voxel in the first relation table; and add the index number related to the first newly added output feature to the second relation table.

[0270] In some possible embodiments, the above-mentioned operation unit 1120 is further configured to: construct a first relation table and a second relation table according to a reference table, the reference table including a first column and a second column, the L entries in the first column corresponding to the coordinates of L voxels, N voxels belonging to L voxels, the N entries in the second column corresponding to the index numbers of N input features, the N entries corresponding to N voxels, and the remaining (LN) entries in the second column containing invalid values.

[0271] In some possible embodiments, the above-mentioned arithmetic unit 1120 is further configured to: update the reference table according to the first failed voxel and / or the first newly added voxel, wherein in the updated reference table, the table entries corresponding to the coordinates and input features of the first failed voxel are invalid values, and / or, in the updated reference table, the coordinates of the first newly added voxel are written in one table entry of the first column, and the index number of the first newly added input feature is written in the table entry corresponding to the coordinates of the first newly added voxel in the second column.

[0272] In some possible embodiments, when Y is greater than 1, the above-mentioned operation unit 1120 is specifically used to: write the index numbers of the input features corresponding to the Y first target voxels into the (N+1)th row of the Y first relation sub-tables according to the updated reference table, where the Y first target voxels are voxels within the receptive field of the convolution kernel centered on the first newly added voxel; when the index number of the input feature corresponding to the second target voxel in the updated reference table is a valid value, determine that the row number corresponding to the second target voxel in the updated reference table is the first target row number, where the second target voxel corresponds to the center of the first convolution kernel within the receptive field of the Y convolution kernels, including the first newly added voxel; and write a valid value into the table entry corresponding to the first target row number of the first relation sub-table corresponding to the first convolution kernel.

[0273] In some possible embodiments, the above-mentioned operation unit 1120 is specifically used to: write the index numbers of the output features corresponding to the Y first target voxels into the (N+1)th row of the Y second relation sub-tables according to the table entry positions of the table entries in the first relation table which contain the index numbers of the input features corresponding to the Y first target voxels respectively.

[0274] In some possible embodiments, the above-mentioned operation unit 1120 is specifically used to: determine the row number corresponding to the first failure voxel in the updated reference table as the second target row number according to the updated reference table; write invalid values ​​into the table entries corresponding to the second target row number of the Y first relational sub-tables; when the index number of the input feature corresponding to the third target voxel in the updated reference table is a valid value, determine the row number corresponding to the third target voxel in the updated reference table as the third target row number, wherein the third target voxel corresponds to the center of the second convolutional kernel within the receptive field of the Y convolutional kernels, which includes the first failure voxel; and write invalid values ​​into the table entries corresponding to the third target row number of the first relational sub-table corresponding to the second convolutional kernel.

[0275] In some possible embodiments, the above-described apparatus 1100 may further include a determining unit 1130, which determines the number of newly added and / or invalid points in the second point cloud data compared to the first point cloud data before the calculation unit 1120 performs sparse convolution operation on the second point cloud data based on the difference between the first point cloud data and the second point cloud data.

[0276] In some possible embodiments, the first point cloud data and the second point cloud data are acquired by sensors mounted on the vehicle. Before the computing unit 1120 performs sparse convolution operation on the second point cloud data based on the difference between the first point cloud data and the second point cloud data, the determining unit 1130 is also used to determine that the speed of the vehicle is less than or equal to a second threshold.

[0277] This application also proposes a data processing apparatus, which includes a processor and a memory, wherein the processor and the memory are connected together, the memory is used to store program code, and the processor is used to call the program code to execute any of the data processing methods proposed in this application.

[0278] Furthermore, embodiments of this application also propose a vehicle that includes any of the data processing devices proposed in embodiments of this application.

[0279] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0280] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0281] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0282] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0283] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0284] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0285] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of data processing, characterized by, The method includes: Acquire first point cloud data, which includes N points and N input features corresponding to the N points, where N is greater than 0; Acquire second point cloud data, which includes M points and M input features corresponding to the M points, where M is greater than 0; Based on the difference between the first point cloud data and the second point cloud data, a sparse convolution operation is performed on the second point cloud data.

2. The method of claim 1, wherein, The positions of the N points are represented by the coordinates of N voxels. The first point cloud data is used to construct a first relation table and a second relation table. The first relation table represents the correspondence between the N voxels and the N input features, and the second relation table represents the correspondence between the N voxels and the N output features. The step of performing sparse convolution operation on the second point cloud data based on the difference between the first point cloud data and the second point cloud data includes: Update the first relationship table and the second relationship table based on the differences between the first point cloud data and the second point cloud data; Based on the updated first and second relation tables, perform sparse convolution operations on the second point cloud data.

3. The method of claim 2, wherein, The number of entries in the first relation table and the number of entries in the second relation table are pre-configured.

4. The method according to claim 2 or 3, characterized in that, The number of convolution kernels used for the sparse convolution operation is Y. When Y is greater than 1, the first relation table includes Y first relation sub-tables, and the Y first relation sub-tables correspond to the Y convolution kernels. The second relation table includes Y second relation sub-tables, and the Y second relation sub-tables correspond to the Y convolution kernels. The N entries of the Y first relation sub-tables contain the index numbers of the N input features, and the N entries of the Y second relation sub-tables contain the index numbers of the N output features.

5. The method of claim 4, wherein, The positions of the M points are represented by the coordinates of the M voxels. Updating the first relation table and the second relation table based on the differences between the first point cloud data and the second point cloud data includes: Update the first relation table and the second relation table based on the first newly added voxel, and / or update the first relation table based on the first failed voxel, wherein the first newly added voxel is a voxel among the M voxels that does not belong to the N voxels, and the first failed voxel is a voxel among the N voxels that does not belong to the M voxels.

6. The method of claim 5, wherein, The first newly added voxel corresponds to the first newly added input feature and the first newly added output feature respectively. The step of updating the first relationship table according to the first newly added voxel and / or the first invalid voxel includes: adding the index number related to the first newly added input feature to the first relationship table, and / or, invalidating the content of the table entry related to the first invalid voxel in the first relationship table. The step of updating the second relation table based on the first newly added voxel includes: adding the index number related to the first newly added output feature to the second relation table.

7. The method of claim 6, wherein, The method further includes: Based on the reference table, construct the first relation table and the second relation table. The reference table includes a first column and a second column. The L entries in the first column correspond to the coordinates of L voxels, and the N voxels belong to the L voxels. The N entries in the second column correspond to the index numbers of the N input features, and the N entries correspond to the N voxels. The remaining (LN) entries in the second column contain invalid values.

8. The method of claim 7, wherein, The method further includes: The reference table is updated based on the first failed voxel and / or the first newly added voxel. In the updated reference table, the entries containing the coordinates and input features corresponding to the first failed voxel are invalid values, and / or, in the updated reference table, the coordinates of the first newly added voxel are written in one entry of the first column, and the index number of the first newly added input feature is written in the entry of the second column corresponding to the coordinates of the first newly added voxel.

9. The method of claim 8, wherein, When Y is greater than 1, adding the index number related to the first newly added input feature to the first relationship table includes: According to the updated reference table, the index numbers of the input features corresponding to the Y first target voxels are written into the (N+1)th row of the Y first relation sub-tables. The Y first target voxels are voxels within the receptive field of the convolution kernel centered on the first newly added voxel. When the index number of the input feature corresponding to the second target voxel in the updated reference table is a valid value, the row number corresponding to the second target voxel in the updated reference table is determined to be the first target row number, and the second target voxel corresponds to the center of the first convolutional kernel within the receptive field of the Y convolutional kernels, which includes the first newly added voxel. Write a valid value into the table entry corresponding to the first target row number of the first relational sub-table corresponding to the first convolution kernel.

10. The method according to claim 8 or 9, characterized in that, Adding the index number associated with the first newly added output feature to the second relationship table includes: Based on the position of the table entry in the first relation table that contains the index numbers of the input features corresponding to the Y first target voxels, the index numbers of the output features corresponding to the Y first target voxels are written into the (N+1)th row of the Y second relation sub-tables.

11. The method according to any one of claims 8 to 10, characterized in that, The step of invalidating the contents of the table entries in the first relationship table related to the first failed voxel includes: Based on the updated reference table, the row number corresponding to the first failed voxel in the updated reference table is determined to be the second target row number; Write invalid values ​​into the table entries corresponding to the second target row numbers of the Y first relation sub-tables; When the index number of the input feature corresponding to the third target voxel in the updated reference table is a valid value, the row number corresponding to the third target voxel in the updated reference table is determined to be the third target row number, and the third target voxel corresponds to the center of the second convolution kernel within the receptive field of the Y convolution kernels, which includes the first failed voxel. Write an invalid value into the table entry corresponding to the third target row number of the first relational sub-table corresponding to the second convolution kernel.

12. The method according to any one of claims 1 to 11, characterized in that, Before performing sparse convolution operations on the second point cloud data based on the difference between the first point cloud data and the second point cloud data, the method further includes: The number of newly added and / or invalid points in the second point cloud data compared to the first point cloud data is determined to be less than or equal to a first threshold.

13. The method according to any one of claims 1 to 12, characterized in that, The first point cloud data and the second point cloud data are acquired through sensors mounted on the vehicle. Before performing sparse convolution operations on the second point cloud data based on the differences between the first point cloud data and the second point cloud data, the method further includes: The speed of the vehicle is obtained, and the speed of the vehicle is less than or equal to a second threshold.

14. An apparatus for data processing, characterized by The device includes: The acquisition unit is used to acquire first point cloud data, which includes N points and N input features corresponding to the N points, where N is greater than 0; and to acquire second point cloud data, which includes M points and M input features corresponding to the M points, where M is greater than 0. The computation unit is used to perform sparse convolution operation on the second point cloud data based on the difference between the first point cloud data and the second point cloud data.

15. The apparatus of claim 14, wherein, The positions of the N points are represented by the coordinates of N voxels. The first point cloud data is used to construct a first relation table and a second relation table. The first relation table represents the correspondence between the N voxels and the N input features, and the second relation table represents the correspondence between the N voxels and the N output features. The computation unit is specifically used for: Update the first relationship table and the second relationship table based on the differences between the first point cloud data and the second point cloud data; Based on the updated first and second relation tables, perform sparse convolution operations on the second point cloud data.

16. The apparatus of claim 15, wherein, The number of entries in the first relation table and the number of entries in the second relation table are pre-configured.

17. The apparatus of claim 14 or 16, wherein, The number of convolution kernels used for the sparse convolution operation is Y. When Y is greater than 1, the first relation table includes Y first relation sub-tables, and the Y first relation sub-tables correspond to the Y convolution kernels. The second relation table includes Y second relation sub-tables, and the Y second relation sub-tables correspond to the Y convolution kernels. The N entries of the Y first relation sub-tables contain the index numbers of the N input features, and the N entries of the Y second relation sub-tables contain the index numbers of the N output features.

18. The apparatus of claim 17, wherein, The positions of the M points are represented by the coordinates of the M voxels, and the computation unit is specifically used for: Update the first relation table and the second relation table based on the first newly added voxel, and / or update the first relation table based on the first failed voxel, wherein the first newly added voxel is a voxel among the M voxels that does not belong to the N voxels, and the first failed voxel is a voxel among the N voxels that does not belong to the M voxels.

19. The apparatus of claim 18, wherein, The first newly added voxel corresponds to the first newly added input feature and the first newly added output feature respectively. The operation unit is specifically used to: add the index number related to the first newly added input feature to the first relation table, and / or, invalidate the contents of the table entries related to the first invalid voxel in the first relation table. And add the index number associated with the first newly added output feature to the second relationship table.

20. The apparatus of claim 19, wherein, The arithmetic unit is also used for: Based on the reference table, construct the first relation table and the second relation table. The reference table includes a first column and a second column. The L entries in the first column correspond to the coordinates of L voxels, and the N voxels belong to the L voxels. The N entries in the second column correspond to the index numbers of the N input features, and the N entries correspond to the N voxels. The remaining (LN) entries in the second column contain invalid values.

21. The apparatus of claim 20, wherein, The arithmetic unit is also used for: The reference table is updated based on the first failed voxel and / or the first newly added voxel. In the updated reference table, the entries containing the coordinates and input features corresponding to the first failed voxel are invalid values, and / or, in the updated reference table, the coordinates of the first newly added voxel are written in one entry of the first column, and the index number of the first newly added input feature is written in the entry of the second column corresponding to the coordinates of the first newly added voxel.

22. The apparatus of claim 21, wherein, When Y is greater than 1, the arithmetic unit is specifically used for: According to the updated reference table, the index numbers of the input features corresponding to the Y first target voxels are written into the (N+1)th row of the Y first relation sub-tables. The Y first target voxels are voxels within the receptive field of the convolution kernel centered on the first newly added voxel. When the index number of the input feature corresponding to the second target voxel in the updated reference table is a valid value, the row number corresponding to the second target voxel in the updated reference table is determined to be the first target row number, and the second target voxel corresponds to the center of the first convolutional kernel within the receptive field of the Y convolutional kernels, which includes the first newly added voxel. Write a valid value into the table entry corresponding to the first target row number of the first relational sub-table corresponding to the first convolution kernel.

23. The apparatus of claim 21 or 22, wherein, The arithmetic unit is specifically used for: Based on the position of the table entry in the first relation table that contains the index numbers of the input features corresponding to the Y first target voxels, the index numbers of the output features corresponding to the Y first target voxels are written into the (N+1)th row of the Y second relation sub-tables.

24. The apparatus of any one of claims 21-23, wherein, The arithmetic unit is specifically used for: Based on the updated reference table, the row number corresponding to the first failed voxel in the updated reference table is determined to be the second target row number; Write invalid values ​​into the table entries corresponding to the second target row numbers of the Y first relation sub-tables; When the index number of the input feature corresponding to the third target voxel in the updated reference table is a valid value, the row number corresponding to the third target voxel in the updated reference table is determined to be the third target row number, and the third target voxel corresponds to the center of the second convolution kernel within the receptive field of the Y convolution kernels, which includes the first failed voxel. Write an invalid value into the table entry corresponding to the third target row number of the first relational sub-table corresponding to the second convolution kernel.

25. The apparatus of any one of claims 14-24, wherein, The device further includes a determining unit, which determines that, before the calculation unit performs sparse convolution operation on the second point cloud data based on the difference between the first point cloud data and the second point cloud data, the determining unit is used to determine that the number of newly added and / or invalid points in the second point cloud data compared to the first point cloud data is less than or equal to a first threshold.

26. The apparatus of any one of claims 14-25, wherein, The first point cloud data and the second point cloud data are acquired by sensors mounted on the vehicle. Before the computing unit performs sparse convolution operation on the second point cloud data based on the difference between the first point cloud data and the second point cloud data, the determining unit is used to determine that the speed of the vehicle is less than or equal to a second threshold.

27. An apparatus for data processing, characterized by It includes a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to perform the method as described in any one of claims 1 to 13.

28. A vehicle characterized by Includes the apparatus as described in any one of claims 14 to 27.

29. A chip system, characterized by The chip system is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected via lines; the interface circuits are used to receive signals from the memory of the electronic device and send the signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1 to 13.

30. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method as described in any one of claims 1 to 13.

31. A computer program product, characterised in that, It includes instructions that, when executed by a processor, cause the method of any one of claims 1 to 13 to be performed.

Images