Shape fitting-based parcel specification detection method

Abstract

The present application relates to the technical field of logistics, and discloses a shape fitting-based parcel specification detection method. In the method, a target detection model is used for performing target detection on an RGB image of a parcel detection region to determine a parcel region where a parcel waiting for detection is located, then depth data of the parcel region is extracted and converted to obtain parcel point cloud data, then a plurality of point clouds are randomly selected from the parcel point cloud data to respectively fit a plurality of reference geometric shapes, the confidence level of each fitted reference geometric shape is obtained by evaluating the degree of matching between the remaining parcel point cloud data and each fitted reference geometric shape, and finally, specification parameters of said parcel are calculated on the basis of the reference geometric shape having the highest confidence level. The method combines an image algorithm and point cloud processing technology, so as to robustly process point cloud data of various qualities, thereby quickly and accurately identifying various standard and irregular parcel specifications, and thus significantly improving the sorting efficiency and application range of logistics automation systems.

Description

A Package Size Detection Method Based on Shape Fitting Technical Field

[0001] This application relates to the field of logistics technology, and in particular to a method for detecting package specifications based on shape fitting. Background Technology

[0002] In modern logistics systems, automated parcel sorting and processing are crucial for improving logistics efficiency, and the accuracy and efficiency of parcel detection directly impact the subsequent sorting and processing results. Current common parcel detection technologies primarily employ two-dimensional (2D) image detection. However, most parcels are three-dimensional, and 2D image detection technology struggles to accurately and completely capture their three-dimensional shape features. This limits its effectiveness, allowing it to perform well only on standard-sized and standard-shaped parcels. In real-world logistics scenarios, besides standard parcels, there are many irregularly shaped or extra-standard-sized parcels. 2D image detection technology often fails to accurately identify these irregularly shaped parcels, thus failing to meet the high efficiency and accuracy requirements of modern logistics. Summary of the Invention

[0003] This application addresses the aforementioned problems and technical requirements by proposing a package specification detection method based on shape fitting. The technical solution of this application is as follows:

[0004] A package specification detection method based on shape fitting, the package specification detection method comprising:

[0005] Use an RGBD smart depth camera to acquire RGB images and depth data of the package detection area;

[0006] The object detection model is used to perform object detection on the RGB image to determine the package area where the package to be detected is located, and then the depth data of the package area is extracted and converted to obtain the package point cloud data;

[0007] Multiple point clouds are randomly selected from the package point cloud data as sample points, and various reference geometric shapes are fitted based on the selected sample points;

[0008] The degree of matching between other package point cloud data and each fitted reference geometry is evaluated to obtain the confidence level of each fitted reference geometry; the higher the degree of matching between other package point cloud data and the reference geometry, the higher the confidence level.

[0009] The specifications of the package to be inspected are calculated based on the point cloud data of the package and the reference geometry with the highest confidence.

[0010] A further technical solution involves obtaining the confidence level for each fitted reference geometry, including:

[0011] The point cloud set P is formed by selecting all point clouds that conform to the geometric constraints of the reference geometry from the package point cloud data. ψ ;

[0012] P point cloud collection ψ The point clouds are clustered based on location, and the largest connected point cloud set P is obtained by extracting the category containing the most point clouds. max ;

[0013] Based on the maximum connected point cloud set P max The confidence level is determined by the number of point clouds included.

[0014] A further technical solution involves using a reference geometry that includes planar, cylindrical, and spherical shapes. Detecting whether any point in the bounding point cloud data conforms to the geometric constraints of the reference geometry includes:

[0015] For the fitted planar shape, if the angle between the point cloud normal vector and the fitted planar shape normal vector is less than the angle threshold β1, the point cloud is determined to conform to the geometric constraints of the planar shape; otherwise, the point cloud is determined not to conform to the geometric constraints of the planar shape.

[0016] For the fitted cylindrical shape, determine the intersection point of the projection plane of the point cloud, which is perpendicular to the central axis of the fitted cylindrical shape, and the central axis. When the angle between the vector from the intersection point to the point cloud and the normal vector of the point cloud is less than the angle threshold β2, and the distance between the point cloud and the central axis of the fitted cylindrical shape is less than the radius of the cross section of the cylinder, it is determined that the point cloud conforms to the geometric constraints of the cylindrical shape; otherwise, it is determined that the point cloud does not conform to the geometric constraints of the cylindrical shape.

[0017] For the fitted sphere shape, if the angle between the vector from the center of the fitted sphere shape to the point cloud and the normal vector of the point cloud is less than the angle threshold β3, and the distance between the point cloud and the center of the fitted sphere shape is less than the radius of the sphere, then the point cloud is determined to conform to the geometric constraints of the sphere shape; otherwise, the point cloud is determined not to conform to the geometric constraints of the sphere shape.

[0018] A further technical solution involves using a reference geometry, including a planar shape, and fitting the planar shape based on selected sample points, including:

[0019] Three non-collinear point clouds were randomly selected from the package point cloud data as sample points p1, p2 and p3;

[0020] The cross product of vectors is used to calculate the plane normal vector of the candidate plane formed by sample points p1, p2, and p3. in, It is the vector from sample point p1 to sample point p2. It is the vector from sample point p1 to sample point p3;

[0021] When the angle between the point cloud normal vectors of sample points p1, p2 and p3 and the plane normal vector n of the candidate plane is less than the angle threshold β1, the candidate plane formed by sample points p1, p2 and p3 is taken as the fitted plane shape; otherwise, the step of randomly selecting three non-collinear point clouds from the wrapped point cloud data as sample points p1, p2 and p3 is repeated.

[0022] A further technical solution involves using a reference geometry, including a cylindrical shape, fitting a cylindrical shape based on selected sample points, and determining the geometric feature parameters of the fitted cylindrical shape, including:

[0023] Two point clouds were randomly selected from the package point cloud data as sample points p4 and p5;

[0024] The normal vector of the candidate central axis ν is obtained by calculating the vector product of the point cloud normal vector n4 of sample point p4 and the point cloud normal vector n5 of sample point p5, and the projection plane Pv perpendicular to the normal vector of the candidate central axis ν is determined.

[0025] Determine the projection line L'4 of the line L4 containing the point cloud normal vector n4 of sample point p4 onto the projection plane Pv, and determine the projection line L'5 of the line L5 containing the point cloud normal vector n5 of sample point p5 onto the projection plane Pv. Determine the intersection point c of the projection lines L'4 and L'5. The line along the normal vector of the candidate central axis ν and passing through the intersection point c is taken as the candidate central axis ν.

[0026] Determine the projection point p'4 of sample point p4 in projection plane Pv, and determine the projection point p'5 of sample point p5 in projection plane Pv. Use the average of the Euclidean distances between the intersection point c and the projection points p'4 and p'5 as the radius of the candidate cylinder cross section.

[0027] When the difference between the Euclidean distance between the projection point p'4 and the intersection point c and the radius of the candidate cylinder cross section is less than the distance threshold λ1, and the difference between the Euclidean distance between the projection point p'5 and the intersection point c and the radius of the candidate cylinder cross section is less than the distance threshold λ1, and the angle between the vector from the intersection point c to the projection point p'4 and the point cloud normal vector n4 of the sample point p4 is less than the angle threshold β2, and the angle between the vector from the intersection point c to the projection point p'5 and the point cloud normal vector n5 of the sample point p5 is less than the angle threshold β2, the cylinder shape determined based on the candidate central axis and the radius of the candidate cylinder cross section will be used as the fitted cylinder shape; otherwise, the step of randomly selecting two point clouds from the wrapped point cloud data as sample points p4 and p5 will be repeated.

[0028] A further technical solution involves using a reference geometry, including a sphere shape, fitting the sphere shape based on selected sample points, and determining the geometric feature parameters of the fitted sphere shape, including:

[0029] Two point clouds are randomly selected from the package point cloud data as sample points p6 and p7. The line L6 passing through sample point p6 and along the direction of the point cloud normal vector n6 of sample point p6 is calculated, as well as the line L7 passing through sample point p7 and along the direction of the point cloud normal vector n7 of sample point p7 is calculated.

[0030] Calculate the shortest line segment between line L6 and line L7, and use the midpoint of the shortest line segment as the candidate center of the sphere;

[0031] The average of the Euclidean distances between the candidate sphere center and sample points p6 and p7 is taken as the radius of the candidate sphere.

[0032] When the difference between the Euclidean distance between the candidate sphere center and sample point p6 and the radius of the candidate sphere is less than the distance threshold λ2, and the difference between the Euclidean distance between the candidate sphere center and sample point p7 and the radius of the candidate sphere is less than the distance threshold λ2, and the angle between the vector from the candidate sphere center to sample point p6 and the point cloud normal vector n6 of sample point p6 is less than the angle threshold β3, and the angle between the vector from the candidate sphere center to sample point p7 and the point cloud normal vector n7 of sample point p7 is less than the angle threshold β3, the sphere shape determined by the candidate sphere center and the candidate sphere radius will be used as the fitted sphere shape; otherwise, the step of randomly selecting two point clouds from the wrapped point cloud data as sample points p6 and p7 will be repeated.

[0033] The further technical solution involves calculating the specification parameters of the package to be inspected based on the package point cloud data and the reference geometry with the highest confidence level, including:

[0034] When the reference geometry with the highest confidence is a planar shape, determine the maximum connected point cloud set P. max The distribution area of ​​the point cloud on the fitted planar shape and the distribution area along the plane normal vector direction of the planar shape are used to obtain three-dimensional size information. The specification parameters of the package to be detected include: the package shape is a cube and the package size is the calculated three-dimensional size information.

[0035] When the reference geometry with the highest confidence is a cylinder, determine the maximum connected point cloud set P. max The distribution area of ​​the point cloud along the central axis of the fitted cylinder shape is used to determine the length of the cylinder, and the specifications of the package to be detected are obtained, including: the package shape is a cylinder shape, the radius of the package bottom surface is the radius of the cross section of the fitted cylinder shape, and the package length is the calculated cylinder length.

[0036] When the reference geometry with the highest confidence is a sphere, the specifications of the package to be detected include: the package shape is a sphere and the package radius is the radius of the fitted sphere.

[0037] The further technical solution involves converting the package point cloud data, including:

[0038] The depth data of the packaged area is converted into three-dimensional point cloud data using the internal parameters of the RGBD smart depth camera, and the initial point cloud data is obtained by equidistant sampling.

[0039] Radius filtering is completed by filtering out point clouds with fewer than a threshold number of neighboring point clouds in the initial point cloud data obtained from sampling. The neighboring point clouds of each point cloud include all point clouds within the filtering radius of the point cloud.

[0040] For any point cloud in the initial point cloud data after radius filtering, calculate the average Euclidean distance μ and the label difference σ between the point cloud and its K nearest point clouds, and obtain the distance threshold max = μ + ε * σ. When the average Euclidean distance μ between the point cloud and its K nearest point clouds is greater than max, filter out the point cloud; otherwise, retain the point cloud. ε is the threshold coefficient.

[0041] After traversing all the point clouds in the initial point cloud data that has undergone radius filtering, the wrapped point cloud data is obtained.

[0042] The further technical solution involves extracting depth data from the package area and converting it into package point cloud data, including:

[0043] The depth data of the package area is extracted, and the depth data that is much larger than the suspension height of the RGBD smart depth camera is filtered out. The edges of the remaining depth data are then subjected to morphological processing to obtain the noise-removed depth data of the package area.

[0044] Statistical analysis is performed on the noise-removed depth data within the package area. The height estimate of the package to be detected is obtained based on the reference plane position of the calibrated package detection area. Depth data in the noise-removed depth data within the package area that deviates from the height estimate by a degree reaching the deviation threshold are filtered out to obtain the processed depth data of the package area. The processed depth data of the package area is then converted to obtain package point cloud data.

[0045] The further technical solution is that the target detection model used is designed based on the PP-PicoDet series target detection model. The target detection model uses a lightweight LCNet neural network as the feature extraction layer, uses PicoHeadV2 backbone network as the target detection head structure and combines it with the LCPAN module.

[0046] During model training, data augmentation techniques and batch processing strategies such as batch random adjustment of image size and normalization are used to expand the training sample set. A momentum optimizer combined with cosine decay and linear warm-up strategies is used to adjust the learning rate.

[0047] The beneficial technical effects of this application are:

[0048] This application discloses a package specification detection method based on shape fitting. This method utilizes RGB images and depth data combined with a target detection model to extract package point cloud data. Then, it constructs a shape fit based on different reference geometries and evaluates the matching degree, ultimately selecting the reference geometry with the highest confidence to determine the package shape and obtain specification parameters. This method can robustly process point cloud data of various qualities. By combining image algorithms and point cloud processing techniques, it can quickly and accurately identify various standard and irregular package specifications, significantly improving the sorting efficiency and applicability of automated logistics systems. Attached Figure Description

[0049] Figure 1 is a flowchart of a package specification detection method according to an embodiment of this application.

[0050] Figure 2 is a schematic diagram of constructing a candidate cylinder shape using two selected sample points.

[0051] Figure 3 is a schematic diagram of constructing a sphere shape using two selected sample points. Detailed Implementation

[0052] The specific embodiments of this application will be further described below with reference to the accompanying drawings.

[0053] This application discloses a package specification detection method based on shape fitting. Please refer to the flowchart shown in Figure 1. The package specification detection method includes:

[0054] Step 1: Use an RGBD smart depth camera to acquire RGB images and depth data of the package detection area. The RGBD smart depth camera is suspended above the package detection area and vertically downwards towards the package detection area to acquire RGB images and depth data. In actual application scenarios, the package specification detection operation is performed during the package sorting process, so the RGBD smart depth camera is suspended above the sorting conveyor belt and acquires RGB images and depth data at the sorting conveyor belt.

[0055] RGBD smart depth cameras typically have a large field of view, so the entire field of view of an RGBD smart depth camera is not usually used as the package detection area. Instead, the package detection area, i.e. the area where the package may appear, is pre-defined. Then, the region of interest, i.e. the RGB image and depth data of the package detection area, is extracted from the RGB image and depth data directly obtained from the RGBD smart depth camera. This allows background data to be filtered out, reducing the amount of subsequent data processing.

[0056] Step 2: Use the object detection model to perform object detection on the RGB image of the package detection area to determine the package area where the package to be detected is located.

[0057] The object detection model is pre-trained. It can be used to detect and locate packages in the RGB image of the package detection area, thereby locating the target detection box of the package to be detected and determining the package area where the package to be detected is located.

[0058] The object detection model used in this step is based on the PP-PicoDet series of object detection models. This model is a state-of-the-art lightweight model that not only boasts excellent detection performance but also can be efficiently deployed on mobile devices. This object detection model uses a lightweight LCNet neural network as its feature extraction layer, characterized by a scaling factor of 0.35 and three feature maps at different scales, optimized for computational efficiency by adjusting the network's depth and width. The model uses the PicoHeadV2 backbone network as its object detection head structure, a highly efficient feature extraction structure. Its core is a convolutional feature extraction module called PicoFeat. This module can process input feature maps with 96 channels through two convolutional layers while maintaining the same number of channels in the output feature map. The model also incorporates an LCPAN module as a feature extraction module, capable of outputting 96-channel feature maps, providing rich feature representations for subsequent object detection tasks. To enhance feature expressiveness, batch normalization and Squeeze-and-Excitation (SE) channel attention mechanisms are employed. Furthermore, this module supports multi-scale feature fusion, effectively integrating features at different scales through a Feature Pyramid Network (FPN). Class prediction and bounding box regression share the same feature representations; this design aims to reduce the number of model parameters while maintaining high performance.

[0059] When training this object detection model, a certain amount of package sample data was first collected to construct a training sample set. The collected package sample data included standard and irregularly shaped packages of various shapes and sizes. Then, data augmentation techniques and batch processing strategies such as batch random image resizing and normalization were used to expand the training sample set. The data augmentation techniques used included random cropping, random flipping, and color distortion. Each batch was set to 12 samples, and a strategy of data shuffling and discarding the last incomplete batch was enabled to enhance the model's robustness to different image variations. In addition, 6 worker threads were set up to process the data in parallel to improve training efficiency. In terms of training parameter settings, the initial base learning rate was set to 0.01 and adjusted through two scheduling strategies: first, a linear warm-up strategy, which gradually increased the learning rate from 0.1 times the base learning rate to full speed in the first 500 steps of training to stabilize the performance in the early stages of training; and second, a cosine decay strategy, which simulated the decay trend of the cosine function in the following training cycles to gradually reduce the learning rate, helping the model to fine-tune in the later stages of training and avoid overfitting. For the optimizer, a momentum optimizer with a momentum parameter of 0.9 was chosen, which helps accelerate convergence and reduce oscillations during training. Furthermore, to control model complexity and prevent overfitting, L2 regularization with a regularization factor of 0.00004 was introduced. By penalizing the sum of squared weight parameters, it encourages the model to learn a more concise representation. The training process was set to run for 600 epochs to ensure the model had sufficient time to learn and optimize its performance. During the evaluation and testing phases, the model underwent data processing procedures such as image decoding, scaling, normalization, and batch imputation to ensure the accuracy of the evaluation results.

[0060] Step 3: Then extract the depth data of the package area and convert it to obtain package point cloud data.

[0061] After extracting the target detection box of the package to be detected in step 2 to determine the package area, the depth data within the package area can be extracted, and then combined with the internal parameters of the RGBD smart depth camera to obtain the package point cloud data.

[0062] (1) To improve the efficiency of subsequent data processing and reduce interference from noisy data, in one embodiment, the depth data within the package area is first preprocessed before being converted into package point cloud data. The preprocessing performed on the depth data within the package area includes:

[0063] The depth data of the package area is extracted. After filtering out the depth data that is much larger than the suspension height of the RGBD smart depth camera, the edges of the remaining depth data are morphologically processed to obtain the noise-removed depth data of the package area.

[0064] Then, statistical analysis is performed on the depth data after noise removal within the package area. The height estimate of the package to be detected is obtained based on the reference plane position of the calibrated package detection area. Depth data in the depth data after noise removal within the package area that deviates from the height estimate to a deviation threshold are filtered out to obtain the processed depth data of the package area. The processed depth data of the package area is then converted to obtain package point cloud data.

[0065] The reference plane position where the package detection area is located is pre-calibrated. The method is as follows: the depth data of the package detection area is obtained by using an RGBD intelligent depth camera when there are no packages in the package detection area. Then, the depth data of the package detection area when there are no packages in the package detection area is converted into three-dimensional point cloud data by using the internal parameters of the RGBD intelligent depth camera. Finally, the reference plane position where the package detection area is located is obtained by fitting the data using the least squares algorithm.

[0066] (2) In order to further improve the efficiency of subsequent data processing and reduce the interference caused by noisy data, in one embodiment, the 3D point cloud data obtained by converting the depth data of the wrapped area using the internal parameters of the RGBD smart depth camera is not directly used as the wrapped point cloud data, but further post-processing operations are also performed:

[0067] After converting the depth data of the enclosed area into 3D point cloud data using the internal parameters of the RGBD smart depth camera, initial point cloud data is obtained by equidistant sampling. This operation helps reduce the amount of computation, and the sampling distance can be customized.

[0068] Then, radius filtering is performed. Specifically, point clouds with fewer than a certain number of neighboring point clouds in the initial sampled point cloud data are filtered out. Radial filtering has been completed. The neighboring point clouds of each point cloud include all point clouds within the filtering radius of the point cloud. In one example, the filtering radius is set to 3cm and the threshold is 5. Therefore, if the number of other point clouds within a 3cm radius of a point cloud is less than 5, that point will be filtered out.

[0069] Further statistical outlier filtering is performed. Specifically, for any point cloud in the initial point cloud data after radius filtering, the average Euclidean distance μ and the label difference σ between the point cloud and its K nearest point clouds are calculated, and the distance threshold max = μ + ε * σ is obtained. When the average Euclidean distance μ between the point cloud and its K nearest point clouds is greater than max, the point cloud is filtered out; otherwise, the point cloud is retained. ε is the threshold coefficient. In one example, the threshold coefficient ε = 2.5, and K is 8.

[0070] After traversing all point clouds in the initial point cloud data that has undergone radius filtering, the wrapped point cloud data is obtained. The radius filtering and statistical outlier filtering described above can identify and remove noise or outliers in the dataset. Through two-step filtering, higher quality wrapped point cloud data can be obtained, which is beneficial to improving the accuracy and efficiency of subsequent processing tasks.

[0071] Step 4: Randomly select multiple point clouds from the package point cloud data as sample points, and fit various reference geometric shapes based on the selected sample points.

[0072] In reality, most packages are rectangular. Depending on the application scenario, common irregularly shaped packages include cylindrical barrel-shaped packages and spherical packages. Therefore, in one embodiment, the detection is specifically targeted at these package shapes. Thus, the reference geometries in this embodiment include planar, cylindrical, and spherical shapes. The methods for fitting these different reference geometries based on selected sample points are different and are described below:

[0073] (1) Fitting the plane shape based on the selected sample points

[0074] First, three non-collinear point clouds are randomly selected from the package point cloud data as sample points p1, p2 and p3;

[0075] The cross product of vectors is used to calculate the plane normal vector of the candidate plane formed by sample points p1, p2, and p3. in, It is the vector from sample point p1 to sample point p2. It is the vector from sample point p1 to sample point p3.

[0076] When the angle between the point cloud normal vector of sample point p1 and the plane normal vector n of the candidate plane is less than the angle threshold β1, and the angle between the point cloud normal vector of sample point p2 and the plane normal vector n of the candidate plane is less than the angle threshold β1, and the angle between the point cloud normal vector of sample point p3 and the plane normal vector n of the candidate plane is less than the angle threshold β1, it is determined that sample points p1, p2, and p3 all satisfy the orientation constraints of the constructed candidate plane. Then, the candidate plane formed by sample points p1, p2, and p3 is taken as the fitted plane shape. Otherwise, the step of randomly selecting three non-collinear point clouds from the wrapped point cloud data as sample points p1, p2, and p3 is repeated. The angle threshold β1 is customizable.

[0077] (2) Fitting the cylinder shape based on the selected sample points

[0078] Referring to Figure 2, randomly select two point clouds from the package point cloud data as sample points p4 and p5.

[0079] First, calculate the vector product of the point cloud normal vector n4 of sample point p4 and the point cloud normal vector n5 of sample point p5 to obtain the normal vector. Then, determine the projection plane Pv perpendicular to this normal vector. Project the line L4 containing the point cloud normal vector n4 of sample point p4 onto the projection plane Pv to obtain the projection line L'4. Project the line L5 containing the point cloud normal vector n5 of sample point p5 onto the projection plane Pv to obtain the projection line L'5. Determine the intersection point c of the projection lines L'4 and L'5 on the projection plane Pv. The line passing through the intersection point c and along the normal vector is the candidate central axis ν.

[0080] Determine the projection point p'4 of sample point p4 onto the projection plane Pv, and determine the projection point p'5 of sample point p5 onto the projection plane Pv. Then, calculate the average of the Euclidean distances between the intersection point c and the projection points p'4 and p'5, respectively, as the candidate cylinder cross-sectional radius r. Based on the candidate central axis ν and the candidate cylinder cross-sectional radius r, the shape of a candidate cylinder with infinite length can be determined, as shown by the dashed line in Figure 2.

[0081] When the difference between the Euclidean distance between the projection point p'4 and the intersection point c (i.e., the Euclidean distance between sample point p4 and the candidate central axis ν) and the radius r of the candidate cylinder's cross-section is less than the distance threshold λ1, and the difference between the Euclidean distance between the projection point p'5 and the intersection point c (i.e., the Euclidean distance between sample point p5 and the candidate central axis ν) and the radius r of the candidate cylinder's cross-section is less than the distance threshold λ1, it is determined that sample points p4 and p5 are both near the surface of the constructed candidate cylinder shape, and therefore both sample points p4 and p5 satisfy the distance constraint. This distance threshold λ1 is customizable.

[0082] When the angle between the vector from intersection point c to projection point p'4 and the point cloud normal vector n4 of sample point p4 is less than the angle threshold β2, and the angle between the vector from intersection point c to projection point p'5 and the point cloud normal vector n5 of sample point p5 is less than the angle threshold β2, it can be determined that both sample points p4 and p5 satisfy the direction constraint. This angle threshold β2 is customizable.

[0083] If sample points p4 and p5 both satisfy the distance and orientation constraints respectively, it indicates that the candidate cylinder shape is reasonable. The cylinder shape determined by the candidate central axis ν and the cross-sectional radius r of the candidate cylinder will then be used as the final fitted cylinder shape. Otherwise, the step of randomly selecting two point clouds from the wrapped point cloud data as sample points p4 and p5 will be repeated.

[0084] (3) Fit the shape of the sphere based on the selected sample points.

[0085] Referring to Figure 3, randomly select two point clouds from the package point cloud data as sample points p6 and p7, and calculate the straight line L6 passing through sample point p6 and along the direction of the point cloud normal vector n6 of sample point p6, and calculate the straight line L7 passing through sample point p7 and along the direction of the point cloud normal vector n7 of sample point p7.

[0086] Calculate the shortest line segment H6H7 between line L6 and line L7, and take the midpoint of the shortest line segment H6H7 as the candidate center of the sphere O. The shortest line segment H6H7 is perpendicular to line L6 with the foot of the perpendicular at H6, and the shortest line segment H6H7 is perpendicular to line L7 with the foot of the perpendicular at H7.

[0087] The average Euclidean distance between the candidate sphere center O and sample points p6 and p7 is taken as the radius R of the candidate sphere. A candidate sphere can be constructed based on the candidate sphere center O and the candidate sphere radius R.

[0088] If the difference between the Euclidean distance between the candidate sphere center O and sample point p6 and the radius R of the candidate sphere is less than the distance threshold λ2, and the difference between the Euclidean distance between the candidate sphere center O and sample point p7 and the radius R of the candidate sphere is also less than the distance threshold λ2, then sample points p6 and p7 are determined to be near the surface of the constructed candidate sphere, and therefore both sample points p6 and p7 satisfy the distance constraint. This distance threshold λ2 can be customized.

[0089] When the angle between the vector from the candidate sphere center O to the sample point p6 and the point cloud normal vector n6 of the sample point p6 is less than the angle threshold β3, and the angle between the vector from the candidate sphere center O to the sample point p7 and the point cloud normal vector n7 of the sample point p7 is less than the angle threshold β3, it can be determined that both sample points p6 and p7 satisfy the direction constraint. This angle threshold β3 is customizable.

[0090] When it is determined that sample points p6 and p7 satisfy the distance constraint and the orientation constraint respectively, it means that the candidate sphere is reasonable. Then, the sphere shape determined by the candidate sphere center O and the candidate sphere radius R is taken as the sphere shape obtained by final fitting. Otherwise, the step of randomly selecting two point clouds from the wrapped point cloud data as sample points p6 and p7 is repeated.

[0091] Step 5: Evaluate the degree of matching between other package point cloud data and each fitted reference geometry, and obtain the confidence score for each fitted reference geometry. A higher degree of matching between other package point cloud data and a reference geometry results in a higher confidence score, indicating that the package point cloud data better matches the characteristics of that reference geometry. The method for evaluating the degree of matching between other package point cloud data and each reference geometry is similar, mainly including the following steps: Evaluation is performed from multiple dimensions for any reference geometry:

[0092] (1) Select all point clouds that conform to the geometric constraints of the reference geometry from the package point cloud data to form a point cloud set P. ψ The specific screening methods differ for the three different reference geometries, and are described below:

[0093] For the fitted planar shape, if the angle between the normal vector of any point cloud in the wrapped point cloud data and the normal vector of the fitted planar shape is less than the angle threshold β1, the point cloud is determined to conform to the geometric constraints of the planar shape; otherwise, the point cloud is determined not to conform to the geometric constraints of the planar shape. This allows for the selection of a set of point clouds P that conform to the geometric constraints of the planar shape. ψ .

[0094] For the fitted cylindrical shape, for any point cloud in the bounding point cloud data, first determine the projection plane of the point cloud perpendicular to the central axis of the fitted cylindrical shape. Then determine the intersection point of the projection plane and the central axis. If the angle between the vector from the intersection point to the point cloud and the point cloud normal vector is less than the angle threshold β2, and the distance between the point cloud and the central axis of the fitted cylindrical shape (this distance is the distance from the intersection point of the projection plane and the central axis to the point cloud) is less than the cross-sectional radius of the cylinder, then the point cloud is determined to conform to the geometric constraints of the cylindrical shape. Otherwise, the point cloud is determined not to conform to the geometric constraints of the cylindrical shape. Thus, a set of point clouds P that conform to the geometric constraints of the cylindrical shape can be selected. ψ .

[0095] For the fitted sphere shape, for any point cloud in the bounding point cloud data, if the angle between the vector from the center of the fitted sphere shape to the point cloud and the normal vector of the point cloud is less than the angle threshold β3, and the distance between the point cloud and the center of the fitted sphere shape is less than the radius of the fitted sphere, then the point cloud is determined to conform to the geometric constraints of the sphere shape; otherwise, the point cloud is determined not to conform to the geometric constraints of the sphere shape. This allows for the selection of a set of point clouds P that conform to the geometric constraints of the sphere shape. ψ .

[0096] (2) For point cloud set P ψ The point clouds are clustered based on location, and the largest connected point cloud set P is obtained by extracting the category containing the most point clouds. max .

[0097] (3) Based on the maximum connected point cloud set P max The confidence level is determined by the number of point clouds included, which is the final point cloud set P that only considers geometric constraints. ψ The point cloud that constitutes the largest connected component in terms of shape is selected to ensure the integrity and continuity of the shape. The calculation method for different point cloud quantities and confidence levels can be customized.

[0098] Step 6: Calculate the specifications of the package to be inspected based on the package point cloud data and the reference geometry with the highest confidence level.

[0099] When the reference geometry with the highest confidence is a planar shape, the shape of the package to be detected can be determined to be rectangular. Then, the dimensions of the package to be detected can be further determined, including: determining the largest connected point cloud set P. max The distribution regions of the point cloud on the fitted planar shape and along the plane normal vector direction of the planar shape are used to obtain the three-dimensional size information as the wrapping size. The specific method for calculating the three-dimensional size information based on the distribution can refer to existing practices, which will not be elaborated here.

[0100] When the reference geometry with the highest confidence level is a cylinder, the shape of the package to be detected can be determined to be cylindrical. The cross-sectional dimensions of the cylinder can be determined, but the length is not yet known. Therefore, the maximum connected point cloud set P can be further determined. max The distribution area of ​​the point cloud along the central axis of the fitted cylinder shape determines the cylinder length. Subsequently, the radius of the enclosing base is determined to be the cross-sectional radius of the fitted cylinder shape, and the enclosing length is the calculated cylinder length. The specific method for calculating the cylinder length based on the distribution can refer to existing practices, and will not be elaborated here.

[0101] When the reference geometry with the highest confidence is a sphere, the shape of the package to be detected can be determined to be a sphere, and the radius of the package can also be determined to be the radius of the sphere obtained by fitting the sphere shape.

[0102] The above descriptions are merely preferred embodiments of this application, and this application is not limited to the above embodiments. It is understood that other improvements and variations that can be directly derived or conceived by those skilled in the art without departing from the spirit and concept of this application should be considered to be included within the protection scope of this application.

Claims

1. A method for detecting package specifications based on shape fitting, characterized in that, The package specification detection method includes: Use an RGBD smart depth camera to acquire RGB images and depth data of the package detection area; The object detection model is used to perform object detection on the RGB image to determine the package area where the package to be detected is located, and then the depth data of the package area is extracted and converted to obtain the package point cloud data; Multiple point clouds are randomly selected from the package point cloud data as sample points, and various reference geometric shapes are fitted based on the selected sample points; The degree of matching between other package point cloud data and each fitted reference geometry is evaluated to obtain the confidence level of each fitted reference geometry; the higher the degree of matching between other package point cloud data and the reference geometry, the higher the confidence level. The specifications of the package to be inspected are calculated based on the point cloud data of the package and the reference geometry with the highest confidence.

2. The parcel specification detection method according to claim 1, characterized in that, The confidence scores for each fitted reference geometry include: The point cloud set P is formed by selecting all point clouds that conform to the geometric constraints of the reference geometry from the package point cloud data. ψ ; P point cloud collection ψ The point clouds are clustered based on location, and the largest connected point cloud set P is obtained by extracting the category containing the most point clouds. max ; Based on the maximum connected point cloud set P max The confidence level is determined by the number of point clouds included.

3. The parcel specification detection method according to claim 2, characterized in that, The reference geometry includes planar, cylindrical, and spherical shapes. Detecting whether any point in the bounding point cloud data conforms to the geometric constraints of the reference geometry includes: For the fitted planar shape, if the angle between the point cloud normal vector and the fitted planar shape normal vector is less than the angle threshold β1, the point cloud is determined to conform to the geometric constraints of the planar shape; otherwise, the point cloud is determined to not conform to the geometric constraints of the planar shape. For the fitted cylindrical shape, determine the intersection point of the projection plane of the point cloud, which is perpendicular to the central axis of the fitted cylindrical shape, and the central axis. When the angle between the vector from the intersection point to the point cloud and the normal vector of the point cloud is less than the angle threshold β2, and the distance between the point cloud and the central axis of the fitted cylindrical shape is less than the radius of the cross section of the cylinder, it is determined that the point cloud conforms to the geometric constraints of the cylindrical shape; otherwise, it is determined that the point cloud does not conform to the geometric constraints of the cylindrical shape. For the fitted sphere shape, if the angle between the vector from the center of the fitted sphere shape to the point cloud and the normal vector of the point cloud is less than the angle threshold β3, and the distance between the point cloud and the center of the fitted sphere shape is less than the radius of the sphere, then the point cloud is determined to conform to the geometric constraints of the sphere shape; otherwise, the point cloud is determined not to conform to the geometric constraints of the sphere shape.

4. The parcel specification detection method according to claim 1, characterized in that, The reference geometry includes planar shapes, and the planar shape fitted based on selected sample points includes: Three non-collinear point clouds were randomly selected from the package point cloud data as sample points p1, p2 and p3; The cross product of vectors is used to calculate the plane normal vector of the candidate plane formed by sample points p1, p2, and p3. in, It is the vector from sample point p1 to sample point p2. It is the vector from sample point p1 to sample point p3; When the angle between the point cloud normal vectors of sample points p1, p2 and p3 and the plane normal vector n of the candidate plane is less than the angle threshold β1, the candidate plane formed by sample points p1, p2 and p3 is taken as the fitted plane shape; otherwise, the step of randomly selecting three non-collinear point clouds from the wrapped point cloud data as sample points p1, p2 and p3 is repeated.

5. The parcel specification detection method according to claim 1, characterized in that, The reference geometry includes a cylindrical shape. The geometric feature parameters of the fitted cylindrical shape, determined based on selected sample points, include: Two point clouds were randomly selected from the package point cloud data as sample points p4 and p5; The normal vector of the candidate central axis ν is obtained by calculating the vector product of the point cloud normal vector n4 of sample point p4 and the point cloud normal vector n5 of sample point p5, and the projection plane Pv perpendicular to the normal vector of the candidate central axis ν is determined. Determine the projection line L'4 of the line L4 containing the point cloud normal vector n4 of sample point p4 onto the projection plane Pv, and determine the projection line L'5 of the line L5 containing the point cloud normal vector n5 of sample point p5 onto the projection plane Pv. Determine the intersection point c of the projection lines L'4 and L'5. The line along the normal vector of the candidate central axis ν and passing through the intersection point c is taken as the candidate central axis ν. Determine the projection point p'4 of sample point p4 in projection plane Pv, and determine the projection point p'5 of sample point p5 in projection plane Pv. Use the average of the Euclidean distances between the intersection point c and the projection points p'4 and p'5 as the radius of the candidate cylinder cross section. When the difference between the Euclidean distance between the projection point p'4 and the intersection point c and the radius of the candidate cylinder's cross section is less than the distance threshold λ1, and the difference between the Euclidean distance between the projection point p'5 and the intersection point c and the radius of the candidate cylinder's cross section is less than the distance threshold λ1, and the angle between the vector from the intersection point c to the projection point p'4 and the point cloud normal vector n4 of the sample point p4 is less than the angle threshold β2, and the angle between the vector from the intersection point c to the projection point p'5 and the point cloud normal vector n5 of the sample point p5 is less than the angle threshold β2, the cylinder shape determined based on the candidate central axis and the radius of the candidate cylinder's cross section will be used as the fitted cylinder shape; otherwise, the step of randomly selecting two point clouds from the wrapped point cloud data as sample points p4 and p5 will be repeated.

6. The parcel specification detection method according to claim 1, characterized in that, The reference geometry includes a sphere shape. The geometric feature parameters of the fitted sphere shape, determined based on selected sample points, include: Two point clouds are randomly selected from the package point cloud data as sample points p6 and p7. The line L6 passing through sample point p6 and along the direction of the point cloud normal vector n6 of sample point p6 is calculated, as well as the line L7 passing through sample point p7 and along the direction of the point cloud normal vector n7 of sample point p7 is calculated. Calculate the shortest line segment between line L6 and line L7, and use the midpoint of the shortest line segment as the candidate center of the sphere; The average of the Euclidean distances between the candidate sphere center and sample points p6 and p7 is taken as the radius of the candidate sphere. When the difference between the Euclidean distance between the candidate sphere center and sample point p6 and the radius of the candidate sphere is less than the distance threshold λ2, and the difference between the Euclidean distance between the candidate sphere center and sample point p7 and the radius of the candidate sphere is less than the distance threshold λ2, and the angle between the vector from the candidate sphere center to sample point p6 and the point cloud normal vector n6 of sample point p6 is less than the angle threshold β3, and the angle between the vector from the candidate sphere center to sample point p7 and the point cloud normal vector n7 of sample point p7 is less than the angle threshold β3, the sphere shape determined by the candidate sphere center and the candidate sphere radius will be used as the fitted sphere shape; otherwise, the step of randomly selecting two point clouds from the wrapped point cloud data as sample points p6 and p7 will be repeated.

7. The parcel specification detection method according to claim 3, characterized in that, The specifications of the package to be inspected are calculated based on the point cloud data of the package and the reference geometry with the highest confidence level. These specifications include: When the reference geometry with the highest confidence is a planar shape, determine the maximum connected point cloud set P. max The distribution area of ​​the point cloud on the fitted planar shape and the distribution area along the plane normal vector direction of the planar shape are used to obtain three-dimensional size information. The specification parameters of the package to be detected include: the package shape is a cube and the package size is the calculated three-dimensional size information. When the reference geometry with the highest confidence is a cylinder, determine the maximum connected point cloud set P. max The distribution area of ​​the point cloud along the central axis of the fitted cylindrical shape is used to determine the length of the cylinder. The specifications of the package to be detected are obtained, including: the package shape is a cylindrical shape, the radius of the package bottom surface is the radius of the cross section of the fitted cylindrical shape, and the package length is the calculated length of the cylinder. When the reference geometry with the highest confidence is a sphere, the specifications of the package to be detected include: the package shape is a sphere and the package radius is the radius of the fitted sphere shape.

8. The parcel specification detection method according to claim 1, characterized in that, The converted package point cloud data includes: The depth data of the packaged area is converted into three-dimensional point cloud data using the internal parameters of the RGBD smart depth camera, and the initial point cloud data is obtained by equidistant sampling. Radius filtering is completed by filtering out point clouds with fewer than a threshold number of neighboring point clouds in the initial point cloud data obtained from sampling. The neighboring point clouds of each point cloud include all point clouds within the filtering radius of the point cloud. For any point cloud in the initial point cloud data after radius filtering, calculate the average Euclidean distance μ and the label difference σ between the point cloud and its K nearest point clouds, and obtain the distance threshold max = μ + ε*σ. When the average Euclidean distance μ between the point cloud and its K nearest point clouds is greater than max, the point cloud is filtered out; otherwise, the point cloud is retained. ε is the threshold coefficient. After traversing all the point clouds in the initial point cloud data that has undergone radius filtering, the wrapped point cloud data is obtained.

9. The parcel specification detection method according to claim 8, characterized in that, The depth data of the extracted package area is converted to obtain package point cloud data, including: The depth data of the package area is extracted, and the depth data that is much larger than the suspension height of the RGBD smart depth camera is filtered out. The edges of the remaining depth data are then subjected to morphological processing to obtain the noise-removed depth data of the package area. Statistical analysis is performed on the depth data after noise removal within the package area. The height estimate of the package to be detected is obtained based on the reference plane position of the calibrated package detection area. Depth data in the noise-removed depth data within the package area that deviates from the height estimate by a deviation threshold are filtered out to obtain the processed depth data of the package area. Then, the processed depth data of the package area is converted to obtain package point cloud data.

10. The parcel specification detection method according to claim 1, characterized in that, The target detection model used is designed based on the PP-PicoDet series target detection model. The target detection model uses a lightweight LCNet neural network as the feature extraction layer, uses a PicoHeadV2 backbone network as the target detection head structure, and combines it with the LCPAN module. During model training, data augmentation techniques and batch processing strategies such as batch random adjustment of image size and normalization are used to expand the training sample set. A momentum optimizer combined with cosine decay and linear warm-up strategies is used to adjust the learning rate.

Images