Material sorting method and material sorting apparatus

Abstract

A material sorting method, specifically comprising: acquiring an X-ray image and a three-dimensional image of materials to be sorted; on the basis of at least one of the X-ray image and the three-dimensional image, identifying a material state of each material, wherein the material state includes a non-adhesion state and an adhesion state, and the adhesion state includes a state in which the material comes into contact with at least one of the remaining materials; for the X-ray image and the three-dimensional image, if material state identification has been executed, performing image segmentation on the basis of the material state of each material, and if the material state identification has not been executed, directly performing image segmentation on each material; on the basis of the material state of each material, matching each material in the X-ray image subjected to image segmentation with a corresponding material in the three-dimensional image subjected to image segmentation; and on the basis of a material matching result between the X-ray image and the three-dimensional image, performing material sorting. Further provided is a material sorting apparatus.

Description

Material sorting methods and material sorting devices

[0001] This disclosure claims priority to Chinese Patent Application No. 202411629953.4, filed on November 14, 2024, the contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of material sorting, and more specifically, to material sorting methods and material sorting apparatus. Background Technology

[0003] Material sorting refers to the process of classifying and selecting materials based on their specific properties or standards during production, logistics, and warehousing. For example, materials to be sorted are uniformly fed to the sorting area via a conveyor belt; then, X-ray imaging technology is used to scan the materials and identify the X-ray images; a computer analyzes this data, determines the blowing strategy, and transmits instructions to the blowing execution system; finally, the blowing execution system controls the opening and closing of the jet valves and the blowing time according to the instructions to blow the materials to the target sorting position.

[0004] In the material sorting process, ensuring high accuracy in material identification is crucial, which typically depends on the ability to accurately segment images of each material. Some materials are extremely similar in atomic number and color, making it difficult to clearly distinguish their differences using X-ray images. When similar materials with very similar atomic numbers and colors adhere to each other during transport, relevant X-ray image segmentation algorithms may fail to handle this accurately. For example, they may struggle to accurately segment images of adhered materials, or over-segment long, strip-shaped adhered materials into multiple independent parts. Summary of the Invention

[0005] In view of the above problems, this disclosure provides a material sorting method and a material sorting device.

[0006] According to a first aspect of this disclosure, a material sorting method is provided, comprising: acquiring an X-ray image and a three-dimensional image of the materials to be sorted; identifying the material state of each material based on at least one of the X-ray image and the three-dimensional image, wherein the material state includes a non-adhesive state and an adhesive state, and the adhesive state includes a state in contact with at least one other material; for the X-ray image and the three-dimensional image, if material state identification has been performed, performing image segmentation based on the material state of each material, and if material state identification has not been performed, directly performing image segmentation on each material; matching each material in the image-segmented X-ray image with the corresponding material in the image-segmented three-dimensional image based on the material state of each material; and sorting the materials according to the material matching results between the X-ray image and the three-dimensional image.

[0007] According to embodiments of this disclosure, material sorting based on the material matching results between the X-ray image and the three-dimensional image includes at least one of the following: if the material matching results show that at least one of the two successfully matched materials is in an adhered state and at least one has not undergone image segmentation processing, then a first sorting strategy is executed; if the material matching fails in either the X-ray image or the three-dimensional image, then the first sorting strategy is executed on the failed-matching material; if the material matching results show that at least one of the two successfully matched materials is in an adhered state and has already undergone image segmentation processing, then a second sorting strategy is executed; if the material matching results show that both successfully matched materials are in a non-adhesive state, then the second sorting strategy is executed.

[0008] According to embodiments of this disclosure, matching each material in the image-segmented X-ray image with its corresponding material in the image-segmented three-dimensional image based on the material state of each material includes: determining the correspondence between the materials in the X-ray image and the three-dimensional image based on the X-ray data of the X-ray image and the point cloud data of the three-dimensional image; for any two materials that have a correspondence between the X-ray image and the three-dimensional image, if at least one of the two materials is in an adhesive state, then a first matching strategy is executed; if the two materials are not in an adhesive state, then a second matching strategy is executed.

[0009] According to an embodiment of this disclosure, executing the first matching strategy includes: calculating the overlap of the two materials, wherein the overlap is determined based on at least one of the two-dimensional projected shape and the two-dimensional projected area of ​​the materials; and determining that the two materials with a corresponding relationship are successfully matched when the overlap is greater than or equal to a first value.

[0010] According to an embodiment of this disclosure, when the overlap is less than the first value, at least one of the two materials is identified as the first target material; the first target material is merged with the remaining N materials that are adjacent to it in the same image, where N is an integer greater than or equal to 1; the first target material is merged with at least one specific material to obtain merged material, wherein the merge degree between the specific material and the first target material is greater than or equal to a second value, and the merging process includes marking multiple materials as the same material for re-matching.

[0011] According to an embodiment of this disclosure, when the first target material is a material in the three-dimensional image, performing a merging degree detection on the first target material and the other N materials that are adjacent in the same image includes: performing a high continuity detection on the first target material and the N materials respectively, wherein the high continuity is positively correlated with the merging degree.

[0012] According to embodiments of this disclosure, determining at least one of the two materials as the first target material includes: comparing the two-dimensional projected area of ​​the material in an X-ray image with the two-dimensional projected area of ​​the material in a three-dimensional image; and selecting the material with the smaller two-dimensional projected area as the first target material.

[0013] According to embodiments of this disclosure, executing the second matching strategy includes: confirming a successful match when a single material in the X-ray image and a single material in the three-dimensional image form a one-to-one correspondence. If a single material in one of the X-ray images and the three-dimensional image corresponds to multiple materials in the other image, the multiple materials are merged into a single material to form a one-to-one correspondence, and the successful match is confirmed.

[0014] According to embodiments of this disclosure, for the X-ray image and the three-dimensional image, if material state recognition has been performed, image segmentation is performed based on the material state of each material; if material state recognition has not been performed, image segmentation is performed directly on each material, including: registering the X-ray image with a two-dimensional depth image, the two-dimensional depth image being obtained based on the three-dimensional image; based on the material state of each material, segmenting two-dimensional material images of each material from the registered X-ray image to obtain a third material set; segmenting depth material images of each material from the registered depth image to obtain a fourth material set; wherein, based on the material state of each material, matching each material in the image-segmented X-ray image with the corresponding material in the image-segmented three-dimensional image includes: using the material image in the third material set as a reference, matching the material image in the fourth material set.

[0015] According to an embodiment of this disclosure, executing the second sorting strategy includes: for any two materials that are successfully matched, obtaining the centroid and boundary information of the materials in the three-dimensional image; and performing blowing based on the centroid and boundary information of the materials in the three-dimensional image.

[0016] According to an embodiment of this disclosure, obtaining the centroid of the material in the three-dimensional image includes: calculating the sum of the heights of all pixels corresponding to the material in the three-dimensional image; weighted summing of the width coordinates of all pixels and then dividing by the sum of the heights to obtain the centroid width coordinates; and weighted summing of the length coordinates of all pixels and then dividing by the sum of the heights to obtain the centroid length coordinates.

[0017] According to an embodiment of this disclosure, the weighted summation includes: multiplying the height of each pixel point as a weight by its coordinate value to obtain a weighted result, wherein the coordinate value includes a width coordinate value or a length coordinate value; and summing the weighted results of all the pixels.

[0018] According to embodiments of this disclosure, executing the second sorting strategy further includes: identifying the category of the second target material using the X-ray image and the three-dimensional image, wherein the second target material is a material to be identified that conforms to the second sorting strategy; wherein blowing based on the centroid and boundary information of the material in the three-dimensional image includes: determining blowing parameters based on the category of the second target material; and blowing according to the blowing parameters based on the centroid and boundary information of the material in the three-dimensional image.

[0019] According to embodiments of this disclosure, identifying the category of a second target material based on the X-ray image and the three-dimensional image includes: identifying the X-ray image to obtain a first confidence score for the second target material, wherein the first confidence score represents the score of the second target material belonging to a first category; identifying the three-dimensional image to obtain a second confidence score for the second target material, wherein the second confidence score represents the score of the second target material belonging to the first category; performing a weighted calculation based on the first confidence score and its first weight, and the second confidence score and its second weight, to obtain a third confidence score; if the third confidence score is greater than or equal to a third value, the second target material is determined to belong to the first category, otherwise it is determined to belong to the second category.

[0020] According to embodiments of this disclosure, identifying the category of a second target material based on the X-ray image and the three-dimensional image includes: inputting the X-ray image into a first network branch of an image recognition model to extract a first feature; inputting the three-dimensional image into a second network branch of the image recognition model to extract a second feature, wherein the image recognition model is constructed based on a deep learning network; concatenating the first feature and the second feature, and inputting the concatenated feature into a fully connected layer of the image recognition model; and having the classification layer of the image recognition model process the output of the fully connected layer to identify the category of the second target material.

[0021] Another aspect of this disclosure provides a material sorting device, characterized in that it includes: a conveying module for conveying materials to be sorted; an X-ray image acquisition module for capturing X-ray images of the materials to be sorted; a three-dimensional image acquisition module for capturing three-dimensional images of the materials to be sorted; a control module for acquiring the X-ray images and the three-dimensional images, and executing the material sorting method as described in any of the preceding embodiments to generate a sorting instruction; and a blowing module for blowing the materials leaving the conveying module in response to the sorting instruction from the control module. Attached Figure Description

[0022] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0023] Figure 1 schematically illustrates the structure of a material sorting apparatus according to an embodiment of the present disclosure;

[0024] Figure 2 schematically illustrates a flowchart of a material sorting method according to an embodiment of the present disclosure;

[0025] Figure 3 schematically illustrates a flowchart of material matching according to an embodiment of the present disclosure;

[0026] Figure 4 schematically illustrates a flowchart of material matching according to another embodiment of the present disclosure;

[0027] Figure 5 schematically illustrates a flowchart of the execution of a second sorting strategy according to an embodiment of the present disclosure;

[0028] Figure 6 schematically illustrates a flowchart for obtaining the centroid of a material in a three-dimensional image according to an embodiment of the present disclosure;

[0029] Figure 7 schematically illustrates a flowchart of material matching and nozzle opening / closing time calculation based on point cloud data according to an embodiment of the present disclosure;

[0030] Figure 8 schematically illustrates a flowchart of material matching based on depth images and calculation of nozzle opening and closing time according to an embodiment of the present disclosure;

[0031] Figure 9 schematically illustrates a flowchart for identifying the category of a second target material according to an embodiment of the present disclosure;

[0032] Figure 10 schematically illustrates a flowchart for identifying the category of a second target material according to another embodiment of the present disclosure;

[0033] Figure 11 schematically illustrates a block diagram of an electronic device suitable for implementing a material sorting method according to an embodiment of the present disclosure.

[0034] The reference numerals in the above figures are as follows: 100, conveying module; 210, three-dimensional image acquisition module; 220, X-ray image acquisition module; 300, jetting module; 400, sorting bin; 500, control module.

[0035] It should be noted that, for clarity, the dimensions of the overall / partial structure or the overall / partial region in the drawings used to describe the embodiments of this disclosure may be enlarged or reduced, i.e., these drawings are not drawn to actual scale. Detailed Implementation

[0036] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0037] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0038] Figure 1 schematically shows a structural diagram of a material sorting apparatus according to an embodiment of the present disclosure.

[0039] As shown in Figure 1, the material sorting device includes a conveying module 100, a three-dimensional image acquisition module 210, an X-ray image acquisition module 220, a spraying module 300, a sorting bin 400, and a control module 500. The conveying module 100 is used to convey the material to be sorted; the X-ray image acquisition module 220 is used to capture X-ray images of the material to be sorted; the three-dimensional image acquisition module 210 is used to capture three-dimensional images of the material to be sorted; the control module 500 is used to acquire the X-ray images and the three-dimensional images, and execute the material sorting method provided in any embodiment of this disclosure to generate sorting instructions; the spraying module 300 is used to spray the material leaving the conveying module 100 in response to the sorting instructions from the control module 500.

[0040] For example, the conveying module 100 can be one or a combination of a horizontally arranged conveyor belt, an inclined conveyor belt, and an angled inclined slide. Hereinafter, unless otherwise stated, the conveying module 100 is a horizontally arranged conveyor belt. Materials of different types and particle sizes are distributed along the length direction (i.e., the conveying direction) and width direction on the conveying module 100. Due to the conveyor belt's certain operating speed, different materials are scattered on the conveyor belt, and when transported to the end position, the materials are thrown out of the conveyor belt in a projectile motion. Referring to the coordinate system in the accompanying drawings, the length direction of the conveying module 100 is designated as the first direction (y-direction), the width direction as the second direction (x-direction), and the height direction as the third direction (z-direction).

[0041] For example, the X-ray image acquisition module 220 may include an X-ray source and an X-ray detector. For instance, the X-ray source may be positioned above the material to be sorted, while the X-ray detector may be positioned below the material, such as below the conveyor belt of the conveyor module 100. The X-ray detector receives X-rays that have penetrated the material and converts them into an X-ray image.

[0042] For example, in the sorting of coal and gangue, an X-ray source can transmit dual-energy X-rays through the coal and gangue. The dual-energy detector acquires an analog electrical signal that reflects the intensity of the transmitted X-rays. By performing analog-to-digital conversion, images of the coal and gangue are obtained. Then, based on the different attenuation intensities of the transmitted X-rays, sorting parameters are calculated from the images, thereby identifying and sorting the coal and gangue.

[0043] The dual-energy detector is capable of simultaneously detecting both the high-energy and low-energy components of transmitted X-rays. The control module 500 may include a display device (not shown) configured to display transmission intensity data generated using the high-energy and low-energy components of the X-rays. That is, in some optional embodiments of this disclosure, transmission intensity data can be determined and corresponding transmission images generated solely based on either the high-energy or low-energy component of the X-rays transmitted through coal and gangue.

[0044] For example, the 3D image acquisition module 210 may include a 3D structured light camera. This camera projects light with a specific pattern (such as striped or grid-like light) onto the surface of a material. By analyzing the deformation of the light pattern reflected from the material surface, the 3D shape of the material is calculated to obtain a 3D image. For example, the 3D image acquisition module 210 may include a line laser binocular stereo camera. This camera uses a linear laser as a light source, projects a straight laser beam, and is equipped with two cameras that capture the projection of the laser line onto the material surface from different angles. By analyzing the positional differences of the laser line in the images captured by the two cameras, the 3D information of the material is calculated to obtain a 3D image.

[0045] Exemplarily, the blowing module 300 includes one or more nozzles, such as a plurality of nozzles arranged in an array. The blowing module 300 may also include a solenoid valve and a gas supply device, each nozzle being connected to the solenoid valve, which may be a high-frequency solenoid valve. The solenoid valve communicates with the gas supply device (not shown). The gas supply device contains compressed gas to provide a gas source for the nozzles to blow different types of materials.

[0046] In this embodiment, the blowing module 300 is located below the material movement trajectory. In this disclosure, the material movement trajectory refers to the movement trajectory of the material after it leaves the end of the conveying module 100. In this embodiment, by placing the blowing module 300 below the movement trajectory, a blowing force can be applied to the material, reducing energy consumption while improving the impact of different blowing positions on sorting accuracy. For example, when the material reaches the blowing position, the blowing module 300 can instantly spray a high-pressure airflow for a specific duration, changing the material's movement trajectory through the airflow, causing the material to fall into the corresponding sorting bin 400.

[0047] In some embodiments, the blowing pressure can also be controlled by adjusting the number of nozzles. For example, when the material area is large, a larger number of nozzles can be used to blow the material surface, and when the material area is small, a smaller number of nozzles can be used to blow the material surface, so that the blowing distance (horizontal movement distance) of materials of different volumes and weights is maintained within a certain range.

[0048] In some embodiments, during the sorting process, the control module 500 can adjust the blowing strategy in real time, such as increasing or decreasing the pressure of the nozzle to change the intensity of the airflow, thereby affecting the flight distance and direction of the material; adjust the spray angle of the nozzle so that the airflow direction is perpendicular to the tangent at a certain point on the material's flight trajectory; and adjust the start and stop times of the blowing to match the speed and position of the material passing through the nozzle.

[0049] The material sorting method of this disclosure will be described in detail below based on the material sorting device described in FIG1, with reference to FIG2 to FIG10.

[0050] Figure 2 schematically illustrates a flowchart of a material sorting method according to an embodiment of the present disclosure.

[0051] As shown in Figure 2, this embodiment includes:

[0052] In operation S210, X-ray images and three-dimensional images of the material to be sorted are acquired.

[0053] Referring to Figure 1, in this embodiment of the present disclosure, the installation height of the X-ray image acquisition module 220 and the three-dimensional image acquisition module 210 can be adjusted according to the width of the conveyor belt, the viewing angle, etc. For example, a fan-shaped X-ray perpendicular to the bearing surface of the conveyor belt covers the entire width of the conveyor belt. Taking the position of the X-ray source as a reference, the left side (inlet) direction in Figure 1 is considered the front, and the right side (outlet) direction in Figure 1 is considered the rear. A three-dimensional structured light camera is added in front of the X-ray source, and the imaging range of the three-dimensional structured light camera covers the entire width of the conveyor belt. Thus, the X-ray image acquisition module 220 captures X-ray images of the material on the conveyor belt, and the three-dimensional image acquisition module 210 captures three-dimensional images of the material on the conveyor belt. It can be understood that the X-ray image acquisition module 220 and the three-dimensional image acquisition module 210 can respectively capture images of a partial or complete area on the conveyor belt.

[0054] In operation S220, based on at least one of the X-ray image and the three-dimensional image, the material state of each material is identified, wherein the material state includes a non-adhesive state and an adhesive state, and the adhesive state includes a state in contact with at least one other material.

[0055] For example, the system can be pre-programmed to identify either X-ray images or 3D images, or simultaneously. As another example, the control module 500 can perform image quality assessments upon receiving both X-ray and 3D images to determine if their quality meets the criteria for material state recognition. Based on the image quality assessment results, it decides whether to recognize the X-ray image, the 3D image, or both. For instance, it can use one or more metrics such as sharpness, contrast, and signal-to-noise ratio to characterize image quality, with a pre-set quality threshold. Images with quality greater than or equal to the quality threshold are selected for recognition. If the quality of both the X-ray and 3D images is below the quality threshold, the image with the higher quality is selected for recognition.

[0056] For X-ray images and 3D images where material state recognition has not been performed, the contours of each material in the image are directly detected, and then image segmentation is performed on each contour.

[0057] For example, an adhered state means that adjacent materials are in contact with each other on the conveyor belt, such as one material being above another. A non-adhesive state means that a single material has a certain gap with the surrounding materials, such as that their projections on the bearing surface of the conveyor belt do not intersect.

[0058] In some embodiments, for example, a pre-trained deep learning model can be used to process X-ray images and / or 3D images to effectively identify the material state of each material. For example, the deep learning model includes an edge detection module and an adhesion state recognition module. The edge detection module can run an edge detection algorithm (such as the Canny edge detection method) to identify the edge contours (e.g., represented by a set of pixels) of each material in the input image. Then, the edge contours of each identified material output by the edge detection module are input to the adhesion state recognition module. The adhesion state recognition module can be built based on a convolutional neural network and is capable of predicting whether adhesion exists based on the edge contours of each material. The input images include X-ray images and / or 3D images.

[0059] For example, graphics recognition methods, such as edge detection, contour analysis, and convex hull analysis, can be used to identify the material state of each material. For instance, detecting convex defects in materials within X-ray and / or 3D images effectively identifies the adhesion state of each material. In image processing and computer vision, a convex defect typically refers to a portion of an object's contour or surface that protrudes or extends further than the surrounding area. When two materials adhere, convex defects may manifest as bulges or irregular protrusions on the material surface due to their stacking. Effectively identifying convex defects helps assess the adhesion state of each material. Besides detecting convex defects, adhesion can also be identified by detecting boundary overlap, grayscale differences, or shape features.

[0060] When operating S230, for X-ray images and 3D images, if material state recognition has been performed, image segmentation is performed based on the material state of each material; if material state recognition has not been performed, image segmentation is performed directly on each material.

[0061] In some embodiments, the X-ray beam emitted by the X-ray source inevitably passes through a conveyor belt during scanning. This conveyor belt is composed of a continuous ring of uniform material, but its joints are bonded together using a vulcanization process. The vulcanized portion shows little difference in imaging compared to thin, small materials in low-energy X-ray images, but a greater difference in high-energy X-ray images. Therefore, to reduce the interference of the vulcanized portion on image segmentation, high-energy X-ray images are preferred for image segmentation in the following description.

[0062] For example, regardless of whether it's an X-ray image or a 3D image, if the material state of each material in the image is not identified, the contours of each material are directly detected, and image segmentation is performed based on the material contours. If the material state of each material in the image is identified, different image segmentation operations are performed depending on whether the material is in an adhesive state. Specifically, image segmentation of adhesive materials in this disclosure refers to performing multiple segmentation operations. First, the overall contour containing multiple materials can be segmented to obtain an overall material image. Then, segmentation operations are performed on the overall material image to obtain images of each individual material. Image segmentation of non-adhesive materials in this disclosure refers to performing a single segmentation operation, that is, directly segmenting the image of the non-adhesive material.

[0063] For example, X-ray images have already undergone material state recognition. During the processing of X-ray images, X-ray image processing algorithms (such as the Otsu algorithm, adaptive thresholding algorithm, etc.) can be used to detect the material contours. Then, by detecting the depth of the convex defects in the contours, non-adhesive and adherent materials can be distinguished. For non-adhesive materials, the independent contour of each material can be clearly identified. Once its contour is detected, it is considered to meet the segmentation conditions, and region segmentation is performed directly based on these contours. Adhesive materials, on the other hand, are considered as a single independent contour, which usually contains multiple materials. For adhesive materials, for example, the multiple adhesive materials can be segmented from the X-ray image as a whole. Based on the obtained overall material image, the materials can be further segmented into individual units. This can be achieved through concave point detection and matching algorithms. This algorithm first identifies the connection points between the materials, i.e., concave points, and then determines the optimal segmentation path based on the characteristics of the concave points using certain criteria, further segmenting each material in the adhesive material. The segmentation conditions can be flexibly set according to the parameters of the concave point detection and concave point matching algorithm. For example, if the optimal segmentation path is not determined, the segmentation conditions are not met, and each material in the adhesive material will no longer be segmented.

[0064] For example, in the process of processing a 3D image for material state recognition, a 3D image processing algorithm (such as a 3D Canny boundary detector) can be used to detect the 3D contours of each material. For materials in a non-adhesive state, independent 3D contours can be detected, while multiple materials in an adherent state are considered as a single independent 3D contour. For multiple materials in an adherent state, they are first separated from the 3D image as a whole, and then the overall material image is input into a 3D convolutional neural network for voxel-level image segmentation, further dividing the materials into individual units. Alternatively, a discriminator based on deep learning algorithms can be provided to evaluate the accuracy of the segmentation of the material image output by the 3D convolutional neural network. If the evaluation result shows inaccurate segmentation, the segmentation condition is not met, and the original state is restored without further segmentation. This 3D convolutional neural network is pre-trained using image samples of materials in an adherent state, enabling it to learn the complex boundaries between material adhesions and provide high-precision segmentation results. The discriminator can be trained synchronously with the 3D convolutional neural network. For materials in a non-adhesive state, once their contours are detected, the segmentation condition is considered met, and region segmentation is performed directly based on these contours.

[0065] For the same material, since the position and shape displayed by X-ray images and 3D images may differ, the two images of the same material are matched before being used for further sorting.

[0066] In operation S240, based on the material state of each material, each material in the image-segmented X-ray image is matched with the corresponding material in the image-segmented three-dimensional image.

[0067] In some embodiments, the three-dimensional structured light camera is adjusted according to the pixel size of the X-ray image acquisition module 220 so that its spatial resolution in the second direction of the conveyor belt is consistent with the X-ray pixel size, or is an integer multiple or fractional multiple of the pixel size. Simultaneously, the sampling frequency of the three-dimensional structured light camera is set to be consistent with the sampling frequency of the X-ray image acquisition module 220. After the equipment parameters such as the sampling frequency are fixed, during the material transport process in the conveying module 100, the material first passes through the acquisition area of ​​the three-dimensional structured light camera, and a three-dimensional image is acquired at time t1. Then, after traveling a fixed distance, it passes through the acquisition area of ​​the X-ray image acquisition module 220, and a dual-energy X-ray image is acquired at time t2. Since the acquisition time difference (the difference between time t1 and time t2) between the three-dimensional structured light camera and the X-ray image acquisition module 220 is fixed, the control module 500 can correct the temporal offset between the three-dimensional image and the X-ray image based on the fixed time difference, that is, determine the three-dimensional image acquired at time t1 and the X-ray image acquired at time t2 as images with the same material set. Operation S240 matches the three-dimensional image and the X-ray image that have been corrected in terms of temporal offset.

[0068] For example, for any given material, preprocessing can be performed on its 3D image and X-ray image through operations such as translation, rotation, and scaling. Then, matching can be performed using positional similarity, shape similarity, and whether the two-dimensional projected area is greater than a corresponding threshold. Positional similarity reflects the proximity of the material's center in the 3D and X-ray images; shape similarity reflects the degree of overlap of the material's boundaries in the 3D and X-ray images; and the two-dimensional projected area reflects the difference in the material's two-dimensional area between the 3D and X-ray images. The X-ray image is a top-down 2D image of the material, and the size of the area occupied by the material is considered its two-dimensional projected area in the X-ray image.

[0069] In some embodiments, operation S240 may include determining the correspondence between materials in the X-ray image and the three-dimensional image based on X-ray data of the X-ray image and point cloud data of the three-dimensional image; for any two materials that have a correspondence between the X-ray image and the three-dimensional image, if at least one of the two materials is in an adhesive state, then a first matching strategy is executed; if the two materials are not in an adhesive state, then a second matching strategy is executed.

[0070] For example, if only one of the X-ray images and the 3D image is used for material state recognition, in which case at least one of the two materials is in an adherent state, the material in one image is marked as adherent, while the corresponding material in the other image has no state mark; in which case the two materials are not in an adherent state, the material in one image is marked as non-adhesive, while the corresponding material in the other image has no state mark.

[0071] For example, both X-ray images and 3D images perform material state recognition. In this case, at least one of the two materials is in an adhesive state, which means that the material in one image is marked as adhesive, while the corresponding material in the other image is marked as either adhesive or non-adhesive. In this case, the two materials are not in an adhesive state, which means that the materials in both images are marked as non-adhesive.

[0072] According to embodiments of this disclosure, by designing different matching strategies for different types of material states, the overall matching quality can be improved and errors that may occur in subsequent processing can be reduced.

[0073] In operation of S250, material sorting is performed based on the material matching results between X-ray images and 3D images. For example, the material category can be identified based on the material matching results, and a blowing strategy can be determined, thereby completing the sorting by executing the blowing process.

[0074] According to embodiments of this disclosure, three-dimensional information can be used to more accurately capture the continuity of materials in three-dimensional space, and material image segmentation and sorting can be performed by combining X-ray information. For example, for materials with similar atomic numbers and indistinct color features, but different material shapes and surface roughness, or different three-dimensional shapes, the sorting accuracy can be improved.

[0075] Figure 3 schematically illustrates a flowchart of material matching according to an embodiment of the present disclosure.

[0076] As shown in Figure 3, this embodiment is one example of operating S240, including:

[0077] In operation S310, the correspondence between materials in the X-ray image and the three-dimensional image is determined based on the X-ray data of the X-ray image and the point cloud data of the three-dimensional image.

[0078] For example, a correspondence can be determined based on positional information in X-ray and 3D images. Specifically, when the conveyor belt is running at a constant speed, the material presented in the X-ray and 3D images is the same, so the position of the same material relative to the conveyor belt is consistent in different images. After preprocessing and aligning the two images, materials with the same relative position are identified from the X-ray and point cloud data. Feature points, such as the center, corners, intersections, texture points, or edge points of the material, or one or more such features, are searched. A feature matching algorithm is used to match the consistency of the feature points, and a correspondence between the X-ray and 3D images is established based on the matching results. Alternatively, a correspondence can be established based on the material's grayscale value, color, geometric constraints, or structural similarity. The correspondence can be characterized by a matching score. When the matching score is higher than a certain value (e.g., above 80 out of 100), the same material identifier is assigned to establish the correspondence.

[0079] For example, two-dimensional material images of each material are extracted from the X-ray image to obtain a first material set. For materials in an adhered state, if the segmentation condition is met, image segmentation is performed to obtain a two-dimensional material image. Point cloud material images of each material are extracted from the three-dimensional image to obtain a second material set. For materials in an adhered state, if the segmentation condition is met, image segmentation is performed to obtain a point cloud material image. Determining the correspondence between materials in the X-ray image and the three-dimensional image includes determining the correspondence between the material images in the first material set and the second material set.

[0080] For any two materials that correspond to each other in the X-ray image and the 3D image, if at least one of the materials is in an adhered state, then the first matching strategy is executed, such as operation S320.

[0081] In operation S320, for any two materials that correspond to each other in the X-ray image and the 3D image, the overlap between the two materials is calculated. This overlap is determined based on at least one of the two-dimensional projected shape and two-dimensional projected area of ​​the materials. The overlap refers to a measure of the overlapping area between two or more materials in the X-ray image and the 3D image, and can be represented by a number between 0 and 1. For example, an overlap of 1 indicates that the two-dimensional projected shape and / or two-dimensional projected area of ​​the materials completely overlap. The X-ray image itself is a two-dimensional image, which presents the two-dimensional projected shape and two-dimensional projected area of ​​the materials.

[0082] In operation S330, when the overlap is greater than or equal to the first value, it is determined that two materials with a corresponding relationship have been successfully matched.

[0083] For example, for two-dimensional material images and point cloud material images that have a corresponding relationship, the overlap of the materials displayed by the two is calculated; wherein, when the overlap is greater than or equal to a first value, it is determined that the two-dimensional material image and the point cloud material image that have a corresponding relationship are successfully matched. At this time, it can be considered that the two-dimensional material images and point cloud material images of the two successfully matched materials are actually two-dimensional material images and point cloud material images obtained respectively based on the same material entity.

[0084] For example, the overlap can be calculated by only the shape similarity between two materials, or by only the two-dimensional projected area error between two materials.

[0085] In some embodiments, the overlap can be obtained by simultaneously utilizing the two-dimensional projected shape and two-dimensional projected area of ​​the materials to calculate the intersection of the two material projection regions. First, the three-dimensional material image is projected onto a two-dimensional plane (such as the xoy plane, i.e., the conveyor belt's bearing surface). The two-dimensional projected area of ​​the material in the three-dimensional image is calculated, and the boundary contour of the material is characterized using the coordinates of the xoy plane. The two-dimensional plane displayed in the X-ray image is the xoy plane; therefore, the area of ​​the material and its boundary contour in the X-ray image are directly calculated. The overlapping area can be calculated using the coordinates of the two contours (e.g., pixels with the same coordinates on the boundary contours are considered overlapping), and the intersection area is obtained. The overlap can be obtained by dividing the intersection area by the minimum two-dimensional projected area. The minimum two-dimensional projected area is the smaller two-dimensional projected area of ​​the two materials in the X-ray image and the three-dimensional image. The first value can be selected based on the type of material, sorting requirements, or production experience; for example, it can be 0.8, 0.9, etc., without specific limitations here.

[0086] When the overlap is less than the first value, operations S340 to S360 are executed.

[0087] In operation S340, at least one of the two materials is identified as the first target material.

[0088] In some embodiments, the two-dimensional projected area of ​​the material in the X-ray image and the two-dimensional projected area of ​​the material in the three-dimensional image can be compared, and the material with the smaller two-dimensional projected area can be selected as the first target material. The reason is that only when the two-dimensional projected area is smaller is it necessary to merge the two-dimensional projected areas to increase the total area.

[0089] In operation S350, the first target material is combined with the remaining N materials that are adjacent to it in the same image to perform a degree of merging detection, where N is an integer greater than or equal to 1.

[0090] For example, merging degree refers to the degree to which two materials meet specific merging criteria, such as whether the two materials are physically close or have similar properties. For instance, merging degree can be measured by at least one parameter such as atomic number, physical distance, dimensional properties, and color similarity. Merging degree can be represented by a number between 0 and 1. For example, a merging degree of 1 indicates that the first target material has identical atomic numbers and other parameters to the other N materials located adjacent to it in the same image. Merging degree detection is used to assess the similarity between the first target material and its surrounding materials in order to determine whether to merge them.

[0091] In operation S360, the first target material is merged with at least one specific material to obtain merged material. The degree of merging between the specific material and the first target material is greater than or equal to a second value. The merging process includes marking multiple materials as the same material for re-matching, and fusing the images of these multiple materials. For example, fusing the boundary contour of the specific material in an X-ray image with the boundary contour of the first target material, or fusing the point cloud of the specific material in a 3D image with the point cloud of the first target material. The second value is a preset threshold used to determine whether two materials meet the merging criteria. For example, the second value can be set to 0.7.

[0092] Further elaborating on Figure 3, if the image segmentation is accurate when the material is in a non-adhesive state, then the two-dimensional projected area in the X-ray image and the two-dimensional projected area in the three-dimensional image are consistent, and the overlap is greater than or equal to the first value. When the material is in an adhesive state, several situations may exist:

[0093] 1. If multiple materials in an adhered state are correctly segmented twice in both X-ray and 3D images, then the overlap of each material during matching is greater than or equal to the first value.

[0094] 2. If multiple materials in an adhered state are not further segmented after the first segmentation operation to obtain the overall image in the X-ray image, and are also not further segmented after the first segmentation operation to obtain the overall image in the 3D image, then the overlap of the two overall images is greater than or equal to the first value when matching.

[0095] The first matching strategy may require merging processes in the following situations:

[0096] 3. If multiple materials in an adhered state are correctly segmented in the X-ray image but not in the 3D image, the overlap between the two images is calculated to be less than a first value based on the correctly segmented 2D material image and the unsegmented 3D material image. Then, the multiple 2D material images are merged in the X-ray image. Then, operation S310 and subsequent operations are executed repeatedly until the multiple materials in the adhered state are successfully matched between the two images as a whole.

[0097] 4. If multiple materials in an adhered state are not segmented twice in the X-ray image but are correctly segmented twice in the 3D image, then the overlap between the 3D material image obtained from the correct segmentation and the 2D material image that was not segmented twice is calculated to be less than a first value. In the 3D image, the multiple 3D material images are merged. Then, operation S310 and subsequent operations are executed repeatedly until the multiple materials in the adhered state are successfully matched between the two images as a whole.

[0098] 5. Multiple materials in an adhered state are correctly segmented in the X-ray image, but segmentation is incorrect in the 3D image. Alternatively, multiple materials in an adhered state are incorrectly segmented in the X-ray image, but are correctly segmented in the 3D image. Or, multiple materials in an adhered state are incorrectly segmented in the X-ray image, and segmentation is also incorrect in the 3D image.

[0099] In the fifth scenario, if a material segmented from one image cannot be identified as a corresponding material in another image, the match is considered to have failed. Alternatively, if a material segmented from one image can be identified as a corresponding material in another image, but the calculated overlap is less than a first value, and after multiple (e.g., 3 times, just an example) merging processes and re-matching, the overlap is still less than the first value, then the match is considered to have failed.

[0100] As shown in Figure 3, after executing operation S360, the loop returns to execute operation S310 and subsequent operations for rematching. Rematching includes both successful and unsuccessful matches within a predetermined number of loops (e.g., 3 times, for example only).

[0101] For successful matches, the image is marked as "materials in an adhered state have not undergone image segmentation." For failed matches, the image is marked as "match failed." Merging is used as a fallback strategy; successful matches can obtain other images that might have been incorrectly segmented, thus reducing the probability of incorrect matches. For example, incorrect matches may involve two material images that are not the same material entity still matching successfully. For material A in an adhered state, merging can be prioritized over successful matches. That is, if material A matches successfully with another image, and is determined to be merged with the adjacent failed-match material B, then merging is performed.

[0102] According to embodiments of this disclosure, multiple incorrectly segmented material images can be merged through merging processing. For example, if materials in an adhesive state are incorrectly segmented, the merging processing can restore the overall material image before the secondary segmentation. This helps to pre-screen incorrectly segmented material images and perform targeted processing afterward, avoiding subsequent identification and blowing errors, which could lead to sorting errors.

[0103] In other embodiments, not all successfully matched images are marked as "materials in an adhered state have not undergone image segmentation processing". For example, when the merged material image is the overall material image before secondary segmentation, it is marked as "materials in an adhered state have not undergone image segmentation processing"; when the successfully matched merged material image is not the overall material image before secondary segmentation, it can be marked as "successful match". For example, if the material image of the same material entity in an X-ray image or a 3D image is incorrectly segmented into multiple images (the first target image is one of them), the merging process can successfully match the 2D and 3D material images of the same material entity.

[0104] In some embodiments, when the first target material is a material in a three-dimensional image, performing a merging degree detection on the first target material and the other N materials that are adjacent to it in the same image includes: performing a high continuity detection on the first target material and the N materials respectively, where the high continuity is positively correlated with the merging degree.

[0105] For example, materials adjacent to the first target material are materials located nearby.

[0106] In some embodiments, performing height continuity detection on the first target material and N materials respectively includes: calculating the height difference between the first target material and each of the N materials.

[0107] For example, height continuity refers to the smoothness of the connection between materials in terms of height. For instance, if the height difference between two materials is below a certain value, the smoothness of their connection is high. For example, the height continuity index can be obtained as the ratio of the absolute value of the height difference between materials to a reference value (such as the average height of the materials). For example, a height continuity index less than 0.1 indicates that the two materials are continuous. Assuming material A has a height of 5 cm, material B has a height of 3 cm, a height difference of 2 cm, an average height of 4 cm, and a height continuity index of 0.5, then the smoothness of the connection between materials A and B is low, and their degree of merging is low.

[0108] According to embodiments of this disclosure, by using highly continuous detection, it is possible to more accurately determine whether materials can be combined, thereby improving identification accuracy.

[0109] In some embodiments, when the first target material is a material in an X-ray image, performing a merging degree detection on the first target material and the other N materials located adjacent to it in the same image includes: firstly determining the first target material and one or more materials adjacent to it, calculating the physical distance, relative atomic number similarity, and color similarity of the image information of each material, and assigning values ​​to obtain the merging degree.

[0110] For example, firstly, the physical distance between the first target material and each adjacent material is calculated. The physical distance can be calculated using the shortest distance between the boundary contours of the materials, or the distance between the center coordinates of the materials. When the first target material and its adjacent materials are obtained from a secondary segmentation of the overall material image, the shortest distance between the boundary contours is 0; otherwise, the Euclidean distance between the center coordinates is calculated. Then, the relative atomic number similarity between the first target material and each adjacent material is calculated, followed by the color similarity. For example, if the color vector of tissue A is (120, 130, 140) and the color vector of adjacent tissue B1 is (122, 128, 139), then the color similarity s(A, B1) = 0.98. The physical distance and color similarity are assigned values ​​respectively, and then summed to obtain the merging degree. The closer the physical distance, the higher the weight; the larger the relative atomic number, the higher the weight; and the greater the color similarity, the higher the weight.

[0111] In some embodiments, when the first target material is a material in an X-ray image, performing a merging degree detection on the first target material and the other N materials located adjacent to it in the same image further includes: performing a reverse search to see if the material in the three-dimensional image still has a corresponding relationship with the other N materials; if a one-to-many correspondence exists, performing a high continuity detection on the material itself in the three-dimensional image. Specifically, merging the first target material with at least one specific material to obtain merged material includes: if the high continuity detection result indicates a merging degree greater than or equal to a second value, then merging the first target material with at least one specific material to obtain merged material; if the high continuity detection result indicates a merging degree less than the second value, then segmenting the material in the three-dimensional image according to the segmentation result of the X-ray image, and then matching the segmented materials one by one.

[0112] In some embodiments, executing the second matching strategy includes: confirming successful matching when a single material in an X-ray image and a single material in a three-dimensional image form a one-to-one correspondence; and confirming successful matching when a single material in one of the X-ray images and the three-dimensional image has a correspondence with multiple materials in the other image, and a one-to-one correspondence is formed after merging the multiple materials into a single material.

[0113] For example, material state identification is performed only on X-ray images to determine whether the materials are adhered. Different matching strategies are then executed based on the adhesion determination result. When executing the second matching strategy, if the material is determined to be non-adhesive, the X-ray image segmentation algorithm does not further segment it into multiple materials. Then, it identifies whether a single material in the X-ray image partially or completely overlaps with one or more materials in the 3D image. For example, if the overlap is greater than 0.1, a correspondence is determined, and all point cloud material images that intersect with the X-ray material image (e.g., overlap greater than 0.1) are identified. If only one point cloud material overlaps, both are considered correctly segmented, achieving a one-to-one matching. If more than one point cloud material overlaps, it indicates missegmentation of the point cloud data. In this case, these missegmented point cloud materials are merged to ensure that the merged single material correctly matches the single material based on the X-ray image.

[0114] Figure 4 schematically illustrates a flowchart of material matching according to another embodiment of the present disclosure.

[0115] As shown in Figure 4, in this embodiment, operations S410 to S420 are one embodiment of operation S230, and operation S430 is one embodiment of operation S240, specifically including:

[0116] In operation S410, the X-ray image is registered with a two-dimensional depth image, which is obtained based on the three-dimensional image.

[0117] In operation S420, based on the material state of each material, a two-dimensional material image of each material is extracted from the registered X-ray image to obtain a third material set; a depth material image of each material is segmented from the registered two-dimensional depth image to obtain a fourth material set; wherein, for materials in an adhesive state, if the segmentation condition is met, image segmentation is performed to obtain a depth material image.

[0118] In operation S430, the material image in the third material set is used as a reference to match the material image in the fourth material set.

[0119] For example, the projection transformation matrix from the X-ray image to the depth image is obtained in advance. After receiving the depth image and the X-ray image, the control module 500 registers the X-ray image and the depth image through the projection transformation matrix, then performs image segmentation on the registered X-ray image to separate the adhered materials, and then matches the materials in the depth image with the materials in the registered X-ray image one by one.

[0120] Using the material images in the third material set as a benchmark means that all material images in the third material set are assumed to be correctly segmented. When the material at the same location in the depth image does not match the material in the X-ray image, the material in the depth image is re-segmented based on the material image segmented from the X-ray image. This allows the use of a highly accurate X-ray image segmentation algorithm to provide a reference for depth image segmentation, simplifying the decision-making process and reducing the time and resources spent on decision-making during the matching process.

[0121] According to embodiments of this disclosure, based on the results of image segmentation, materials in X-ray images are matched with corresponding materials in three-dimensional images, which can effectively overcome the limitations of X-ray segmentation and make full use of the three-dimensional characteristics of three-dimensional data (such as point cloud data) to achieve higher-precision material segmentation and matching.

[0122] In some embodiments, material sorting based on material matching results between X-ray images and three-dimensional images includes at least one of the following:

[0123] If the material matching results show that at least one of the two successfully matched materials is in an adhered state, and at least one has not undergone image segmentation processing, then the first sorting strategy is executed. If any material fails to match in the X-ray image or 3D image, then the first sorting strategy is executed on the material that failed to match. "Not undergoing image segmentation processing" means that after the first segmentation operation, the overall image was not correctly segmented a second time. As in the merging process of the first matching strategy above, successfully matched materials are marked as "adhered materials have not undergone image segmentation processing." Unmatched materials are marked as "match failed." Similarly, if multiple adhered materials are not correctly segmented a second time in both the X-ray image and the 3D image, they are marked as "adhered materials have not undergone image segmentation processing."

[0124] If the material matching results show that at least one of the two successfully matched materials is in an adhered state, and image segmentation processing has already been performed, then the second sorting strategy is executed. If the material matching results show that neither of the two successfully matched materials is in an adhered state, then the second sorting strategy is executed. Specifically, the two successfully matched materials are considered the same target material entity, and the second sorting strategy is executed on this target material entity. The image segmentation processing here involves performing a correct secondary segmentation after obtaining the overall image from the first segmentation operation.

[0125] According to embodiments of this disclosure, a variety of sorting strategies are provided, which can take into account object state, image segmentation process and material matching results, and apply targeted sorting strategies to improve material sorting effect.

[0126] For example, if at least one of any two successfully matched materials is in an adhered state, and at least one has not undergone secondary image segmentation, then it can be assumed that at least one or two materials were matched in an adhered state. In this case, due to the difficulty in determining the centroid of each individual material, it may be difficult to determine the accurate blowing position, and also difficult to perform accurate category identification. If any material in the X-ray image or 3D image fails to match, there may also be segmentation errors, which will also lead to difficulty in determining the centroid, inability to blow, and difficulty in performing accurate category identification. Therefore, the first sorting strategy can be to place the target material that conforms to the strategy into a specific location, and then it can be put back into the conveyor belt, or it can be manually sorted. For example, in the scenario of sorting concentrate and tailings, three sorting bins are set up, corresponding to the concentrate sorting bin, the tailings sorting bin, and the sorting bin that conforms to the first sorting strategy. As shown in Figure 1, the material that conforms to the first sorting strategy falls directly into its corresponding sorting bin by gravity. The concentrate and tailings are sorted into the concentrate sorting bin and the tailings sorting bin by blowing.

[0127] The second sorting strategy will be explained in more detail below.

[0128] Figure 5 schematically illustrates a flowchart of the execution of a second sorting strategy according to an embodiment of the present disclosure.

[0129] As shown in Figure 5, this embodiment is one example of implementing the second sorting strategy. Any two materials that are successfully matched are considered to be the same target material entity.

[0130] The S510 is used to acquire the centroid and boundary information of the material in the 3D image.

[0131] For example, when extracting point cloud data from a 3D image, boundary detection algorithms (such as the Canny boundary detector, Sobel operator, Laplacian operator, etc.) can be used to identify material boundaries. Boundary information can be represented by 3D coordinates. The centroid can be determined by calculating the weighted average position of the material pixels, for example, by weighted summation of the coordinates of each voxel (3D pixel).

[0132] In operation S520, blowing is performed based on the centroid and boundary information of the material in the 3D image. This operation blows the target material entity.

[0133] In some embodiments, the control module 500 can more efficiently determine the opening and closing time and spraying position of the control valve nozzle based directly on the centroid and boundary information of the material in the three-dimensional image.

[0134] In other embodiments, the centroid and boundary information of the material in the three-dimensional image can be projected onto the same two-dimensional plane as the X-ray image, and correction can be performed based on the projected centroid and boundary information. The control module 500 can determine the opening and closing time and spraying position of the control valve nozzle based on the corrected X-ray image.

[0135] For example, based on the boundary contour information of a 3D image projected onto a 2D plane (such as the X-ray plane), deformation correction (such as affine transformation or non-rigid transformation) is used to adjust the boundary contour of the material in the X-ray image to ensure the accuracy of the boundary. Furthermore, the centroid of the material in the X-ray image is kept consistent based on the centroid of the 3D image projected onto the 2D plane.

[0136] In terms of identification, there is a wide variety of ores, with significant differences in color, shape, density, and composition. Some concentrates and tailings are extremely similar in atomic number and color, but there are considerable differences in the surface roughness or three-dimensional shape. The closer the atomic number of the concentrate and tailings, the more difficult it is to identify them using dual-energy X-ray imaging. Images acquired at high speeds by visible light cameras, due to inherent defects such as reflection and dispersion, are difficult to accurately capture the detailed features of the mineral surface roughness. X-ray imaging and visible light imaging are 2D imaging and cannot capture the three-dimensional shape of the ore. Therefore, X-ray and visible light images have limited ability to identify these ores, thus affecting the effective sorting process.

[0137] In blow-through separation, X-ray imaging uses a fan-shaped beam. When the beam obliquely strikes the ore on the conveyor belt, it causes a shift between the imaged position and the actual position of the ore. This shift is particularly noticeable when the ore is thick. Similarly, increasing the oblique penetration distance further amplifies the centroid position deviation. Since the nozzle needs to be precisely aligned based on the ore position in the image, this deviation directly affects the accuracy of the blow-through operation. Visible light imaging technology is often affected by interference factors such as dirt and water stains on the conveyor belt surface in practical applications. These interferences blur the image boundary between the ore boundary and the conveyor belt surface, making it difficult to accurately determine the ore position. This ambiguity also negatively impacts the accuracy of the blow-through operation. Therefore, three-dimensional imaging technology can serve as an effective supplement to X-ray imaging-based material sorting, overcoming the centroid position deviation in X-ray images through correction operations.

[0138] According to embodiments of this disclosure, three-dimensional imaging technology is used to accurately correct the centroid offset of X-ray images. The boundaries of each material in the three-dimensional image are used as a reference to correct the X-ray image, serving as the boundaries for opening the valve nozzle. This allows for the calculation of accurate blowing commands, achieving precise control of the air valve and reducing the error rate in material sorting. Furthermore, when identifying material categories based on X-ray images, the corrected centroid and material boundaries can be used as feature information for material category identification, providing richer information for recognition and improving accuracy.

[0139] Figure 6 schematically illustrates a flowchart for obtaining the centroid of a material in a three-dimensional image according to an embodiment of the present disclosure.

[0140] As shown in Figure 6, this embodiment is one example of operating S510, including:

[0141] In operation S610, the sum of the heights of all pixels corresponding to the material in the 3D image is calculated.

[0142] In operation S620, the width coordinates of all pixels are summed by weight and then divided by the sum of their heights to obtain the centroid width coordinates; the length coordinates of all pixels are summed by weight and then divided by the sum of their heights to obtain the centroid length coordinates.

[0143] In some embodiments, the weighted summation includes: multiplying the height of each pixel by its coordinate value as a weight to obtain a weighted result, wherein the coordinate value includes a width coordinate value or a length coordinate value; and summing the weighted results of all pixels.

[0144] Specifically, assuming the material has a uniform density, the mass of a single pixel is equal to the density multiplied by the pixel's height. Therefore, the pixel's height is directly proportional to the mass of the material within that pixel. Using the height information of the 3D image as the material's mass information, the centroid of the material's top-view can be obtained using the following formula:

[0145] The height of each pixel in each material block is denoted as m. i i refers to the i-th pixel, where i is an integer greater than or equal to 1.

[0146] Calculate the sum of the heights of all pixels in the material, denoted as M = Σm i .

[0147] Calculate the weighted average of the material width coordinates, using height as the weight, to obtain the centroid width coordinate X, denoted as:

[0148] Calculate the weighted average of the material's y-coordinates, using height as the weight, to obtain the centroid width coordinate Y, denoted as .

[0149] According to embodiments of this disclosure, the centroid and boundary information of X-ray materials can be corrected using the height and boundary information of a three-dimensional image. This obtains accurate centroid and boundary information of the material, enabling precise material positioning and allowing for the calculation of more accurate injection schemes.

[0150] In some embodiments, performing the second sorting strategy further includes: using X-ray images and three-dimensional images to identify the category of the second target material, wherein the second target material is a material to be identified that conforms to the second sorting strategy.

[0151] Among them, the blowing based on the centroid and boundary information of the material in the X-ray image includes:

[0152] The blowing parameters are determined based on the type of the second target material. For example, different materials may require different blowing pressures or modes. The appropriate nozzle and blowing force (i.e., blowing parameters) on the blowing module can be intelligently selected according to the type of material.

[0153] Based on the centroid and boundary information of the material in the X-ray image, the blowing is performed according to the blowing parameters.

[0154] For example, firstly, the control module 500 uses the center of mass position to calculate the material's trajectory. Next, the control module 500 predicts the exact time the material will reach the spraying position based on the trajectory, and further determines the start time for spraying based on the material's boundary information. When the material is about to reach the spraying position, the control module 500 sends a signal to the sorting module to start spraying, based on the predicted time.

[0155] The process of implementing the second sorting strategy is further illustrated below with Figures 7 and 8.

[0156] Figure 7 schematically illustrates a flowchart of material matching and nozzle opening / closing time calculation based on point cloud data according to an embodiment of the present disclosure.

[0157] Referring to Figures 3 and 7, and in conjunction with Figures 2, 5, and 6, point cloud data is extracted from the 3D images, and image segmentation is performed using the PointNet++ algorithm (operation S220) to obtain a label image (i.e., a point cloud material image). Image segmentation is then performed on the X-ray image (operation S220) to obtain a label image (i.e., a 2D material image). The point cloud material image and the 2D material image are matched one-to-one to achieve material matching between the two types of images (operation S230). Based on the point cloud segmentation results from the PointNet++ algorithm, the centroid position and boundary calculations are performed on each material (operation S510), and the centroid position and boundary corrections are performed on the corresponding materials in the X-ray image (operation S520). This achieves accurate correspondence between the ore and the valve nozzle position, enabling accurate calculation of nozzle opening and closing times and precise ore injection according to the injection parameters.

[0158] Figure 8 schematically illustrates a flowchart of material matching based on depth images and calculation of nozzle opening and closing times according to an embodiment of the present disclosure.

[0159] Referring to Figures 4 and 8, and in conjunction with Figures 2, 5, and 6, the point cloud data is converted into a depth image to project the three-dimensional spatial information onto a two-dimensional plane. Then, the depth image is registered with the X-ray image using a projection transformation matrix (operation S410). Image segmentation is performed on the registered X-ray and depth images respectively (operation S420). The segmented two-dimensional material images and depth material images are matched one-to-one (operation S430). Based on the depth information of each material, the centroid position and boundary are calculated (operation S510). The centroid position and boundary of the corresponding material in the X-ray image are corrected (operation S520). Therefore, the nozzle opening and closing time can be accurately calculated based on the accurate centroid position and boundary. Subsequently, the blowing parameters can be determined based on the material type; and blowing can be performed according to the blowing parameters based on the material's centroid and boundary information.

[0160] According to embodiments of this disclosure, accurate identification and classification of different types of target materials can be achieved, improving the accuracy and efficiency of sorting operations. Determining the blowing parameters based on the category of the target material allows for better adaptation to different material conditions, thus improving sorting precision.

[0161] X-ray imaging (such as dual-energy X-ray imaging) can effectively distinguish substances with large differences in atomic number, but the closer the atomic numbers are, the lower the sorting effect of X-rays. For example, some ores have similar colors, but their surface unevenness varies significantly. Color cameras cannot accurately represent the unevenness information of the ore surface due to issues such as image reflection and dispersion. However, three-dimensional imaging technology can accurately measure the height information of the ore surface, thus accurately reflecting the unevenness information of the ore surface and compensating for the shortcomings of dual-energy X-ray imaging and color imaging.

[0162] In addition, some concentrates and tailings have significant shape differences. For example, most concentrates of ores such as phosphate rock are cubic in shape, while irregularly shaped tailings are mostly irregularly shaped. Because X-ray imaging is a projection imaging method, the shape of the image is a projection of a three-dimensional shape, lacking height information and unable to capture three-dimensional shape features.

[0163] For minerals with similar atomic numbers and indistinct color characteristics, but differences in particle shape, surface roughness, or three-dimensional shape, the height information in 3D images effectively compensates for the shortcomings of X-ray imaging and color camera imaging in this regard. For ores with significant differences in texture or three-dimensional shape, introducing 3D images will effectively improve the accuracy of identification. Therefore, the following examples, illustrated in Figures 9 and 10, illustrate an implementation of material classification from two different dimensions using X-ray and 3D images.

[0164] Figure 9 schematically illustrates a flowchart for identifying the category of a second target material according to an embodiment of the present disclosure.

[0165] As shown in Figure 9, this embodiment includes:

[0166] In operation S910, a first confidence score for the second target material is obtained by identifying the X-ray image, wherein the first confidence score represents the score by which the second target material belongs to a first category. For example, the identified image is a two-dimensional X-ray image of the second target material segmented from the X-ray image.

[0167] In operation S920, a second confidence score for the second target material is obtained by recognizing the 3D image, where the second confidence score represents the score by which the second target material belongs to the first category. For example, the recognized object is a 3D point cloud image of the second target material segmented from the 3D image.

[0168] In operation S930, a weighted calculation is performed based on the first confidence score and its first weight, and the second confidence score and its second weight, to obtain the third confidence score.

[0169] In operation S940, if the third confidence score is greater than or equal to the third value, the second target material is determined to be in the first category; otherwise, it is in the second category.

[0170] Exemplarily, take the material as ore, the first category as concentrate, and the second category as tailings for example.

[0171] The grade of the ore is continuous. The ore passes through an X-ray image recognition algorithm (such as Faster R-CNN, YOLO network model, U-Net network model) to obtain the confidence score score_Xray of being recognized as concentrate. Set the first threshold threXray_high and the second threshold threXray_low. When score_Xray > threXray_high, score_Xray = 1; when score_Xray < threXray_low, score_Xray = 0. Among them, the first confidence score is score_Xray = 1 or score_Xray = 0. If score_Xray is between threXray_high and threXray_low, it is mapped to a value between 0 and 1.

[0172] At the same time, the point cloud data is recognized to classify the ore into two categories. For example, bumpy tailings, smooth concentrate, or cubic concentrate, and irregularly shaped tailings. The ore passes through a point cloud recognition algorithm (such as PointNet algorithm, VoxelNet, etc.) to obtain the confidence score score_PointCloud of the ore being recognized as concentrate. To convert the continuous confidence score into a discrete class label, set the third threshold threPointClou_low and the fourth threshold threPointCloud_high. When score_PointCloud > threPointCloud_high, score_PointCloud = 1. When score_PointCloud < threPointCloud_low, score_PointCloud = 0. Among them, the second confidence score is score_PointCloud = 1 or score_PointCloud = 0. If score_PointCloud is between threPointClou_high and threPointClou_low, it is mapped to a value between 0 and 1.

[0173] First, a logical OR operation (represented by ||) is performed on score_Xray and score_PointCloud. If either score is true (i.e., greater than 0), the entire logical OR expression is true. Then, score_Xray is multiplied by the first weight and then divided by the difference between threXray_high and threXray_low for standardization. score_PointCloud is multiplied by the second weight and then divided by the difference between threPointCloud_high and threPointCloud_low for standardization. The two normalized scores are then added together to obtain a composite score. Therefore, let the third confidence score be denoted as score, and it is calculated using the following formula:

[0174] The first weight 'a' and the second weight 'b' are the optimal parameter combinations obtained through a prior parameter optimization algorithm. For example, 'a' and 'b' are numbers between 0 and 1.

[0175] Finally, operation S940 is executed to determine the final material category identification result.

[0176] According to embodiments of this disclosure, the X-ray image recognition algorithm focuses on the internal structure and composition of the material, while the point cloud recognition algorithm focuses on the external shape and surface features of the material. Therefore, by analyzing and recognizing the material from two different dimensions using X-ray images and three-dimensional images, the recognition accuracy can be improved.

[0177] In some embodiments, considering that the adhesion of materials affects the accuracy of secondary image segmentation, the material state can be used as one of the input features of the recognition algorithm during recognition.

[0178] For example, during the training and recognition process of the X-ray image recognition algorithm, the feature value of materials in an adhered state is set to 1, while the feature value of materials in a non-adhesive state is set to 0. Simultaneously, for materials in an adhered state, a concave point detection algorithm is used to extract the features of their edge concave points. These features, along with the overall image features of the two-dimensional material image, are input into the X-ray image recognition algorithm to obtain the recognition result. Furthermore, during training, optimal first threshold `threXray_high` and optimal second threshold `threXray_low` can be found based on the material state, as well as optimal first weight `a` and optimal second weight `b`. During recognition, at least one of the first threshold `threXray_high` and the second threshold `threXray_low` can be dynamically changed based on the material state.

[0179] For example, during the training and recognition process of the point cloud recognition algorithm, the feature value of materials in an adhered state is set to 1, while the feature value of materials in a non-adhesive state is set to 0. Simultaneously, for materials in an adhered state, the height continuity features of their edges are extracted using height difference detection. These features, along with the overall image features of the two-dimensional material image, are input into the point cloud recognition algorithm to obtain the recognition result. Similarly, during training, at least one of the optimal matching third threshold `threPointCloud_low` and fourth threshold `threPointCloud_high` can be found based on the material state.

[0180] The X-ray image recognition algorithm and the point cloud recognition algorithm can be trained simultaneously. During training, the optimal first weight 'a' and second weight 'b' can be found based on the material state. During recognition, at least one of the first weight 'a' and second weight 'b' can be dynamically changed based on the material state.

[0181] In some embodiments, in a recognition scenario with three or more categories, a two-dimensional ray image of the second target material can be recognized to obtain a fourth confidence score for each category, and a three-dimensional point cloud image of the second target material can be recognized to obtain a fifth confidence score for each category. Then, the fourth and fifth confidence scores for each category are added together to obtain a sixth confidence score for each category. Finally, the category with the highest score is selected as the final recognition category.

[0182] According to embodiments of this disclosure, the physical state of materials can be taken into account during training and recognition, thereby improving recognition accuracy. Based on corresponding thresholds and weights that dynamically change the material state, targeted recognition of materials in different states can be achieved more effectively.

[0183] Figure 10 schematically illustrates a flowchart for identifying the category of a second target material according to another embodiment of the present disclosure.

[0184] As shown in Figure 10, this embodiment includes:

[0185] In operation S1010, the X-ray image is input into the first network branch of the image recognition model to extract the first feature; the first network branch can be constructed based on a convolutional neural network for processing two-dimensional images.

[0186] In operation S1020, the three-dimensional image is input into the second network branch of the image recognition model to extract the second feature. The image recognition model is constructed based on a deep learning network.

[0187] In this approach, the second network branch can be used to extract features from 3D image point cloud data, yielding rich features in the three-dimensional dimension. This second network branch can be constructed based on a convolutional neural network used for processing 3D images. Alternatively, the 3D image can be first converted into a depth image, and then the second network branch can be used to extract features from the depth image, transforming 3D data processing into 2D data processing. This reduces the complexity of the data processing by the second network branch, which can also be constructed based on a convolutional neural network used for processing 2D images.

[0188] In operation S1030, the first and second features are concatenated, and the concatenated features are input into the fully connected layer of the image recognition model; for example, the Concat operation can be used to concatenate the first and second features. The Concat operation is a data structure operation that joins two or more arrays or feature vectors together.

[0189] In operation S1040, the classification layer of the image recognition model processes the output of the fully connected layer to identify the category of the second target material.

[0190] For example, X-ray images and 3D images are used as inputs to the two branches of a deep learning model, respectively, to extract their respective features. The feature maps of the X-ray and 3D images are then concatenated to obtain a richer feature representation. This representation is then transformed into a new feature space through a fully connected layer. In the classification layer, a softmax function is used to convert the output of the fully connected layer into a probability distribution to obtain the final recognition result. For instance, in a binary classification scenario, the classification layer outputs predicted probabilities of concentrate and tailings, and the higher probability is taken as the recognition result. In a three-class classification scenario, the classification layer outputs predicted probabilities of concentrate, medium-grade ore, and tailings, and the higher probability is taken as the recognition result. It is understood that this disclosure does not limit the number of categories output by the classification layer; it can output predicted probabilities corresponding one-to-one with multiple pre-set categories.

[0191] According to embodiments of this disclosure, a first network branch and a second network branch are specifically responsible for extracting corresponding features from the two types of images, enabling the utilization of information unique to each image. Therefore, by fusing multimodal data from X-ray and 3D images, comprehensive capture of target material features is achieved, enhancing recognition accuracy.

[0192] For example, an initial image recognition model can be obtained in advance, and the image recognition model can be trained based on the material image sample set. The trained image recognition model is then applied to operations S1010 to S1040.

[0193] In some embodiments, considering that the adhesion of materials affects the accuracy of secondary image segmentation, the material state can also be used as one of the input features of the image recognition model during recognition.

[0194] For example, in the training phase of the image recognition model, the material image sample set includes material image samples in an adhered state and material image samples in a non-adhesive state. During training and recognition, the feature value of the adhered material is set to 1, and the feature value of the non-adhesive material is set to 0. In operation S1030, in addition to splicing the first and second features, the feature values ​​1 or 0 representing the material state are also spliced. The spliced ​​features are then processed sequentially through a fully connected layer and a classification layer to output the material category.

[0195] According to embodiments of this disclosure, the adhesion state of materials may affect the accuracy of secondary image segmentation, thereby impacting the recognition results. By introducing feature values ​​of the material state, richer feature representations can be learned, allowing the model to consider the material state during training and recognition, thus improving the accuracy of recognizing different material states.

[0196] Figure 11 schematically illustrates a block diagram of an electronic device suitable for implementing a material sorting method according to an embodiment of the present disclosure. The electronic device shown in Figure 11 is one embodiment of a control module 500.

[0197] As shown in FIG11, an electronic device 1100 according to an embodiment of the present disclosure includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage portion 1108 into a random access memory (RAM) 1103. The processor 1101 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1101 may also include onboard memory for caching purposes. The processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0198] RAM 1103 stores various programs and data required for the operation of electronic device 1100. Processor 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Processor 1101 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1102 and / or RAM 1103. It should be noted that programs may also be stored in one or more memories other than ROM 1102 and RAM 1103. Processor 1101 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0199] According to embodiments of this disclosure, the electronic device 1100 may further include an input / output (I / O) interface 1105, which is also connected to a bus 1104. The electronic device 1100 may also include one or more of the following components connected to the I / O interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. A removable medium 1111, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1110 as needed so that computer programs read from it can be installed into the storage section 1108 as needed.

[0200] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0201] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 1102 and / or RAM 1103 and / or one or more memories other than ROM 1102 and RAM 1103 described above.

[0202] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0203] When the computer program is executed by the processor 1101, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0204] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1109, and / or installed from the removable medium 1111. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0205] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 1109, and / or installed from removable medium 1111. When the computer program is executed by processor 1101, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0206] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0207] The above-described one or more embodiments have the following beneficial effects: Addressing the limitation of a single X-ray image segmentation algorithm in accurately segmenting images, this method comprehensively acquires X-ray images and 3D images, considering the material states of each material. X-ray image segmentation and 3D image segmentation are performed separately. Then, the materials in the X-ray images are matched with the corresponding materials in the 3D images, and material sorting is performed based on the matching results. Therefore, it can more accurately capture the continuity of materials in three-dimensional space using 3D information. Combining X-ray information for material image segmentation and sorting can improve sorting accuracy, for example, for materials with similar atomic numbers, indistinct color characteristics, but differences in shape and surface roughness, or differences in three-dimensional shape.

[0208] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0209] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0210] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A material sorting method, comprising: Acquire X-ray and 3D images of the material to be sorted; Based on at least one of the X-ray image and the three-dimensional image, the material state of each material is identified, wherein the material state includes a non-adhesive state and an adhesive state, and the adhesive state includes a state in contact with at least one other material. For the X-ray image and the three-dimensional image, if material state recognition has been performed, image segmentation is performed based on the material state of each material; if material state recognition has not been performed, image segmentation is performed directly on each material. Based on the material state of each material, each material in the image-segmented X-ray image is matched with the corresponding material in the image-segmented three-dimensional image. Material sorting is performed based on the material matching results between the X-ray image and the three-dimensional image.

2. The method of claim 1, wherein, Material sorting based on the material matching results between the X-ray image and the three-dimensional image includes at least one of the following: If the material matching result shows that at least one of the two successfully matched materials is in an adhesive state and at least one has not undergone image segmentation processing, then the first sorting strategy is executed. If any material in the X-ray image or the three-dimensional image fails to match, the first sorting strategy is executed on the material that fails to match. If the material matching result shows that at least one of the two successfully matched materials is in an adhesive state and has already undergone image segmentation processing, then the second sorting strategy is executed. If the material matching results show that any two successfully matched materials are in a non-adhesive state, then the second sorting strategy is executed.

3. The method of claim 1, wherein, Based on the material state of each material, matching each material in the image-segmented X-ray image with the corresponding material in the image-segmented three-dimensional image includes: Based on the X-ray data of the X-ray image and the point cloud data of the three-dimensional image, the correspondence between the materials in the X-ray image and the three-dimensional image is determined; For any two materials that correspond to each other in the X-ray image and the three-dimensional image, if at least one of the two materials is in an adhesive state, then the first matching strategy is executed; if the two materials are not in an adhesive state, then the second matching strategy is executed.

4. The method of claim 3, wherein, Executing the first matching strategy includes: Calculate the overlap between the two materials, wherein the overlap is determined based on at least one of the two-dimensional projected shape and the two-dimensional projected area of ​​the materials; When the overlap is greater than or equal to the first value, it is determined that the two materials with a corresponding relationship have been successfully matched.

5. The method of claim 4, wherein, When the overlap is less than the first value, At least one of the two materials is identified as the first target material; The first target material is combined with the other N materials that are adjacent to it in the same image to perform a degree of merging detection, where N is an integer greater than or equal to 1; The first target material is merged with at least one specific material to obtain merged material, wherein the degree of merging between the specific material and the first target material is greater than or equal to a second value, and the merging process includes marking multiple materials as the same material for re-matching.

6. The method of claim 5, wherein, When the first target material is a material in the three-dimensional image, the merging degree detection of the first target material with the other N materials in the same adjacent position in the image includes: The first target material and the N materials are respectively subjected to high continuity detection, and the high continuity is positively correlated with the degree of merging.

7. The method of claim 5, wherein, Determining at least one of the two materials as the first target material includes: Compare the two-dimensional projected area of ​​the material in the X-ray image with the two-dimensional projected area of ​​the material in the three-dimensional image; The material with the smaller two-dimensional projected area is selected as the first target material.

8. The method of claim 3, wherein, Executing the second matching strategy includes: When a single material in the X-ray image and a single material in the three-dimensional image form a one-to-one correspondence, the matching is confirmed to be successful. If a single material in one of the X-ray images and the three-dimensional image corresponds to multiple materials in the other image, then the multiple materials are merged into a single material to form a one-to-one correspondence, and the matching is confirmed to be successful.

9. The method of claim 2, wherein, For the X-ray image and the three-dimensional image, if material state recognition has been performed, image segmentation is performed based on the material state of each material; if material state recognition has not been performed, image segmentation is performed directly on each material, including: The X-ray image is registered with a two-dimensional depth image, which is obtained based on the three-dimensional image; Based on the material state of each material, a two-dimensional material image of each material is segmented from the registered X-ray image to obtain a third material set; a depth material image of each material is directly segmented from the registered depth image to obtain a fourth material set. Specifically, matching each material in the image-segmented X-ray image with its corresponding material in the image-segmented three-dimensional image, based on the material state of each material, includes: Based on the material images in the third material set, match the material images in the fourth material set.

10. The method of claim 2, wherein, Executing the second sorting strategy includes: for any two materials that are successfully matched, Obtain the centroid and boundary information of the material in the three-dimensional image; The blowing is performed based on the centroid and boundary information of the material in the three-dimensional image.

11. The method of claim 10, wherein, Obtaining the centroid of the material in the three-dimensional image includes: Calculate the sum of the heights of all pixels corresponding to the material in the three-dimensional image; The centroid width coordinates are obtained by weighted summation of the width coordinates of all the pixels and then divided by the sum of the heights; the centroid length coordinates are obtained by weighted summation of the length coordinates of all the pixels and then divided by the sum of the heights.

12. The method of claim 11, wherein, The weighted summation includes: The height of each pixel is used as a weight, and its coordinate value is multiplied to obtain a weighted result, wherein the coordinate value includes a width coordinate value or a length coordinate value. Sum the weighted results of all the pixels.

13. The method of claim 10, wherein, Executing the second sorting strategy also includes: The category of the second target material is identified using the X-ray image and the three-dimensional image, wherein the second target material is the material to be identified that conforms to the second sorting strategy; The blowing based on the centroid and boundary information of the material in the three-dimensional image includes: The injection parameters are determined based on the type of the second target material; Based on the centroid and boundary information of the material in the three-dimensional image, the material is sprayed according to the spraying parameters.

14. The method of claim 13, wherein, The categories of the second target material identified based on the X-ray image and the three-dimensional image include: The X-ray image is used to obtain a first confidence score for the second target material, wherein the first confidence score represents the score by which the second target material belongs to a first category; The second confidence score of the second target material is obtained by identifying the three-dimensional image, wherein the second confidence score represents the score of the second target material belonging to the first category; A third confidence score is obtained by weighting the first confidence score and its first weight, and the second confidence score and its second weight. If the third confidence score is greater than or equal to the third value, the second target material is determined to be in the first category; otherwise, it is in the second category.

15. The method of claim 14, wherein, The categories of the second target material identified based on the X-ray image and the three-dimensional image include: The X-ray image is input into the first network branch of the image recognition model to extract the first feature; The three-dimensional image is input into the second network branch of the image recognition model to extract the second feature, wherein the image recognition model is constructed based on a deep learning network; The first feature and the second feature are concatenated, and the concatenated feature is input into the fully connected layer of the image recognition model; The classification layer of the image recognition model processes the output of the fully connected layer to identify the category of the second target material.

16. A material sorting device, comprising: Conveying module, used to transport materials to be sorted; The X-ray image acquisition module is used to capture X-ray images of the material to be sorted. A three-dimensional image acquisition module is used to capture three-dimensional images of the material to be sorted; A control module is used to acquire the X-ray image and the three-dimensional image, and execute the material sorting method according to any one of claims 1 to 15 to generate sorting instructions; The blowing module is used to blow the material leaving the conveying module in response to the sorting command of the control module.

Images