Surface mount technology (SMT) material quantity statistical method and apparatus

Abstract

An SMT material quantity statistical method and apparatus. The method comprises: photographing SMT materials to obtain an original image, and accurately cropping an image of a material placement area from the original image, such that background information irrelevant to material quantity statistics is effectively removed; determining a material size of the SMT materials by using the material area image with no background information contained therein, and selecting a matching material detection model on the basis of the material size, wherein the matching material detection model can accurately output a detection result that reflects the quantity of materials of that size; and then collecting statistics on the basis of the detection result, so as to obtain the quantity of the SMT materials. The method improves the pertinence of SMT material quantity statistics, and is applicable to statistical scenarios of SMT materials of various sizes, thereby improving the accuracy of SMT material quantity statistics.

Description

A method and apparatus for counting the quantity of SMT materials

[0001] This application claims priority to Chinese Patent Application No. 202411897822.4, filed on December 20, 2024, entitled "A Method and Apparatus for Counting the Quantity of SMT Materials", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of industrial material counting technology, and in particular to a method and apparatus for counting the quantity of SMT materials. Background Technology

[0003] In modern manufacturing, the accuracy of quantity statistics for SMT (Surface Mount Technology) materials, such as resistors and integrated circuits, plays a crucial role in controlling production costs, ensuring product quality, and improving production efficiency.

[0004] Traditional methods for counting the quantity of SMT materials mainly rely on manual operation, such as counting them one by one by hand. Since SMT materials are small in size and numerous, they are easily affected by human negligence or fatigue, which can lead to deviations in the statistical results, i.e., low accuracy. Summary of the Invention

[0005] The purpose of this application is to provide a method and apparatus for SMT material quantity counting, so as to improve the accuracy of SMT material quantity counting. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of this application provide a method for counting the quantity of SMT materials, the method comprising:

[0007] Acquire raw images of SMT materials by photographing them;

[0008] Images of all SMT material placement areas are cropped from the original image and used as material area images;

[0009] The dimensions of the SMT material are determined based on the material area image.

[0010] Based on the detection results of the material region image using the target material detection model, the quantity of the SMT material is statistically obtained; wherein, the target material detection model is a material detection model corresponding to the size of the SMT material.

[0011] In one possible implementation, determining the size of the SMT material based on the material area image includes:

[0012] Based on the visual features of the material region image, the material regions where each SMT material is located in the material region image are detected; wherein, the visual features are used to represent the contours in the material region image and the positions of the contours;

[0013] The size of the SMT material is determined based on the area of ​​each material region.

[0014] In one possible implementation, determining the size of the SMT material based on the area of ​​each of the material regions includes:

[0015] The area of ​​each of the material regions is statistically analyzed; wherein the statistical value is used to represent the area of ​​a single SMT material;

[0016] If the area represented by the statistical value is greater than the upper limit area threshold, then the size of the SMT material is determined to be the first size;

[0017] If the area represented by the statistical value is not greater than the upper limit area threshold, then the size of the SMT material is determined to be the second size;

[0018] The number of SMT materials is calculated by statistically analyzing the detection results of the material region image based on the target material detection model, including:

[0019] If the size of the SMT material is the first size, the number of SMT materials is obtained by counting the detection results of the material area image based on the first material detection model, wherein the first material detection model is a material detection model for SMT material detection based on visual features.

[0020] If the size of the SMT material is the second size, the number of SMT materials is obtained by counting the detection results of the material area image based on the second material detection model, wherein the second material detection model is a material detection model for SMT material detection based on semantic features.

[0021] In one possible implementation, the first material detection model is a target rotational regression head model, and the second material detection model is a target semantic category regression head model;

[0022] The step of detecting the material region where each SMT material is located in the material region image based on the visual features of the material region image includes:

[0023] The material region image is input into the target rotary regression head model so that the target rotary regression head model extracts the visual features of the material region image and outputs the material region where each SMT material in the material region image is located based on the visual features.

[0024] The quantity of SMT materials is obtained by statistically analyzing the detection results of the material region image based on the first material detection model, including:

[0025] The number of material regions output by the target rotary regression head model is determined as the number of SMT materials;

[0026] The method further includes:

[0027] If the size of the SMT material is the second size, the material region image is input to the target semantic category regression head model so that the target semantic category regression head model extracts the semantic features of the material region image and outputs the material center point of each SMT material in the material region image based on the semantic features;

[0028] The quantity of SMT materials is obtained by statistically analyzing the detection results of the material region image based on the second material detection model, including:

[0029] The number of material center points output by the target semantic category regression head model is determined as the number of SMT materials;

[0030] The target rotating rectangle regression head model is trained using a first sample image, where the size of the SMT material in the first sample image is greater than the upper limit area threshold, and the first sample image is marked with the material region where each of the SMT materials in the first sample image is located; the target semantic category regression head model is trained using a second sample image, where the center point of each of the SMT materials in the second sample image is marked.

[0031] In one possible implementation, the target rotated rectangle regression head model is pre-trained by:

[0032] The first sample image is input into the original rotating rectangle regression head model, so that the original rotating regression head model extracts the visual features of the first sample image, and outputs all predicted regions containing objects in the first sample image, as well as the predicted category and confidence level of each predicted region based on the visual features; wherein, the predicted category is used to represent the category of the object present in the predicted region, and the confidence level is used to represent the accuracy of the category;

[0033] A first loss is calculated based on the first difference between each predicted region and the material region where each SMT material is located as marked in the first sample image; wherein, the first loss is positively correlated with the first difference;

[0034] A second loss is calculated based on the second difference between the predicted category and the target category of the predicted region; wherein, the target category is the category of objects existing in the same region as the predicted region in the first sample image. If the predicted category is the same as the target category, the second difference is negatively correlated with the confidence of the predicted region; if the predicted category is different from the target category, the second difference is positively correlated with the confidence of the predicted region, and the second loss is positively correlated with the second difference.

[0035] Adjust the parameters of the original rotating rectangular regression head model in the direction of gradient descent of the first loss and the second loss to obtain the target rotating rectangular regression head model;

[0036] The target semantic category regression head model is pre-trained in the following ways:

[0037] The second sample image is input into the original semantic category regression head model so that the target semantic category regression head model extracts the semantic features of the second sample image and outputs the predicted material center point of each SMT material in the material region image based on the semantic features.

[0038] A third loss is calculated based on the third difference between the predicted material center point and the material center point of each SMT material marked in the second sample image; wherein the third loss is positively correlated with the third difference.

[0039] Adjust the parameters of the original semantic category regression head model in the direction of the third loss gradient descent to obtain the target semantic category regression head model.

[0040] In one possible implementation, the method further includes:

[0041] Among multiple first preset models, the model that satisfies the first preset condition is selected as the target rotation regression head model, and among multiple second preset models, the model that satisfies the second preset condition is selected as the target semantic category regression head model; wherein, the multiple first preset models are trained based on the same first sample image, and the structures of different first preset models are different, and the multiple second preset models are trained based on the same second sample image, and the structures of different second preset models are different.

[0042] In one possible implementation, the step of outputting the center point of each SMT material in the material region image based on the semantic features includes:

[0043] Based on the semantic features of the material region image, a material center point mask image is output; wherein, the material center point mask is used to represent the material center point of each SMT material in the material region image, the size of the material center point mask image is the same as that of the material region image, and each element is used to represent the confidence level of pixels in the material region image that are at the same position as the material center point;

[0044] Determining the number of material center points output by the target semantic category regression head model as the number of SMT materials includes:

[0045] The material center point mask image is pooled to obtain a pooled material center point mask image;

[0046] In the pooled material center point mask image, determine the maximum value that is greater than the preset screening threshold, and determine the number of the obtained maximum values ​​as the number of SMT materials.

[0047] In one possible implementation, the first sample image and the second sample image are pre-annotated in the following ways:

[0048] Acquire sample material area images obtained by photographing SMT sample materials; wherein each sample material area image is obtained by photographing SMT sample materials of different sizes, and each sample material area image includes multiple SMT sample materials, and the different sizes of SMT sample materials include a first size material and a second size material, the first size material is an SMT sample material with a size greater than the upper limit area threshold, and the second size material is an SMT sample material with a size not greater than the upper limit area threshold;

[0049] The center point of each SMT sample material in the sample material area image is marked to obtain the original sample image;

[0050] In the original sample image, the original sample image obtained by taking a picture of the material of the first size is determined as the first preprocessed sample image. The material area where the SMT sample material is located is marked in the first preprocessed sample image to obtain the first sample image.

[0051] The original sample image is determined as the second sample image, or other images in the original sample image other than the first preprocessed sample image are determined as the second sample image.

[0052] In one possible implementation, determining the original sample image as the second sample image, or determining other images in the original sample image besides the first preprocessed sample image as the second sample image, includes:

[0053] The original sample image is determined as the second preprocessed sample image, or other images in the original sample image other than the first preprocessed sample image are determined as the second preprocessed sample image;

[0054] According to the category of the SMT sample material captured, each of the second preprocessed sample images is uniformly sampled to obtain the sampling result, which is used as the second sample image;

[0055] The step of marking the material region where the SMT sample material is located in the first preprocessed sample image to obtain the first sample image includes:

[0056] Mark the material region where the SMT sample material is located in the first preprocessed sample image to obtain the third preprocessed sample image;

[0057] According to the category of the SMT sample material captured, each of the third preprocessed sample images is uniformly sampled to obtain the sampling result, which is used as the first sample image.

[0058] In one possible implementation, the step of cropping an image of all SMT material placement areas from the original image as a material area image includes:

[0059] Based on the principle of maximizing the variance of gray values ​​between the foreground and background regions after binarization, the original image is subjected to gray-level binarization to obtain a preprocessed image; wherein, the original image is an X-ray image;

[0060] Determine the boundary lines of grayscale values ​​in the preprocessed image;

[0061] The sub-image within the boundary line is cropped from the preprocessed image to serve as the material area image.

[0062] In one possible implementation, the step of calculating the quantity of SMT materials based on the detection results of the material region image using the second material detection model includes:

[0063] If the area represented by the statistical value is greater than the lower limit area threshold, the number of SMT materials is statistically obtained based on the detection results of the unmagnified material area image using the second material detection model.

[0064] If the area represented by the statistical value is not greater than the lower limit area threshold, the number of SMT materials is statistically obtained based on the detection results of the magnified material area image by the second material detection model.

[0065] In one possible implementation, detecting the material region where each SMT material in the material region image is located includes:

[0066] The candidate regions of all potential SMT materials in the material region image are detected, along with the confidence level of each candidate region; wherein, the confidence level is used to represent the probability that an SMT material exists in the candidate region;

[0067] In the candidate region, the region with a confidence level not lower than a preset confidence threshold is determined as the material region where each SMT material in the material region image is located.

[0068] Secondly, embodiments of this application provide an SMT material quantity counting device, the device comprising:

[0069] Image acquisition module, used to acquire raw images obtained by photographing SMT materials;

[0070] The image cropping module is used to crop images of all SMT material placement areas from the original image, as material area images;

[0071] The size determination module is used to determine the size of the SMT material based on the material area image;

[0072] The quantity statistics module is used to count the quantity of the SMT material based on the detection results of the material area image by the target material detection model; wherein, the target material detection model is a material detection model corresponding to the size of the SMT material.

[0073] In one possible implementation, the size determination module includes:

[0074] The first size determination submodule is used to detect the material region where each SMT material is located in the material region image based on the visual features of the material region image; wherein, the visual features are used to represent the outline in the material region image and the position of the outline;

[0075] The second size determination submodule is used to determine the size of the SMT material based on the area of ​​each material region.

[0076] In one possible implementation, the second dimension determining submodule includes:

[0077] The first size determination unit is used to obtain statistical values ​​of the area of ​​each of the material regions; wherein the statistical values ​​are used to represent the area of ​​a single SMT material;

[0078] The second size determination unit is used to determine the size of the SMT material as the first size if the area represented by the statistical value is greater than the upper limit area threshold.

[0079] The third size determination unit is used to determine the size of the SMT material as the second size if the area represented by the statistical value is not greater than the upper limit area threshold.

[0080] The quantity statistics module includes:

[0081] The first quantity statistics submodule is used to count the quantity of SMT materials based on the detection results of the material area image by the first material detection model if the size of the SMT material is the first size. The first material detection model is a material detection model for SMT material detection based on visual features.

[0082] The second quantity statistics submodule is used to count the quantity of SMT materials based on the detection results of the material area image by the second material detection model if the size of the SMT material is the second size. The second material detection model is a material detection model for SMT material detection based on semantic features.

[0083] In one possible implementation, the first material detection model is a target rotational regression head model, and the second material detection model is a target semantic category regression head model;

[0084] The first size determination submodule includes:

[0085] The first size determination unit is used to input the material area image into the target rotary regression head model, so that the target rotary regression head model extracts the visual features of the material area image and outputs the material area where each SMT material in the material area image is located based on the visual features.

[0086] The first quantity statistics submodule includes:

[0087] The first quantity statistics unit is used to determine the number of material regions output by the target rotary regression head model, which is taken as the number of SMT materials;

[0088] The device further includes:

[0089] The center point output module is used to input the material region image into the target semantic category regression head model if the size of the SMT material is the second size, so that the target semantic category regression head model extracts the semantic features of the material region image and outputs the material center point of each SMT material in the material region image based on the semantic features.

[0090] The second quantity statistics submodule includes:

[0091] The second quantity statistics unit is used to determine the number of material center points output by the target semantic category regression head model, which is taken as the number of SMT materials;

[0092] The target rotating rectangle regression head model is trained using a first sample image, where the size of the SMT material in the first sample image is greater than the upper limit area threshold, and the first sample image is marked with the material region where each of the SMT materials in the first sample image is located; the target semantic category regression head model is trained using a second sample image, where the center point of each of the SMT materials in the second sample image is marked.

[0093] In one possible implementation, the target rotated rectangle regression head model is pre-trained by:

[0094] The first sample image is input into the original rotating rectangle regression head model, so that the original rotating regression head model extracts the visual features of the first sample image, and outputs all predicted regions containing objects in the first sample image, as well as the predicted category and confidence level of each predicted region based on the visual features; wherein, the predicted category is used to represent the category of the object present in the predicted region, and the confidence level is used to represent the accuracy of the category;

[0095] A first loss is calculated based on the first difference between each predicted region and the material region where each SMT material is located as marked in the first sample image; wherein, the first loss is positively correlated with the first difference;

[0096] A second loss is calculated based on the second difference between the predicted category and the target category of the predicted region; wherein, the target category is the category of objects existing in the same region as the predicted region in the first sample image. If the predicted category is the same as the target category, the second difference is negatively correlated with the confidence of the predicted region; if the predicted category is different from the target category, the second difference is positively correlated with the confidence of the predicted region, and the second loss is positively correlated with the second difference.

[0097] Adjust the parameters of the original rotating rectangular regression head model in the direction of gradient descent of the first loss and the second loss to obtain the target rotating rectangular regression head model;

[0098] The target semantic category regression head model is pre-trained in the following ways:

[0099] The second sample image is input into the original semantic category regression head model so that the target semantic category regression head model extracts the semantic features of the second sample image and outputs the predicted material center point of each SMT material in the material region image based on the semantic features.

[0100] A third loss is calculated based on the third difference between the predicted material center point and the material center point of each SMT material marked in the second sample image; wherein the third loss is positively correlated with the third difference.

[0101] Adjust the parameters of the original semantic category regression head model in the direction of the third loss gradient descent to obtain the target semantic category regression head model.

[0102] In one possible implementation, the device further includes:

[0103] The model selection module is used to determine, from multiple first preset models, the model that meets the first preset condition as the target rotation regression head model, and from multiple second preset models, the model that meets the second preset condition as the target semantic category regression head model; wherein, the multiple first preset models are trained based on the same first sample image, and the different first preset models have different structures, and the multiple second preset models are trained based on the same second sample image, and the different second preset models have different structures.

[0104] In one possible implementation, the center point output module includes:

[0105] The first center point output submodule is used to output a material center point mask image based on the semantic features of the material region image; wherein, the material center point mask is used to represent the material center point of each SMT material in the material region image, the size of the material center point mask image is the same as that of the material region image, and each element is used to represent the confidence level of pixels with the same position in the material region image as material center points;

[0106] The second quantity statistics submodule includes:

[0107] The third quantity statistics unit is used to pool the material center point mask image to obtain the pooled material center point mask image.

[0108] The fourth quantity statistics unit is used to determine the maximum value greater than the preset screening threshold in the pooled material center point mask image, and to determine the number of the obtained maximum values ​​as the quantity of the SMT material.

[0109] In one possible implementation, the first sample image and the second sample image are pre-annotated in the following ways:

[0110] Acquire sample material area images obtained by photographing SMT sample materials; wherein each sample material area image is obtained by photographing SMT sample materials of different sizes, and each sample material area image includes multiple SMT sample materials, and the different sizes of SMT sample materials include a first size material and a second size material, the first size material is an SMT sample material with a size greater than the upper limit area threshold, and the second size material is an SMT sample material with a size not greater than the upper limit area threshold;

[0111] The center point of each SMT sample material in the sample material area image is marked to obtain the original sample image;

[0112] In the original sample image, the original sample image obtained by taking a picture of the material of the first size is determined as the first preprocessed sample image. The material area where the SMT sample material is located is marked in the first preprocessed sample image to obtain the first sample image.

[0113] The original sample image is determined as the second sample image, or other images in the original sample image other than the first preprocessed sample image are determined as the second sample image.

[0114] In one possible implementation, determining the original sample image as the second sample image, or determining other images in the original sample image besides the first preprocessed sample image as the second sample image, includes:

[0115] The original sample image is determined as the second preprocessed sample image, or other images in the original sample image other than the first preprocessed sample image are determined as the second preprocessed sample image;

[0116] According to the category of the SMT sample material captured, each of the second preprocessed sample images is uniformly sampled to obtain the sampling result, which is used as the second sample image;

[0117] The step of marking the material region where the SMT sample material is located in the first preprocessed sample image to obtain the first sample image includes:

[0118] Mark the material region where the SMT sample material is located in the first preprocessed sample image to obtain the third preprocessed sample image;

[0119] According to the category of the SMT sample material captured, each of the third preprocessed sample images is uniformly sampled to obtain the sampling result, which is used as the first sample image.

[0120] In one possible implementation, the image cropping module includes:

[0121] The first image cropping submodule is used to perform grayscale binarization on the original image based on the principle of maximizing the variance of grayscale values ​​between the foreground region and the background region after binarization, to obtain a preprocessed image; wherein, the original image is an X-ray image;

[0122] The second image cropping submodule is used to determine the grayscale value boundary line in the preprocessed image;

[0123] The third image cropping submodule is used to crop the sub-image within the boundary line from the preprocessed image as the material area image.

[0124] In one possible implementation, the second quantity statistics submodule includes:

[0125] The fifth quantity statistics unit is used to calculate the quantity of SMT materials based on the detection results of the unenlarged material area image of the second material detection model if the area represented by the statistical value is greater than the lower limit area threshold.

[0126] The sixth quantity statistics unit is used to calculate the quantity of SMT materials based on the detection results of the magnified material area image of the second material detection model if the area represented by the statistical value is not greater than the lower limit area threshold.

[0127] In one possible implementation, the first size determination submodule includes:

[0128] The second size determination unit is used to detect the candidate regions where all potential SMT materials are located in the material region image, and the confidence level of each candidate region; wherein, the confidence level is used to represent the probability that an SMT material exists in the candidate region;

[0129] The third size determination unit is used to determine the region in the candidate region where the confidence level is not lower than a preset confidence threshold, as the material region where each SMT material in the material region image is located.

[0130] Thirdly, embodiments of this application provide a method for SMT material quantity statistics, the method comprising:

[0131] Acquire raw images of SMT materials by photographing them;

[0132] The dimensions of the SMT material are determined based on the original image;

[0133] Based on the detection results of the original image using the target material detection model, the quantity of the SMT material is calculated; wherein, the target material detection model is a material detection model corresponding to the size of the SMT material.

[0134] Fourthly, embodiments of this application provide an SMT material quantity counting device, the device comprising:

[0135] The acquisition module is used to acquire raw images obtained by photographing SMT materials;

[0136] A determining module is used to determine the dimensions of the SMT material based on the original image;

[0137] The statistics module is used to count the quantity of SMT materials based on the detection results of the original image by the target material detection model; wherein the target material detection model is a material detection model corresponding to the size of the SMT materials.

[0138] Fifthly, embodiments of this application provide an electronic device, including:

[0139] Memory, used to store computer programs;

[0140] The processor, when executing a program stored in memory, implements any of the above-described material quantity statistics methods.

[0141] Sixthly, embodiments of this application provide a counting machine, which includes an image acquisition device, a processor, and a display;

[0142] The image acquisition device is used to obtain raw images of SMT materials by photographing them;

[0143] The processor is configured to acquire the original image obtained by the image acquisition device by capturing SMT materials; crop the image of all material placement areas from the original image as a material area image; determine the size of the SMT materials based on the material area image; count the number of SMT materials based on the detection results of the material area image using a target material detection model; and send the number of SMT materials to the display; wherein the target material detection model is a material detection model corresponding to the size of the SMT materials.

[0144] The display is used to show the quantity of the SMT materials.

[0145] Seventhly, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the material quantity statistics methods described above.

[0146] Beneficial effects of the embodiments in this application:

[0147] This application provides a material quantity counting method and apparatus. It obtains an original image of the SMT material by capturing images of the material, and then precisely crops the image of the SMT material placement area from the original image, effectively removing background information irrelevant to the SMT material quantity counting. Using the image of the material area without background information, the material size of the SMT material is determined. Based on the material size, a matching material detection model is selected. This matching material detection model can accurately output detection results reflecting the quantity of SMT materials of that size. Based on the detection results, the quantity of SMT materials is counted, improving the relevance of SMT material quantity counting and making it applicable to various SMT material size counting scenarios, thereby improving the accuracy of SMT material quantity counting.

[0148] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0149] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0150] Figure 1 is a flowchart illustrating the SMT material quantity statistics method provided in an embodiment of this application;

[0151] Figure 2 is a first schematic diagram of the SMT material provided in the embodiment of this application;

[0152] Figure 3 is a second schematic diagram of the SMT material provided in the embodiment of this application;

[0153] Figure 4 is a schematic diagram of image preprocessing provided in an embodiment of this application;

[0154] Figure 5 is a schematic diagram of sample annotation provided in the embodiments of this application;

[0155] Figure 6 is a schematic diagram of the depth model provided in an embodiment of this application;

[0156] Figure 7 is a schematic diagram of the hardware system provided in an embodiment of this application;

[0157] Figure 8 is a schematic diagram of the data acquisition module provided in an embodiment of this application;

[0158] Figure 9 is a schematic diagram of the data acquisition module acquiring data according to an embodiment of this application;

[0159] Figure 10 is a schematic diagram of the structure of the construction counting model module provided in the embodiment of this application;

[0160] Figure 11 is a schematic diagram of the SMT material quantity statistics module provided in an embodiment of this application;

[0161] Figure 12 is a flowchart illustrating the counting process of the SMT material quantity statistics module provided in an embodiment of this application;

[0162] Figure 13 is a schematic diagram of the first structure of the SMT material quantity counting device provided in the embodiment of this application;

[0163] Figure 14 is a schematic diagram of a second structure of the SMT material quantity counting device provided in an embodiment of this application;

[0164] Figure 15 is a structural schematic diagram of the material counting machine provided in an embodiment of this application;

[0165] Figure 16 is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0166] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments in this application are within the scope of protection of this application.

[0167] With the rapid development of modern information technology, a wide variety of electronic products have entered the public eye. Among them, circuit boards, as key components controlling the core functions of electronic products, are particularly important in their design and manufacturing process. The construction of circuit boards involves a large number and diverse range of materials, such as resistors and inductors. Therefore, to improve the manufacturing efficiency of circuit boards, surface mount technology (SMT) is often used in the production process. In SMT production lines, SMT materials (i.e., surface mount technology materials), such as resistors and inductors, are used very frequently. Therefore, small-particle materials such as resistors and inductors need to be frequently issued and received. Reasonable material inventory management is essential to accurately track changes in the quantity of small-particle materials such as resistors and inductors during these frequent issuances and receipts for management purposes. Regular and comprehensive material inventory management not only allows for real-time monitoring of material availability, preventing common supply chain issues such as material shortages, loss, and leakage, but also provides decision support for production line optimization, thereby improving production efficiency.

[0168] When managing the inventory of small-particle materials such as resistors and inductors, since these materials are often stored in material trays and are small in size, each material tray will contain a large number of materials in order to save space. Therefore, how to count the number of SMT materials on the material tray (hereinafter referred to as material quantity count) is a difficult problem.

[0169] In related technologies, the quantity of materials can be counted manually one by one, or an image recognition algorithm can be used. In this way, a large number of different types of SMT materials are manually extracted in advance, and the shape, size and other features are captured. The image of the material tray containing the components is then photographed. The extracted features are used to identify the components from the image of the material tray and the quantity of materials is counted.

[0170] However, the above methods have the following problems. First, the statistical method of manually counting each component is costly and inefficient, and is also subject to human error. Second, the aforementioned method using image recognition algorithms requires manual design and feature extraction. Due to the uncertainty of the shape and size of materials, the discreteness of their distribution, and the fact that materials may be very close to each other or multiple SMT materials may be connected together, this type of recognition method has a high false negative rate, that is, a low accuracy.

[0171] Based on this, this application proposes an SMT material quantity counting method to improve the accuracy of SMT material quantity counting. Referring to Figure 1, which is a flowchart illustrating the SMT material quantity counting method provided in this application, the method includes:

[0172] S101, acquire the raw image obtained by photographing the SMT material.

[0173] S102, crop all SMT material placement areas from the original image to obtain the material area image.

[0174] S103, determine the size of the SMT material based on the material area image.

[0175] S104: Based on the detection results of the target material detection model on the material area image, the quantity of SMT materials is statistically obtained.

[0176] The target material detection model is a material detection model corresponding to the size of the SMT material.

[0177] By applying the above embodiments, an original image of the SMT material is obtained by capturing the original image, and the image of the SMT material placement area is accurately cropped from the original image. This effectively removes background information that is irrelevant to the SMT material quantity statistics. Using the material area image without background information, the material size of the SMT material is determined, and a matching material detection model is selected based on the material size. This matching material detection model can accurately output detection results that reflect the quantity of SMT materials of that size. Then, based on the detection results, the quantity of SMT materials is counted, which improves the targeting of SMT material quantity statistics and is applicable to SMT material statistics scenarios of various sizes, thereby improving the accuracy of SMT material quantity statistics.

[0178] The following will provide a detailed explanation of the aforementioned steps S101-S104:

[0179] In step S101, there are multiple SMT materials, and SMT materials of the same size are placed in the same location.

[0180] The material area images in this article can be obtained by acquiring the actual material provided by the user for statistical analysis using an X-ray image acquisition device, or by acquiring the material using a visible light image acquisition device. Referring to Figure 2, which is a first schematic diagram of SMT material provided in this embodiment, 201 is a material area image obtained from the left side of an SMT roll, and 202 is the SMT material in material area image 201. Referring to Figure 3, which is a second schematic diagram of SMT material provided in this embodiment, 301 is a material area image obtained from the left side of another SMT roll, and 302 is the SMT material in material area image 301. For ease of understanding and illustration, the figures only show a relatively sparse arrangement of SMT material; in actual applications, the arrangement of SMT material will be much denser.

[0181] In step S102, during the process of capturing the material area image, background areas other than SMT materials may inevitably be captured. If the background area also contains materials identical to or similar to the SMT materials, these SMT materials or objects in the background area may be incorrectly included in the material count, leading to inaccurate material quantity statistics. On the other hand, when the material area image contains too many background images, this not only increases the complexity of image processing but also reduces the efficiency of material quantity statistics.

[0182] Based on this, in order to improve the accuracy of material quantity statistics, reduce the complexity of image processing, and thus improve the efficiency of material quantity statistics, images of all material placement areas can be cropped from the original image as material area images. The placement area can be the area where the SMT roll is located, the area where the SMT tray is located, or any area where the SMT material is located. No specific limitation is made here.

[0183] In one possible embodiment, a preset cutting method can be set based on the work experience of professional technicians to cut out the area where the material is placed. However, this method has low applicability and may result in the material area image being included in the background area image, leading to a low accuracy rate in material quantity counting. Therefore, in another possible embodiment, to improve the accuracy rate of material quantity counting, the aforementioned step S102 includes steps S1021, S1022, and S1023, specifically:

[0184] S1021, based on the principle of maximizing the variance of gray values ​​between the foreground and background regions after binarization, the original image is subjected to gray-level binarization to obtain a preprocessed image. The original image is an X-ray image.

[0185] One approach is to determine a grayscale threshold based on the principle of maximizing the variance of grayscale values ​​between the foreground and background regions after binarization. This grayscale threshold is then used to perform grayscale binarization on the original image to obtain a preprocessed image. For example, a function relating the grayscale variance to the grayscale threshold can be constructed, and the grayscale value at which the variance is maximized can be calculated and used as the grayscale threshold. Alternatively, the variance of the grayscale values ​​can be calculated for each of the multiple grayscale values ​​until the grayscale value at which the variance is maximized is obtained, and this value can then be used as the grayscale threshold.

[0186] S1022, Determine the boundary line of grayscale values ​​in the preprocessed image.

[0187] The setting of this dividing line is quite flexible. It can be the outer rectangular boundary of the area where the SMT material is located, or it can be a boundary of other shapes. There are no specific limitations here.

[0188] S1023, Extract the sub-image within the boundary line from the preprocessed image as the material area image.

[0189] Taking the boundary line as the outer rectangular boundary of the area where the SMT material is located as an example, the sub-image within the outer rectangular boundary is cropped from the preprocessed image as the material area image. The cropped image is the non-material area image.

[0190] Referring to Figure 4, which is a schematic diagram of image preprocessing provided in an embodiment of this application, taking the material area image as an example of photographing the material contained in an SMT roll, 401 is the original image obtained by photographing the SMT roll, and 402 is the material area image obtained after preprocessing.

[0191] By applying the above embodiments and utilizing grayscale binarization technology, based on the principle of maximizing the variance of grayscale values ​​between the foreground and background regions, the boundary between the material region and the background becomes clearer. On this basis, the grayscale value boundary line is further determined, and the sub-image within the boundary line is cropped as the material region image. This removes redundant background information, reduces noise interference, and lowers the complexity of image processing. This not only improves the efficiency of material quantity statistics but also improves the accuracy of material quantity statistics.

[0192] In step S103, the size of the SMT material is used to indicate the material's specifications, i.e., whether the SMT material is a large-size or small-size material. In one possible embodiment, a professional can set a threshold based on their experience, and use this threshold and an image recognition algorithm to determine the material's size; however, this method has low accuracy.

[0193] In another possible embodiment, visual features of the material area image can be extracted. These visual features represent the contours in the material area image and the location of the contours. Based on the contours in the material area image and the location of the contours, the size of the SMT material is determined. The aforementioned step S103 includes steps S1031 and S1032, specifically:

[0194] S1031, based on the visual features of the material area image, detect the material area where each SMT material is located in the material area image.

[0195] S1032, determine the size of the SMT material based on the area of ​​each material region.

[0196] The visual features of a material region image are the macroscopic features of the material, such as its outline and color. After obtaining these visual features, the material regions where each material in the image is located are detected based on these features.

[0197] When determining the size of SMT materials based on the area of ​​each material region, in one possible embodiment, the size of the SMT materials can be determined by using calculation software or a mathematical model, based on a preset conversion ratio between material region area and size, and the measured area of ​​each material region, combined with the aforementioned conversion ratio. The preset conversion ratio is set according to the product design requirements, production process specifications, and relevant parameters of the SMT materials.

[0198] In another possible embodiment, the size of the SMT material can be determined based on the statistical values ​​of the area of ​​each material region. The aforementioned step S1032 includes steps S10321, S10322, and S10323, specifically:

[0199] S10321, Statistical values ​​of the area of ​​each material region are obtained.

[0200] The statistical value is used to represent the area of ​​a single SMT material.

[0201] S10322, If the area represented by the statistical value is greater than the upper limit area threshold, then the size of the SMT material is determined as the first size.

[0202] S10323, if the area represented by the statistical value is not greater than the upper limit area threshold, then the size of the SMT material is determined to be the second size.

[0203] In one possible embodiment, statistical values ​​of each material region can be obtained. These statistical values ​​represent the area of ​​a single material. For example, if the material region is represented in the form of a material frame, the statistical value can be the average area of ​​all detected material frames, or the median area of ​​all detected material frames, etc.

[0204] In another possible embodiment, the statistical values ​​of the diagonal length, perimeter, and side length of each material area can be obtained. Taking the material area as represented by a material frame as an example, the statistical value can be the maximum value of the diagonal length of all material frames or the maximum value of the side length of all material frames.

[0205] The upper limit area threshold can be set according to the user's actual needs or experience. This application does not impose any restrictions on this. It is understood that if the user needs to control the efficiency within a high range, a lower upper limit area threshold can be set. If the user needs to control the accuracy within a high range, a higher upper limit area threshold can be set.

[0206] The first dimension indicates that the SMT material is a large-size material, and the second dimension indicates that the SMT material is a small-size material. If the area represented by the statistical value is greater than the upper area threshold, the SMT material is determined to be the first dimension, i.e., a large-size material. If the area represented by the statistical value is not greater than the upper area threshold, the SMT material is determined to be the second dimension, i.e., a small-size material.

[0207] In one possible embodiment, after acquiring the material area image, the image can be input into an algorithm model pre-set by the user based on experience. This algorithm model extracts visual features and detects the material area of ​​the SMT material based on these features. However, user experience is often limited, making experience-based algorithm models difficult to apply to various scenarios.

[0208] Based on this, in another possible embodiment, the material region image can be input into a depth model, which extracts visual features based on the material region image and outputs the material region where the SMT material is located in the material region image according to the visual features. In this embodiment, the depth model is a target rotation regression head model, and the aforementioned step S1031 may include step S10311, specifically:

[0209] S10311, Input the material area image into the target rotary regression head model so that the target rotary regression head model extracts the visual features of the material area image, and outputs the material area where each SMT material is located in the material area image based on the visual features.

[0210] In this embodiment, the material region is represented in the form of a material box. The target rotation regression head model is pre-trained, and the specific training method will be described below, without further details. It is understood that although the material region in this embodiment is represented in the form of a material box, this does not limit the material region in this application to be represented in the form of a material box. In other possible embodiments, it can also be represented in other forms besides a material box, such as a mask or vertex coordinates. For ease of description, the following explanation only uses the case where the material region is represented in the form of a material box as an example. The principle is the same for cases where the material region is represented in other forms, and will not be elaborated upon here.

[0211] By applying the above embodiments, the material region image is input into the target rotational regression head model. The model extracts visual features from the image and accurately outputs all material regions containing material in the image based on these features. This effectively reduces detection errors caused by experience-based algorithm models, improving the accuracy of material region detection and consequently, the accuracy of material quantity statistics. Furthermore, outputting material regions based on a depth model eliminates the need for manual intervention, automating material quantity statistics and reducing labor costs.

[0212] Considering the possibility of erroneously detecting areas that are not material locations as SMT material locations, leading to a higher number of material frames than the actual number of materials, directly determining the number of material frames as the number of SMT materials could result in an excessive number of materials being counted. Therefore, in one possible embodiment, the regions in the material region image are filtered, and the filtered regions are used as the material regions. In this embodiment, step S1031 may specifically include steps S10312 and S10313, specifically:

[0213] S10312 detects the candidate regions of all potential SMT materials in the material region image, as well as the confidence level of each candidate region.

[0214] S10313, determine the region with a confidence level not lower than the preset confidence threshold in the candidate region, and use it as the material region where each SMT material in the material region image is located.

[0215] In order to ensure the independence between material regions and reduce material quantity statistics errors caused by inaccurate detection, the potential regions of all potential SMT materials in the material region image can be detected before the candidate regions are detected. Then, the potential regions are filtered. Specifically, regions that meet the preset non-overlapping conditions can be identified as candidate regions.

[0216] The specific method for determining the candidate regions that meet the preset non-overlap condition can be achieved by performing a non-maximum suppression (NMS) operation on the candidate regions. Therefore, the preset non-overlap condition can be understood as follows: during the NMS operation, the cross-union ratio between any two retained candidate regions is lower than a preset threshold. Similarly, this threshold can be set according to the user's actual needs or experience.

[0217] The specific method for determining the region that meets the preset non-overlapping condition in the candidate region can also be implemented in a way other than NMS operation. For example, the area of ​​the overlapping region can be calculated. If the area is greater than the preset area threshold, the overlapping condition is determined to be met. If it is not greater than the preset area threshold, the overlapping condition is determined not to be met.

[0218] The confidence level is used to represent the probability that SMT materials are present in the initial region. The confidence level threshold can be set according to the user's actual needs or experience, and this application does not impose any restrictions on it. It is understood that if the user needs to control the false negative rate to a low range, a lower confidence level threshold can be set; if the user needs to control the false positive rate to a low range, a higher confidence level threshold can be set.

[0219] By applying the above embodiments, by detecting the candidate regions where all potential materials are located in the material region image and assigning a confidence level to each candidate region, and by setting a preset confidence threshold, regions with a confidence level not lower than the threshold are selected as the material regions where all materials are located in the material region image. This process effectively eliminates low-probability false detection regions (i.e., regions where no materials exist), improves the accuracy of detection, reduces material quantity statistics errors caused by detection inaccuracies, and further improves the accuracy of material quantity statistics.

[0220] In step S104, a material detection model is selected based on the size of the SMT material to accurately detect the quantity of material of that size. Step S104 includes steps S1041 and S1042, specifically:

[0221] S1041, if the size of the SMT material is the first size, then based on the detection results of the material area image by the first material detection model, the quantity of SMT materials is obtained.

[0222] The first material detection model is a material detection model based on visual features.

[0223] S1042, if the size of the SMT material is the second size, then based on the detection results of the material area image by the second material detection model, the quantity of SMT materials is obtained.

[0224] The second material detection model is a material detection model based on semantic features.

[0225] Both semantic and visual features are directly derived from the material region image. When the area of ​​an SMT material exceeds the upper area threshold, it is considered to be of the first size, i.e., a large-sized material. Therefore, the number of SMT materials can be determined based on the macroscopic visual features of the material region image. When the area of ​​an SMT material is not greater than the upper area threshold, it is considered to be of the second size, i.e., a small-sized material. Therefore, the number of SMT materials can be determined based on the pixel-level semantic features of the material region image.

[0226] The material detection model can be any model capable of detecting results that characterize the quantity of materials. In one possible embodiment, the first material detection model is a target rotational regression head model, and the second material detection model is a target semantic category regression head model.

[0227] When the first material detection model is the target rotating regression head model, the aforementioned step S1041 includes step S10411, specifically:

[0228] S10411, determine the number of material regions output by the target rotary regression head model, as the quantity of materials.

[0229] When the second material detection model is the target semantic category regression head model, if the size of the SMT material is the second size, the material region image is input into the target semantic category regression head model so that the target semantic category regression head model extracts the semantic features of the material region image and outputs the material center point of each SMT material in the material region image based on the semantic features.

[0230] The aforementioned step S1042 includes step S10421, specifically:

[0231] S10421, determine the number of material center points output by the target semantic category regression head model, and use them as the number of SMT materials.

[0232] Applying the above embodiments, when the SMT materials are large-sized, these materials can be accurately represented by macroscopic visual features. Therefore, it can be assumed that the quantity of large-sized materials can be accurately detected based on visual features. Thus, the quantity of materials can be directly determined based on the number of material regions identified by visual features. When the SMT materials are small-sized, the material region image is input into the target semantic category regression head model. The model extracts semantic features from the material region image and accurately outputs the center point of each material based on these features. By counting the number of material center points, the total number of materials can be directly obtained. This method eliminates the need for manual intervention and uses different quantity counting methods for different sizes of SMT materials, improving both the automation level and accuracy of material quantity counting.

[0233] In one possible embodiment, a material center point mask is determined based on the semantic features of the material region image. The number of materials is obtained by the area ratio of the mask regions. For example, given a reference mask area A and a mask region area B, the number of materials in that mask region can be estimated based on the value of B / A. The mask region is then divided equally, and the center of each equally divided mask region is taken as the material center. This method relies on the area ratio of the mask regions to estimate the number of materials. This area-based estimation method may lead to a decrease in accuracy when the shape and size of SMT materials vary greatly or when SMT materials are densely arranged. Furthermore, the practice of equally dividing the mask region to obtain the material center may lead to inaccurate material center positioning when the SMT materials are unevenly distributed or have irregular shapes, thus affecting the accuracy of material quantity counting.

[0234] Based on this, in another possible embodiment, the aforementioned steps output the material center point of each SMT material in the material region image based on semantic features, specifically including:

[0235] Based on the semantic features of the material region image, output the material center point mask image.

[0236] The material center point mask is used to represent the material center point of each SMT material in the material area image. The size of the material center point mask image is the same as that of the material area image, and each element is used to represent the confidence level of the pixel with the same position in the material area image as the material center point.

[0237] Based on this, the aforementioned step S10421 specifically includes:

[0238] The material center point mask image is pooled to obtain the pooled material center point mask image.

[0239] In the pooled material center point mask image, determine the maximum value greater than the preset screening threshold, and use the number of determined maximum values ​​as the number of materials.

[0240] In this embodiment, a material center point mask is used to represent the aforementioned material center points, and a material center point mask image is used to represent the material center point image. Maximum values ​​are determined in the pooled material center point mask image. Maximum values ​​higher than a preset screening threshold are identified and recorded as valid maximum points. The number of valid maximum points is counted as the number of materials, and the position of each valid maximum point is the material center point of each material in the material region image. It is understood that in this embodiment, a maximum value is recorded as a valid maximum point in response to a maximum value exceeding the preset screening threshold; therefore, the preset screening threshold can be considered as a response threshold. The preset screening threshold can be set according to the user's actual needs or experience, and this application does not impose any restrictions on it.

[0241] By applying the above embodiments, the material center point mask image is determined based on the semantic features of the material area image. By performing pooling processing on the material center point mask image, data redundancy is effectively reduced, and noise interference is also effectively reduced. In the pooled material center point mask image, the maximum value is determined by setting a preset filtering threshold, which reduces the inaccuracy caused by adhesion phenomenon, thereby accurately obtaining the real material center point and accurately counting the number of materials. The result is accurate, and the accuracy of material quantity counting is improved.

[0242] In some scenarios, small-sized SMT materials may contain SMT materials with an area much smaller than the upper area threshold. For these excessively small SMT materials, it is difficult to accurately detect the center point of the material directly based on the original material area image. Therefore, in one possible embodiment, a lower area threshold can be preset. The aforementioned step S1042 may also include steps S10422 and S10423, specifically:

[0243] S10422, If the area represented by the statistical value is greater than the lower limit area threshold, then based on the detection results of the unmagnified material area image by the second material detection model, the number of SMT materials is statistically obtained.

[0244] S10423, if the area represented by the statistical value is not greater than the lower limit area threshold, the quantity of SMT materials is statistically obtained based on the detection results of the magnified material area image by the second material detection model.

[0245] The lower limit area threshold can be set according to the user's actual needs or experience. This application does not restrict this, but the lower limit area threshold should be much smaller than the upper limit area threshold.

[0246] Enlarging an image of a material region can be achieved by upsampling the image, which involves using interpolation algorithms (such as bilinear interpolation, bicubic interpolation, or more advanced algorithms) to increase the number of pixels and thus improve image resolution. Alternatively, other methods can be employed to enlarge the image, such as image enhancement techniques to improve visual quality, making SMT material features clearer and easier to detect. Image enhancement techniques can involve adjusting parameters like brightness, contrast, and sharpening, or applying filters to reduce noise and interference. In some cases, multiple methods can be combined to achieve the best magnification effect; specific limitations are not specified here.

[0247] Applying the above embodiments, the method determines whether the area represented by the statistical value is greater than a lower area threshold. If the area is greater than the lower area threshold, the quantity of materials is counted based on the detection results of the unmagnified material region image by the second material detection model, saving computational resources and improving the speed of material quantity counting. When the area represented by the statistical value is not greater than the lower area threshold, i.e., when the image details are blurry or the material distribution is dense, the method magnifies the material region image to obtain a clearer and easier-to-detect image. Based on the detection results of the magnified material region image by the second material detection model, the quantity of materials is counted more accurately. This method not only improves the accuracy of material quantity counting but also enhances its applicability, enabling it to handle material region images of different sizes and complexities, effectively improving the accuracy of material quantity counting.

[0248] The following will explain in detail how to train the aforementioned target rotation rectangle regression head model and target semantic category regression head model. The target rotation rectangle regression head model is trained using the first sample image, and the target semantic category regression head model is trained using the second sample image. The following will elaborate on the two stages of sample preprocessing and model training:

[0249] I. Sample Preprocessing

[0250] The material size of the SMT material in the first sample image is greater than the upper limit area threshold. The first sample image is marked with the material area where each SMT material is located. The material size of the SMT material in the second sample image is not greater than the upper limit area threshold. The second sample image is marked with the material center point of each SMT material in the second sample image.

[0251] The first and second sample images were pre-annotated using the following methods:

[0252] S201, acquire the image of the sample material area obtained by photographing the SMT sample material.

[0253] Each sample material region image is obtained by photographing SMT sample materials of different sizes, and each sample material region image includes multiple SMT sample materials. The different sizes of materials include first-size materials and second-size materials. The first-size materials are sample materials whose size is greater than the upper limit area threshold, and the second-size materials are sample materials whose size is not greater than the upper limit area threshold. The first-size materials are sample materials with the first size, and the second-size materials are sample materials with the second size.

[0254] S202, mark the center point of each SMT sample material in the sample material area image to obtain the original sample image.

[0255] The center point of the SMT sample material can be obtained by manual marking or by other marking methods, and no specific limitation is made here.

[0256] S203, in the original sample image, determine the original sample image obtained by shooting the material of the first size as the first preprocessed sample image, mark the material area where the sample material is located in the first preprocessed sample image, and obtain the first sample image.

[0257] S204, determine the original sample image as the second sample image, or determine the other images in the original sample image other than the first preprocessed sample image as the second sample image.

[0258] In other words, when training the target rotation regression head model, images of material regions labeled with the locations of materials of various sizes are used for training. When training the target semantic category regression head model, images labeled with the center points of each small-sized material, images labeled with the center points of each large-sized material, or images labeled with the center points of each small-sized material can be used for training. Alternatively, images labeled with the center points of both large-sized and small-sized materials can be used for training.

[0259] Taking an image labeled with the material areas of various sizes as the first sample image and an image labeled with the center points of each small size material as the second sample image as an example, see Figure 5. Figure 5 is a schematic diagram of sample labeling provided in this embodiment. In Figure 5, 501 and 502 are sample images obtained by X-Ray equipment from real SMT rolls. The center points of each sample material in the sample images are labeled. The material center point labeling of sample image 501 results in labeled sample image 5011, and the material center point labeling of sample image 502 results in labeled sample image 5021. Secondary labeling is performed on the first-size materials. The materials in sample image 502 and the labeled sample image 5021 are first-size materials, and the material areas of each sample material in the sample images are labeled. Taking the material area as a material frame, with the material frame being a rectangle as an example, secondary labeling is performed on the labeled sample image 5021 to obtain the secondary-labeled sample image 5022. This application is applicable to SMT material scenarios with small sizes and large quantities. For ease of understanding and illustration, the diagram only shows a relatively sparse arrangement of SMT components, while in actual applications, the arrangement of SMT components will be much denser.

[0260] Applying the above embodiments, firstly, multiple images of sample material regions obtained from photographing SMT sample materials are acquired. These images contain sample materials of various sizes. The center points of these materials are labeled to obtain the original sample images, ensuring sample diversity and enabling the trained model to better adapt to materials of different sizes. Based on material size, the original sample images are divided into two groups: one group consists of first-size materials (size greater than the upper area threshold) captured as the first preprocessed sample images; the other group consists of images of first-size materials (size greater than the upper area threshold) and second-size materials (size not greater than the upper area threshold), used as the second sample images. The material regions where the sample materials in the first preprocessed sample images are located are labeled to obtain the first sample images. The model is trained using both the first and second sample images, helping the model learn the characteristics of materials of different sizes. In summary, this method improves sample quality, enhances the accuracy and generalization ability of the trained model, is applicable to various material quantity statistics scenarios, and improves the accuracy of material quantity statistics.

[0261] Since SMT materials exist in various categories, the size mentioned in this article refers to the specification of the SMT material, and the category refers to the type of SMT material. For example, resistors and inductors are different categories of SMT materials, and large-size resistors and small-size resistors are different sizes of SMT materials. To further improve the generalization ability of the trained model, in one possible embodiment, the labeled sample images can be uniformly sampled according to the category of the sample material, and the sampled sample images are used as the first sample image and the second sample image. The above step S203 includes steps S2031 and S2032, specifically:

[0262] S2031, the original sample image is determined as the second preprocessed sample image, or, other images in the original sample image besides the first preprocessed sample image are determined as the second preprocessed sample image.

[0263] S2032, according to the category of the SMT sample material captured, uniformly sample each second preprocessed sample image to obtain the sampling result as the second sample image.

[0264] The above step S204 includes steps S2041 and S2042, specifically:

[0265] S2041, mark the material area where the SMT sample material is located in the first preprocessed sample image to obtain the third preprocessed sample image.

[0266] S2042, according to the category of the SMT sample material captured, uniformly sample each third preprocessed sample image to obtain the sampling result, which is used as the first sample image.

[0267] By applying the above embodiments, each second preprocessed sample image and each third preprocessed sample image are uniformly sampled according to the category of the sample material being photographed. This allows the obtained second sample images and first sample images to comprehensively and evenly reflect the sample material of each category, enhancing the generalization ability of the model trained using the first and second sample images. This enables the entire model to exhibit superior performance when facing sample materials of different categories and sizes, thereby making the material quantity statistics method applicable to various material statistics scenarios of different sizes and improving the accuracy of material quantity statistics.

[0268] Using the above method, the first sample image and the second sample image can be obtained. These two images together constitute the training set for the training model in this application. To further improve model performance, in one possible embodiment, a validation set can also be set. The validation set is obtained in the same way as the training set. The specific usage of the validation set will be described below and will not be repeated here. However, the obtained training set and validation set contain each real material category, and the samples in the training set and validation set do not overlap. If the number of samples in a certain category is insufficient, the sampling requirements of the training set will be prioritized.

[0269] II. Model Training

[0270] The target rotation rectangle regression head model is pre-trained using the following methods:

[0271] S301, the first sample image is input into the original rotating rectangle regression head model so that the original rotating regression head model extracts the visual features of the first sample image and outputs the prediction regions containing all objects in the first sample image, as well as the prediction category and confidence of each prediction region based on the visual features.

[0272] The prediction category is used to represent the category of objects existing in the prediction area, and the confidence level is used to represent the accuracy of the category. The object categories include SMT materials and non-SMT materials.

[0273] S302, calculate the first loss based on the first difference between each predicted region and the material region where each material is located as marked in the first sample image.

[0274] Among them, the first loss is positively correlated with the first difference.

[0275] S303, calculate the second loss based on the second difference between the predicted category and the target category of the predicted region.

[0276] The target category is the category of objects in the same region as the predicted region in the first sample image. If the predicted category is the same as the target category, the second difference is negatively correlated with the confidence of the predicted region. If the predicted category is different from the target category, the second difference is positively correlated with the confidence of the predicted region, and the second loss is positively correlated with the second difference.

[0277] S304. Adjust the parameters of the original rotated rectangular regression head model in the direction of gradient descent of the first and second losses to obtain the target rotated rectangular regression head model.

[0278] The target semantic category regression head model is pre-trained in the following ways:

[0279] S401, the second sample image is input into the original semantic category regression head model so that the target semantic category regression head model can extract the semantic features of the second sample image and output the predicted material center point of each material in the material region image based on the semantic features.

[0280] S402, calculate the third loss based on the third difference between the predicted material center point and the material center point of each material marked in the second sample image.

[0281] Among them, the third loss is positively correlated with the third difference.

[0282] S403, adjust the parameters of the original semantic category regression head model in the direction of the third loss gradient descent to obtain the target semantic category regression head model.

[0283] It is understandable that in this paper, the first loss is positively correlated with the first difference, the second difference is positively correlated with the confidence level of the prediction region, and the second loss is positively correlated with the second difference. Taking the positive correlation between the first loss and the first difference as an example, positive correlation means that, when other factors affecting the first loss remain unchanged except for the first difference, the first loss increases monotonically with the increase of the first difference. The monotonous increase in this paper can refer to strict monotonous increase or non-strict monotonous increase.

[0284] During the training of the above model, the first difference can be calculated using IOU (Intersection over Union) loss, and the second and third differences can be calculated using cross-entropy loss.

[0285] During the training of deep learning models, due to the iterative nature of training algorithms (such as gradient descent) and the potential for randomness, each training iteration may yield different combinations of model parameters. These different combinations of model parameters may correspond to different performances of the model on the training data. Therefore, it is common practice to try multiple different training configurations to generate multiple sets of training parameters.

[0286] To evaluate the model performance under these different parameter combinations, the aforementioned validation set can be used to determine whether the model is overfitting, underfitting, or has achieved a good generalization ability. Then, based on the model's performance on the validation set, the best set is selected as the final parameters of the deep learning model. These parameters will be used for the final deployment of the model to perform prediction or classification tasks in practical applications.

[0287] Applying the above embodiments, the target rotating rectangle regression head model, by comprehensively considering the matching degree between the predicted region and the actual material region, as well as the accuracy of the predicted category, and continuously optimizing the model parameters using gradient descent, achieves accurate prediction of the material location and category in the image. Meanwhile, the target semantic category regression head model focuses on predicting the material center point. By minimizing the difference between the predicted material center point and the actual labeled center point, it further improves the accuracy of material center identification. The target rotating rectangle regression head model and the target semantic category regression head model obtained through the above training methods significantly improve the accuracy of material quantity statistics.

[0288] During the model training phase, to ensure the model is applicable to various material statistics scenarios of different sizes and categories, multiple deep models with different structures are trained. A single model includes an image encoder, an image decoder, a rotated rectangle regression head, or a semantic category classification head. The structure is shown in Figure 6, which is a schematic diagram of the deep model provided in this embodiment. As shown in Figure 6, the deep model includes an image encoder, an image decoder, a rotated rectangle regression head, or a semantic category classification head. The image encoder is a feature extractor composed of convolutional layers, pooling layers, normalization layers, and activation layers. Its function is to downsample and map the input image to an abstract feature space. The image decoder is a feature interpreter composed of convolutional layers, normalization layers, activation layers, and upsampling layers. Its function is to progressively decode the abstract features extracted by the encoder into the required task output. The output head converts the features output by the image decoder into the required material bounding box coordinates or material center point mask.

[0289] Deep models can be divided into two categories: a first preset model and a second preset model. The first preset model includes a rotated rectangle regression head, and the second preset model includes a semantic category classification head. Multiple first preset models are trained on the same first sample image, and the structures of different first preset models are different. Similarly, multiple second preset models are trained on the same second sample image, and the structures of different second preset models are different.

[0290] In the structure of the first preset model, the image encoder can be a feature extraction module of a common object detection network, including but not limited to SSD (Single Shot MultiBox Detector), R-CNN (Regions with Convolutional Neural Networks), YOLO (You Only Look Once), and the YOLO series of real-time object detection algorithms. The image decoder can be a convolutional layer or a feature pyramid, etc. The rotated rectangle regression head is composed of multiple sub-networks combined with convolutional layers.

[0291] In the structure of the second pre-defined model, the image encoder can be a common downsampling network, including but not limited to ResNet (a residual network), VGGNet (a deep convolutional neural network), MobileNet (a lightweight convolutional neural network), and other common backbone networks and their improved structures, as well as self-built deep convolutional networks. The image decoder can be an upsampling module of common networks such as FCN (Fully Convolutional Networks for semantic segmentation), Unet (a deep learning network structure for image segmentation), DeepLab (a deep learning model for image segmentation), or a self-built upsampling network. The semantic category classifier is a softmax (a function) or sigmoid (a function) activation layer.

[0292] The first preset model, including the rotating rectangular regression head, is based on a target detection framework. It performs well for counting SMT material categories such as long, irregularly shaped, multi-peaked, and large-sized materials, but its performance is poor for counting small-sized, adhered, and numerous SMT material categories. The second preset model, including the semantic category classification head, is based on a semantic segmentation framework. It performs well for counting small-sized and adhered SMT material categories. When using the model to count the number of materials, the appropriate model can be selected according to the size of the SMT materials.

[0293] As mentioned above, the depth model includes multiple first preset models and multiple second preset models. For materials of similar size, different first preset models or different second preset models may be applicable due to factors such as material category and shape. Therefore, in one possible embodiment, when performing material quantity statistics, the applicable first preset model or second preset model can be selected according to preset conditions as the target rotational regression head model or the target semantic category regression head model, specifically including:

[0294] Among multiple first-preset models, the model that satisfies the first-preset condition is selected as the target rotation regression head model, and among multiple second-preset models, the model that satisfies the second-preset condition is selected as the target semantic category regression head model.

[0295] The first and second preset conditions can be set according to the user's actual needs or experience, and this application does not impose any restrictions on them.

[0296] By applying the above embodiments, by comparing multiple models with different structures and determining the model that meets specific conditions as the target model, the accuracy and generalization ability of the model can be significantly improved, thereby improving the accuracy of subsequent image processing and material quantity statistics.

[0297] This application also provides another method for SMT material quantity statistics, the method including the following steps:

[0298] Step 1: Obtain raw images of SMT materials by photographing them;

[0299] Step 2: Determine the dimensions of the SMT materials based on the original image;

[0300] Step 3: Based on the detection results of the original image using the target material detection model, the quantity of SMT materials is obtained; where the target material detection model is the material detection model corresponding to the size of the SMT materials.

[0301] Step one is equivalent to step S101 mentioned above, and step three is equivalent to step S104 mentioned above, which will not be repeated here.

[0302] In step two, in one possible embodiment, the size of the SMT material can be determined based on the original image in accordance with the manner described in steps S102 and S103 above.

[0303] In another possible embodiment, the size of the SMT material can be determined by image segmentation and pixel statistics, specifically:

[0304] Image segmentation algorithms, such as semantic segmentation models based on deep learning, are used to segment the original image, accurately separating the SMT material region from the background region and other non-material regions, resulting in an image containing only SMT materials.

[0305] In this image, pixel statistics are performed on each SMT material area to calculate its pixel area. Simultaneously, image calibration technology is used to establish a conversion relationship between image pixels and actual physical dimensions. For example, given the pixel size and actual size of a reference object of known dimensions in the image, the conversion ratio between pixels and actual units such as millimeters can be derived.

[0306] Based on the calculated pixel area of ​​the material region and the pixel-to-physical size conversion relationship, the actual area of ​​each SMT material is calculated. For materials with regular shapes (such as rectangles, circles, etc.), their length, width, or diameter can be derived from the area formula. For materials with irregular shapes, the equivalent size method can be used to approximate them as regular shapes before calculating the size, or the size can be determined by establishing an empirical model of the size and area of ​​irregular shapes.

[0307] By applying the above embodiments, an original image of the SMT material is obtained by taking a picture. Based on the original image, the material size of the SMT material is determined. Based on the material size, a matching material detection model is selected. The matching material detection model can accurately output the detection result reflecting the quantity of SMT materials of that size. Then, based on the detection result, the quantity of SMT materials is counted, which improves the pertinence of SMT material quantity statistics and is applicable to SMT material statistics scenarios of various sizes, thereby improving the accuracy of SMT material quantity statistics.

[0308] To more clearly explain the aforementioned material quantity statistics method, the following will provide a detailed description of the hardware system, hardware modules, and hardware units that execute this method:

[0309] Referring to Figure 7, which is a schematic diagram of the hardware system provided in the embodiment of this application, it includes a data acquisition module 701, a counting model construction module 702, and a material quantity statistics module 703.

[0310] The data acquisition module 701 is used to execute the various method steps in the aforementioned sample preprocessing.

[0311] The counting model construction module 702 is used to execute the various method steps in the aforementioned model training.

[0312] The material quantity statistics module 703 is used to execute the aforementioned steps S101-S104 or steps one-three. For detailed descriptions of the methods and steps executed by each module, please refer to the foregoing, and they will not be repeated here.

[0313] Referring to Figure 8, which is a schematic diagram of the structure of the data acquisition module provided in the embodiment of this application, the data acquisition module 701 includes a data collection unit 7011, a data labeling unit 7012, a data classification unit 7013, and a data division unit 7014.

[0314] The data collection unit 7011 is used to perform the aforementioned step S201.

[0315] The data labeling unit 7012 is used to perform the aforementioned steps S201, S202 and S203.

[0316] The data classification unit 7013 is used to perform the aforementioned step S203.

[0317] The data partitioning unit 7014 is used to execute the aforementioned steps S2031-S2042. For a detailed description of the method steps executed by each unit, please refer to the foregoing, and it will not be repeated here.

[0318] Referring to Figure 9, which is a schematic diagram of the data acquisition module acquiring data according to an embodiment of this application, taking the sample image 501 and sample image 502 in Figure 5 above as an example, the data collection unit 7011 collects the sample image 501 and sample image 502.

[0319] After the data annotation unit 7012 annotates each sample image, it obtains the annotated sample image 5011 and the annotated sample image 5021.

[0320] The data classification unit 7013 classifies the sample materials according to their size, dividing them into large-size sample materials (i.e., first-size materials) and small-size sample materials (i.e., second-size materials). The materials in sample image 502 and the labeled sample image 5021 are first-size materials. The material area is still represented by a material frame. Taking a rectangular frame as an example, the labeled sample image 5021 is labeled again to obtain the labeled sample image 5022.

[0321] Data partitioning unit 7014 uniformly samples the labeled sample images according to the category of sample materials to obtain training set 901 and validation set 902.

[0322] Referring to Figure 10, which is a schematic diagram of the structure of the counting model construction module provided in the embodiment of this application, as shown in Figure 10, the counting model construction module 702 includes a counting model construction unit 7021 and a counting model training unit 7022.

[0323] The counting model construction unit 7021 is used to construct the original rotated rectangle regression head model and the original semantic category regression head model in the aforementioned steps S301 and S401. The counting model training unit 7022 is used to execute the aforementioned steps S301-S304 and S401-S403. For detailed descriptions of the methods and steps executed by each unit, please refer to the foregoing, and they will not be repeated here.

[0324] Referring to Figure 11, which is a schematic diagram of the structure of the SMT material quantity statistics module provided in the embodiment of this application, the material quantity statistics module 703 includes a data preprocessing unit 7031, a material size estimation unit 7032, a model reasoning unit 7033, and a result filtering unit 7034.

[0325] The data preprocessing unit 7031 is used to execute the aforementioned steps S1021-S1023, the material size estimation unit 7032 is used to execute the aforementioned steps S1031 and S1032, the model reasoning unit 7033 is used to execute the aforementioned steps S104, S1041 and S1042, and the result filtering unit 7034 is used to execute step S10421. For a detailed description of the method steps executed by each unit, please refer to the foregoing, and it will not be repeated here.

[0326] Referring to Figure 12, which is a flowchart of the SMT material quantity statistics module for counting provided in the embodiment of this application, taking the original image 401 in Figure 4 above as an example, the data preprocessing unit 7031 performs material area cropping on the original image 401 to obtain the material area image 402.

[0327] The material size estimation unit 7032 inputs the material region image 402 into the target rotated rectangle regression head model 1201 to obtain all material regions containing materials in the material region image. For materials whose material region area statistics are greater than the upper limit area threshold, the number of material regions can be taken as the number of materials 1203. For materials whose material region area statistics are not greater than the upper limit area threshold, the material region image is input into the target semantic category regression head model 1202 to obtain the material center point mask image, and the number of material center point masks in the material center point mask image is taken as the number of materials.

[0328] The result filtering unit 7034 is used to filter the material center point mask image by pooling to obtain the maximum value, and the number of maximum value points obtained by filtering is taken as the number of materials 1204.

[0329] Corresponding to the aforementioned SMT material quantity counting method, this application embodiment also provides an SMT material quantity counting device. Referring to Figure 13, Figure 13 is a schematic diagram of the first structure of the SMT material quantity counting device provided in this application embodiment, including:

[0330] Image acquisition module 1301 is used to acquire raw images obtained by photographing SMT materials.

[0331] Image cropping module 1302 is used to crop images of all SMT material placement areas from the original image, as material area images.

[0332] The size determination module 1303 is used to determine the size of SMT materials based on the material area image.

[0333] The quantity statistics module 1304 is used to count the quantity of SMT materials based on the detection results of the material area image by the target material detection model.

[0334] The target material detection model is a material detection model corresponding to the size of the SMT material.

[0335] By applying the above embodiments, original images of SMT materials are obtained by capturing images of the materials. The image of the material placement area is then precisely cropped from the original image, effectively removing background information irrelevant to the SMT material quantity statistics. Using the material area image without background information, the material size of the SMT materials is determined. Based on the material size, a matching material detection model is selected. This matching material detection model can accurately output detection results reflecting the quantity of SMT materials of that size. Based on the detection results, the quantity of SMT materials is then counted, improving the targeting of SMT material quantity statistics. This method is applicable to SMT material quantity statistics scenarios of various sizes, thereby improving the accuracy of SMT material quantity statistics.

[0336] In one possible implementation, the size determination module includes:

[0337] The first size determination submodule is used to detect the material region where each SMT material is located in the material region image based on the visual features of the material region image; wherein, the visual features are used to represent the outline in the material region image and the position of the outline;

[0338] The second size determination submodule is used to determine the size of the SMT material based on the area of ​​each material region.

[0339] In one possible implementation, the second dimension determining submodule includes:

[0340] The first size determination unit is used to obtain statistical values ​​of the area of ​​each of the material regions; wherein the statistical values ​​are used to represent the area of ​​a single SMT material;

[0341] The second size determination unit is used to determine the size of the SMT material as the first size if the area represented by the statistical value is greater than the upper limit area threshold.

[0342] The third size determination unit is used to determine the size of the SMT material as the second size if the area represented by the statistical value is not greater than the upper limit area threshold.

[0343] The quantity statistics module includes:

[0344] The first quantity statistics submodule is used to count the quantity of SMT materials based on the detection results of the material area image by the first material detection model if the size of the SMT material is the first size. The first material detection model is a material detection model for SMT material detection based on visual features.

[0345] The second quantity statistics submodule is used to count the quantity of SMT materials based on the detection results of the material area image by the second material detection model if the size of the SMT material is the second size. The second material detection model is a material detection model for SMT material detection based on semantic features.

[0346] In one possible implementation, the first material detection model is a target rotational regression head model, and the second material detection model is a target semantic category regression head model;

[0347] The first size determination submodule includes:

[0348] The first size determination unit is used to input the material area image into the target rotary regression head model, so that the target rotary regression head model extracts the visual features of the material area image and outputs the material area where each SMT material in the material area image is located based on the visual features.

[0349] The first quantity statistics submodule includes:

[0350] The first quantity statistics unit is used to determine the number of material regions output by the target rotary regression head model, which is taken as the number of SMT materials;

[0351] The device further includes:

[0352] The center point output module is used to input the material region image into the target semantic category regression head model if the size of the SMT material is the second size, so that the target semantic category regression head model extracts the semantic features of the material region image and outputs the material center point of each SMT material in the material region image based on the semantic features.

[0353] The second quantity statistics submodule includes:

[0354] The second quantity statistics unit is used to determine the number of material center points output by the target semantic category regression head model, which is taken as the number of SMT materials;

[0355] The target rotating rectangle regression head model is trained using a first sample image, where the size of the SMT material in the first sample image is greater than the upper limit area threshold, and the first sample image is marked with the material region where each of the SMT materials in the first sample image is located; the target semantic category regression head model is trained using a second sample image, where the center point of each of the SMT materials in the second sample image is marked.

[0356] In one possible implementation, the target rotated rectangle regression head model is pre-trained by:

[0357] The first sample image is input into the original rotating rectangle regression head model, so that the original rotating regression head model extracts the visual features of the first sample image, and outputs all predicted regions containing objects in the first sample image, as well as the predicted category and confidence level of each predicted region based on the visual features; wherein, the predicted category is used to represent the category of the object present in the predicted region, and the confidence level is used to represent the accuracy of the category;

[0358] A first loss is calculated based on the first difference between each predicted region and the material region where each SMT material is located as marked in the first sample image; wherein, the first loss is positively correlated with the first difference;

[0359] A second loss is calculated based on the second difference between the predicted category and the target category of the predicted region; wherein, the target category is the category of objects existing in the same region as the predicted region in the first sample image. If the predicted category is the same as the target category, the second difference is negatively correlated with the confidence of the predicted region; if the predicted category is different from the target category, the second difference is positively correlated with the confidence of the predicted region, and the second loss is positively correlated with the second difference.

[0360] Adjust the parameters of the original rotating rectangular regression head model in the direction of gradient descent of the first loss and the second loss to obtain the target rotating rectangular regression head model;

[0361] The target semantic category regression head model is pre-trained in the following ways:

[0362] The second sample image is input into the original semantic category regression head model so that the target semantic category regression head model extracts the semantic features of the second sample image and outputs the predicted material center point of each SMT material in the material region image based on the semantic features.

[0363] A third loss is calculated based on the third difference between the predicted material center point and the material center point of each SMT material marked in the second sample image; wherein the third loss is positively correlated with the third difference.

[0364] Adjust the parameters of the original semantic category regression head model in the direction of the third loss gradient descent to obtain the target semantic category regression head model.

[0365] In one possible implementation, the device further includes:

[0366] The model selection module is used to determine, from multiple first preset models, the model that meets the first preset condition as the target rotation regression head model, and from multiple second preset models, the model that meets the second preset condition as the target semantic category regression head model; wherein, the multiple first preset models are trained based on the same first sample image, and the different first preset models have different structures, and the multiple second preset models are trained based on the same second sample image, and the different second preset models have different structures.

[0367] In one possible implementation, the center point output module includes:

[0368] The first center point output submodule is used to output a material center point mask image based on the semantic features of the material region image; wherein, the material center point mask is used to represent the material center point of each SMT material in the material region image, the size of the material center point mask image is the same as that of the material region image, and each element is used to represent the confidence level of pixels with the same position in the material region image as material center points;

[0369] The second quantity statistics submodule includes:

[0370] The third quantity statistics unit is used to pool the material center point mask image to obtain the pooled material center point mask image.

[0371] The fourth quantity statistics unit is used to determine the maximum value greater than the preset screening threshold in the pooled material center point mask image, and to determine the number of the obtained maximum values ​​as the quantity of the SMT material.

[0372] In one possible implementation, the first sample image and the second sample image are pre-annotated in the following ways:

[0373] Acquire sample material area images obtained by photographing SMT sample materials; wherein each sample material area image is obtained by photographing SMT sample materials of different sizes, and each sample material area image includes multiple SMT sample materials, and the different sizes of SMT sample materials include a first size material and a second size material, the first size material is an SMT sample material with a size greater than the upper limit area threshold, and the second size material is an SMT sample material with a size not greater than the upper limit area threshold;

[0374] The center point of each SMT sample material in the sample material area image is marked to obtain the original sample image;

[0375] In the original sample image, the original sample image obtained by taking a picture of the material of the first size is determined as the first preprocessed sample image. The material area where the SMT sample material is located is marked in the first preprocessed sample image to obtain the first sample image.

[0376] The original sample image is determined as the second sample image, or other images in the original sample image other than the first preprocessed sample image are determined as the second sample image.

[0377] In one possible implementation, determining the original sample image as the second sample image, or determining other images in the original sample image besides the first preprocessed sample image as the second sample image, includes:

[0378] The original sample image is determined as the second preprocessed sample image, or other images in the original sample image other than the first preprocessed sample image are determined as the second preprocessed sample image;

[0379] According to the category of the SMT sample material captured, each of the second preprocessed sample images is uniformly sampled to obtain the sampling result, which is used as the second sample image;

[0380] The step of marking the material region where the SMT sample material is located in the first preprocessed sample image to obtain the first sample image includes:

[0381] Mark the material region where the SMT sample material is located in the first preprocessed sample image to obtain the third preprocessed sample image;

[0382] According to the category of the SMT sample material captured, each of the third preprocessed sample images is uniformly sampled to obtain the sampling result, which is used as the first sample image.

[0383] In one possible implementation, the image cropping module includes:

[0384] The first image cropping submodule is used to perform grayscale binarization on the original image based on the principle of maximizing the variance of grayscale values ​​between the foreground region and the background region after binarization, to obtain a preprocessed image; wherein, the original image is an X-ray image;

[0385] The second image cropping submodule is used to determine the grayscale value boundary line in the preprocessed image;

[0386] The third image cropping submodule is used to crop the sub-image within the boundary line from the preprocessed image as the material area image.

[0387] In one possible implementation, the second quantity statistics submodule includes:

[0388] The fifth quantity statistics unit is used to calculate the quantity of SMT materials based on the detection results of the unenlarged material area image of the second material detection model if the area represented by the statistical value is greater than the lower limit area threshold.

[0389] The sixth quantity statistics unit is used to calculate the quantity of SMT materials based on the detection results of the magnified material area image of the second material detection model if the area represented by the statistical value is not greater than the lower limit area threshold.

[0390] In one possible implementation, the first size determination submodule includes:

[0391] The second size determination unit is used to detect the candidate regions where all potential SMT materials are located in the material region image, and the confidence level of each candidate region; wherein, the confidence level is used to represent the probability that an SMT material exists in the candidate region;

[0392] The third size determination unit is used to determine the region in the candidate region where the confidence level is not lower than a preset confidence threshold, as the material region where each SMT material in the material region image is located.

[0393] Corresponding to the aforementioned SMT material quantity counting method, this application embodiment also provides an SMT material quantity counting device. Referring to Figure 14, Figure 14 is a second structural schematic diagram of the SMT material quantity counting device provided in this application embodiment, including:

[0394] The acquisition module 1401 is used to acquire the original image obtained by photographing SMT materials;

[0395] The determining module 1402 is used to determine the size of the SMT material based on the original image;

[0396] The statistics module 1403 is used to count the number of SMT materials based on the detection results of the original image by the target material detection model; wherein the target material detection model is a material detection model corresponding to the size of the SMT materials.

[0397] By applying the above embodiments, an original image of the SMT material is obtained by taking a picture. Based on the original image, the material size of the SMT material is determined. Based on the material size, a matching material detection model is selected. The matching material detection model can accurately output the detection result reflecting the quantity of SMT materials of that size. Then, based on the detection result, the quantity of SMT materials is counted, which improves the pertinence of SMT material quantity statistics and is applicable to SMT material statistics scenarios of various sizes, thereby improving the accuracy of SMT material quantity statistics.

[0398] Corresponding to the aforementioned SMT material quantity counting method, this application embodiment also provides a material counting machine. Referring to Figure 15, Figure 15 is a structural schematic diagram of the material counting machine provided in this application embodiment. The material counting machine 1500 includes an image acquisition device 1501, a processor 1502, and a display 1503.

[0399] Image acquisition device 1501 is used to obtain raw images by photographing SMT materials.

[0400] The processor 1502 is configured to acquire the original image obtained by the image acquisition device by capturing SMT materials; crop the image of all material placement areas from the original image as a material area image; determine the size of the SMT materials based on the material area image; count the number of SMT materials based on the detection result of the material area image by a target material detection model, and send the number of SMT materials to the display; wherein the target material detection model is a material detection model corresponding to the size of the SMT materials.

[0401] The display is used to show the quantity of the SMT materials.

[0402] The image acquisition device 1501 establishes a communication connection with the processor 1502, and the processor 1502 establishes a communication connection with the display 1503.

[0403] The steps performed by the image acquisition device 1501 and the processor 1502 are the aforementioned steps S101-S105, the details of which have been explained above and will not be repeated here.

[0404] The display can be any device with display capabilities, such as digital displays, computers, and televisions, etc., without specific limitations in this document. When displaying the quantity of SMT materials, it can be displayed in the form of charts, text, or images, and will not be listed exhaustively here.

[0405] By applying the above embodiments, original images of SMT materials are obtained through photography. The image of the SMT material placement area is then precisely cropped from the original image, effectively removing background information irrelevant to SMT material quantity statistics. Using the image of the material area without background information, the size of the SMT materials is determined. Based on the material size, a matching material detection model is selected. This matching model accurately outputs detection results reflecting the quantity of SMT materials of that size. Based on the detection results, the quantity of SMT materials is then counted, improving the specificity of SMT material quantity statistics. This method is applicable to various SMT material size counting scenarios, thereby improving the accuracy of SMT material quantity statistics. After the SMT material quantity is counted, it is sent to a display for display, increasing the transparency of SMT material management and providing timely and accurate data support for SMT material management.

[0406] This application also provides an electronic device, as shown in FIG16, including:

[0407] Memory 1601 is used to store computer programs;

[0408] When processor 1602 executes a program stored in memory 1601, it performs the following steps:

[0409] Acquire the original image obtained by photographing SMT materials; crop the image of all SMT material placement areas from the original image as material area images; determine the size of the SMT materials based on the material area images; and count the number of SMT materials based on the detection results of the material area images using a target material detection model; wherein, the target material detection model is a material detection model corresponding to the size of the SMT materials.

[0410] or,

[0411] Acquire an original image of the SMT material by photographing it; determine the size of the SMT material based on the original image; and count the number of the SMT material based on the detection results of the original image using a target material detection model; wherein the target material detection model is a material detection model corresponding to the size of the SMT material.

[0412] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 1602, the communication interface, and the memory 1601 communicating with each other via the communication bus.

[0413] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or a bus of one size.

[0414] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0415] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0416] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0417] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described material quantity statistics methods.

[0418] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the material quantity statistics methods described above.

[0419] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0420] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0421] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0422] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for counting the quantity of SMT materials, characterized in that, The method includes: Acquire raw images of SMT materials by photographing them; Images of all SMT material placement areas are cropped from the original image and used as material area images; The dimensions of the SMT material are determined based on the material area image. Based on the detection results of the material region image using the target material detection model, the quantity of the SMT material is statistically obtained; wherein, the target material detection model is a material detection model corresponding to the size of the SMT material.

2. The method according to claim 1, characterized in that, Determining the size of the SMT material based on the material area image includes: Based on the visual features of the material region image, the material regions where each SMT material is located in the material region image are detected; wherein, the visual features are used to represent the contours in the material region image and the positions of the contours; The size of the SMT material is determined based on the area of ​​each material region.

3. The method according to claim 2, characterized in that, Determining the size of the SMT material based on the area of ​​each material region includes: The area of ​​each of the material regions is statistically analyzed; wherein the statistical value is used to represent the area of ​​a single SMT material; If the area represented by the statistical value is greater than the upper limit area threshold, then the size of the SMT material is determined to be the first size; If the area represented by the statistical value is not greater than the upper limit area threshold, then the size of the SMT material is determined to be the second size; The number of SMT materials is calculated by statistically analyzing the detection results of the material region image based on the target material detection model, including: If the size of the SMT material is the first size, the number of SMT materials is obtained by counting the detection results of the material area image based on the first material detection model, wherein the first material detection model is a material detection model for SMT material detection based on visual features. If the size of the SMT material is the second size, the number of SMT materials is obtained by counting the detection results of the material area image based on the second material detection model, wherein the second material detection model is a material detection model for SMT material detection based on semantic features.

4. The method according to claim 3, characterized in that, The first material detection model is a target rotational regression head model, and the second material detection model is a target semantic category regression head model; The step of detecting the material region where each SMT material is located in the material region image based on the visual features of the material region image includes: The material region image is input into the target rotary regression head model so that the target rotary regression head model extracts the visual features of the material region image and outputs the material region where each SMT material in the material region image is located based on the visual features. The quantity of SMT materials is obtained by statistically analyzing the detection results of the material region image based on the first material detection model, including: The number of material regions output by the target rotary regression head model is determined as the number of SMT materials; The method further includes: If the size of the SMT material is the second size, the material region image is input to the target semantic category regression head model so that the target semantic category regression head model extracts the semantic features of the material region image and outputs the material center point of each SMT material in the material region image based on the semantic features; The quantity of SMT materials is obtained by statistically analyzing the detection results of the material region image based on the second material detection model, including: The number of material center points output by the target semantic category regression head model is determined as the number of SMT materials; The target rotating rectangle regression head model is trained using a first sample image, where the size of the SMT material in the first sample image is greater than the upper limit area threshold, and the first sample image is marked with the material region where each of the SMT materials in the first sample image is located; the target semantic category regression head model is trained using a second sample image, where the center point of each of the SMT materials in the second sample image is marked.

5. The method according to claim 4, characterized in that, The target rotation rectangle regression head model is pre-trained using the following methods: The first sample image is input into the original rotating rectangle regression head model, so that the original rotating regression head model extracts the visual features of the first sample image, and outputs all predicted regions containing objects in the first sample image, as well as the predicted category and confidence level of each predicted region based on the visual features; wherein, the predicted category is used to represent the category of the object present in the predicted region, and the confidence level is used to represent the accuracy of the category; A first loss is calculated based on the first difference between each predicted region and the material region where each SMT material is located as marked in the first sample image; wherein, the first loss is positively correlated with the first difference; A second loss is calculated based on the second difference between the predicted category and the target category of the predicted region; wherein, the target category is the category of objects existing in the same region as the predicted region in the first sample image. If the predicted category is the same as the target category, the second difference is negatively correlated with the confidence of the predicted region; if the predicted category is different from the target category, the second difference is positively correlated with the confidence of the predicted region, and the second loss is positively correlated with the second difference. Adjust the parameters of the original rotating rectangular regression head model in the direction of gradient descent of the first loss and the second loss to obtain the target rotating rectangular regression head model; The target semantic category regression head model is pre-trained in the following ways: The second sample image is input into the original semantic category regression head model so that the target semantic category regression head model extracts the semantic features of the second sample image and outputs the predicted material center point of each SMT material in the material region image based on the semantic features. A third loss is calculated based on the third difference between the predicted material center point and the material center point of each SMT material marked in the second sample image; wherein the third loss is positively correlated with the third difference. Adjust the parameters of the original semantic category regression head model in the direction of the third loss gradient descent to obtain the target semantic category regression head model.

6. The method according to claim 4, characterized in that, The method further includes: Among multiple first preset models, the model that satisfies the first preset condition is selected as the target rotation regression head model, and among multiple second preset models, the model that satisfies the second preset condition is selected as the target semantic category regression head model; wherein, the multiple first preset models are trained based on the same first sample image, and the structures of different first preset models are different, and the multiple second preset models are trained based on the same second sample image, and the structures of different second preset models are different.

7. The method according to claim 4, characterized in that, The step of outputting the center point of each SMT material in the material region image based on the semantic features includes: Based on the semantic features of the material region image, a material center point mask image is output; wherein, the material center point mask is used to represent the material center point of each SMT material in the material region image, the size of the material center point mask image is the same as that of the material region image, and each element is used to represent the confidence level of pixels in the material region image that are at the same position as the material center point; Determining the number of material center points output by the target semantic category regression head model as the number of SMT materials includes: The material center point mask image is pooled to obtain a pooled material center point mask image; In the pooled material center point mask image, determine the maximum value that is greater than the preset screening threshold, and determine the number of the obtained maximum values ​​as the number of SMT materials.

8. The method according to claim 4, characterized in that, The first sample image and the second sample image were pre-annotated in the following ways: Acquire sample material area images obtained by photographing SMT sample materials; wherein each sample material area image is obtained by photographing SMT sample materials of different sizes, and each sample material area image includes multiple SMT sample materials, and the different sizes of SMT sample materials include a first size material and a second size material, the first size material is an SMT sample material with a size greater than the upper limit area threshold, and the second size material is an SMT sample material with a size not greater than the upper limit area threshold; The center point of each SMT sample material in the sample material area image is marked to obtain the original sample image; In the original sample image, the original sample image obtained by taking a picture of the material of the first size is determined as the first preprocessed sample image. The material area where the SMT sample material is located is marked in the first preprocessed sample image to obtain the first sample image. The original sample image is determined as the second sample image, or other images in the original sample image other than the first preprocessed sample image are determined as the second sample image.

9. The method according to claim 8, characterized in that, The step of determining the original sample image as the second sample image, or determining other images in the original sample image other than the first preprocessed sample image as the second sample image, includes: The original sample image is determined as the second preprocessed sample image, or other images in the original sample image other than the first preprocessed sample image are determined as the second preprocessed sample image; According to the category of the SMT sample material captured, each of the second preprocessed sample images is uniformly sampled to obtain the sampling result, which is used as the second sample image; The step of marking the material region where the SMT sample material is located in the first preprocessed sample image to obtain the first sample image includes: Mark the material region where the SMT sample material is located in the first preprocessed sample image to obtain the third preprocessed sample image; According to the category of the SMT sample material captured, each of the third preprocessed sample images is uniformly sampled to obtain the sampling result, which is used as the first sample image.

10. The method according to claim 1, characterized in that, The image of all SMT material placement areas cropped from the original image, as the material area image, includes: Based on the principle of maximizing the variance of gray values ​​between the foreground and background regions after binarization, the original image is subjected to gray-level binarization to obtain a preprocessed image; wherein, the original image is an X-ray image; Determine the boundary lines of grayscale values ​​in the preprocessed image; The sub-image within the boundary line is cropped from the preprocessed image to serve as the material area image.

11. The method according to claim 3, characterized in that, The number of SMT materials is calculated based on the detection results of the material region image using the second material detection model, including: If the area represented by the statistical value is greater than the lower limit area threshold, the number of SMT materials is statistically obtained based on the detection results of the unmagnified material area image using the second material detection model. If the area represented by the statistical value is not greater than the lower limit area threshold, the number of SMT materials is statistically obtained based on the detection results of the magnified material area image by the second material detection model.

12. The method according to claim 2, characterized in that, The detection obtains the material regions where each SMT material is located in the material region image, including: The candidate regions of all potential SMT materials in the material region image are detected, along with the confidence level of each candidate region; wherein, the confidence level is used to represent the probability that an SMT material exists in the candidate region; In the candidate region, the region with a confidence level not lower than a preset confidence threshold is determined as the material region where each SMT material in the material region image is located.

13. A method for counting the quantity of SMT materials, characterized in that, The method includes: Acquire raw images of SMT materials by photographing them; The dimensions of the SMT material are determined based on the original image; Based on the detection results of the original image using the target material detection model, the quantity of the SMT material is calculated; wherein, the target material detection model is a material detection model corresponding to the size of the SMT material.

14. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-12 or 13.

15. A material counting machine, characterized in that, The counting machine includes an image acquisition device, a processor, and a display; The image acquisition device is used to obtain raw images of SMT materials by photographing them; The processor is configured to acquire the original image obtained by the image acquisition device by capturing the SMT material; crop the image of all material placement areas from the original image as the material area image; and determine the size of the SMT material based on the material area image. Based on the detection results of the material area image using the target material detection model, the quantity of the SMT material is calculated and sent to the display; wherein, the target material detection model is a material detection model corresponding to the size of the SMT material; The display is used to show the quantity of the SMT materials.

Images