PCB via defect detection method and apparatus based on multi-model fusion, and device and medium

Abstract

A PCB via defect detection method and apparatus based on multi-model fusion, and a device and a medium. The method comprises: acquiring a set of sample images to be processed, and processing said set of sample images to obtain a set of target sample images (S110); performing pruning training on a plurality of deep learning models to be trained on the basis of the set of target sample images, so as to obtain target deep learning models respectively corresponding to said plurality of deep learning models (S120), wherein the target deep learning models comprise a target key point model, a target detection model and a target segmentation model, and model structures of said plurality of deep learning models are different from each other; and establishing a target fusion model on the basis of the target deep learning models, and inputting an image to be detected into the target fusion model, so as to determine a via defect detection result corresponding to said image (S130).

Description

Multi-model fusion method, device, equipment and medium for PCB through-hole defect detection

[0001] This application claims priority to Chinese Patent Application No. 202510293846.7, filed with the Chinese Patent Office on March 13, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and for example to a method, apparatus, device and medium for detecting PCB through-hole defects using multi-model fusion. Background Technology

[0003] With the rapid development of the electronics manufacturing industry, printed circuit boards (PCBs), as the foundation and core component of electronic products, are crucial to the performance and reliability of these products. Through-holes (Vias) on PCBs are key structures used to connect different layers and are widely used in the manufacture of multilayer circuit boards. During PCB production, through-holes may develop defects due to various reasons, such as uneven hole diameter, hole wall cracks, and copper foil peeling. These defects directly affect the electrical performance and reliability of the circuit and may even lead to product failure.

[0004] However, traditional through-hole defect detection methods rely heavily on manual visual inspection and traditional optical inspection equipment, which have shortcomings such as low efficiency and great susceptibility to human factors. Summary of the Invention

[0005] This application provides a method, apparatus, device, and medium for detecting PCB via defects through multi-model fusion. By fusing multiple models and combining them with pruning optimization, the advantages of multiple models in defect detection can be fully utilized. Furthermore, while ensuring high accuracy, the inference speed can be significantly improved to meet real-time requirements, thus solving the problem of decreased model inference speed in related technologies.

[0006] This application provides a multi-model fusion method for detecting PCB via defects, which includes:

[0007] Obtain the set of sample images to be processed, and process the set of sample images to be processed to obtain the target set of sample images;

[0008] Based on the target sample image set, multiple deep learning models to be trained are pruned and trained to obtain target deep learning models corresponding to the multiple deep learning models to be trained, wherein the target deep learning models include target key point models, target detection models and target segmentation models; the model structures of the multiple deep learning models to be trained are different from each other;

[0009] A target fusion model is established based on the target deep learning model, and the image to be detected is input into the target fusion model to determine the detection result of the through hole defect corresponding to the image to be detected.

[0010] This application provides a multi-model fusion PCB via defect detection device, which includes:

[0011] The sample image determination module is configured to acquire a set of sample images to be processed and process the set of sample images to be processed to obtain a target sample image set.

[0012] The model training module is configured to perform pruning training on multiple deep learning models to be trained based on the target sample image set, to obtain target deep learning models corresponding to the multiple deep learning models to be trained, wherein the target deep learning models include target key point models, target detection models and target segmentation models; the model structures of the multiple deep learning models to be trained are different from each other;

[0013] The defect detection module is configured to establish a target fusion model based on the target deep learning model, input the image to be detected into the target fusion model, and determine the through-hole defect detection result corresponding to the image to be detected.

[0014] This application provides an electronic device, the electronic device comprising:

[0015] At least one processor; and,

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the multi-model fusion PCB via defect detection method according to any embodiment of this application.

[0018] This application provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the multi-model fusion PCB via defect detection method described in any embodiment of this application. Attached Figure Description

[0019] Figure 1 is a flowchart of a multi-model fusion PCB via defect detection method according to an embodiment of this application.

[0020] Figure 2 is a flowchart of the key point detection model provided according to an embodiment of this application.

[0021] Figure 3 is a schematic diagram of the channel scaling factor provided according to an embodiment of this application.

[0022] Figure 4 is a flowchart of a multi-model fusion PCB via defect detection method according to an embodiment of this application.

[0023] Figure 5 is a schematic diagram of the target fusion model processing procedure provided according to an embodiment of this application.

[0024] Figure 6 is a schematic diagram of the information fusion module processing procedure provided according to an embodiment of this application.

[0025] Figure 7 is a schematic diagram of the processing procedure of the prompt information interaction module provided in the embodiment of this application.

[0026] Figure 8 is a schematic diagram of a multi-model fusion PCB via defect detection device provided according to an embodiment of this application.

[0027] Figure 9 is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort should fall within the scope of protection of this application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Example 1:

[0031] Figure 1 is a flowchart of a multi-model fusion PCB via defect detection method provided in an embodiment of this application. This embodiment is applicable to the automatic detection of via defects on a PCB based on acquired PCB images during PCB production. This method can be executed by a multi-model fusion PCB via defect detection device, which can be implemented in hardware and / or software and can be configured in an electronic device. As shown in Figure 1, the method includes:

[0032] S110. Obtain the set of sample images to be processed, and process the set of sample images to be processed to obtain the target sample image set.

[0033] The set of sample images to be processed can be the unprocessed raw images obtained after scanning the PCB. The target set of sample images can be understood as the collection of sample images obtained after processing the raw images.

[0034] The PCB is scanned by X-ray equipment to obtain an unprocessed PCB image. The image set to be processed is then processed to obtain a target image set. For example, the images in the image set to be processed may be cleaned and enhanced to obtain the target image set.

[0035] Based on the above technical solution, the step of processing the sample image set to be processed to obtain the target sample image set includes: annotating the sample image set to be processed using image annotation software to obtain a sample image set containing annotation labels; dividing the sample image set containing annotation labels into a sample image set and a test image set according to a preset ratio; and determining the target sample image set based on the sample image set and the test image set.

[0036] Image annotation software can be used to annotate images to be annotated, such as the LabelMe software. Annotation labels can be pre-set image labels, including keypoint labels, object detection labels, and grayscale labels. The preset ratio can be understood as a pre-set proportion used to divide the sample set. The sample image set can be the set of images used to train the model; correspondingly, the test image set can be the set of images used to test the model.

[0037] Choose appropriate image annotation software according to the requirements, such as LabelImg, LabelMe, VOTT and other annotation software. The labels are divided into three categories: key points, object detection and mask. Then, use the rotation annotation software to annotate the images in the sample image set to be processed, and after annotation, divide the sample image set with annotation labels into a sample image set and a test image set according to a preset ratio, for example, it can be divided into a sample image set and a test set in a 7:3 ratio.

[0038] Based on the above technical solution, determining the target sample image set based on the sample image set and the test image set includes: adjusting the image sizes in the sample image set and the test image set based on preset image sizes, and performing random horizontal flipping processing; converting the images that have undergone random horizontal flipping processing into tensor format, and performing normalization processing to obtain the target sample image set.

[0039] Preset image size can be understood as the pre-defined size of the input image. Tensor format can be the format used to describe the image using tensors.

[0040] The data is preprocessed by resizing the input image to 1024×1024 and then randomly flipping it horizontally with a probability of 0.5 to convert it to Tensor format. After conversion, standardization and normalization are performed. For example, using the Python environment, the `torchvision.Resize()` method is used to resize the image. Next, the `torchvision.RandomCrop()` method is applied to randomly crop the image, followed by the `torchvision.RandomHorizontalFlip()` method, which performs a horizontal random flip with a 50% probability. Then, the processed image is converted to Tensor format using the `transforms.ToTensor()` method, with initial normalization performed during this process. Finally, the `transforms.Normalize()` function is used to complete the image data standardization process, ensuring that the data meets predefined normalization requirements. The details of the normalization algorithm include: Where μ represents the image mean, x represents the image tensor, σ represents the standard deviation, and x max x represents the maximum value of the image tensor. min This represents the minimum value of the image tensor. This represents the standardized image tensor. This represents the normalized image tensor.

[0041] S120. Based on the target sample image set, perform pruning training on multiple deep learning models to be trained to obtain a target deep learning model corresponding to the multiple deep learning models to be trained.

[0042] The deep learning model to be trained can be an initialized deep learning model, which may include a keypoint model, a detection model, and a segmentation model to be trained. The target deep learning model includes a target keypoint model, a target detection model, and a target segmentation model; the model structures of the multiple deep learning models to be trained are different from each other.

[0043] Keypoint, detection, and segmentation models were established using LiteHRnet, YOLOv8, and Unet models, respectively. These models were further initialized, and a structured pruning training framework was designed to prune and train them. It should be noted that, as shown in Figure 2, since the size of the via is very small compared to the overall image, an additional decoder head was added to the output layer of the LiteHRnet model. This decoder head outputs an additional resolution map, using deconvolution to ultimately output a heatmap of the original size. Since the keypoint detection model is only used to locate target positions, it can be understood as a multi-target, single-type keypoint detection model; therefore, a bottom-up framework detection logic was adopted. Referring to the CenterNet model, the output of LiteHRnet is a heatmap feature map, and the training labels are the heatmap feature map and a mask. The mask is generated based on the coordinates corresponding to the maximum value of each Gaussian kernel in the heatmap label. The FocalLoss loss function is used to calculate the heatmap feature loss. Assuming the input image is... The output Heatmap is FocalLoss can be represented as:

[0044] ;in, and These are hyperparameters, set to 2 and 4 respectively.

[0045] YOLOv8 uses the most lightweight version, YOLOv8-s. Its labels consist of three parts: bounding box parameters, class labels, and confidence scores. Based on grid cells, each grid predicts multiple bounding boxes. The label for each bounding box includes: the bounding box center point, the bounding box width and height, as well as the confidence score and the probability distribution of the class. Its format is... YOLOv8's loss function consists of several parts, including: Localization Loss: This measures the difference between the predicted bounding box and the ground truth bounding box. It is typically calculated using the Intersection over Union (IoU) and Mean Square Error (MSE). ;

[0046] Confidence loss measures the difference between predicted and actual confidence levels. It is commonly calculated using binary cross-entropy (BCE). ;

[0047] Classification loss measures the difference between the predicted and true classes, and is typically calculated using cross-entropy loss. Therefore, the final combination of loss functions can be determined as follows:

[0048] ;

[0049] The segmentation model UNet uses ResNet50 as its encoder, and labels are applied to the regions at the via locations. It employs a binary classification method to extract the regions at the via locations, and its loss function is:

[0050] .

[0051] Based on the above technical solution, the step of pruning and training multiple deep learning models to be trained based on the target sample image set includes: determining the scaling factors of multiple channels corresponding to the batch normalization layer in the multiple deep learning models to be trained; performing regularization processing on the scaling factors of the multiple channels; and performing channel pruning on the batch normalization layer according to the regularization processing result.

[0052] Batch normalization layers can be used to normalize the input of each layer of a neural network, which can solve the problem of input data distribution drift during training.

[0053] In deep learning models, all batch normalization layers are identified, and for each layer, its corresponding scaling factor is extracted. It's important to note that the scaling factor determines the importance of each channel in the feature map. Therefore, during model training, a regularization term is added to the loss function. Based on the regularized scaling factor, a pruning threshold is determined; channels below the threshold are pruned. For example, in deep learning, batch normalization (BN) layers are commonly used techniques to accelerate training and improve model generalization. The core operation of batch normalization layers is to standardize the input of each layer, ensuring its mean is 0 and its variance is 1. Batch normalization layers typically include two important learnable parameters: the scaling factor. and offset factor .

[0054] To implement pruning in the batch normalization layer, these parameters are regularized to reduce unimportant channels and weights. Batch normalization layer pruning primarily selects channels to be pruned by evaluating the importance of each channel. Through L1 regularization, the model can automatically prune unimportant channels. The parameters are compressed to zero, thus achieving channel pruning. For an input tensor... The operation process of the batch standardization layer can be represented by the following formula: ;in, It is input Mean across batches It is the variance of the input. It is a very small constant used to prevent the denominator from being zero. The subsequent batch normalization layer will then scale and offset the normalized input: ;in, It is the scaling factor for each channel. It is the offset factor for each channel.

[0055] The goal of batch normalization layer pruning is to adjust the scaling factor through regularization. Channels that contribute little to the model are removed. This is done by evaluating the importance of each channel using L1 regularization and then... The value determines which channels can be pruned. Less important channels will be pruned, thus reducing the computational cost and number of parameters in the network. This is illustrated in Figure 3. For each channel in the batch normalization layer... Its scaling factor Its importance can be assessed by adding L1 regularization. The goal of L1 regularization is to make... The weights of channels with smaller values ​​tend to zero. The L1 regularization term can be expressed as: The L1 regularization term is applied to the loss function through the gradient descent optimization process, causing some less important regularization terms to be applied. It is reduced or reduced to zero. The total loss function becomes: .

[0056] Based on the above technical solution, after performing channel pruning on the batch normalization layer according to the regularization processing result, the method further includes: determining the optimal reconstruction matrix and optimal model parameters based on minimizing the reconstruction error function; and deploying the feature reconstruction layer after the batch normalization layer according to the optimal reconstruction matrix and the optimal model parameters.

[0057] Minimizing the reconstruction error function can be used to determine the loss function for the feature reconstruction layer. The reconstruction matrix is ​​used to transform the pruned feature maps back to a form close to the original feature maps, thereby reducing the information loss caused by pruning. The feature reconstruction layer is a structure of three concatenated convolutional layers.

[0058] A reconstruction error function is constructed to measure the difference between the output of the pruned model and the original model output. The reconstruction matrix is ​​then solved using optimization algorithms (such as gradient descent and least squares) to minimize the reconstruction error function. Simultaneously, the model parameters need to be updated to ensure good performance after pruning and reconstruction. A feature reconstruction layer is then designed based on the optimal reconstruction matrix and applied after the batch normalization layer following pruning. It should be noted that during channel pruning, some information is removed, affecting the model's accuracy. Therefore, the feature map reconstruction step aims to recover lost feature information to maintain model performance. For the information lost after pruning, a reconstruction network is designed to fit the feature maps before and after pruning, recovering the lost information. Assume the original feature map output of the network before pruning is... After pruning, the resulting feature map , Some channels are pruned. 'b' represents the batch dimension, indicating the number of samples input at once. Inputting multiple samples at once during training improves computational efficiency. 'c' represents the channel dimension, indicating the number of channels in the input data or the number of feature maps. For Convolutional Neural Networks (CNNs), it typically represents the number of output feature maps of the convolutional layers. 'h' represents the height dimension, indicating the height of the image or feature map. 'w' represents the width dimension, indicating the width of the image or feature map.

[0059] The goal is to analyze the feature maps after pruning. Reconstruct the feature map to obtain a result that is as close as possible to the original feature map. The feature map, the feature map after pruning The input to the reconstruction network Ref is typically intended to produce output features similar to the original model after the pruned features are reconstructed.

[0060] ;

[0061] in, This represents the error term, indicating the portion that cannot be completely recovered through linear combination. For the reconstruction module, the structure of the pruned feature map is transformed to be the same as that of the original feature map. It should be noted that the feature reconstruction layer is a three-layer convolutional structure, using a concatenated structure of 1×1 convolutions, 3×3 convolutions, and 1×1 convolutions to transform the pruned features. The spatial structure, by minimizing the error term The sum of squares, i.e., minimizing the reconstruction error: ;in This represents the Frobenius norm, which is the square root of the sum of the squares of the elements of a matrix. Let represent the network parameters before pruning. By minimizing the above loss function, the optimal reconstruction matrix can be obtained. and model parameters Therefore, after training the reconstructed network, it is usually necessary to fine-tune the pruned network to ensure that the fusion of the reconstructed network and the main network does not affect the model performance.

[0062] S130. Establish a target fusion model based on the target deep learning model, and input the image to be detected into the target fusion model to determine the through hole defect detection result corresponding to the image to be detected.

[0063] The target fusion model can be understood as a deep learning model obtained by combining target deep learning models. The image to be detected can be a PCB image that needs to be inspected for through-hole defects.

[0064] After the target deep learning model is trained, a target fusion model can be constructed based on multiple target deep learning models. The image to be detected is then input into the target fusion model to obtain the through-hole defect detection result output by the target fusion model.

[0065] The technical solution of this application embodiment involves acquiring a set of sample images to be processed, processing the set to obtain a target sample image set, and then pruning and training multiple deep learning models to be trained based on the target sample image set to obtain target deep learning models corresponding to the multiple deep learning models to be trained. Finally, a target fusion model is established based on the target deep learning models, and the image to be detected is input into the target fusion model to determine the detection result of the via defect corresponding to the image to be detected. Based on the above technical solution, by fusing multiple models and combining pruning optimization, the advantages of multiple models in defect detection can be fully utilized, and the inference speed can be significantly improved while ensuring high accuracy, thus meeting real-time requirements.

[0066] Example 2:

[0067] Figure 4 is a flowchart of a multi-model fusion method for detecting PCB via defects provided in an embodiment of this application. This embodiment illustrates the technical solution of establishing a target fusion model based on the target deep learning model, based on the above technical solution. As shown in Figure 4, the method includes:

[0068] S210. Remove the output layer from the target key point model and the target segmentation model, and input the hidden layer results from the target key point model and the target segmentation model into the prompt information fusion module to obtain the fused prompt features output by the prompt information fusion module.

[0069] The hidden layer result can be the output of the last hidden layer in the keypoint model and the segmentation model. The information fusion module can be understood as a processing model configured to perform feature fusion on the output features of the target keypoint model and the target segmentation model.

[0070] The overall fusion structure of the fusion model is shown in Figure 5. The keypoint detection model and the segmentation model are loaded with pruned pre-trained parameters, and the output layer is removed. The features of the last hidden layer are used as output and input into the Prompt fusion module to generate multimodal cue priors. During training, the keypoint detection model and the segmentation model freeze their parameters and do not participate in network parameter updates. They act as multimodal feature generators. The Prompt fusion module fuses the keypoint information and mask information as multimodal cue features. These features are then fused with the multi-level output features of the YOLOv8 Backbone through the Prompt interaction module. The design goal of the Prompt interaction module is to establish effective connections between different modal features, so that the keypoint information and mask information can provide more accurate contextual information for the YOLOv8 detection model, thereby enhancing the model's ability to identify through-hole defects.

[0071] It should be noted that the structure of the Prompt fusion module is shown in Figure 6. Since the output features of the segmentation model and the keypoint model are misaligned in both spatial and channel dimensions, a 1×1 convolution is first used to transform the channels of both features to ensure they have the same number of channels. Next, a 3×3 convolutional block enhances the features, aiming to extract richer feature information and improve feature representation. The enhanced features are then adjusted for spatial resolution through spatial transformation, aligning the spatial scales of the two features. However, spatial transformation may introduce some interference and unnecessary noise; therefore, another 3×3 convolutional block is used afterward to suppress the interference caused by spatial transformation, ensuring feature quality and effectiveness. To avoid suboptimal results that may occur with static fusion, a channel attention mechanism is introduced to dynamically adjust the importance of the fused features. Before feature concatenation, the input features first undergo global average pooling to obtain global information, then undergo channel dimensionality reduction through a 1×1 convolution, and subsequently generate dynamic attention weights using the Softmax activation function. These weights reflect the importance of each channel and are multiplied by the original input features, thereby dynamically selecting and enhancing key feature information. The two dynamically selected features are then concatenated and fused. 1×1 and 3×3 convolutional blocks are applied to the concatenated features to further aggregate and enhance them. The resulting fused cue features contain complementary information from both keypoint detection and segmentation tasks, providing more accurate and richer prior information for subsequent detection models.

[0072] S220. Obtain the detection output features of the target detection model, and process the fused prompt features and the detection output features based on the prompt information interaction module to obtain the target detection features.

[0073] The detected output features can be multi-level output features corresponding to the image, processed by the object detection model. The prompt information interaction module can be understood as a neural network layer that integrates these features.

[0074] The structure of the Prompt interaction module is shown in Figure 7. This module aims to effectively fuse the fused prompt features with the multi-level output features of the YOLOv8 Backbone. The Prompt interaction module consists of two main parts: an attention module and a feed-forward network (FFN). First, the first-level features in the YOLOv8 Backbone are concatenated with the fused prompt features. The concatenated features are then input into the attention module, which follows the basic attention mechanism of the Transformer structure. In the attention module, the concatenated features are first processed by a 3×3 convolutional block to enhance their expressive power. Then, channel separation and shape transformation operations are performed to generate the query vector (Q) and key vector (K). Next, the concatenated features are aggregated and shape transformed by a 1×1 convolution to obtain the value vector (V). After the attention operation, the Q, K, and V vectors are fused with the relevant prompt information and the original features. At this point, the fused features are residually connected with the input features to form the final feature representation containing the embedded prompt information. Next, the features embedded with the prompt information are fed into a feedforward network (FFN) consisting of gated transposed convolutions. This module decouples the input features using a block concatenation of two 1×1 and 3×3 convolutional blocks, resulting in two branch features. One branch is passed through a sigmoid function to obtain gated attention information, which is then multiplied by the other branch feature and aggregated through a 1×1 convolution. Finally, it is concatenated with the input feature residuals to obtain the final output feature. In this way, the Prompt interaction module can effectively fuse prompt information from different models, providing more semantically rich and spatially expressive features for subsequent detection tasks.

[0075] S230, The target detection model outputs a through-hole defect detection label based on the target detection features.

[0076] The target detection model ultimately outputs via defect detection labels corresponding to the images based on target detection features. It should be noted that comparative experiments were conducted to verify the effectiveness of the pruning compression method, and a self-built test set was used for evaluation. First, a baseline model (YOLOV8-S) was trained, and several key performance indicators on the test set were recorded, including detection accuracy, number of parameters, and inference time. The baseline model is a complete network without pruning, and after sufficient training, its detection performance was ensured to reach the optimal level. According to the experimental results, the baseline model achieved an accuracy of 83.32%, a parameter count of 7.2 M, and an inference time of 302.23 ms / image. After obtaining the optimal performance of the baseline model, the pruning compression method was applied to optimize the model. The pruning operation includes two parts: channel pruning and feature reconstruction. After pruning, the computational complexity and number of parameters were significantly reduced, with the number of parameters decreasing by 73.2% and the inference time decreasing from 302.23 ms / image to 63.3 ms / image. Although pruning significantly reduces computation and the number of parameters, the model's performance degrades to some extent due to the loss of some information during the pruning process. In our experiments, the accuracy of the pruned model decreased by approximately 1.05%.

[0077] In the multi-model fusion process provided in this application embodiment, the keypoint detection model and the segmentation model are first pruned and trained in the same way. The pruned keypoint detection model and the segmentation model are used as feature generators to extract keypoint information and segmentation mask information from each model, respectively. After the Prompt fusion module fuses these two features, a fused feature containing multimodal information is generated. These fused features are input into the target detection model (such as YOLOv8) for processing. In multi-model fusion, the parameters of the pruned pre-trained keypoint detection model and the segmentation model are frozen to prevent them from being updated during training, and they only provide information as feature generators. This ensures that the feature information they provide is not affected by the training process, ensuring the stability of the multimodal features. When fusing the multi-level output features of the YOLOv8 Backbone with the fused cue features, the Prompt interaction module combines the features of the two to generate the final feature representation used for detection. Through multi-model fusion, the detection accuracy of the pruned model is significantly improved. Compared with a single pruned model, the detection accuracy after multi-model fusion is improved by 13.87%. Furthermore, the fused model not only restored accuracy but also achieved a better balance between inference speed and memory usage. The inference time of the multi-model fusion model is not much different from the baseline model, and the inference latency is reduced by 20% compared to the single pruned model.

[0078] The technical solution of this application embodiment involves acquiring a set of sample images to be processed, processing the set to obtain a target sample image set, and then pruning and training multiple deep learning models to be trained based on the target sample image set to obtain target deep learning models corresponding to the multiple deep learning models to be trained. Finally, a target fusion model is established based on the target deep learning models, and the image to be detected is input into the target fusion model to determine the detection result of the via defect corresponding to the image to be detected. Based on the above technical solution, by fusing multiple models and combining pruning optimization, the advantages of multiple models in defect detection can be fully utilized, and the inference speed can be significantly improved while ensuring high accuracy, thus meeting real-time requirements.

[0079] Example 3:

[0080] Figure 8 is a schematic diagram of a multi-model fusion PCB via defect detection device provided in an embodiment of this application. As shown in Figure 8, the device includes: a sample image determination module 810, a model training module 820, and a defect detection module 830, wherein...

[0081] The sample image determination module 810 is configured to acquire a set of sample images to be processed and process the set of sample images to be processed to obtain a target sample image set.

[0082] The model training module 820 is configured to perform pruning training on multiple deep learning models to be trained based on the target sample image set, to obtain a target deep learning model corresponding to the multiple deep learning models to be trained, wherein the target deep learning model includes a target key point model, a target detection model, and a target segmentation model; the model structures of the multiple deep learning models to be trained are different from each other;

[0083] The defect detection module 830 is configured to establish a target fusion model based on the target deep learning model, input the image to be detected into the target fusion model, and determine the through-hole defect detection result corresponding to the image to be detected.

[0084] Based on the above technical solution, the model training module 820 is configured to determine the scaling factors of multiple channels in the multiple deep learning models to be trained, which are respectively corresponding to the batch normalization layer; perform regularization processing on the scaling factors of the multiple channels; and perform channel pruning on the batch normalization layer according to the regularization processing result.

[0085] Based on the above technical solution, the model training module 820 is configured to determine the optimal reconstruction matrix and optimal model parameters based on minimizing the reconstruction error function; and to deploy the feature reconstruction layer after the batch normalization layer according to the optimal reconstruction matrix and the optimal model parameters; wherein, the feature reconstruction layer is a three-layer concatenated convolution structure.

[0086] Based on the above technical solution, the defect detection module 830 is configured to remove the output layer from the target key point model and the target segmentation model, and input the hidden layer results from the target key point model and the target segmentation model to the prompt information fusion module to obtain the fused prompt features output by the prompt information fusion module; and establish the target fusion model based on the fused prompt features and the target detection model.

[0087] Based on the above technical solution, the defect detection module 830 is configured to acquire the detection output features of the target detection model, and process the fused prompt features and the detection output features based on the prompt information interaction module to obtain target detection features; the target detection model outputs a through-hole defect detection label according to the target detection features.

[0088] Based on the above technical solution, the sample image determination module 810 is configured to annotate the sample image set to be processed based on image annotation software to obtain a sample image set containing annotation labels; wherein, the annotation labels include key point labels, target detection labels and grayscale labels;

[0089] The sample image set containing labels is divided into a sample image set and a test image set according to a preset ratio, and the target sample image set is determined based on the sample image set and the test image set.

[0090] Based on the above technical solution, the sample image determination module 810 is configured to adjust the image size of the sample image set and the test image set based on a preset image size, and perform random horizontal flipping processing; convert the image that has undergone random horizontal flipping processing into tensor mode format, and perform normalization processing to obtain the target sample image set.

[0091] The technical solution of this application embodiment involves acquiring a set of sample images to be processed, processing the set to obtain a target sample image set, and then pruning and training multiple deep learning models to be trained based on the target sample image set to obtain target deep learning models corresponding to the multiple deep learning models to be trained. Finally, a target fusion model is established based on the target deep learning models, and the image to be detected is input into the target fusion model to determine the detection result of the via defect corresponding to the image to be detected. Based on the above technical solution, by fusing multiple models and combining pruning optimization, the advantages of multiple models in defect detection can be fully utilized, and the inference speed can be significantly improved while ensuring high accuracy, thus meeting real-time requirements.

[0092] The multi-model fusion PCB via defect detection device provided in this application embodiment can execute the multi-model fusion PCB via defect detection method provided in any embodiment of this application, and has the corresponding functional modules for executing the method.

[0093] Example 4:

[0094] Figure 9 illustrates a schematic diagram of an electronic device 10 that can be used to implement embodiments of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0095] As shown in Figure 9, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0096] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0097] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs several of the methods and processes described above, such as a multi-model fusion method for detecting PCB via defects.

[0098] In some embodiments, the multi-model fusion PCB via defect detection method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the multi-model fusion PCB via defect detection method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the multi-model fusion PCB via defect detection method by any other suitable means (e.g., by means of firmware).

[0099] The various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0100] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0101] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0102] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0103] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0104] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0105] It should be understood that the various processes shown above can be used to rearrange, add, or delete steps. For example, the multiple steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

Claims

1. A multi-model fusion method for detecting through-hole defects in printed circuit boards (PCBs), comprising: Obtain the set of sample images to be processed, and process the set of sample images to be processed to obtain the target set of sample images; Based on the target sample image set, multiple deep learning models to be trained are pruned and trained to obtain target deep learning models corresponding to the multiple deep learning models to be trained, wherein the target deep learning models include target key point models, target detection models and target segmentation models; the model structures of the multiple deep learning models to be trained are different from each other; a decoding head using deconvolution is added to the output layer of the target key point detection model; A target fusion model is established based on the target deep learning model, and the image to be detected is input into the target fusion model to determine the detection result of the through hole defect corresponding to the image to be detected; The step of establishing a target fusion model based on the target deep learning model includes: removing the output layers from the target keypoint model and the target segmentation model, and inputting the hidden layer results from the target keypoint model and the target segmentation model into the prompt information fusion module to obtain the fused prompt features output by the prompt information fusion module; obtaining the detection output features of the target detection model, and processing the fused prompt features and the detection output features based on the prompt information interaction module to obtain target detection features; the target detection model outputs a through-hole defect detection label based on the target detection features; the prompt information interaction module includes an attention module and a feedforward network; the attention module is configured to process the spliced ​​features to obtain features embedded with prompt information; the feedforward network is configured to process the features embedded with prompt information to obtain the target detection features; the spliced ​​features are obtained by splicing the detection output features and the fused prompt features.

2. The method according to claim 1, wherein, The step of pruning and training multiple deep learning models based on the target sample image set includes: Determine the scaling factors of multiple channels in the multiple deep learning models to be trained, each corresponding to a batch normalization layer; The scaling factors of the multiple channels are regularized, and the channels of the batch normalization layer are pruned based on the regularization results.

3. The method according to claim 2, after performing channel pruning on the batch normalization layer based on the regularization result, further includes: The optimal reconstruction matrix and optimal model parameters are determined by minimizing the reconstruction error function; The feature reconstruction layer is deployed after the batch normalization layer according to the optimal reconstruction matrix and the optimal model parameters; wherein the feature reconstruction layer is a three-layer concatenated convolution structure.

4. The method according to claim 1, wherein, The process of processing the set of sample images to be processed to obtain the target set of sample images includes: The image set to be processed is labeled using image annotation software to obtain a set of image sets with annotation labels; wherein, the annotation labels include key point labels, target detection labels and grayscale labels; The sample image set containing labels is divided into a sample image set and a test image set according to a preset ratio, and the target sample image set is determined based on the sample image set and the test image set.

5. The method according to claim 4, wherein, Determining the target sample image set based on the sample image set and the test image set includes: The image sizes in the sample image set and the test image set are adjusted based on preset image sizes, and then randomly horizontally flipped. The images that have undergone random horizontal flipping are converted into tensor format and then normalized to obtain the target sample image set.

6. A multi-model fusion printed circuit board (PCB) through-hole defect detection device, comprising: The sample image determination module is configured to acquire a set of sample images to be processed and process the set of sample images to be processed to obtain a target sample image set. The model training module is configured to perform pruning training on multiple deep learning models to be trained based on the target sample image set, to obtain target deep learning models corresponding to the multiple deep learning models to be trained, wherein the target deep learning models include a target key point model, a target detection model, and a target segmentation model; the model structures of the multiple deep learning models to be trained are different from each other; a decoding head using deconvolution is added to the output layer of the target key point detection model; The defect detection module is configured to establish a target fusion model based on the target deep learning model, input the image to be detected into the target fusion model, and determine the detection result of the through hole defect corresponding to the image to be detected; The defect detection module is configured to remove the output layers from the target keypoint model and the target segmentation model, and input the hidden layer results from the target keypoint model and the target segmentation model into the prompt information fusion module to obtain the fused prompt features output by the prompt information fusion module; obtain the detection output features of the target detection model, and process the fused prompt features and the detection output features based on the prompt information interaction module to obtain the target detection features; the target detection model outputs a via defect detection label based on the target detection features; the prompt information interaction module includes an attention module and a feedforward network; the attention module is configured to process the spliced ​​features to obtain features embedded with prompt information; the feedforward network is configured to process the features embedded with prompt information to obtain the target detection features; the spliced ​​features are obtained by splicing the detection output features and the fused prompt features.

7. An electronic device, the electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the multi-model fusion method for detecting through-hole defects in printed circuit boards (PCBs) according to any one of claims 1-5.

8. A computer-readable storage medium storing computer instructions for causing a processor to execute the multi-model fusion method for detecting through-hole defects in printed circuit boards (PCBs) according to any one of claims 1-5.

Images