Defect detection method, model training method, electronic apparatus and storage medium

Abstract

Provided in the present disclosure are a defect detection method, a model training method, an electronic apparatus and a storage medium. The defect detection method comprises: acquiring an image to be subjected to detection of a product to be subjected to detection, wherein the image to be subjected to detection comprises a plurality of periodically arranged sub-images; determining a period offset of the image to be subjected to detection; on the basis of the period offset, performing cyclic image shifting on the image to be subjected to detection in a first direction, so as to obtain a first shifted image, and performing cyclic image shifting on the image to be subjected to detection in a second direction opposite the first direction, so as to obtain a second shifted image; on the basis of the image to be subjected to detection, the first shifted image and the second shifted image, generating a difference-fused image; merging the image to be subjected to detection and the difference-fused image, so as to obtain a merged image; and using a neural network model to process the merged image, so as to obtain a defect detection result.

Description

Defect detection methods, model training methods, electronic devices and storage media Technical Field

[0001] This disclosure relates to the field of detection, and in particular to a defect detection method, a model training method, an electronic device, and a storage medium. Background Technology

[0002] To detect defects in products such as display panels, images of the product are acquired and input into a neural network model. The neural network model then processes the images to detect the defects present in the product. Summary of the Invention

[0003] In a first aspect of this disclosure, a defect detection method is provided, comprising: acquiring an image of a product to be inspected, wherein the image to be inspected includes a plurality of sub-images arranged periodically; determining a periodic offset of the image to be inspected; performing a cyclic image offset on the image to be inspected in a first direction according to the periodic offset to obtain a first offset image, and performing a cyclic image offset on the image to be inspected in a second direction opposite to the first direction to obtain a second offset image; generating a difference fusion image based on the image to be inspected, the first offset image, and the second offset image; merging the image to be inspected and the difference fusion image to obtain a merged image; and processing the merged image using a neural network model to obtain a defect detection result.

[0004] In some embodiments, generating a difference fusion image based on the image to be detected, the first offset image, and the second offset image includes: generating a first difference image based on the difference between the image to be detected and the first offset image; generating a second difference image based on the difference between the image to be detected and the second offset image; and generating the difference fusion image based on the first difference image and the second difference image.

[0005] In some embodiments, generating the difference fusion image based on the first difference image and the second difference image includes: performing minimum value fusion on the first difference image and the second difference image to obtain the difference fusion image.

[0006] In some embodiments, generating the first difference image includes: blurring the image to be detected to obtain a blurred image to be detected; blurring the first offset image to obtain a first blurred image; and generating the first difference image based on the difference between the blurred image to be detected and the first blurred image. Generating the second difference image includes: blurring the second offset image to obtain a second blurred image; and generating the second difference image based on the difference between the blurred image to be detected and the second blurred image.

[0007] In some embodiments, performing a cyclic image offset on the image to be detected in a first direction to obtain a first offset image includes: determining the coordinates of a first target pixel in the image to be detected corresponding to the m-th pixel based on the sum of the coordinates of the m-th pixel in the first offset image and the periodic offset, where 1 ≤ m ≤ M, and M is the total number of pixels in the first offset image; and setting the pixel value of the m-th pixel as the pixel value of the first target pixel. Performing a cyclic image offset on the image to be detected in a second direction opposite to the first direction to obtain a second offset image includes: determining the coordinates of a second target pixel in the image to be detected corresponding to the n-th pixel based on the difference between the coordinates of the n-th pixel in the second offset image and the periodic offset, where 1 ≤ n ≤ N, and N is the total number of pixels in the second offset image; and setting the pixel value of the n-th pixel as the pixel value of the second target pixel.

[0008] In some embodiments, the periodic offset includes a row offset and a column offset; the row coordinate of the first target pixel is determined by the sum of the row coordinate of the m-th pixel and the row offset, and the number of rows h of the pixel matrix of the image to be detected; the column coordinate of the first target pixel is determined by the sum of the column coordinate of the m-th pixel and the column offset, and the number of columns w of the pixel matrix of the image to be detected; the row coordinate of the second target pixel is determined by the difference between the row coordinate of the n-th pixel and the row offset, and the number of rows h; the column coordinate of the second target pixel is determined by the difference between the column coordinate of the n-th pixel and the column offset, and the number of columns w.

[0009] In some embodiments, when the sum of the row coordinates and the row offset of the m-th pixel is greater than or equal to 0 and less than or equal to h-1, the row coordinates of the first target pixel are the sum of the row coordinates and the row offset of the m-th pixel; when the sum of the row coordinates and the row offset of the m-th pixel is less than 0, the row coordinates of the first target pixel are the sum of the sum of the row coordinates and the row offset of the m-th pixel and the number of rows h; when the sum of the row coordinates and the row offset of the m-th pixel is greater than h-1, the row coordinates of the first target pixel are the difference between the sum of the row coordinates and the row offset of the m-th pixel and the number of rows h.

[0010] In some embodiments, when the sum of the column coordinates and the column offset of the m-th pixel is greater than or equal to 0 and less than or equal to w-1, the column coordinates of the first target pixel are the sum of the column coordinates and the column offset of the m-th pixel; when the sum of the column coordinates and the column offset of the m-th pixel is less than 0, the column coordinates of the first target pixel are the sum of the sum of the column coordinates and the column offset of the m-th pixel and the number of columns w; when the sum of the column coordinates and the column offset of the m-th pixel is greater than w-1, the column coordinates of the first target pixel are the difference between the sum of the column coordinates and the column offset of the m-th pixel and the number of columns w.

[0011] In some embodiments, when the difference between the row coordinates and the row offset of the nth pixel is greater than or equal to 0 and less than or equal to h-1, the row coordinates of the second target pixel are the difference between the row coordinates and the row offset of the nth pixel; when the difference between the row coordinates and the row offset of the nth pixel is less than 0, the row coordinates of the second target pixel are the sum of the difference between the row coordinates and the row offset of the nth pixel and the number of rows h; when the difference between the row coordinates and the row offset of the nth pixel is greater than h-1, the row coordinates of the first target pixel are the difference between the difference between the row coordinates and the row offset of the nth pixel and the number of rows h.

[0012] In some embodiments, when the difference between the column coordinates and the column offset of the nth pixel is greater than or equal to 0 and less than or equal to w-1, the column coordinates of the second target pixel are the difference between the column coordinates and the column offset of the nth pixel; when the difference between the column coordinates and the column offset of the nth pixel is less than 0, the column coordinates of the second target pixel are the sum of the difference between the column coordinates and the column offset of the nth pixel and the number of columns w; when the sum of the column coordinates and the column offset of the nth pixel is greater than w-1, the column coordinates of the second target pixel are the difference between the difference between the column coordinates and the column offset of the nth pixel and the number of columns w.

[0013] In some embodiments, determining the period offset of the image to be detected includes: calculating a first autocorrelation coefficient matrix of the image to be detected; generating a mask matrix, wherein the size of the mask matrix is ​​the same as the size of the first autocorrelation coefficient matrix; obtaining a second autocorrelation coefficient matrix of the image to be detected based on the first autocorrelation coefficient matrix and the mask matrix; and determining the period offset of the image to be detected based on the second autocorrelation coefficient matrix.

[0014] In some embodiments, determining the period offset of the image to be detected based on the second autocorrelation coefficient matrix includes: selecting the element with the largest value in the second autocorrelation coefficient matrix as the target element; and determining the period offset based on the coordinate value of the target element if the value of the target element is greater than a predetermined threshold.

[0015] In some embodiments, determining the periodic offset based on the coordinate values ​​of the target element includes: using the row coordinates of the target element as the row offset in the periodic offset; and using the column coordinates of the target element as the column offset in the periodic offset.

[0016] In some embodiments, generating the mask matrix includes: setting a first mask vector, wherein the length of the first mask vector is equal to the number of rows of the first autocorrelation coefficient matrix; setting a second mask vector, wherein the length of the second mask vector is equal to the number of columns of the first autocorrelation coefficient matrix; and obtaining the mask matrix based on the first mask vector and the second mask vector, wherein the elements in a specified region of the mask matrix are 0.

[0017] In some embodiments, if the row coordinate of the k-th element in the mask matrix is ​​greater than a first threshold, the value of the k-th element is 0. The first threshold is the product of the number of rows in the mask matrix and a first coefficient, 1 ≤ k ≤ K, where K is the total number of elements in the mask matrix. If the row coordinate of the k-th element is less than or equal to the first threshold and the column coordinate of the k-th element is greater than a second threshold and less than a third threshold, the value of the k-th element is 0. The second threshold is the product of a specified number of columns and the first coefficient, and the third threshold is the difference between the number of columns in the mask matrix and the second threshold. The specified number of columns is the number of rows in the mask matrix. The value of the k-th element is half the number of columns in the matrix. When the row coordinate of the k-th element is less than or equal to a fourth threshold and the column coordinate of the k-th element is less than or equal to a fifth threshold, the value of the k-th element is 0. The fourth threshold is the product of the number of rows in the mask matrix and the second coefficient, and the fifth threshold is the product of the specified number of columns and the second coefficient, where the second coefficient is less than the first coefficient. When the row coordinate of the k-th element is less than or equal to a fourth threshold and the column coordinate of the k-th element is greater than or equal to a sixth threshold, the value of the k-th element is 0. The sixth threshold is the difference between the number of columns in the mask matrix and the fifth threshold.

[0018] In some embodiments, when parameter i is greater than the first threshold, the i-th element in the first mask vector is 0, where parameter i is a natural number not greater than the length of the first mask vector; when parameter i is less than or equal to the first threshold, the value of the i-th element in the first mask vector is determined based on the difference between the length of the first mask vector and parameter i; when parameter j is greater than the second threshold and less than the third threshold, the j-th element in the second mask vector is 0, where parameter j is a natural number not greater than the length of the second mask vector; when parameter j is less than or equal to the second threshold, the value of the j-th element in the second mask vector is determined based on the difference between the specified column number and parameter j; when parameter j is greater than or equal to the third threshold, the value of the j-th element in the second mask vector is determined based on the difference between parameter j and the specified column number.

[0019] In some embodiments, obtaining the mask matrix based on the first mask vector and the second mask vector includes: calculating the vector product of the first mask vector and the second mask vector to obtain the mask matrix.

[0020] In some embodiments, obtaining the second autocorrelation coefficient matrix of the image to be detected based on the first autocorrelation coefficient matrix and the mask matrix includes: performing a dot product between the first autocorrelation coefficient matrix and the mask matrix to obtain the second autocorrelation coefficient matrix.

[0021] In some embodiments, calculating the first autocorrelation coefficient matrix of the image to be detected includes: expanding the image to be detected to obtain a target image including the image to be detected; and calculating the autocorrelation coefficient of the target image to obtain the first autocorrelation coefficient matrix.

[0022] In some embodiments, expanding the image to be detected includes: converting the image to be detected into a grayscale image; blurring the grayscale image to obtain a blurred image; and expanding the blurred image to obtain the target image.

[0023] In some embodiments, expanding the blurred image includes filling the edges of the blurred image with pixels having a plurality of predetermined pixel values ​​to obtain a target image.

[0024] In some embodiments, the number of rows in the pixel matrix of the target image is twice the number of rows in the pixel matrix of the image to be detected; and the number of columns in the pixel matrix of the target image is twice the number of columns in the pixel matrix of the image to be detected.

[0025] In some embodiments, merging the image to be detected and the difference fusion image to obtain a merged image includes: performing channel merging on the image to be detected and the difference fusion image to obtain the merged image.

[0026] In a second aspect of this disclosure, a model training method is provided, comprising: determining a periodic offset of a sample image, wherein the sample image includes a plurality of sub-images arranged periodically; performing a cyclic image offset on the sample image in a first direction according to the periodic offset to obtain a first offset image, and performing a cyclic image offset on the sample image in a second direction opposite to the first direction to obtain a second offset image; generating a difference fusion image based on the sample image, the first offset image, and the second offset image; merging the sample image and the difference fusion image to obtain a merged image; processing the merged image using a neural network model to obtain a defect detection result; determining a loss function based on the defect detection result and the annotation information of the sample image; and training the neural network model based on the loss function.

[0027] In a third aspect of this disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute instructions stored in the memory to implement the method as described in any of the above embodiments.

[0028] In a fourth aspect of this disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any of the above embodiments.

[0029] In a fifth aspect of this disclosure, a computer program product is provided, including computer instructions, wherein the computer instructions, when executed by a processor, implement the method as described in any of the above embodiments.

[0030] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 is a schematic image of a product to be tested according to an embodiment of this disclosure;

[0033] Figure 2 is a flowchart illustrating a defect detection method according to an embodiment of this disclosure;

[0034] Figure 3 is a flowchart illustrating a method for determining period offset according to an embodiment of this disclosure;

[0035] Figure 4 is a schematic diagram of a first autocorrelation coefficient matrix according to an embodiment of the present disclosure;

[0036] Figure 5 is a schematic diagram of a mask matrix according to an embodiment of this disclosure;

[0037] Figure 6 is a schematic diagram of the second autocorrelation coefficient matrix according to an embodiment of this disclosure;

[0038] Figure 7 is a flowchart illustrating a first offset image generation method according to an embodiment of this disclosure;

[0039] Figure 8 is a schematic diagram of an offset image according to an embodiment of the present disclosure;

[0040] Figure 9 is a flowchart illustrating a second offset image generation method according to an embodiment of this disclosure;

[0041] Figure 10 is a schematic diagram of an offset image according to another embodiment of the present disclosure;

[0042] Figure 11 is a flowchart illustrating a difference fusion image generation method according to an embodiment of this disclosure;

[0043] Figure 12 is a schematic diagram of a first difference image according to an embodiment of the present disclosure;

[0044] Figure 13 is a schematic diagram of a second difference image according to an embodiment of the present disclosure;

[0045] Figure 14 is a schematic diagram of a difference fusion image according to an embodiment of the present disclosure;

[0046] Figure 15 is a flowchart illustrating a difference fusion image generation method according to another embodiment of this disclosure;

[0047] Figure 16 is a schematic flowchart of a model training method according to an embodiment of the present disclosure;

[0048] Figure 17 is a schematic flowchart of a model training method according to another embodiment of this disclosure;

[0049] Figure 18 is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0050] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0051] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0052] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0053] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0054] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0055] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0056] In related technologies, neural network models are used to process images of products under inspection to detect defects. However, because defects in these products can be of various types, such as scratches, stains, and cracks, the areas corresponding to defects in the image exhibit significant visual diversity. Therefore, neural network models, relying solely on image recognition, cannot accurately detect defects, hindering the effective improvement of defect detection accuracy.

[0057] The inventors noted that images of products to be inspected, such as display panels, typically include multiple areas arranged periodically. For example, as shown in Figure 1, image 10 of the product to be inspected includes 25 identical sub-images, thus giving image 10 a visually distinct periodicity. As shown in Figure 1, due to defects in the product to be inspected, image 10 contains an abnormal region 11. Because this abnormal region 11 is small and irregularly shaped, it may not be effectively identified.

[0058] The inventors discovered through research that, as shown in Figure 1, since the 25 sub-images included in the image 10 of the product to be inspected are the same, by comparing the sub-image with the abnormal region 11 with other sub-images in the image 10 of the product to be inspected, the abnormal region 11 can be effectively identified, thereby detecting defects in the product to be inspected.

[0059] Accordingly, this disclosure provides a fault detection method that can effectively detect defects in the product by utilizing the periodic features of the image of the product to be tested, and effectively improve the defect detection accuracy of the neural network model.

[0060] Figure 2 is a schematic flowchart of a defect detection method according to an embodiment of the present disclosure. In some embodiments, the following defect detection method is performed by an electronic device, including steps 21-26.

[0061] In step 21, an image of the product to be inspected is obtained, wherein the image to be inspected includes multiple sub-images arranged periodically.

[0062] For example, the products to be tested include display panels, integrated circuit chips, microelectromechanical systems (MEMS), optical components, and intermediate material films in the manufacturing process of display panels, integrated circuit chips, MEMS, or optical components using photomasks.

[0063] For example, a display panel includes a plurality of neatly arranged pixel units, each pixel unit including a first sub-pixel unit for displaying red, a second sub-pixel unit for displaying blue, and a third sub-pixel unit for displaying green. If the image of each pixel unit is taken as a sub-image in the image of the display panel, it will be found that the image of the display panel includes a plurality of sub-images arranged periodically.

[0064] In step 22, the period offset of the image to be detected is determined.

[0065] It should be noted that the period offset of the image to be detected is used to represent the period value of the image. For example, if the image to be detected is defect-free, each pixel in the image is cyclically offset to the left according to the period offset to obtain a first offset image, which is the same as the image to be detected. As another example, if the image to be detected is defect-free, each pixel in the image to be detected is cyclically offset to the right according to the period offset to obtain a second offset image, which is the same as the image to be detected.

[0066] Figure 3 is a flowchart illustrating a method for determining the period offset according to an embodiment of the present disclosure, including steps 31-34.

[0067] In step 31, the first autocorrelation coefficient matrix of the image to be detected is calculated.

[0068] In some embodiments, the image to be detected is expanded to obtain a target image that includes the image to be detected. Next, the autocorrelation coefficient of the target image is calculated to obtain a first autocorrelation coefficient matrix.

[0069] It should be noted that by expanding the image to be detected, the edge regions of the image to be detected can be effectively involved in the autocorrelation calculation, so that the first autocorrelation coefficient matrix can accurately reflect the autocorrelation characteristics of the image to be detected.

[0070] Furthermore, to facilitate the calculation of the autocorrelation coefficient, the number of rows and columns of the pixel matrix of the image to be detected needs to be increased. For example, to facilitate the calculation of the autocorrelation coefficient of the target image using the Fourier transform and inverse Fourier transform functions, the image to be detected is expanded so that the number of rows and columns of the pixel matrix of the obtained target image is an integer power of 2.

[0071] In some embodiments, the step of expanding the image to be detected includes the following.

[0072] 1) Convert the image to be detected into a grayscale image.

[0073] For example, if the image to be detected is an RGB image with 3 channels, a grayscale image is obtained by calculating the average value of these 3 channels.

[0074] 2) Blur the grayscale image to obtain a blurred image.

[0075] It should be noted that blurring grayscale images can effectively remove image noise and improve the robustness of autocorrelation calculation.

[0076] In some embodiments, blurring may include Gaussian blur, box blur, dual blur, and other similar techniques. For example, the Gaussian blur parameters used in Gaussian blur processing can be selected based on the image noise. For instance, Gaussian blur parameters could be: a blur kernel size of 9 and a root mean square error of 3.

[0077] 3) Expand the blurred image to obtain the target image.

[0078] In some embodiments, pixels with multiple predetermined pixel values ​​are filled at the edges of the blurred image to obtain the target image.

[0079] For example, the pixel value of the pixels filled in a blurred image is 0.

[0080] For example, the size of the image to be detected is h×w, meaning the number of rows in the pixel matrix of the image to be detected is h and the number of columns is w. Correspondingly, the size of the blurred image is also h×w. Pixels with a value of 0 are filled into a predetermined column on the right side of the blurred image, and pixels with a value of 0 are filled into a predetermined row at the bottom of the blurred image to obtain the target image.

[0081] In some embodiments, the number of rows in the pixel matrix of the target image is twice the number of rows in the pixel matrix of the image to be detected, and the number of columns in the pixel matrix of the target image is twice the number of columns in the pixel matrix of the image to be detected.

[0082] For example, the size of the image to be detected is h×w, meaning the pixel matrix of the image to be detected has h rows and w columns. The size of the target image is 2h×2w, meaning the pixel matrix of the target image has 2h rows and 2w columns.

[0083] In some embodiments, the first autocorrelation coefficient matrix of the target image can be calculated using the following formula (1): A(i,j)=∑ u ∑ v G BP (u,v)·G BP (u+i,v+j) (1)

[0084] In formula (1), G BP (u,v) represents the pixel in the u-th row and v-th column of the pixel matrix of the target image, G BP (u+i,v+j) represents the pixel in the (u+i)th row and (v+j)th column of the pixel matrix of the target image. A(i,j) is the autocorrelation coefficient between the pixel of the target image and the target image itself after shifting by i rows and j columns.

[0085] In some embodiments, the first autocorrelation coefficient matrix of the target image can also be calculated using the fast Fourier transform, as shown in the following formula (2).

[0086] In formula (2), fft2 is the two-dimensional fast Fourier transform function, ifft2 is the two-dimensional fast Fourier inverse transform function, and G... BP Let A be the target image, A be the first autocorrelation coefficient matrix, and F be the... * Let F be the conjugate complex number.

[0087] It should be noted that, due to the symmetry of the autocorrelation coefficient matrix, only the autocorrelation coefficient matrix of the upper half of the target image needs to be calculated. For example, if the size of the target image is 2h×2w, then the size of the first autocorrelation coefficient matrix is ​​h×2w.

[0088] For example, the first autocorrelation coefficient matrix is ​​shown in Figure 4.

[0089] In step 32, a mask matrix is ​​generated, wherein the size of the mask matrix is ​​the same as the size of the first autocorrelation coefficient matrix.

[0090] For example, if the size of the first autocorrelation coefficient matrix is ​​h×2w, then the size of the mask matrix is ​​also h×2w.

[0091] It should be noted that the mask matrix is ​​used to shield unwanted periodic points in the first autocorrelation coefficient matrix, thereby effectively improving the stability of period detection.

[0092] In some embodiments, the step of generating the mask matrix includes the following.

[0093] 1) Set the first mask vector, where the length of the first mask vector is equal to the number of rows in the first autocorrelation coefficient matrix.

[0094] For example, if the size of the first autocorrelation coefficient matrix is ​​h×2w, then the length of the first mask vector is h.

[0095] It should be noted that since the size of the mask matrix is ​​the same as the size of the first autocorrelation coefficient matrix, the length of the first mask vector is also equal to the number of rows of the mask matrix.

[0096] In some embodiments, when parameter i is greater than the first threshold Th1, the value of the i-th element in the first mask vector is 0, where parameter i is a natural number not greater than the length of the first mask vector. When parameter i is less than or equal to the first threshold, the value of the i-th element in the first mask vector is determined based on the difference between the length of the first mask vector and parameter i.

[0097] It should be noted that each number in the vector is called an element. For example, if the first mask vector contains 10 numbers, then the first mask vector contains 10 elements, and the i-th number in the first mask vector is called the i-th element of the first mask vector.

[0098] In some embodiments, the first threshold Th1 is the product of the length of the first mask vector and the first coefficient.

[0099] For example, the length of the first mask vector is h, and the first coefficient is s. max Then the first threshold Th1 is as shown in formula (3). Th1=h×s max (3)

[0100] It should be noted that the first coefficient s max Less than 1. For example, the first coefficient s max It is 0.5.

[0101] In some embodiments, when parameter i is less than or equal to a first threshold, the step of determining the value of the i-th element in the first mask vector based on the difference between the length of the first mask vector and parameter i includes the following:

[0102] 1. Calculate the difference between the length of the first mask vector and the parameter i to obtain the first intermediate value.

[0103] 2. Calculate the ratio of the first intermediate value to the length of the first mask vector to obtain the second intermediate value.

[0104] 3. Calculate the square root of the second intermediate value to obtain the value of the i-th element in the first mask vector.

[0105] For example, the first mask vector M h As shown in formula (4).

[0106] 2) Set a second mask vector, where the length of the second mask vector is equal to the number of columns in the first autocorrelation coefficient matrix.

[0107] For example, if the size of the first autocorrelation coefficient matrix is ​​h×2w, then the length of the second mask vector is 2w.

[0108] It should be noted that since the size of the mask matrix is ​​the same as the size of the first autocorrelation coefficient matrix, the length of the second mask vector is also equal to the number of columns of the mask matrix.

[0109] In some embodiments, when parameter j is greater than the second threshold Th2 and less than the third threshold Th3, the j-th element in the second mask vector is 0, where parameter j is a natural number not greater than the length of the second mask vector. When parameter j is less than or equal to the second threshold Th2, the value of the j-th element in the second mask vector is determined based on the difference between the specified column number and parameter j. When parameter j is greater than or equal to the third threshold Th3, the value of the j-th element in the second mask vector is determined based on the difference between parameter j and the specified column number.

[0110] In some embodiments, the second threshold Th2 is the product of a specified number of columns and a first coefficient, where the specified number of columns is half the length of the second mask vector. The third threshold Th3 is the difference between the length of the second mask vector and the second threshold.

[0111] For example, if the length of the second mask vector is 2w, then the specified number of columns is w, and the first coefficient is s. max Then the second threshold Th2 is shown in formula (5), and the third threshold Th3 is shown in formula (6). Th2=w×s max (5) Th3=2×ww×s max (6)

[0112] In some embodiments, when parameter j is less than or equal to the second threshold Th2, the step of determining the value of the j-th element in the second mask vector based on the difference between the specified column number and parameter j includes the following.

[0113] 1. Calculate the difference between the specified number of columns and parameter j to obtain a third intermediate value.

[0114] 2. Calculate the ratio of the third intermediate value to the specified column number to obtain the fourth intermediate value.

[0115] 3. Calculate the square root of the fourth intermediate value to obtain the value of the j-th element in the second mask vector.

[0116] In some embodiments, when parameter j is greater than or equal to the third threshold Th3, the step of determining the value of the j-th element in the second mask vector based on the difference between parameter j and the specified column number includes the following.

[0117] 1. Calculate the difference between parameter j and the specified column number to obtain the fifth intermediate value.

[0118] 2. Calculate the ratio of the fifth intermediate value to the specified column number to obtain the sixth intermediate value.

[0119] 3. Calculate the square root of the sixth intermediate value to obtain the value of the j-th element in the second mask vector.

[0120] For example, the second mask vector M w As shown in formula (7).

[0121] 3) Based on the first mask vector and the second mask vector, obtain the mask matrix, where the elements in the specified region of the mask matrix take the value of 0.

[0122] In some embodiments, the vector product of the first mask vector and the second mask vector is calculated to obtain the mask matrix.

[0123] For example, the mask matrix M is shown in formula (8).

[0124] In formula (8), the symbol T is used to denote transpose.

[0125] It should be noted that the first mask vector M h The length is h, and the second mask vector is M. w If the length of the mask matrix is ​​2w, then the size of the mask matrix M is h×2w.

[0126] Figure 5 is a schematic diagram of a mask matrix according to an embodiment of this disclosure.

[0127] It should be noted that, according to the above formulas (4), (7) and (8), there are regions in the mask matrix M where all elements have a value of 0, namely regions 51 and 52 in Figure 5.

[0128] For example, if the row coordinate of the k-th element in the mask matrix M is greater than the first threshold Th1, the value of the k-th element is 0, 1≤k≤K, where K is the total number of elements in the mask matrix. In this case, the k-th element is located in region 51 in Figure 5.

[0129] For example, if the row coordinate of the k-th element is less than or equal to the first threshold Th1, and the column coordinate of the k-th element is greater than the second threshold Th2 and less than the third threshold Th3, the value of the k-th element is 0. In this case, the k-th element is located in region 52 in Figure 5.

[0130] The inventors noted that, as shown in Figure 4, the autocorrelation coefficients in the upper left and upper right regions of the first autocorrelation coefficient matrix negatively impact the stability of periodic detection. Therefore, as shown in Figure 5, further adjustments to the mask matrix are necessary.

[0131] In some embodiments, if the row coordinate of the k-th element is less than or equal to the fourth threshold Th4 and the column coordinate of the k-th element is less than or equal to the fifth threshold Th5, the value of the k-th element is 0. In this case, the k-th element is located in region 53 in Figure 5.

[0132] In some embodiments, the fourth threshold Th4 is the product of the number of rows in the mask matrix and the second coefficient, where the second coefficient is less than the first coefficient. The fifth threshold Th5 is the product of a specified number of columns and the second coefficient, where the specified number of columns is half the number of columns in the mask matrix.

[0133] For example, if the number of columns in the mask matrix is ​​2w, then specify the number of columns as w and the second coefficient as s. min Then the fourth threshold Th4 is shown in formula (9), and the fifth threshold Th5 is shown in formula (10). Th4=h×s min (9) Th5=w×s min (10)

[0134] For example, the first coefficient s max The second coefficient is 0.5. min It is 0.2.

[0135] In some embodiments, if the row coordinate of the k-th element is less than or equal to the fourth threshold Th4 and the column coordinate of the k-th element is greater than or equal to the sixth threshold Th6, the value of the k-th element is 0. In this case, the k-th element is located in region 54 in Figure 5.

[0136] In some embodiments, the sixth threshold Th6 is the difference between the number of columns in the mask matrix and the fifth threshold.

[0137] For example, if the number of columns in the mask matrix is ​​2w, the sixth threshold Th6 is as shown in formula (11). Th6=2×ww×s min (11)

[0138] It should be noted that, as shown in Figure 5, Th1>Th4, Th6>Th3>Th2>Th5.

[0139] In other words, by using the following formula (12), regions 53 and 54, in the mask matrix M, are further defined so that all elements have a value of 0.

[0140] In step 33, the second autocorrelation coefficient matrix of the image to be detected is obtained based on the first autocorrelation coefficient matrix and the mask matrix.

[0141] In some embodiments, the second autocorrelation coefficient matrix is ​​obtained by multiplying the first autocorrelation coefficient matrix and the mask matrix by a dot product.

[0142] For example, by performing a dot product between the first autocorrelation coefficient matrix shown in Figure 4 and the mask matrix shown in Figure 5, the second autocorrelation matrix is ​​obtained as shown in Figure 6.

[0143] In step 34, the period offset of the image to be detected is determined based on the second autocorrelation coefficient matrix.

[0144] In some embodiments, the element with the largest value in the second autocorrelation coefficient matrix is ​​selected as the target element. If the value of the target element is greater than a predetermined threshold, the period offset is determined based on the coordinate value of the target element.

[0145] It should be noted that the period offset includes both row offset and column offset.

[0146] For example, the row coordinates of the target element can be used as the row offset in the periodic offset, and the column coordinates of the target element can be used as the column offset in the periodic offset.

[0147] In step 23, based on the periodic offset, the image to be detected is cyclically offset in the first direction to obtain a first offset image, and the image to be detected is cyclically offset in the second direction opposite to the first direction to obtain a second offset image.

[0148] Figure 7 is a flowchart illustrating a first offset image generation method according to an embodiment of the present disclosure, including steps 71-72.

[0149] In step 71, the coordinates of the first target pixel in the image to be detected corresponding to the m-th pixel are determined based on the sum of the coordinates of the m-th pixel in the first offset image and the period offset, where 1≤m≤M and M is the total number of pixels in the first offset image.

[0150] In some embodiments, the row coordinates of the first target pixel are determined by the sum of the row coordinates and row offset of the m-th pixel, and the number of rows h of the pixel matrix of the image to be detected. The column coordinates of the first target pixel are determined by the sum of the column coordinates and column offset of the m-th pixel, and the number of columns w of the pixel matrix of the image to be detected.

[0151] In some embodiments, when the sum of the row coordinates and row offset of the m-th pixel is greater than or equal to 0 and less than or equal to h-1, the row coordinates of the first target pixel are the sum of the row coordinates and row offset of the m-th pixel. When the sum of the row coordinates and row offset of the m-th pixel is less than 0, the row coordinates of the first target pixel are the sum of the sum of the row coordinates and row offset of the m-th pixel and the number of rows h. When the sum of the row coordinates and row offset of the m-th pixel is greater than h-1, the row coordinates of the first target pixel are the difference between the sum of the row coordinates and row offset of the m-th pixel and the number of rows h.

[0152] In some embodiments, when the sum of the column coordinates and column offset of the m-th pixel is greater than or equal to 0 and less than or equal to w-1, the column coordinates of the first target pixel are the sum of the column coordinates and column offset of the m-th pixel. When the sum of the column coordinates and column offset of the m-th pixel is less than 0, the column coordinates of the first target pixel are the sum of the sum of the column coordinates and column offset of the m-th pixel and the number of columns w. When the sum of the column coordinates and column offset of the m-th pixel is greater than w-1, the column coordinates of the first target pixel are the difference between the sum of the column coordinates and column offset of the m-th pixel and the number of columns w.

[0153] For example, I1(i,j) is the m-th pixel in the first offset image, where the row coordinate of the m-th pixel in the pixel matrix of the first offset image is i and the column coordinate is j. I(i2,j2) is the first target pixel in the image to be detected, where the row coordinate of the first target pixel in the pixel matrix of the image to be detected is i2 and the column coordinate is j2.

[0154] Based on the above analysis, I1(i,j) and I(i2,j2) satisfy formula (13) and formula (14).

[0155] In formulas (13) and (14), d x d is the column offset. y This is the row offset.

[0156] In step 72, the pixel value of the m-th pixel is set to the pixel value of the first target pixel.

[0157] In other words, according to the following formula (15), the pixels in the image to be detected I are cyclically offset using row offset and column offset to generate the first offset image I1. I1(i,j)=I(i2,j2) (15)

[0158] Figure 8 is a schematic diagram of an offset image according to an embodiment of the present disclosure. In Figure 8, Figure 81 is the image to be detected, and Figure 82 is the first offset image generated according to formula (15). By comparing Figure 81 and Figure 82, it can be seen that Figure 82 is obtained by cyclically shifting the pixels in Figure 81 to the left.

[0159] Figure 9 is a flowchart illustrating a second offset image generation method according to an embodiment of the present disclosure, including steps 91-92.

[0160] In step 91, the coordinates of the second target pixel in the image to be detected corresponding to the nth pixel are determined based on the difference between the coordinates of the nth pixel in the second offset image and the period offset, where 1≤n≤N and N is the total number of pixels in the second offset image.

[0161] In some embodiments, the row coordinates of the second target pixel are determined by the difference between the row coordinates and row offset of the nth pixel, and the number of rows h. The column coordinates of the second target pixel are determined by the difference between the column coordinates and column offset of the nth pixel, and the number of columns w.

[0162] In some embodiments, when the difference between the row coordinate and row offset of the nth pixel is greater than or equal to 0 and less than or equal to h-1, the row coordinate of the second target pixel is the difference between the row coordinate and row offset of the nth pixel. When the difference between the row coordinate and row offset of the nth pixel is less than 0, the row coordinate of the second target pixel is the sum of the difference between the row coordinate and row offset of the nth pixel and the number of rows h. When the difference between the row coordinate and row offset of the nth pixel is greater than h-1, the row coordinate of the first target pixel is the difference between the difference between the row coordinate and row offset of the nth pixel and the number of rows h.

[0163] In some embodiments, when the difference between the column coordinate and column offset of the nth pixel is greater than or equal to 0 and less than or equal to w-1, the column coordinate of the second target pixel is the difference between the column coordinate and column offset of the nth pixel. When the difference between the column coordinate and column offset of the nth pixel is less than 0, the column coordinate of the second target pixel is the sum of the difference between the column coordinate and column offset of the nth pixel and the number of columns w. When the sum of the column coordinate and column offset of the nth pixel is greater than w-1, the column coordinate of the second target pixel is the difference between the difference between the column coordinate and column offset of the nth pixel and the number of columns w.

[0164] For example, I2(i,j) is the nth pixel in the second offset image, where the row coordinate of the nth pixel in the pixel matrix of the second offset image is i and the column coordinate is j. I(i2,j2) is the second target pixel in the image to be detected, where the row coordinate of the second target pixel in the pixel matrix of the image to be detected is i2 and the column coordinate is j2.

[0165] Based on the above analysis, I2(i,j) and I(i2,j2) satisfy formula (16) and formula (17).

[0166] In formulas (16) and (17), d x d is the column offset. y This is the row offset.

[0167] In step 92, the pixel value of the nth pixel is set to the pixel value of the second target pixel.

[0168] In other words, according to the following formula (18), the pixels in the image to be detected I are cyclically offset using row offset and column offset to generate the second offset image I2. I2(i,j)=I(i2,j2) (18)

[0169] Figure 10 is a schematic diagram of an offset image according to another embodiment of the present disclosure. In Figure 10, Figure 101 is the image to be detected, and Figure 102 is the second offset image generated according to formula (18). By comparing Figure 101 and Figure 102, it can be seen that Figure 102 is obtained by cyclically shifting the pixels in Figure 101 to the right.

[0170] In step 24, a difference fusion image is generated based on the image to be detected, the first offset image, and the second offset image.

[0171] Figure 11 is a flowchart illustrating a difference fusion image generation method according to an embodiment of the present disclosure, including steps 111-113.

[0172] In step 111, a first difference image is generated based on the difference between the image to be detected and the first offset image.

[0173] In some embodiments, the image to be detected is blurred to obtain a blurred image to be detected. A first offset image is blurred to obtain a first blurred image. A first difference image is generated based on the difference between the blurred image to be detected and the first blurred image.

[0174] It should be noted that the image to be detected is blurred to remove noise. Similarly, the first offset image is blurred to remove noise.

[0175] In some embodiments, blurring may include Gaussian blurring, box blurring, double blurring, and other similar processing methods.

[0176] For example, the first difference image D1 is shown in formula (19). D1=|I1B -I B | (19)

[0177] In formula (19), I 1B For the first blurred image, I B The image to be detected is a blurred image.

[0178] For example, the first difference image D1 is shown in Figure 12.

[0179] In step 112, a second difference image is generated based on the difference between the image to be detected and the second offset image.

[0180] In some embodiments, the second offset image is blurred to obtain a second blurred image. A second difference image is generated based on the difference between the blurred image to be detected and the second blurred image.

[0181] It should be noted that the second offset image is blurred in order to remove noise from it.

[0182] In some embodiments, blurring may include Gaussian blurring, box blurring, double blurring, and other similar processing methods.

[0183] For example, the second difference image D2 is shown in formula (20). D2=|I 2B -I B | (20)

[0184] In formula (20), I 2B For the second blurred image, I B The image to be detected is a blurred image.

[0185] For example, the second difference image D2 is shown in Figure 13.

[0186] In step 113, a difference fusion image is generated based on the first difference image and the second difference image.

[0187] In some embodiments, the first difference image and the second difference image are fused using minimum value fusion to obtain a difference fused image.

[0188] For example, the difference fusion image is shown in Equation (21). D(i,j)=min[D1(i,j),D2(i,j)] (21)

[0189] Figure 14 is a schematic diagram of a difference fusion image according to an embodiment of the present disclosure. In Figure 14, Figure 141 is the image to be detected, and Figure 142 is the difference fusion image generated according to formula (21). By comparing Figure 141 and Figure 142, it can be seen that the abnormal areas in the image to be detected are clearly presented in the difference fusion image. Therefore, the difference fusion image can effectively improve the accuracy of defect detection.

[0190] In step 25, the image to be detected and the difference fusion image are merged to obtain a merged image.

[0191] In some embodiments, the image to be detected and the difference fusion image are channel-merged to obtain a merged image.

[0192] For example, the image to be detected is an RGB image with 3 channels, and the difference fusion image is a single-channel image. By merging the channels of the image to be detected and the difference fusion image, the resulting merged image has 4 channels.

[0193] In step 26, the merged image is processed using a neural network model to obtain the defect detection results.

[0194] In some embodiments, the neural network model includes a visual network model and an object detection model. The visual network model is used to extract visual features from the image, and the object detection model is used to analyze the visual features to output defect detection results.

[0195] For example, visual network models can include ResNet, MobileNet, SWIN, and other models used to extract visual features from images. Object detection models can include Faster R-CNN, YOLO, DETR, and other models used for object detection.

[0196] In the defect detection provided in the above embodiments of this disclosure, the periodic offset of the image to be detected is calculated, and the image to be detected is cyclically offset in a first direction and a second direction opposite to the first direction according to the periodic offset to obtain a first offset image and a second offset image. Based on the difference between the image to be detected and the first offset image, and the difference between the image to be detected and the second offset image, a difference fusion image is generated. Then, the image to be detected and the difference fusion image are merged, and the merged image is input into a neural network model so that the neural network model can effectively identify abnormal regions in the image to be detected, thereby detecting defects in the product to be detected.

[0197] Figure 15 is a flowchart illustrating a difference fusion image generation method according to another embodiment of the present disclosure, including steps 151-1512.

[0198] In step 151, the acquired image to be detected I is converted to obtain a grayscale image G, wherein the image to be detected I includes multiple sub-images arranged periodically.

[0199] In step 152, the grayscale image G is blurred to obtain a blurred image G. B .

[0200] In step 153, the blurred image G B The image is then expanded to obtain the target image, and the first autocorrelation coefficient matrix A of the target image is calculated.

[0201] For example, the first autocorrelation coefficient matrix A of the target image can be calculated using the above formula (1) or formula (2).

[0202] In step 154, the first autocorrelation coefficient matrix A and the mask matrix M are multiplied by a dot to obtain the second autocorrelation coefficient matrix A. N .

[0203] For example, the mask matrix M can be generated using the above formulas (8) and (12).

[0204] In step 155, based on the second autocorrelation coefficient matrix A N Determine the period offset d of the image I to be detected. x and d y .

[0205] In some embodiments, the second autocorrelation coefficient matrix A is selected. N The element with the largest value in the set is selected as the target element. If the value of the target element is greater than a predetermined threshold, the period offset is determined based on the coordinate value of the target element.

[0206] For example, the row coordinates of the target element are used as the row offset d in the periodic offset. y The column coordinates of the target element are used as the column offset d in the periodic offset. x .

[0207] In step 156, based on the periodic offset, the image to be detected I is cyclically offset in the first direction to obtain the first offset image I1, and the image to be detected I is cyclically offset in the second direction opposite to the first direction to obtain the second offset image I2.

[0208] For example, a first offset image I1 is obtained according to the scheme provided in any embodiment of FIG7, and a second offset image I2 is obtained according to the scheme provided in any embodiment of FIG9.

[0209] In step 157, the first offset image I1 is blurred to obtain the first blurred image I. 1B .

[0210] In step 158, the second offset image I2 is blurred to obtain the second blurred image I. 2B .

[0211] In step 159, the image to be detected I is blurred to obtain the blurred image to be detected. B .

[0212] In step 1510, the blurred image I to be detected is calculated. B and the first blurred image I 1B The differences are used to generate the first difference image D1.

[0213] For example, the first difference image D1 is obtained using formula (19).

[0214] In step 1511, the blurred image I to be detected is calculated. B Second blurred image I 2B The difference is used to generate a second difference image D2.

[0215] For example, the second difference image D2 is obtained using formula (20).

[0216] In step 1512, the first difference image D1 and the second difference image D2 are fused by minimum value to obtain the difference fused image D.

[0217] For example, the difference fusion image D can be obtained using formula (21).

[0218] Next, the image to be detected, I, and the difference fusion image, D, are merged to obtain the merged image I. D And merge image I D Input the neural network model so that the neural network model can output the detection results.

[0219] In some embodiments, the neural network model used in the above embodiments may be trained using the model training method described in the following embodiments.

[0220] Figure 16 is a schematic flowchart of a model training method according to an embodiment of the present disclosure. In some embodiments, the following model training method is performed by an electronic device, including steps 161-167.

[0221] In step 161, the period offset of the sample image is determined, wherein the sample image comprises multiple sub-images arranged periodically.

[0222] In some embodiments, the period offset of the sample image is determined using the scheme involved in any of the embodiments in FIG3.

[0223] In step 162, the sample image is cyclically offset in the first direction according to the periodic offset to obtain a first offset image, and the sample image is cyclically offset in the second direction opposite to the first direction to obtain a second offset image.

[0224] In some embodiments, a first offset image is obtained according to the scheme provided in any embodiment of FIG7, and a second offset image is obtained according to the scheme provided in any embodiment of FIG9.

[0225] In step 163, a difference fusion image is generated based on the sample image, the first offset image, and the second offset image.

[0226] In some embodiments, a difference fusion image is generated according to the scheme provided in any of the embodiments in FIG11.

[0227] In step 164, the sample image and the difference fusion image are merged to obtain a merged image.

[0228] In some embodiments, the sample image and the difference fusion image are channel-merged to obtain a merged image.

[0229] In step 165, the merged image is processed using a neural network model to obtain the defect detection results.

[0230] In step 166, the loss function is determined based on the defect detection results and the annotation information of the sample images.

[0231] In step 167, the neural network model is trained according to the loss function.

[0232] Figure 17 is a schematic flowchart of a model training method according to another embodiment of the present disclosure, including steps 171-176.

[0233] In step 171, the difference fusion image D is calculated for sample image I.

[0234] In some embodiments, the difference fusion image D is calculated according to the scheme provided in any of the embodiments in FIG15.

[0235] In step 172, the sample image I and the difference fusion image D are merged to obtain the merged image I. D .

[0236] In some embodiments, sample image I and difference fusion image D are channel-merged to obtain merged image I. D .

[0237] In step 173, the visual network model in the neural network model is used to merge the image I. D The process is performed to obtain the visual feature F.

[0238] For example, visual network models can include ResNet, MobileNet, SWIN, and other models used to extract visual features from images.

[0239] In step 174, the target detection model in the neural network model is used to perform target detection on the visual feature F to obtain the defect detection result P.

[0240] For example, object detection models can include Faster R-CNN, YOLO, DETR, and other models used for object detection.

[0241] In step 175, the loss function Loss is determined using the defect detection result P and the annotation information of the sample image.

[0242] In step 176, the visual network model and the object detection model in the neural network model are trained according to the loss function Loss.

[0243] Figure 18 is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure.

[0244] As shown in Figure 18, the electronic device 180 is presented in the form of a general-purpose computing device. The electronic device 180 includes a memory 181, a processor 182, and a bus 183 connecting different system components.

[0245] The memory 181 may include, for example, system memory, non-volatile storage media, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs. The system memory may include volatile storage media, such as random access memory (RAM) and / or cache memory. The non-volatile storage media may store, for example, instructions for a corresponding embodiment of at least one defect detection method or model training method being executed. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, etc.

[0246] The processor 182 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete hardware components such as discrete gates or transistors. Accordingly, each module, such as the acquisition module, the calculation module, and the adjustment module, can be implemented by executing instructions in the central processing unit (CPU) running memory to perform the corresponding steps, or by implementing dedicated circuitry to perform the corresponding steps.

[0247] For example, processor 182 is configured to implement the method involved in any of the embodiments of FIG2, FIG3, FIG7, FIG9, FIG11, FIG15 to FIG17 based on memory-stored instruction execution.

[0248] It should be noted that when the processor 182 executes the method according to any of the embodiments in Figures 2, 3, 7, 9, 11, and 15, the electronic device can serve as a defect detection device. When the processor 182 executes the method according to any of the embodiments in Figures 16 and 17, the electronic device can serve as a model training device.

[0249] Bus 183 can use any of the various bus architectures. For example, bus architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, and the Peripheral Component Interconnect (PCI) bus.

[0250] The interfaces 184, 185, and 186 of the electronic device 180, as well as the memory 181 and processor 182, can be connected via bus 183. Input / output interface 184 provides a connection interface for input / output devices such as monitors, mice, and keyboards. Network interface 185 provides a connection interface for various networked devices. Storage interface 186 provides a connection interface for external storage devices such as floppy disks, USB flash drives, and SD cards.

[0251] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations thereof, can be implemented by computer-readable program instructions.

[0252] These computer-readable program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device to produce a machine, such that execution of the instructions by the processor produces means for implementing the functions specified in one or more boxes of the flowchart and / or block diagram.

[0253] These computer-readable program instructions may also be stored in a computer-readable storage medium. These instructions cause a computer to work in a particular manner to produce an article of manufacture, including instructions that implement the functions specified in one or more boxes in a flowchart and / or block diagram.

[0254] This disclosure may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0255] This disclosure also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method involved in any of the embodiments shown in Figures 2, 3, 7, 9, 11, 15 to 17.

[0256] This disclosure also provides a computer program product, including computer instructions, wherein when executed by a processor, the computer instructions implement the method involved in any of the embodiments shown in Figures 2, 3, 7, 9, 11, 15 to 17.

[0257] In some embodiments, the functional units described above may be implemented as general-purpose processors, programmable logic controllers (PLCs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any suitable combination thereof for performing the functions described herein.

[0258] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0259] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A defect detection method, comprising: Acquire an image of the product to be inspected, wherein the image to be inspected includes multiple sub-images arranged periodically; Determine the period offset of the image to be detected; Based on the periodic offset, the image to be detected is cyclically offset in a first direction to obtain a first offset image, and the image to be detected is cyclically offset in a second direction opposite to the first direction to obtain a second offset image; A difference fusion image is generated based on the image to be detected, the first offset image, and the second offset image; The image to be detected and the difference fusion image are merged to obtain a merged image; The merged image is processed using a neural network model to obtain defect detection results.

2. The defect detection method according to claim 1, wherein, The step of generating a difference fusion image based on the image to be detected, the first offset image, and the second offset image includes: A first difference image is generated based on the difference between the image to be detected and the first offset image; A second difference image is generated based on the difference between the image to be detected and the second offset image; The difference fusion image is generated based on the first difference image and the second difference image.

3. The defect detection method according to claim 2, wherein, Generating the difference fusion image based on the first difference image and the second difference image includes: The first difference image and the second difference image are fused by minimum value fusion to obtain the difference fused image.

4. The defect detection method according to claim 2, wherein, Generating the first difference image includes: The image to be detected is blurred to obtain a blurred image to be detected; The first offset image is blurred to obtain a first blurred image; Based on the difference between the blurred image to be detected and the first blurred image, the first difference image is generated; The generation of the second difference image includes: The second offset image is blurred to obtain a second blurred image; The second difference image is generated based on the difference between the blurry image to be detected and the second blurry image.

5. The defect detection method according to claim 1, wherein, The step of performing a cyclic image offset on the image to be detected in a first direction to obtain a first offset image includes: Based on the sum of the coordinates of the m-th pixel in the first offset image and the periodic offset, the coordinates of the first target pixel in the image to be detected corresponding to the m-th pixel are determined, where 1≤m≤M, and M is the total number of pixels in the first offset image. Set the pixel value of the m-th pixel to the pixel value of the first target pixel; The step of cyclically shifting the image to be detected in a second direction opposite to the first direction to obtain a second shifted image includes: Based on the difference between the coordinates of the nth pixel in the second offset image and the periodic offset, the coordinates of the second target pixel in the image to be detected corresponding to the nth pixel are determined, where 1≤n≤N, and N is the total number of pixels in the second offset image; Set the pixel value of the nth pixel to the pixel value of the second target pixel.

6. The defect detection method according to claim 5, wherein, The period offset includes row offset and column offset; The row coordinates of the first target pixel are determined by the sum of the row coordinates of the m-th pixel and the row offset, and the number of rows h of the pixel matrix of the image to be detected; The column coordinates of the first target pixel are determined by the sum of the column coordinates of the m-th pixel and the column offset, as well as the number of columns w of the pixel matrix of the image to be detected. The row coordinates of the second target pixel are determined by the difference between the row coordinates of the nth pixel and the row offset, and the number of rows h. The column coordinates of the second target pixel are determined by the difference between the column coordinates of the nth pixel and the column offset, and the number of columns w.

7. The defect detection method according to claim 6, wherein, When the sum of the row coordinates of the m-th pixel and the row offset is greater than or equal to 0 and less than or equal to h-1, the row coordinates of the first target pixel are the sum of the row coordinates of the m-th pixel and the row offset. When the sum of the row coordinates and the row offset of the m-th pixel is less than 0, the row coordinates of the first target pixel are the sum of the sum of the row coordinates and the row offset of the m-th pixel and the number of rows h. If the sum of the row coordinates and the row offset of the m-th pixel is greater than h-1, the row coordinates of the first target pixel are the difference between the sum of the row coordinates and the row offset of the m-th pixel and the number of rows h.

8. The defect detection method according to claim 6, wherein, When the sum of the column coordinates and the column offset of the m-th pixel is greater than or equal to 0 and less than or equal to w-1, the column coordinates of the first target pixel are the sum of the column coordinates and the column offset of the m-th pixel. When the sum of the column coordinates and the column offset of the m-th pixel is less than 0, the column coordinates of the first target pixel are the sum of the sum of the column coordinates and the column offset of the m-th pixel and the number of columns w. If the sum of the column coordinates and the column offset of the m-th pixel is greater than w-1, the column coordinates of the first target pixel are the difference between the sum of the column coordinates and the column offset of the m-th pixel and the number of columns w.

9. The defect detection method according to claim 6, wherein, When the difference between the row coordinates of the nth pixel and the row offset is greater than or equal to 0 and less than or equal to h-1, the row coordinates of the second target pixel are the difference between the row coordinates of the nth pixel and the row offset. If the difference between the row coordinates of the nth pixel and the row offset is less than 0, the row coordinates of the second target pixel are the sum of the difference between the row coordinates of the nth pixel and the row offset, and the number of rows h. If the difference between the row coordinates of the nth pixel and the row offset is greater than h-1, the row coordinates of the first target pixel are the difference between the difference between the row coordinates of the nth pixel and the row offset and the number of rows h.

10. The defect detection method according to claim 6, wherein, When the difference between the column coordinates of the nth pixel and the column offset is greater than or equal to 0 and less than or equal to w-1, the column coordinates of the second target pixel are the difference between the column coordinates of the nth pixel and the column offset. When the difference between the column coordinates of the nth pixel and the column offset is less than 0, the column coordinates of the second target pixel are the sum of the difference between the column coordinates of the nth pixel and the column offset and the number of columns w. If the sum of the column coordinates and the column offset of the nth pixel is greater than w-1, the column coordinates of the second target pixel are the difference between the difference between the column coordinates and the column offset of the nth pixel and the number of columns w.

11. The defect detection method of claim 1, wherein, Determining the period offset of the image to be detected includes: Calculate the first autocorrelation coefficient matrix of the image to be detected; Generate a mask matrix, wherein the size of the mask matrix is ​​the same as the size of the first autocorrelation coefficient matrix; The second autocorrelation coefficient matrix of the image to be detected is obtained based on the first autocorrelation coefficient matrix and the mask matrix; The period offset of the image to be detected is determined based on the second autocorrelation coefficient matrix.

12. The defect detection method of claim 11, wherein, The step of determining the period offset of the image to be detected based on the second autocorrelation coefficient matrix includes: Select the element with the largest value in the second autocorrelation coefficient matrix as the target element; If the value of the target element is greater than a predetermined threshold, the period offset is determined based on the coordinate value of the target element.

13. The defect detection method of claim 12, wherein, Determining the period offset based on the coordinate values ​​of the target element includes: The row coordinates of the target element are used as the row offset in the periodic offset; The column coordinates of the target element are used as the column offset in the period offset.

14. The defect detection method according to claim 11, wherein, The generated mask matrix includes: Set a first mask vector, wherein the length of the first mask vector is equal to the number of rows in the first autocorrelation coefficient matrix; Set a second mask vector, wherein the length of the second mask vector is equal to the number of columns of the first autocorrelation coefficient matrix; The mask matrix is ​​obtained based on the first mask vector and the second mask vector, wherein the elements in the specified region of the mask matrix have a value of 0.

15. The defect detection method according to claim 14, wherein, If the row coordinate of the kth element in the mask matrix is ​​greater than the first threshold, the value of the kth element is 0. The first threshold is the product of the number of rows in the mask matrix and the first coefficient, 1≤k≤K, where K is the total number of elements in the mask matrix. If the row coordinate of the kth element is less than or equal to the first threshold and the column coordinate of the kth element is greater than the second threshold and less than the third threshold, the value of the kth element is 0. The second threshold is the product of the specified number of columns and the first coefficient, and the third threshold is the difference between the number of columns of the mask matrix and the second threshold. The specified number of columns is half of the number of columns of the mask matrix. If the row coordinate of the kth element is less than or equal to the fourth threshold and the column coordinate of the kth element is less than or equal to the fifth threshold, the value of the kth element is 0. The fourth threshold is the product of the number of rows of the mask matrix and the second coefficient, and the fifth threshold is the product of the specified number of columns and the second coefficient, and the second coefficient is less than the first coefficient. If the row coordinate of the k-th element is less than or equal to the fourth threshold and the column coordinate of the k-th element is greater than or equal to the sixth threshold, the value of the k-th element is 0, where the sixth threshold is the difference between the number of columns of the mask matrix and the fifth threshold.

16. The defect detection method according to claim 15, further comprising: When parameter i is greater than the first threshold, the i-th element in the first mask vector takes the value of 0, where parameter i is a natural number not greater than the length of the first mask vector. When the parameter i is less than or equal to the first threshold, the value of the i-th element in the first mask vector is determined based on the difference between the length of the first mask vector and the parameter i. When parameter j is greater than the second threshold and less than the third threshold, the j-th element in the second mask vector takes the value of 0, where parameter j is a natural number not greater than the length of the second mask vector; If the parameter j is less than or equal to the second threshold, the value of the j-th element in the second mask vector is determined based on the difference between the specified column number and the parameter j. If the parameter j is greater than or equal to the third threshold, the value of the j-th element in the second mask vector is determined based on the difference between the parameter j and the specified column number.

17. The defect detection method of claim 14, wherein, The step of obtaining the mask matrix based on the first mask vector and the second mask vector includes: The mask matrix is ​​obtained by calculating the vector product of the first mask vector and the second mask vector.

18. The defect detection method of claim 11, wherein, The step of obtaining the second autocorrelation coefficient matrix of the image to be detected based on the first autocorrelation coefficient matrix and the mask matrix includes: The second autocorrelation coefficient matrix is ​​obtained by multiplying the first autocorrelation coefficient matrix and the mask matrix.

19. The defect detection method of claim 11, wherein, The calculation of the first autocorrelation coefficient matrix of the image to be detected includes: The image to be detected is expanded to obtain a target image that includes the image to be detected; Calculate the autocorrelation coefficient of the target image to obtain the first autocorrelation coefficient matrix.

20. The defect detection method of claim 19, wherein, The expansion of the image to be detected includes: Convert the image to be detected into a grayscale image; The grayscale image is blurred to obtain a blurred image; The blurred image is expanded to obtain the target image.

21. The defect detection method of claim 20, wherein, The process of expanding the blurred image includes: The edges of the blurred image are filled with pixels having multiple predetermined pixel values ​​to obtain the target image.

22. The defect detection method according to claim 19, wherein, The number of rows in the pixel matrix of the target image is twice the number of rows in the pixel matrix of the image to be detected; The number of columns in the pixel matrix of the target image is twice the number of columns in the pixel matrix of the image to be detected.

23. The defect detection method according to any one of claims 1-22, wherein, The step of merging the image to be detected and the difference fusion image to obtain a merged image includes: The image to be detected and the difference fusion image are merged by channel merging to obtain the merged image.

24. A model training method, comprising: Determine the periodic offset of a sample image, wherein the sample image comprises multiple sub-images arranged periodically; Based on the periodic offset, the sample image is cyclically offset in a first direction to obtain a first offset image, and the sample image is cyclically offset in a second direction opposite to the first direction to obtain a second offset image; A difference fusion image is generated based on the sample image, the first offset image, and the second offset image; The sample image and the difference fusion image are merged to obtain a merged image; The merged image is processed using a neural network model to obtain defect detection results; Based on the defect detection results and the annotation information of the sample images, a loss function is determined; The neural network model is trained based on the loss function.

25. An electronic device comprising: Memory; A processor, coupled to a memory, configured to implement the method as described in any one of claims 1-23 based on memory-stored instruction execution.

26. A computer-readable storage medium, wherein, A computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-23.

27. A computer program product comprising computer instructions, wherein the computer instructions, when executed by a processor, implement the method as described in any one of claims 1-23.

Images