Multi-camera fusion-based method for detecting surface defects of workpiece, and related device

Abstract

A multi-camera fusion-based method for detecting surface defects of a workpiece, applied to defect detection of a rotary workpiece made of a metal material and having a curved surface shape and high reflectivity to light. The method comprises: using a plurality of industrial cameras to perform video acquisition on the surface of a rotary workpiece so as to generate video data of the surface of the rotary workpiece (S1); decoding the video data to acquire image frames in a video (S2); cropping and stitching the acquired image frames to form an unwrapped image of the entire or partial surface of the workpiece (S3); and performing image processing analysis on the unwrapped image to detect defects on the surface of the workpiece (S4).

Description

A workpiece surface defect detection method based on multi-camera fusion and related equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is related to the Chinese Patent Application No. 202510074527.7 entitled "A workpiece surface defect detection method based on multi-camera fusion and related equipment" filed on January 17, 2025, which is incorporated by reference in its entirety into this application. TECHNICAL FIELD

[0003] The present application relates to the technical field of machine vision detection, and in particular to a workpiece surface defect detection method based on multi-camera fusion and related equipment. BACKGROUND

[0004] In the current industrial detection field, machine vision technology is widely used in product detection. It mainly uses an industrial camera to take pictures of the detection product, which is usually saved in the computer in the form of pictures. The image signal captured by the camera is first converted into a digital signal that can be processed by the computer, and then transmitted to the memory of the computer through a high-speed bus. On this basis, the machine vision software will deeply analyze these digital images to determine the quality of the product and whether it meets the preset standards, so as to achieve the purpose of image detection. Therefore, the imaging effect plays a crucial role in machine vision detection.

[0005] The inventors found that the related art at least has the following problems: when using a face array camera for imaging, if the detection product is a specific high-reflective curved metal object, the change of the surface curvature and the reflection characteristics of the metal body may cause uneven image brightness distribution, increasing the difficulty of image processing. The working principle of a line scanning camera is different from that of a common face array camera. The face array camera can obtain a complete two-dimensional image in a single exposure process, while the line scanning camera only takes a line (the width of the line is usually one pixel) on the object at a time. During the shooting process, the line scanning camera gradually combines these lines to form a complete two-dimensional image through line-by-line scanning. Although the line scanning camera has certain advantages in imaging high-reflective curved objects, it has very high requirements for the installation position in actual application. Any slight position deviation (more than 0.05 mm) may significantly affect the imaging effect.

[0006] In view of the limitations of the face array camera in processing images of high-reflective curved metal objects and the strict requirements of the line scanning camera for the installation position, the present application provides a workpiece surface defect detection method based on multi-camera fusion, which provides a new solution for workpiece detection. SUMMARY

[0007] The purpose of some embodiments of the present application is to provide a multi-camera fusion workpiece surface defect detection method, which is applied to the defect detection of a rotary body workpiece with curved surface shape, high reflectivity to light and metal material, the method comprising: using a plurality of industrial cameras to collect video of the surface of the rotary body workpiece to generate video data of the surface of the rotary body workpiece; decoding the video data to obtain each frame of image in the video; cropping and splicing the obtained frame image to form an expanded image of the overall or partial workpiece surface; and performing image processing analysis on the expanded image to detect defects on the workpiece surface.

[0008] In addition, according to the multi-camera fusion workpiece surface defect detection method provided by the present application, the step of using a plurality of industrial cameras to collect video of the surface of the rotary body workpiece to generate video data of the surface of the rotary body workpiece comprises: arranging a plurality of industrial cameras above or around the rotary body workpiece, each of the industrial cameras being configured to video a specific local position of the rotary body workpiece; starting the industrial cameras to simultaneously video each specific local position of the rotary body workpiece, ensuring that the video range of each camera covers an independent and edge-overlapping area of the surface of the rotary body workpiece; and splicing the local video data videoed by each of the industrial cameras into video data of the surface of the rotary body workpiece.

[0009] In addition, according to the multi-camera fusion workpiece surface defect detection method provided by the present application, the surface of the rotary body workpiece comprises a plurality of rotary surfaces with different diameters, and the specific local position videoed by each of the industrial cameras corresponds to a rotary surface with different diameter; the step of arranging a plurality of industrial cameras above or around the rotary body workpiece, each of the industrial cameras being configured to video a specific local position of the rotary body workpiece comprises: determining the diameters of each of the different rotary surfaces on the surface of the rotary body workpiece; setting a maximum processing frame rate for the rotary surface with the largest diameter, and setting the processing frame rate of the corresponding industrial camera according to the diameter, with the maximum processing frame rate as a reference standard, wherein the rotary surface with larger diameter corresponds to higher processing frame rate, and the rotary surface with smaller diameter corresponds to lower processing frame rate; and configuring the set different processing frame rates in the parameters of the corresponding industrial cameras, so that each of the industrial cameras can video the corresponding rotary surface on the rotary body workpiece according to the set processing frame rate.

[0010] In addition, according to the multi-camera fusion workpiece surface defect detection method provided in the application, the step of decoding the video data to obtain each frame image in the video comprises: receiving a video code stream obtained by encoding and compressing the video data, wherein the video code stream comprises parameter information of the video and compressed image data, and the parameter information comprises video parameters and an encoding format; parsing the video code stream based on the encoding format and the video parameters to obtain image data in the video frame by frame; traversing each frame image data, determining a frame image to be extracted according to a set frame extraction parameter, and the frame extraction parameter comprises a start frame, an end frame, a frame extraction interval and a frame number; and saving each selected frame image as an image file.

[0011] In addition, according to the multi-camera fusion workpiece surface defect detection method provided in the application, the step of decoding the video data to obtain each frame image in the video comprises: receiving a video code stream obtained by encoding and compressing the video data, wherein the video code stream comprises parameter information of the video and compressed image data, and the parameter information comprises video parameters and an encoding format; parsing the video code stream based on the encoding format and the video parameters to obtain image data in the video frame by frame; traversing each frame image data, determining a frame image to be extracted according to a set frame extraction parameter, and the frame extraction parameter comprises a start frame, an end frame, a frame extraction interval and a frame number; and saving each selected frame image as an image file.

[0012] In addition, according to the multi-camera fusion workpiece surface defect detection method provided in the application, the step of decoding the video data to obtain each frame image in the video comprises: receiving a video code stream obtained by encoding and compressing the video data, wherein the video code stream comprises parameter information of the video and compressed image data, and the parameter information comprises video parameters and an encoding format; parsing the video code stream based on the encoding format and the video parameters to obtain image data in the video frame by frame; traversing each frame image data, determining a frame image to be extracted according to a set frame extraction parameter, and the frame extraction parameter comprises a start frame, an end frame, a frame extraction interval and a frame number; and saving each selected frame image as an image file.

[0013] In addition, according to the multi-camera fusion workpiece surface defect detection method provided in the application, the step of decoding the video data to obtain each frame image in the video comprises: receiving a video code stream obtained by encoding and compressing the video data, wherein the video code stream comprises parameter information of the video and compressed image data, and the parameter information comprises video parameters and an encoding format; parsing the video code stream based on the encoding format and the video parameters to obtain image data in the video frame by frame; traversing each frame image data, determining a frame image to be extracted according to a set frame extraction parameter, and the frame Extraction parameter comprises a start frame, an end frame, a frame extraction interval and a frame number; saving each selected frame image as an image file.

[0014] The embodiment of the present application further provides an electronic device, which comprises one or more processors and a memory storing computer program instructions, wherein the computer program instructions, when executed, cause the processor to perform the multi-camera fusion workpiece surface defect detection method according to any one of the above.

[0015] The embodiment of the present application further provides a computer readable storage medium, which stores computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the multi-camera fusion workpiece surface defect detection method according to any one of the above.

[0016] The embodiment of the present application further provides a computer program product, which comprises computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the multi-camera fusion workpiece surface defect detection method according to any one of the above. BRIEF DESCRIPTION OF DRAWINGS

[0017] One or more embodiments are exemplified by pictures in the drawings corresponding thereto, which do not constitute a limitation on the embodiments.

[0018] Fig. 1 is a flowchart of a multi-camera fusion workpiece surface defect detection method according to an embodiment of the present application;

[0019] Fig. 2 is a size diagram of a rotary workpiece according to an embodiment of the present application;

[0020] Fig. 3 is an original picture and a corresponding surface development picture of a part of a rotary workpiece according to an embodiment of the present application;

[0021] Fig. 4 is an original picture and a corresponding surface development picture of another part of a rotary workpiece according to an embodiment of the present application;

[0022] Fig. 5 is a structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] For the purposes of the present application, the technical solutions and advantages thereof are more clearly apparent, the following will be further described in detail in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in the present application can be combined with other embodiments without conflict. Unless otherwise defined, the technical terms or scientific terms involved in the present application should be understood as the usual meaning by those of ordinary skill in the art to which the present application belongs. The terms "one", "a", "an", "the", and similar words involved in the present application do not represent quantity limitation, and can represent singular or plural. The terms "include", "contain", "have", and any variations thereof involved in the present application are intended to cover non-exclusive inclusion; The terms "first", "second", "third", and the like involved in the present application are only to distinguish similar objects, and do not represent a specific order of the objects.

[0024] First of all, it should be pointed out that the surface of the metal material made of the rotary body workpiece has strong reflectivity, which is easy to cause reflection and glare phenomenon when the area array camera is shooting, thereby affecting the clarity and contrast of the image. Moreover, the change of the surface curvature and the reflection characteristics of the metal body will cause the uneven distribution of image brightness, which increases the difficulty of image processing. Therefore, when the area array camera processes the ordinary image of the high-reflective curved surface metal body, the imaging quality is limited, the measurement accuracy is limited, the applicable scene is limited, and the cost and technical difficulty are limited. Therefore, the present application provides a workpiece surface defect detection method and related equipment based on multi-camera fusion, which takes corresponding measures to optimize the image acquisition and processing process according to these characteristics, so as to improve the accuracy and efficiency of defect detection.

[0025] Some embodiments of the present application relate to a workpiece surface defect detection method based on multi-camera fusion, referring to Fig. 1, the method is applied to the defect detection of a rotary body workpiece made of metal material with curved surface shape and high reflectivity to light, the method comprises the following steps: step S1: adopting several industrial cameras to collect video of the surface of the rotary body workpiece to generate video data of the surface of the rotary body workpiece.

[0026] In this embodiment, step S1 specifically comprises: step S101: arranging several industrial cameras above or around the rotary body workpiece, each industrial camera is configured to shoot video of a specific local position of the rotary body workpiece.

[0027] For example, as shown in the design drawing of the rotary workpiece in FIG. 2, the rotary workpiece is in a typical rotary shape, continuously rotating around a fixed axis. The rotary workpiece has multiple rotary surfaces with different diameters. That is, the peripheral surface of the workpiece does not have a uniform diameter, but rather exhibits different diameter sizes at different positions according to design requirements. At one end of the workpiece (e.g., the left end), there is a rotary surface with a larger diameter, and as the peripheral surface extends to the other end (e.g., the right end), the diameter gradually decreases. At the other end of the workpiece, there is a rotary surface with a smaller diameter.

[0028] Since the rotary workpiece has multiple rotary surfaces with different diameters, the shooting height of the industrial camera needs to be adjusted according to the different diameters. When the camera shoots a rotary surface with a larger diameter, in order to completely capture the image of the area, the camera needs to be adjusted to a higher shooting height to expand the field of view. Conversely, when shooting a rotary surface with a smaller diameter, the camera can appropriately lower the shooting height.

[0029] In this embodiment, step S101 specifically includes: step S1011, determining the diameters of each different rotary surface on the surface of the rotary workpiece; step S1012, setting a maximum processing frame rate for the rotary surface with the largest diameter, and setting the processing frame rate of the corresponding industrial camera according to the diameter size based on the maximum processing frame rate as a reference standard, wherein the rotary surface with a larger diameter corresponds to a higher processing frame rate, and the rotary surface with a smaller diameter corresponds to a lower processing frame rate; and step S1013, configuring the different processing frame rates set in the parameters of the corresponding industrial cameras, so that each industrial camera can shoot the corresponding rotary surface on the rotary workpiece according to the set processing frame rate.

[0030] Specifically for step S101, the different diameter sizes of each rotary surface are determined according to the diameter sizes of each rotary surface marked on the design drawing. After determining the diameters of each rotary surface, a suitable maximum processing frame rate needs to be selected. This frame rate is usually determined based on the shooting requirements of the rotary surface with the largest diameter to ensure that the image of the rotary surface can be clearly and continuously captured. The processing frame rate of the corresponding industrial camera is set in proportion to the diameter size of the other rotary surfaces based on the maximum processing frame rate as a reference standard.

[0031] For example, three to four cameras are used to take images of the outer surface of the rotary workpiece shown in FIG. 2, and the imaged pictures are cropped to 1280x30 pixels. This cropping operation significantly improves the processing frame rate of the camera, which can be increased from the standard 216.25 fps to 1000 fps.

[0032] In this working environment, the field of view of the camera is set to 96mm x 76.8mm. Based on this field of view and the cropped pixel size, the calculated single-pixel accuracy is: 96mm / 1280pixel = 0.075mm / pixel, which means that each pixel in the image corresponds to 0.075mm on the actual object.

[0033] To capture video, the camera is then adjusted to video capture mode. Taking the first camera as an example, the camera captures the diameter of the product corresponding to φ87.8mm (approximately φ88mm for calculation, as shown in Figure 2 at mark ①), and the circumference is calculated by the formula π x D, which is 276.32mm. At this time, under the effect of cropping, the video capture speed of the camera is: 1000fps x 0.075mm / pixel = 75mm / s.

[0034] To ensure that the captured video is synchronized with the product rolling, the rotational speed of the outer surface needs to be accurately set. This rotational speed is obtained by multiplying the capture speed 75mm / s by 60 (converting the speed unit from mm / s to mm / min), and then dividing by the circumference of the product 276.32mm. The calculated rotational speed of the outer surface is: 60 x capture speed / circumference = 60 x 75 / 276.32 = 16.285r / min (minutes).

[0035] After setting the above rotational speed, the video recording time can be further determined. This time is obtained by dividing the circumference of the product 276.32mm by the capture speed 75mm / s, and the result is 276.32 / 75 = 3.684 seconds. According to the setting of the first camera, the frame rate settings of the other cameras can be deduced (the frame rate setting can be set in the camera software or by using a programming interface to call the camera configuration interface to complete the setting), to ensure that all cameras capture at the same rotational speed, thereby ensuring the synchronization and accuracy of the entire system. According to the half radius of the revolving surface at mark ① in Figure 2, which is 44mm, and the half radius of the revolving surface at mark ②, which is 22mm, the camera recording parameters are shown in Table 1.

[0036] Table 1 Recording parameters of different diameter revolving surfaces

[0037] In this embodiment, step S102: start the industrial camera to simultaneously capture video of each specific local position of the revolving body workpiece, ensuring that the camera's shooting range covers an independent and edge-overlapping area on the surface of the revolving body workpiece.

[0038] For step S102, specifically, each camera is equipped with a corresponding lens for capturing different areas of the outer surface of the workpiece. The light source provides the necessary illumination to ensure clear images. The workpiece is clamped on a rolling mechanism, and a belt drives the product to rotate. As the product rotates, its outer surface passes through the camera's shooting area one by one. Multiple cameras are installed in fixed positions, with lenses aimed at the rotating workpiece. When the workpiece rotates into the camera's shooting range, the camera captures the image of that area. Since there are multiple cameras, they can capture different parts of the workpiece's outer surface respectively. According to the time the camera needs to record, adjust the speed of the rolling belt to accurately control the rotation speed of the workpiece surface. The camera shooting can be triggered synchronously according to the movement of the belt to ensure that the workpiece surface is shot at the best position. The position and intensity of the light source can be adjusted according to the shooting needs to ensure that clear images can be obtained at different shooting angles. This helps to reduce shadows and reflections, improve image contrast and detail performance.

[0039] It should be noted that the shooting range of each camera needs to be carefully designed to ensure that they cover independent areas of the workpiece surface, and there is a certain edge overlap between these areas. The edge overlap is to achieve seamless connection in the subsequent video data splicing process, avoiding information omission or misplacement.

[0040] For example, the industrial camera can be a high-frame-rate industrial camera with a frame rate of up to 1000 fps. The industrial camera uses USB3.0 transmission speed, which is significantly improved compared to USB2.0 and GigE cameras (Gigabit Ethernet, Gigabit Ethernet industrial camera). With a transmission bandwidth of 5Gbps, combined with a high-speed sensor, under the shooting conditions of 300,000 pixels and 800 frames per second (FPS), the data transmission time can be as low as 1.2 milliseconds. Even under the high-resolution camera of 20 million pixels, it can achieve a transmission speed of up to 18 frames per second. Benefiting from the abundant transmission bandwidth, USB3.0 technology enables a computer to maintain high frame rates while connecting multiple cameras, which undoubtedly improves the flexibility and scalability of the system. Moreover, it performs well in CPU occupancy, which uses direct memory access (DMA) transmission method, meaning that the data transmission process almost does not occupy CPU resources, thus ensuring the efficient operation of the system and the rational use of resources.

[0041] In this embodiment, step S103: splicing the local video data captured by each industrial camera into video data of the workpiece surface.

[0042] For step S103, specifically, it is the key step of multi-camera fusion. Through image processing algorithms, the local video data captured by each camera is spliced to form a video data that can cover the entire or partial surface of the workpiece. In this process, the information of the edge overlap part is used to calibrate and adjust the position between different video frames, ensuring that the spliced video data is continuous in time and space. Before video splicing, the camera needs to be calibrated to obtain the internal and external parameters of the camera. These parameters are used for subsequent image registration and fusion. Ensure that the video streams from different cameras are synchronized in time to avoid misalignment or cracking after splicing.

[0043] For example, video streams are obtained from each camera, each frame of image is preprocessed, including denoising, size adjustment, etc., feature detection algorithms (such as SIFT, SURF or ORB) are used to extract key points and descriptors in each frame of image, and then the best matching points are found out through feature matching algorithms (such as FLANN or BFMatcher). The homography matrix is calculated based on the matching points, and the images captured by different cameras are aligned to the same coordinate system through perspective transformation of the homography matrix. The aligned images are fused to generate a seamless spliced video picture. During video splicing, multi-band fusion, image smoothing and other technologies are generally involved to reduce the visibility of the joint. Therefore, through the above steps and the application of key algorithms, the local video data captured by each camera can be spliced to form a video data that covers the entire or partial surface of the workpiece.

[0044] In this embodiment, step S2: decoding the video data to obtain each frame of image in the video.

[0045] In this embodiment, step S2 specifically includes: step S201: receiving a video code stream after encoding and compression of the video data, the video code stream containing parameter information and compressed image data of the video, and the parameter information including video parameters and encoding format.

[0046] For step S201, specifically, the video code stream is the video data after encoding and compression, which contains the parameter information and actual image data of the video. The parameter information is crucial for the subsequent decoding process, which tells the decoder how to correctly parse and restore the video data. These parameter information usually includes video parameters (such as resolution, frame rate, etc.) and encoding format (such as H.264, H.265, etc.).

[0047] In this embodiment, step S202: based on the encoding format and video parameters, the video code stream is parsed, and the image data in the video is obtained frame by frame.

[0048] In this embodiment, step S202 specifically includes: using the transform coding technology in the video encoding process to perform inverse quantization and inverse transform processing on the image frame data in the video code stream in the video decoding process, mapping the quantized coefficients back to the original or close-to-original transform coefficient values, and converting the transform coefficient values back to the spatial domain to restore the original uncompressed image data; at the same time, using the prediction coding technology in the video encoding process to perform intra-frame prediction on the correlation between adjacent pixels in the same frame image, and inter-frame prediction on the motion information between adjacent frame images, to use the inverse process of the prediction coding technology to reconstruct the image data; frame by frame, the video decoding process is performed to obtain and reconstruct the image data in the video until the entire video code stream decoding is completed.

[0049] Specifically for step S202, in the video encoding process, the transform coding technology transforms the pixel values in the image or video frame, such as discrete cosine transform (DCT) or discrete wavelet transform (DWT) transform coding technology, and quantizes the transformed coefficients to reduce the data volume. In the decoding process, the quantized coefficients first need to be inverse quantized, i.e., the quantized coefficient values are mapped back to the original or close-to-original transform coefficient values. The transform coefficient values obtained after inverse quantization are still in the transform domain (such as the frequency domain), and need to be converted back to the spatial domain (i.e., the pixel domain) through inverse transform processing to restore the original uncompressed image data. The inverse transform processing usually corresponds to the transform algorithm used in the encoding process, such as inverse discrete cosine transform, inverse discrete wavelet transform, etc.

[0050] In order to remove the temporal and spatial redundancy in the video, the video encoding uses prediction coding technology. In the video decoding process, the prediction coding technology in the video encoding process is used for intra-frame prediction and inter-frame prediction. For intra-frame prediction, since there is usually strong correlation between adjacent pixels within the same frame image, the decoder will use this correlation for intra-frame prediction, i.e., predicting the current pixel value according to the decoded adjacent pixel values. By calculating the residual between the predicted value and the true value, and decoding the residual, the accurate value of the current pixel can be obtained. For inter-frame prediction, due to the temporal continuity of the video sequence, there is motion information between adjacent frame images. The decoder will use inter-frame prediction technology to predict the pixel value of the current frame by finding the motion vector between adjacent frames. This usually involves the process of motion estimation and compensation, i.e., generating a predicted image of the current frame according to the decoded adjacent frame image and the motion vector, and then decoding the residual to obtain the accurate image of the current frame.

[0051] The decoder performs the above decoding process frame by frame in the order of frames in the video code stream. For each frame of image data, the inverse quantization, inverse transformation, intra-frame prediction, and inter-frame prediction are sequentially performed to restore the original image data without compression or loss, so that more detailed information and details are retained, and the image is clearer and more realistic. Using the original image data for defect detection can reduce the influence of noise and distortion introduced in the image compression or processing process on the detection result.

[0052] In this embodiment, step S203: traverse each frame of image data, and determine the frame images to be extracted according to the set frame extraction parameters, including the start frame, the end frame, the frame extraction interval, and the frame number.

[0053] Step S204: save each selected frame image as an image file.

[0054] The decoder determines the frame images to be extracted according to the set frame extraction parameters. The frame extraction parameters include the start frame, the end frame, the frame extraction interval, and the frame number. By traversing each frame of image data and screening according to these parameters, the decoder can accurately extract the frame images required by the user. Finally, the decoder saves each selected frame image as an image file. These image files are used for subsequent splicing processing. The saving format can be selected according to user requirements, such as JPEG, PNG, etc.

[0055] In this embodiment, step S3: crop and splice the obtained frame images to form an expanded image of the overall or partial workpiece surface.

[0056] In this embodiment, step S3 specifically includes: step S301: according to the part of the workpiece that needs to be detected for defects, crop the obtained frame images according to the preset cropping parameters; step S302: splice the cropped frame images according to the predetermined splicing order and rules to form an expanded image of the overall or partial workpiece surface; and step S303: save the spliced expanded image of the workpiece surface as an image file.

[0057] Specifically, the part of the workpiece that needs to be detected for defects can be determined first, such as the entire workpiece itself or a certain local position of the workpiece. According to the determined part of the workpiece, the corresponding cropping parameters are preset, such as the size, shape, and position of the cropping area. The obtained frame images can be operated and batch-processed using an image processing tool.

[0058] Before stitching, the stitching order and rules between the cropped multiple frame images need to be determined. If the frame images are extracted in the time order of video recording, the stitching should also be done in this time order to maintain the continuity of the workpiece surface changes. The stitching rules are usually determined according to the shape, texture, defect features of the workpiece surface and detection requirements. If the workpiece surface is flat, the stitching rules may focus on ensuring the accurate alignment of adjacent images in the horizontal or vertical direction. For curved workpieces, the stitching rules need to consider the perspective deformation and curved fitting of the images to ensure that the stitched images remain visually coherent and accurate. The consistency and repeatability of the texture can help determine the overlapping area between adjacent images and serve as a reference for stitching. Defect features are also one of the key factors to be considered in the stitching rules, for example, if the defect appears as a small crack or scratch, the stitching rules should avoid excessive smoothing or image fusion in the defect area.

[0059] According to the needs, further processing and adjustment can be performed on the stitched images, such as color correction, brightness adjustment, contrast enhancement, etc., to improve the image quality and detection effect. After completing the stitching and adjustment, the final workpiece surface unwrapped image is saved as an image file. According to the image resolution and quality parameters, the appropriate image format is selected for saving. The unwrapped image of the workpiece surface after stitching is shown in FIG. 3 and FIG. 4. Among them, FIG. a shows the original image of a part of a rotary workpiece, and FIG. b shows the two-dimensional view of the surface unwrapped after the part.

[0060] As can be seen from FIG. 3 and FIG. 4, the images generated by a series of algorithms have uniform illumination and clear defect contrast, providing reliable guarantee for defect detection and identification.

[0061] In this embodiment, step S4: image processing analysis is performed on the unwrapped image to detect defects on the workpiece surface. The color image can be converted to a grayscale image to simplify the calculation and highlight the defect features in the image. The image segmentation algorithm is used to set an appropriate threshold value to segment the grayscale difference between the defects and the background in the image, and separate the defects from the background. Based on the extracted features, a classifier (such as support vector machine, neural network, etc.) is used to detect and identify the defects in the image. According to the characteristics and types of the defects, they can be classified into different defect categories, such as cracks, scratches, pits, etc. Therefore, by performing image processing analysis on the unwrapped image, the defects on the workpiece surface are automatically detected. This replaces the traditional manual detection method, improves the detection efficiency and accuracy, and reduces the detection cost.

[0062] In this embodiment, the local video data of the plurality of industrial cameras is spliced in real time to form video data of the surface of the rotary workpiece by using the multi-core CPU and the parallel processing technology of the CUDA architecture, and the video data is decoded and spliced to form the expanded image of the surface of the workpiece.

[0063] Specifically, the multi-core CPU can process multiple tasks at the same time, and the processing speed is improved through parallel computing. CUDA (Compute Unified Device Architecture) is a parallel computing architecture launched by NVIDIA, which greatly improves the computing performance by utilizing the processing power of a graphics processing unit (GPU). By combining the multi-core CPU and the parallel processing technology of the CUDA architecture, the video fusion and image processing algorithm of multiple stations can be highly parallelized, and the image splicing speed is significantly accelerated through parallel computing, so as to realize real-time image splicing of the local video data of the plurality of industrial cameras, and provide an efficient and accurate solution for quality detection of the surface of the rotary workpiece.

[0064] In summary, the workpiece surface defect detection method of the multi-camera fusion provided in the embodiments of the present application is applied to defect detection of a rotary workpiece with a curved surface shape, high reflectivity to light, and made of metal material. The method comprises: using a plurality of industrial cameras to collect video of the surface of the rotary workpiece to generate video data of the surface of the rotary workpiece; decoding the video data to obtain each frame of image in the video; cropping and splicing the obtained frame of image to form an expanded image of the overall or local surface of the workpiece; and performing image processing analysis on the expanded image to detect defects on the surface of the workpiece.

[0065] It can be found that, compared with the prior art, the video of the surface of the rotary workpiece is collected by combining the flexible configuration of the plurality of industrial cameras, which can ensure that the image information of the surface of the workpiece is obtained from multiple angles and positions. The high-speed video collection technology is adopted to realize the optical flexibility that cannot be matched by ordinary line-scan cameras. This configuration not only adapts to various complex industrial scenes, but also has a large depth of field feature, which can ensure that clear images can be obtained at different working distances. The outside wall detection of the metal rotary workpiece with a curved surface and high reflectivity has strong universality. By configuring the number of stations and adjusting the focal length, the surface defect detection of rotary workpieces of any length can be easily realized.

[0066] Furthermore, some embodiments of the present application provide an electronic device. The electronic device can be various forms of digital computers, such as a laptop computer, a desktop computer, a workstation, a personal digital assistant, a server, a blade server, a mainframe computer, and so on. The electronic device can also be various forms of mobile devices, such as a personal digital processing, a cellular phone, a smart phone, a wearable device, and other similar computing devices.

[0067] The electronic device includes one or more processors, and a memory storing computer program instructions that, when executed, cause the processors to perform the multi-camera fusion based workpiece surface defect detection method provided by any one or more embodiments described above. FIG. 5 discloses an exemplary structural diagram of the electronic device. As shown in FIG. 5, the electronic device includes one or more processors 1101, a memory 1102, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other by different buses, and can be mounted on a common motherboard or otherwise mounted as desired. The processor can process instructions executed within the electronic device, including instructions stored in the memory or on the memory to display a GUI on an external input / output device, such as a display device coupled to the interface. In some other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple storage devices, if desired. Also, multiple electronic devices can be connected, each device providing part of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). Herein, the components shown, their connections and relationships, and their functions, are merely examples, and are not intended to limit the implementation of the present application described and / or claimed herein.

[0068] The electronic device can also include an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103, and the output device 1104 can be connected by a bus or otherwise, and are connected by a bus in FIG. 5 as an example.

[0069] The input device 1103 can receive input digital or character information, and generate key signal input related to user settings and function controls of the electronic device, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, and the like input device. The output device 1104 can include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor), and the like. The display device can include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display. In some embodiments, the display device can be a touch screen.

[0070] To provide for interaction with a user, the electronic device can be a computer. The computer has a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0071] In some embodiments of the present application, a computer program / instruction is stored on a computer readable medium, and the computer program / instruction is executed by a processor to implement the multi-camera fusion workpiece surface defect detection method provided by any one or more of the above embodiments. The computer readable medium can be included in the electronic device described in the above embodiments; or can exist separately and not be assembled into the device. The computer readable medium carries one or more computer readable instructions.

[0072] The memory 1102 can be used to store non-transitory software programs, non-transitory computer executable programs and modules. The processor 1101 executes various functions and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 1102, so as to implement the program instructions / modules corresponding to the method provided by any one or more of the above embodiments in the embodiments of the present application.

[0073] The memory 1102 can include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required by a function; and the data storage area can store data created according to the use of the electronic device. In addition, the memory 1102 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory 1102 can optionally include a memory disposed remotely with respect to the processor 1101, and these remote memories can be connected to the electronic device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0074] It should be noted that more specific examples of the computer- readable storage medium can include but are not limited to an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer- readable storage medium can be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0075] Computer-readable storage media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device.

[0076] Computer program code for carrying out operations of the present application can be written in one or more programming languages or combinations of languages including object-oriented, such as Java, Smalltalk, C++, conventional procedural programming languages, such as the "C" programming language or similar programming languages. Program code can be executed entirely on a user computer, partially on a user computer, as a standalone software package, partially on a user computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0077] In the above-described embodiments, all or a part thereof can be realized by software, hardware, firmware, or any combination thereof. For example, an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device can be used. In some embodiments, a software program of the present application can be executed by a processor to implement the above steps or functions. Likewise, a software program of the present application (including related data structures) can be stored in a computer-readable recording medium, such as a RAM memory, a magnetic or optical drive, or a floppy disk, and the like. In addition, some steps or functions of the present application can be implemented by hardware, such as a circuit that cooperates with a processor to perform the respective steps or functions.

[0078] Some embodiments of the present application provide a computer program product including one or more computer programs / instructions that, when executed by a processor, wholly or partially implement the processes or functions described in the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server, or data center to another via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk, SSD), etc.

[0079] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0080] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A multi-camera fusion based workpiece surface defect detection method, wherein, The method is applied to defect detection of a rotary workpiece with a curved surface shape, high light reflectivity and a metal material, and comprises the following steps: a plurality of industrial cameras are used to collect videos of the surface of the rotary workpiece to generate video data of the surface of the rotary workpiece; the video data is decoded to obtain each frame image in the video; the obtained frame image is cropped and spliced to form an expanded image of the whole or partial workpiece surface; and the expanded image is subjected to image processing analysis to detect defects on the workpiece surface.

2. The method of claim 1, wherein, The step of collecting videos of the surface of the rotary workpiece by using a plurality of industrial cameras to generate video data of the surface of the rotary workpiece comprises the following steps: a plurality of industrial cameras are arranged above or around the rotary workpiece, each of the industrial cameras is configured to shoot videos of a specific local position of the rotary workpiece; the industrial cameras are started to simultaneously shoot videos of each specific local position of the rotary workpiece, so that the shooting range of each camera covers an independent and edge-overlapping area on the surface of the rotary workpiece; and the local video data shot by each industrial camera is spliced into video data of the surface of the rotary workpiece.

3. The method of claim 2, wherein, The surface of the rotary workpiece comprises a plurality of rotary surfaces with different diameters, and the specific local position shot by each industrial camera corresponds to a rotary surface with a different diameter; the step of arranging a plurality of industrial cameras above or around the rotary workpiece, each of which is configured to shoot videos of a specific local position of the rotary workpiece, comprises the following steps: the diameters of each rotary surface on the surface of the rotary workpiece are determined; the maximum processing frame rate is set for the rotary surface with the largest diameter, and the processing frame rate of the corresponding industrial camera is set according to the diameter, wherein the rotary surface with a larger diameter corresponds to a higher processing frame rate, and the rotary surface with a smaller diameter corresponds to a lower processing frame rate; and the set different processing frame rates are configured in the parameters of the corresponding industrial cameras, so that each industrial camera can shoot the corresponding rotary surface on the rotary workpiece according to the set processing frame rate.

4. The method of claim 1, wherein, The step of decoding the video data to obtain each frame image in the video comprises the following steps: a video bitstream obtained by encoding and compressing the video data is received, the video bitstream contains parameter information of the video and compressed image data, and the parameter information comprises video parameters and an encoding format; the video bitstream is parsed based on the encoding format and the video parameters to obtain image data in the video frame by frame; each frame of image data is traversed, and a frame image to be extracted is determined according to a set frame extraction parameter, the frame extraction parameter comprises a start frame, an end frame, a frame extraction interval and a frame number; and each frame image selected is saved as an image file separately.

5. The method of claim 4, wherein, The step of parsing the video code stream based on the encoding format and the video parameters and obtaining image data in the video frame by frame comprises: using transform coding technology in the video encoding process to perform inverse quantization and inverse transform processing on image frame data in the video code stream in the video decoding process, mapping the quantized coefficients back to original or nearly original transform coefficient values, and converting the transform coefficient values back to the spatial domain to restore the original image data that has not been compressed; at the same time, in the video decoding process, using prediction coding technology in the video encoding process to perform intra-frame prediction on the correlation between adjacent pixels in the same frame image and inter-frame prediction on motion information between adjacent frame images to perform image data reconstruction using the inverse process of the prediction coding technology; performing video decoding processing frame by frame to obtain and reconstruct image data in the video until the entire video code stream decoding is completed.

6. The method of claim 1, wherein, The step of cropping and splicing the obtained frame images to form the developed image of the overall or partial workpiece surface comprises: cropping the obtained frame images according to the preset cropping parameters according to the workpiece part that needs to be detected; splicing the cropped multiple frame images according to the predetermined splicing order and rule to form the developed image of the overall or partial workpiece surface; and saving the spliced developed image of the workpiece surface as an image file.

7. The method of claim 1, wherein, Further comprising: using multi-core CPU and CUDA architecture parallel processing technology to perform real-time image splicing on the local video data of the multiple industrial cameras to form the video data of the surface of the workpiece of revolution, and to decode and splice the video data to form the developed image of the workpiece surface.

8. An electronic device, comprising: The electronic device comprises: one or more processors; and a memory storing computer program instructions which, when executed, cause the processor to perform the multi-camera fused workpiece surface defect detection method of any one of claims 1-7.

9. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions, when executed by the processor, implement the multi-camera fused workpiece surface defect detection method of any one of claims 1-7.

10. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions, when executed by the processor, implement the multi-camera fused workpiece surface defect detection method of any one of claims 1-7.

Images