Multimodal-based steel coil surface defect detection method and apparatus, device, and medium

Abstract

A multimodal-based steel coil surface defect detection method and apparatus, a device, and a medium. The method comprises: acquiring input images of a steel coil surface and task descriptions corresponding to the input images; performing encoding conversion on the input images and the task descriptions, so as to determine discrete sequences; inputting the discrete sequences into an encoder-decoder model to obtain task output sequences; and performing inverse quantization inference on the basis of the task output sequences to determine bounding boxes, masks, and defect description information of the input images, so as to complete steel coil surface defect detection. In the present application, samples of different defect types can be quickly acquired by means of sample pairs formed by the input images and the task descriptions, thereby facilitating rapid iteration of model training. In addition, by means of a multi-modal image set, the bounding boxes, the masks, and the defect description information are directly outputted, thereby also improving the detection accuracy and the range of detection application of steel coil surface defects.

Description

Methods, devices, equipment, and media for detecting surface defects in steel coils based on multimodal methods. Technical Field

[0001] This invention relates to the field of steel coil inspection technology, and in particular to a method, apparatus, equipment and medium for detecting surface defects in steel coils based on multimodal methods. Background Technology

[0002] In modern industrial manufacturing, especially in precision manufacturing and automated production lines, the control of product quality is becoming increasingly stringent. To ensure high-quality product standards, manufacturers need to implement efficient defect detection during the production process.

[0003] However, in related technologies, steel coils are often visually inspected based on image data of a single modality, such as visible light images or infrared images. Such methods are difficult to fully capture the detailed information of product defects, which can easily lead to problems such as insufficient defect recognition accuracy, poor adaptability to complex defect types, and weak stability under different environmental conditions. Summary of the Invention

[0004] This invention provides a method, apparatus, equipment, and medium for detecting surface defects in steel coils based on multimodal methods, in order to solve the technical problems of low detection accuracy and limited detection types in steel coils.

[0005] In a first aspect, this application provides a multimodal steel coil surface defect detection method, comprising: acquiring an input image containing the surface of the steel coil and a task description corresponding to the input image; encoding and converting the input image and the task description to determine a discrete sequence, wherein the task descriptions for different task types employ different encoding methods; inputting the discrete sequence into an encoder-decoder model to obtain a task output sequence; and performing inverse quantization inference based on the task output sequence to determine the detection box, mask, and defect description information of the input image, thereby completing the steel coil surface defect detection.

[0006] In some embodiments of the first aspect, the task descriptions for different task types employ different encoding methods, including: if the task description is an object detection task, then by quantizing continuous image coordinates in the input image, the bounding box of the object is obtained, and the bounding box and the object description are converted into a series of discrete sequences; if the task description is an instance segmentation task, then the polygons corresponding to the instance mask in the input image are predicted in the form of an image coordinate sequence, and the coordinates corresponding to the polygons are quantized into discrete sequences; if the task description is an image description task, then the task description is converted into a text sequence.

[0007] In some embodiments of the first aspect, the method for determining the encoder-decoder model includes:

[0008] A dataset consisting of multiple sample pairs is obtained, each sample pair including the input image and the corresponding task description; the dataset is divided into a training set, a test set, and a validation set, and the encoder-decoder model is a Pix2SeqV2 model; the Pix2SeqV2 model is trained using the training set, and the initial parameters are adjusted through backpropagation to obtain a first encoder-decoder model; the hyperparameters of the first encoder-decoder model are adjusted using the validation set to obtain a second encoder-decoder model; the performance of the second encoder-decoder model is evaluated using the test set, and a third encoder-decoder model with satisfactory generalization performance is output, and the third encoder-decoder model is used as the final encoder-decoder model.

[0009] In some embodiments of the first aspect, the encoder-decoder model includes an encoder, which is based on a Transformer architecture and / or a ConvNeXt architecture; a Head for receiving feature maps from the backbone network or other layers and converting them into task outputs, predicting the output class labels, bounding boxes, and segmentation masks; a Neck, which is an intermediate layer connecting the backbone network and the Head; and a decoder, conditional on a prompt, for generating an output sequence for a single object detection task, wherein during training, the prompt and the desired output are concatenated into a single sequence, and sequence weighting is used to ensure that the decoder is only trained to predict the desired output.

[0010] In some embodiments of the first aspect, the prompt is given and used to guide the model to process and generate input instructions containing different types of data, including images, text, and speech.

[0011] In some embodiments of the first aspect, inverse quantization inference is performed based on the task output sequence to determine the detection box, mask, and defect description information of the input image, including:

[0012] For object detection tasks, the predicted task output sequence is decomposed into multiple token tuples to obtain coordinate tokens and category tokens, and the coordinate tokens are dequantized to obtain detection boxes; for instance segmentation tasks, the coordinate tokens corresponding to each polygon are dequantized and converted into dense masks, and the average of the masks is calculated to obtain a binary mask; for image description tasks, the predicted category tokens are mapped to text to form defect description information.

[0013] In some embodiments of the first aspect, the method further includes: mapping the features of different modalities to a preset feature domain through feature transformation of the image sequence corresponding to the input image and the text sequence corresponding to the task description to form fused features; constructing a defect detection network based on the fused features, and training the defect detection network with a multimodal dataset composed of the fused features represented by the input image and the task description, wherein the defect detection network is used to simultaneously perform target detection, instance segmentation and defect description; and using the trained defect detection network to detect the acquired image to be tested, so as to output the detection box, mask and defect description information of the image to be tested, wherein the image to be tested is the input image.

[0014] In a second aspect, this application provides a multimodal steel coil surface defect detection device, comprising: an acquisition module for acquiring an input image containing the surface of the steel coil and a task description corresponding to the input image; an encoding conversion module for encoding and converting the input image and the task description to determine a discrete sequence, wherein the task descriptions for different task types employ different encoding methods; a task output module for inputting the discrete sequence into an encoder-decoder model to obtain a task output sequence; and a defect detection module for performing inverse quantization inference based on the task output sequence to determine the detection box, mask, and defect description information of the input image, thereby completing the steel coil surface defect detection.

[0015] In a third aspect, this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described multimodal steel coil surface defect detection method.

[0016] In a fourth aspect, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described multimodal steel coil surface defect detection method.

[0017] In the above-mentioned scheme implemented by the multimodal steel coil surface defect detection method, apparatus, equipment, and medium, the method acquires a sample pair formed by an input image containing the steel coil surface and a corresponding task description; encodes and converts the multimodal sample pair to determine a discrete sequence; inputs the discrete sequence into an encoder-decoder model to obtain a task output sequence, wherein the encoder-decoder model is a Pix2SeqV2 model; and performs inverse quantization inference based on the task output sequence to determine the detection box, mask, and defect description information of the input image. On the one hand, this application not only improves the performance of industrial image processing systems but also enhances the automatic detection capability of steel coil surface defects. On the other hand, by using the sample pair formed by the input image and the task description, samples of different defect types can be quickly acquired, which is beneficial for rapid iteration of model training. At the same time, by directly outputting the detection box, mask, and defect description information through a multimodal image set, the detection accuracy and applicability of steel coil surface defects are also improved. Attached Figure Description

[0018] Figure 1 is a flowchart of a multimodal steel coil surface defect detection method according to an embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of a multimodal steel coil surface defect detection device according to an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of the processing structure of a multimodal steel coil surface defect detection device in one embodiment of the present invention;

[0021] Figure 4 is a structural schematic diagram of a computer device according to an embodiment of the present invention;

[0022] Figure 5 is another structural schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] The present invention will now be described in detail through specific embodiments. Please refer to Figure 1, which is a flowchart illustrating a multimodal steel coil surface defect detection method provided by an embodiment of the present invention, including the following steps:

[0025] Step S101: Obtain an input image containing the surface of the steel coil and a task description corresponding to the input image;

[0026] Specifically, a high-resolution camera or image acquisition device is used to acquire a clear image of the steel coil surface. Based on the detection requirements, a corresponding task description is generated for each input image, such as "detecting cracks on the surface of the steel coil" or "identifying rust on the surface of the steel coil".

[0027] In this system, there is a one-to-one correspondence between the input images and the task descriptions, forming sample training pairs. For example, multiple images of the surface of multiple steel coils can be acquired simultaneously to improve detection efficiency.

[0028] Step S102: Encode and convert the input image and task description to determine the discrete sequence, wherein different encoding methods are used for task descriptions of different task types;

[0029] Specifically, the input image is preprocessed, such as through normalization and noise reduction, to improve image quality. Based on the type of task description, an appropriate encoding method is selected to convert the image and task description into a discrete sequence. For example, for object detection tasks, specific encoding rules can be used to encode the target object into a discrete sequence. It should be noted that tasks include, but are not limited to, object detection, instance segmentation, and image description tasks; different tasks require different encoding methods for conversion.

[0030] Step S103: Input the discrete sequence into the encoder-decoder model to obtain the task output sequence. For example, the encoder-decoder model is the Pix2SeqV2 model.

[0031] Specifically, the Pix2SeqV2 model can transform an input discrete sequence into an output sequence, which includes key information such as bounding boxes, masks, and defect descriptions. The encoder of the Pix2SeqV2 model is responsible for extracting image features, and the decoder is responsible for generating the output sequence based on the features. For example, by choosing the Pix2SeqV2 model as the encoder-decoder model, the discrete sequence is input into the model, and the encoder and decoder process it to obtain the task's output sequence.

[0032] In addition, this application can also employ other advanced encoder-decoder models, such as Faster R-CNN and YOLO, to adapt to different detection tasks.

[0033] Step S104: Perform inverse quantization inference based on the task output sequence to determine the detection box, mask, and defect description information of the input image in order to complete the surface defect detection of the steel coil.

[0034] Specifically, dequantization inference can convert the task output sequence back to its original numerical form, thereby extracting key information; bounding boxes are used to locate defect positions, masks are used to identify defect regions, and defect description information is used to describe the characteristics and type of defects. For example, by dequantizing the task output sequence and converting it back to its original numerical form, the bounding boxes, masks, and defect description information of the input image can be determined based on the dequantization result.

[0035] Through the above methods, this application reduces manual intervention and time costs by automating the image acquisition and processing process; it employs the Pix2SeqV2 model and inverse quantization inference technology to accurately identify and describe defects on the surface of steel coils; it also supports multiple task types and encoding methods to adapt to different detection needs and scenarios; furthermore, it provides an intelligent solution for steel coil surface defect detection, promoting technological progress and industrial upgrading in related industries.

[0036] In some embodiments, different encoding methods are used for task descriptions of different task types, including:

[0037] If the task description is an object detection task, then the bounding box of the object is obtained by quantizing the continuous image coordinates in the input image, and the bounding box and the object description are converted into a series of discrete sequences.

[0038] If the task description is an instance segmentation task, then predict the polygons in the input image corresponding to the instance mask in the form of an image coordinate sequence, and quantize the coordinates of the polygons into a discrete sequence.

[0039] If the task description is an image description task, then the task description is converted into a text sequence.

[0040] For example, if the task description is object detection, the continuous coordinates in the image are converted into discrete integer coordinates using a predefined quantization step size; the target objects in the image are traversed, and the bounding box coordinates of each object are quantized; based on the quantized coordinates, the bounding box of each target is determined. The bounding box is usually represented as a quadruple (x_min, y_min, x_max, y_max), representing the coordinates of the top left and bottom right corners, respectively. The coordinates of the bounding box and the description of the target object (such as the category label) are converted into a series of discrete symbols or numbers; this is implemented through an encoding algorithm, such as using one-hot encoding or embedding vectors to represent the discrete sequence.

[0041] For example, if the task is described as an instance segmentation task, a deep learning model (such as yolov11seg) is used to predict the mask of each instance in the input image, and the mask is converted into a polygon representation, usually through an approximation algorithm (such as the Douglas-Peucker algorithm); the vertex coordinates of the polygon are quantized, similar to the quantization process in object detection tasks; and the quantized coordinate sequence is used as the output of the model.

[0042] For example, image description tasks require generating text descriptions related to image content. Natural language processing techniques are used to convert the text descriptions in image description tasks into a series of discrete symbols or word sequences, including but not limited to preprocessing steps such as word segmentation, stop word removal, and stemming.

[0043] Through the above methods, this application can handle different types of image tasks (object detection, instance segmentation, image description), demonstrating strong adaptability. Preprocessing steps such as quantization and polygon approximation reduce data processing complexity and improve processing efficiency.

[0044] In some embodiments, the method for determining the encoder-decoder model includes:

[0045] Obtain a dataset consisting of multiple sets of sample pairs, each set of sample pairs including the input image and the corresponding task description;

[0046] The dataset is divided into training, testing, and validation sets. The Pix2SeqV2 model is trained using the training set, and the first encoder-decoder model is obtained by adjusting the initial parameters through backpropagation. The hyperparameters of the first encoder-decoder model are adjusted using the validation set to obtain the second encoder-decoder model. The performance of the second encoder-decoder model is evaluated using the testing set, and the third encoder-decoder model with satisfactory generalization performance is output. The third encoder-decoder model is used as the final encoder-decoder model.

[0047] Specifically, a large amount of image data is collected, and for each image, a corresponding task description is provided, such as the specific requirements for tasks like object detection and image segmentation. Images and their corresponding task descriptions are paired to form a dataset. The dataset is then randomly divided into three parts: a training set, a test set, and a validation set. A typical division ratio is 70% for training, 20% for testing, and 10% for validation.

[0048] Initialize the parameters of the Pix2SeqV2 model; input the images from the training set into the model and output the corresponding task description prediction results; calculate the loss between the prediction results and the true task descriptions, and adjust the model parameters through the backpropagation algorithm to minimize the loss. For example, the Pix2SeqV2 model is based on an encoder-decoder architecture, which converts images into discrete labeled sequences for prediction. The backpropagation algorithm updates the model parameters by calculating gradients to optimize the prediction results.

[0049] During training, the model's performance is periodically evaluated using a validation set. Based on the performance on the validation set, the model's hyperparameters (such as learning rate and batch size) are adjusted. The training process is repeated until the optimal combination of hyperparameters is found. For example, the validation set is used to evaluate the model's performance under different hyperparameters to select the best combination.

[0050] The trained model is used to make predictions on images in a test set. For example, the test set is used to evaluate the model's generalization ability on unknown data. Based on the performance evaluation results on the test set, the model with satisfactory generalization performance is selected. This model is then saved and deployed as the final encoder-decoder model. For example, the best-performing model is selected for saving and deployment to ensure accuracy in real-world applications.

[0051] Through the above methods, reasonable dataset partitioning and training processes ensured the training efficiency and performance evaluation accuracy of the model. Utilizing the encoder-decoder architecture and backpropagation algorithm of the Pix2SeqV2 model, the mapping relationship between images and task descriptions was successfully learned. By adjusting hyperparameters and performance evaluation on the test set, the model's performance was further optimized, and its generalization ability was improved.

[0052] The final output model has excellent generalization ability and can accurately fulfill the requirements of task description in practical applications, providing strong support for applications in the fields of image processing and computer vision.

[0053] In some embodiments, the encoder-decoder model includes an encoder, which is based on a Transformer architecture and / or a ConvNeXt architecture; a Head, which receives feature maps from the backbone network or other layers and converts them into task outputs, predicting the class label, bounding box, and segmentation mask of the output; a Neck, which is an intermediate layer connecting the backbone network and the Head; and a decoder, which is conditional on a prompt and generates an output sequence for a single object detection task. During training, the prompt and the desired output are concatenated into a single sequence, and sequence weighting is used to ensure that the decoder is only trained to predict the desired output.

[0054] The encoder is based on a Transformer architecture and / or a ConvNeXt architecture. In the Transformer architecture, the encoder transforms each word in the input sequence into a feature vector and generates an encoded output containing the semantic information of the input sequence. In the ConvNeXt architecture, the encoder captures spatial-level features through convolutional layers.

[0055] For example, the Transformer architecture utilizes self-attention to capture long-range dependencies in the input sequence and enhances feature representation through multi-layered stacked encoder blocks. The ConvNeXt architecture, employing modern convolutional neural network design principles, progressively extracts features from images by stacking multiple convolutional blocks. The encoder efficiently extracts feature representations from input data (such as images and text), providing a solid foundation for subsequent task prediction.

[0056] For example, the Head receives feature maps from the backbone network or other layers and transforms them into task output. This includes predicting the class label, bounding box, and segmentation mask for the output. The Head typically contains a series of convolutional layers, pooling layers, and fully connected layers that map the feature maps to a task-relevant output space. For instance, for classification tasks, the Head uses a softmax function to map features to a class distribution; for detection tasks, the Head includes a classification head (for predicting the target class) and a regression head (for predicting the location of the target box). Using the Head, the output can be accurately predicted based on the feature maps, improving the prediction accuracy and robustness of the task.

[0057] For example, the Neck is an intermediate layer connecting the backbone network and the Head, responsible for further processing or adjusting the features extracted by the backbone network. Neck modules have diverse structures, commonly including feature pyramid networks and path aggregation networks. These modules enhance the expressive power of features by fusing feature maps of different scales, improving the model's ability to detect targets at different scales. Neck modules can enhance the expressive power of features and improve the model's detection accuracy and generalization ability for targets at different scales.

[0058] The decoder is conditional on a prompt to generate an output sequence for a single object detection task. During training, the prompt and the desired output are concatenated into a single sequence, using sequence weighting to ensure that the decoder is trained only to predict the desired output.

[0059] For example, decoders are typically based on the Transformer architecture or a similar structure, utilizing self-attention mechanisms and positional encoding to generate output sequences. In object detection tasks, the decoder receives feature maps and prompts from the encoder as input and progressively generates output information such as the location and category of the target.

[0060] Adjust the decoder's structure and parameters according to task requirements and model complexity. For example, in object detection tasks, a Transformer-based decoder can be used to generate output information such as bounding boxes and class labels.

[0061] In this embodiment, the decoder can accurately generate the output sequence for the object detection task based on the prompt and the feature map output by the encoder, improving the prediction accuracy and efficiency of the task. Simultaneously, by using methods such as sequence weighting, it can be ensured that the decoder only focuses on the desired output, further improving the model's performance.

[0062] In some embodiments, prompt is given and used to guide the model to process and generate input instructions containing different types of data, including images, text, and speech.

[0063] In some embodiments, inverse quantization inference is performed based on the task output sequence to determine the detection bounding box, mask, and defect description information of the input image, including:

[0064] For object detection tasks, the predicted task output sequence is decomposed into multiple token tuples to obtain coordinate tokens and category tokens, and the coordinate tokens are dequantized to obtain detection boxes. For instance segmentation tasks, the coordinate tokens corresponding to each polygon are dequantized and converted into dense masks. The average of the masks is calculated to obtain a binary mask. For image description tasks, the predicted category tokens are mapped to text to form defect description information.

[0065] For example, the predicted task output sequence is typically generated by a neural network and contains the target's location and category information. Using a specific decoding method, the output sequence is decomposed into multiple token tuples, each containing a coordinate token and a category token. The coordinate tokens are usually represented in quantized form to reduce storage and transmission overhead. Through dequantization, the coordinate tokens are converted into actual coordinate values, thus obtaining the location of the detection box.

[0066] In instance segmentation tasks, targets are typically represented as polygons. The coordinate tokens of each polygon are dequantized to obtain their actual coordinate values. A dense mask is generated based on the dequantized polygon coordinates. The mask is averaged, and pixels with mask values ​​greater than a certain threshold are set to 1, while the rest are set to 0, resulting in a binary mask. In image description tasks, neural networks typically predict category tokens related to the image content. These category tokens are then mapped to predefined text descriptions to form defect description information.

[0067] By employing key technologies such as token tuple decomposition, inverse quantization, binary mask generation, and text description generation, the above method enables accurate detection and description of targets in images. It can be adjusted and optimized according to specific application scenarios and needs. This solution provides new ideas and methods for target detection, instance segmentation, and image description tasks in the field of computer vision, contributing to the development and application of related technologies.

[0068] In some embodiments, the method further includes: mapping the features of different modalities to a preset feature domain through feature transformation of the image sequence corresponding to the input image and the text sequence corresponding to the task description to form fused features; constructing a defect detection network based on the fused features, and training the defect detection network with a multimodal dataset composed of the fused features represented by the input image and the task description, wherein the defect detection network is used to simultaneously perform object detection, instance segmentation and defect description; and using the trained defect detection network to detect the acquired image to be tested, so as to output the detection box, mask and defect description information of the image to be tested, wherein the image to be tested is the input image.

[0069] For example, the input image is first converted into an image sequence, involving segmenting the image into small patches or extracting key features from the image; simultaneously, the task description is converted into a text sequence, including converting the natural language description into a computer-processable format, such as word vectors or embedding representations. Then, feature transformation methods (such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), etc.) are used to map the features of the image sequence and text sequence to a preset feature domain to form fused features.

[0070] For example, a defect detection network is constructed based on fused features. This network typically includes an input layer, a feature extraction layer, a classification layer, and an output layer. The input layer receives a multimodal dataset, which is a dataset consisting of the input image and the fused features represented by the task description. Through the training process, the parameters of the defect detection network are optimized so that it can accurately identify defects in the image.

[0071] For example, the image to be tested is input into a trained defect detection network. The network processes the input image and outputs a detection box, a mask, and defect description information. The detection box is used to locate the defect in the image, the mask is used to represent the precise shape and extent of the defect, and the defect description information provides a detailed textual description of the defect.

[0072] By fusing image and text features in the above manner, multimodal information is fully utilized, improving the accuracy and robustness of defect detection. The constructed defect detection network can simultaneously perform target detection, instance segmentation, and defect description, achieving the integration and unification of multi-task processing. Furthermore, this solution can be widely applied in fields such as industrial automation, intelligent manufacturing, and quality inspection, providing strong technical support for defect detection and quality control.

[0073] The following is an exemplary description of the multimodal steel coil surface defect detection method provided in the above embodiments through another specific example. The multimodal steel coil surface defect detection method provided in this embodiment includes the following main steps:

[0074] 1) Acquire images of the steel coil surface;

[0075] 2) Convert the input image and task description into an input token;

[0076] 3) Obtain the task output token using the pix2seqv2 encoder-decoder model;

[0077] 4) Reasoning yields the required detection box, mask, and defect description information.

[0078] In section 2), the conversion to input tokens is characterized by encoding both the image input and the task description into discrete token sequences, using different encoding methods for different visual tasks: For object detection tasks, this patent converts bounding boxes and object descriptions into a series of discrete tokens by quantizing continuous image coordinates; for instance segmentation tasks, this patent predicts the polygons corresponding to the instance mask in the form of an image coordinate sequence (i.e., a sequence of polygon vertices determined clockwise from a given polygon vertex), rather than a pixel-by-pixel mask. Similarly, this patent quantizes coordinates into discrete tokens. For image description tasks, this patent directly converts them into text tokens.

[0079] In 3), the pix2seqv2 encoder is characterized in that the image encoder perceives pixels in an image and maps them to hidden representations, specifically instantiated as ConvNeXt, Transformer, or a combination thereof. Transformer-based sequence decoders, widely used in modern language modeling, generate one token at a time, conditional on the previous token and the encoded image representation. This eliminates the complexity and customization (e.g., task-specific heads or necks) of modern neural network architectures for these visual tasks.

[0080] In section 4), the pix2seqv2 decoder is characterized in that the decoder, conditional on a prompt, directly generates output tokens for a single object detection task, so that the model can generate outputs adapted to the task of interest. During training, the model concatenates the prompt and the desired output into a single sequence, using a token weighting scheme to ensure that the decoder is only trained to predict the desired output. During inference, the prompt is given and fixed, so the decoder only needs to generate the rest of the sequence. The training objective is to maximize the likelihood of an image-based token and previous tokens, i.e.:

[0081] Here, X represents the input image, and Y is a sequence of length L related to X. As mentioned earlier, the initial part of sequence Y is a task description, for which the weight W is set to zero so that it is not included in the loss.

[0082] Furthermore, a prompt is an instruction used to guide the model in processing and generating input containing different types of data (such as text, images, audio, etc.). By designing appropriate prompts, users can specify how the model should associate and process data from these different modalities, thereby generating multimodal content.

[0083] The inference process is characterized by decoding the task output tokens, using different decoding methods for different visual tasks: for object detection, the predicted sequence is decomposed into tuples with 5 tokens to obtain coordinate tokens and category tokens, and the coordinate tokens are dequantized to obtain bounding boxes; for instance segmentation, the coordinate tokens corresponding to each polygon are dequantized and then converted into dense masks. Since the model itself is not trained using any geometry-specific regularizers, the output polygon masks may be noisy. To reduce noise, multiple sequences can be sampled, the masks can be averaged, and a binary mask can be obtained by applying a simple threshold; for image description, the predicted discrete tokens are directly mapped to the text.

[0084] As can be seen, in the above scheme, a sample pair is formed by acquiring an input image containing the surface of the steel coil and the corresponding task description; the multimodal sample pair is encoded and converted to determine a discrete sequence; the discrete sequence is input into the encoder-decoder model to obtain the task output sequence, and the encoder-decoder model is a Pix2SeqV2 model; inverse quantization inference is performed based on the task output sequence to determine the detection box, mask, and defect description information of the input image; on the one hand, this application can not only improve the performance of the industrial image processing system, but also improve the automatic detection capability of steel coil surface defects; on the other hand, by using the sample pair formed by the input image and the task description, samples of different defect types can be quickly acquired, which is conducive to the rapid iteration of model training. At the same time, by directly outputting the detection box, mask, and defect description information through the multimodal image set, the detection accuracy and detection applicability of steel coil surface defects are also improved.

[0085] See Figure 3 for details. It is a schematic diagram of the processing structure of a multimodal steel coil surface defect detection device according to an embodiment of the present invention; including:

[0086] Image and text information are input separately. The image is segmented and combined with positional encoding before being input into the encoder network. Simultaneously, the encoded text information is combined with the image information in the encoder. Finally, the encoder information is used by the decoder to obtain the predicted token value.

[0087] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0088] In one embodiment, a multimodal steel coil surface defect detection device is provided, which corresponds one-to-one with the multimodal steel coil surface defect detection method described in the previous embodiment. As shown in Figure 2, the multimodal steel coil surface defect detection device includes an acquisition module 201, an encoding conversion module 202, a task output module 203, and a defect detection module 204. Detailed descriptions of each functional module are as follows:

[0089] The acquisition module 201 is used to acquire an input image containing the surface of the steel coil and a task description corresponding to the input image;

[0090] The encoding conversion module 202 is used to perform encoding conversion on the input image and the task description to determine the discrete sequence, wherein different encoding methods are used for task descriptions of different task types;

[0091] Task output module 203 is used to input discrete sequences into the encoder-decoder model to obtain task output sequences;

[0092] The defect detection module 204 is used to perform inverse quantization inference based on the task output sequence to determine the detection box, mask and defect description information of the input image in order to complete the surface defect detection of the steel coil.

[0093] This invention provides a multimodal steel coil surface defect detection device. It acquires a sample pair consisting of an input image of the steel coil surface and a corresponding task description. The multimodal sample pair is encoded and converted to determine a discrete sequence. This discrete sequence is then input into an encoder-decoder model (a Pix2SeqV2 model) to obtain the task output sequence. Inverse quantization inference is performed based on the task output sequence to determine the detection box, mask, and defect description information of the input image. On one hand, this invention not only improves the performance of industrial image processing systems but also enhances the automatic detection capability of steel coil surface defects. On the other hand, the sample pair formed by the input image and task description allows for the rapid acquisition of samples of different defect types, facilitating rapid iteration of model training. Furthermore, the direct output of the detection box, mask, and defect description information through the multimodal image set also improves the detection accuracy and applicability of steel coil surface defects.

[0094] Specific limitations regarding the multimodal steel coil surface defect detection device can be found in the limitations of the multimodal steel coil surface defect detection method described above, and will not be repeated here. Each module in the aforementioned multimodal steel coil surface defect detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0095] In one embodiment, a computer device is provided, which can be a server, and its internal structure diagram is shown in Figure 4. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a multimodal steel coil surface defect detection method on the server side.

[0096] In one embodiment, a computer device is provided, which can be a client, and its internal structure diagram is shown in Figure 5. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a multimodal steel coil surface defect detection method on the client side.

[0097] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: acquiring an input image containing the surface of a steel coil and a task description corresponding to the input image; encoding and converting the input image and the task description to determine a discrete sequence, wherein different encoding methods are used for task descriptions of different task types; inputting the discrete sequence into an encoder-decoder model to obtain a task output sequence, wherein the encoder-decoder model is a Pix2SeqV2 model; and performing inverse quantization inference based on the task output sequence to determine the detection box, mask, and defect description information of the input image to complete the steel coil surface defect detection.

[0098] The computer device provided in the above embodiments acquires a sample pair formed by an input image containing the surface of a steel coil and a corresponding task description; encodes and converts the multimodal sample pair to determine a discrete sequence; inputs the discrete sequence into an encoder-decoder model to obtain a task output sequence, wherein the encoder-decoder model is a Pix2SeqV2 model; and performs inverse quantization inference based on the task output sequence to determine the detection box, mask, and defect description information of the input image. On the one hand, this application not only improves the performance of industrial image processing systems but also enhances the automatic detection capability of steel coil surface defects. On the other hand, by using the sample pair formed by the input image and the task description, samples of different defect types can be quickly acquired, which is beneficial for rapid iteration of model training. At the same time, by directly outputting the detection box, mask, and defect description information through a multimodal image set, the detection accuracy and applicability of steel coil surface defects are also improved.

[0099] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: acquiring an input image containing the surface of a steel coil and a task description corresponding to the input image; encoding and converting the input image and the task description to determine a discrete sequence, wherein different encoding methods are used for task descriptions of different task types; inputting the discrete sequence into an encoder-decoder model to obtain a task output sequence, wherein the encoder-decoder model is a Pix2SeqV2 model; and performing inverse quantization inference based on the task output sequence to determine the detection box, mask, and defect description information of the input image to complete the detection of defects on the surface of the steel coil.

[0100] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0101] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0102] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0103] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for detecting surface defects in steel coils based on multimodal methods, characterized in that, include: Obtain an input image containing the surface of the steel coil and a corresponding task description of the input image; The input image and the task description are encoded and converted to determine a discrete sequence, wherein different encoding methods are used for the task descriptions of different task types; The discrete sequence is input into the encoder-decoder model to obtain the task output sequence; Based on the task output sequence, inverse quantization inference is performed to determine the detection box, mask, and defect description information of the input image in order to complete the surface defect detection of the steel coil.

2. The method for detecting surface defects of steel coils based on multimodal detection as described in claim 1, characterized in that, Different task descriptions for different task types employ different encoding methods, including: If the task description is an object detection task, then the bounding box of the object is obtained by quantizing the continuous image coordinates in the input image, and the bounding box and the object description are converted into a series of discrete sequences; If the task is described as an instance segmentation task, then the polygons corresponding to the instance mask in the input image are predicted in the form of an image coordinate sequence, and the coordinates corresponding to the polygons are quantized into a discrete sequence. If the task description is an image description task, then the task description is converted into a text sequence.

3. The method for detecting surface defects in steel coils based on multimodal detection as described in claim 1, characterized in that, The method for determining the encoder-decoder model includes: Obtain a dataset consisting of multiple sets of sample pairs, each set of sample pairs including the input image and the corresponding task description; The dataset is divided into a training set, a test set, and a validation set. The encoder-decoder model is a Pix2SeqV2 model. The Pix2SeqV2 model is trained using the training set, and the initial parameters are adjusted through backpropagation to obtain a first encoder-decoder model. The hyperparameters of the first encoder-decoder model are adjusted using the validation set to obtain a second encoder-decoder model. The performance of the second encoder-decoder model is evaluated using the test set, and a third encoder-decoder model with satisfactory generalization performance is output. The third encoder-decoder model is used as the final encoder-decoder model.

4. The method for detecting surface defects of steel coils based on multimodal detection as described in any one of claims 1-3, characterized in that, The encoder-decoder model includes an encoder, which is based on a Transformer architecture and / or a ConvNeXt architecture; a Head, which receives feature maps from the backbone network or other layers and converts them into task outputs, predicting the output class label, bounding box, and segmentation mask; a Neck, which is an intermediate layer connecting the backbone network and the Head; and a decoder, which is conditional on a prompt and generates an output sequence for a single object detection task. During training, the prompt and the desired output are concatenated into a single sequence, and sequence weighting is used to ensure that the decoder is only trained to predict the desired output.

5. The method for detecting surface defects of steel coils based on multimodal analysis as described in claim 4, characterized in that, The prompt is given and is used to guide the model to process and generate input instructions containing different types of data, including images, text, and speech.

6. The method for detecting surface defects of steel coils based on multimodal detection as described in any one of claims 1-3, characterized in that, Based on the task output sequence, inverse quantization inference is performed to determine the detection box, mask, and defect description information of the input image, including: For object detection tasks, the predicted task output sequence is decomposed into multiple token tuples to obtain coordinate tokens and category tokens, and the coordinate tokens are dequantized to obtain detection boxes; for instance segmentation tasks, the coordinate tokens corresponding to each polygon are dequantized and converted into dense masks, and the average of the masks is calculated to obtain a binary mask; for image description tasks, the predicted category tokens are mapped to text to form defect description information.

7. The method for detecting surface defects of steel coils based on multimodal detection as described in any one of claims 1-3, characterized in that, Also includes: The image sequence corresponding to the input image and the text sequence corresponding to the task description are mapped to a preset feature domain through feature transformation to form fused features; Based on the fusion features, a defect detection network is constructed, and the defect detection network is trained using a multimodal dataset composed of the fusion features represented by the input image and the task description. The defect detection network is used to simultaneously perform object detection, instance segmentation, and defect description. The trained defect detection network is used to detect the acquired image to be tested, so as to output the detection box, mask, and defect description information of the image to be tested, wherein the image to be tested is the input image.

8. A multimodal steel coil surface defect detection device, characterized in that, include: The acquisition module is used to acquire an input image containing the surface of the steel coil and a task description corresponding to the input image; The encoding conversion module is used to encode the input image and the task description to determine a discrete sequence, wherein the task descriptions for different task types adopt different encoding methods; The task output module is used to input the discrete sequence into the encoder-decoder model to obtain the task output sequence; The defect detection module is used to perform inverse quantization inference based on the task output sequence to determine the detection box, mask and defect description information of the input image, so as to complete the surface defect detection of the steel coil.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images