Method and computer device for training neural network for object detection of industrial images

Abstract

Various embodiments concern a method of training a neural network model for object detection of industrial images comprising: obtaining a pre-trained object detection model, wherein the pre-trained object detection model is pre-trained using a source domain; bootstrap training of the pre-trained object detection model to obtain an intermediate bootstrapped trained object detection model by: sampling an industrial image from a dataset; segmenting the industrial image into semantic regions using an image segmentation model; sample two regions of the semantic regions and sample two crops from each region of the two regions; training the pre-trained object detection model using the two regions and the two crops from each region to obtain the intermediate bootstrapped trained object detection model and using the intermediate bootstrapped trained object detection model to train the neural network for object detection of industrial images.

Description

METHOD AND COMPUTER DEVICE FOR TRAINING NEURAL NETWORK FOR OBJECT DETECTION OF INDUSTRIAL IMAGESTECHNICAL FIELD

[0001] Various aspects of this disclosure relate to methods and computing device for training a neural network model for object detection of industrial images.BACKGROUND

[0002] In industrial automation, object detection (i.e. identification and localization of objects in an image) is a common computer vision task. From quality inspection to waste sorting and assembly / disassembly, having high-performance and reliable object detection models is crucial to many tasks. However, most neural network models readily available are trained on public datasets like Common Objects in Context (COCO), which only includes everyday objects (e.g. food, animals or vehicles). Thus, in order to yield decent results on industrial images, these neural network models need to be fine-tuned on the dataset of interest such as a dataset of industrial images. These datasets are typically much smaller than public datasets as annotation and finetuning is a costly process.

[0003] Accordingly, efficient and cost-effective approaches for object detection are desirable.SUMMARY

[0004] Various embodiments concern a method of training a neural network model for object detection of industrial images comprising: obtaining a pre-trained object detection model, wherein the pre-trained object detection model is pre-trained using a source domain; bootstrap training of the pre-trained object detection model to obtain an intermediate bootstrapped trained object detection model by: sampling an industrial image from a dataset; segmenting the industrial image into semantic regions using an image segmentation model; sample two regions of the semantic regions and sample two crops from each region of the two regions; training the pre-trained object detection model using the two regions and the two crops from each region to obtain the intermediate bootstrapped trained object detection model andusing the intermediate bootstrapped trained object detection model to train the neural network for object detection of industrial images.

[0005] According to one embodiment, the method includes training the pre-trained object detection model using the two regions and the two crops from each region to obtain the intermediate bootstrapped trained object detection model by: training the pre-trained object detection model that the two crops from a same region of the two regions are have similar features and crops from a different region of the two regions have different features.

[0006] According to one embodiment, the method includes fine-tuning the intermediate bootstrapped trained object detection model using a target domain, wherein the target domain comprises images of industrial parts.

[0007] According to one embodiment, in the source domain comprises images of everyday objects.

[0008] According to one embodiment, the pre-trained object detection model comprises a backbone for extracting features of an image and a head for mapping the features for prediction.

[0009] According to one embodiment, in the image segmentation model is a Segment Anything Model (SAM).

[0010] According to one embodiment, the bootstrap training is unsupervised.

[0011] According to one embodiment, the industrial image is an engine and objects for detection are screws and / or gears.

[0012] According to one embodiment, the neural network model is a convolutional neural network (CNN).

[0013] Various embodiments concern a computer device for training a neural network model for object detection of industrial images comprising: a processor, a memory, the memory storing at least one program code, the at least one program code loaded and executed by the processor to: obtain a pre-trained object detection model, wherein the pre-trained object detection model is pre-trained using a source domain; bootstrap train of the pre-trained object detection model to obtain an intermediate bootstrapped trained object detection model by: sampling an industrial image from a dataset; segmenting the industrial image into semantic regions using an image segmentation model; sample two regions of the semantic regions and sample two crops from each region of the two regions; training the pre-trained object detection model using the two regions and the two crops from each region to obtain the intermediatebootstrapped trained object detection model and using the intermediate bootstrapped trained object detection model to train the neural network for object detection of industrial images.

[0014] According to one embodiment, the processor is further configured to: train the pretrained object detection model using the two regions and the two crops from each region to obtain the intermediate bootstrapped trained object detection model by: training the pre-trained object detection model that the two crops from a same region of the two regions are have similar features and crops from a different region of the two regions have different features.

[0015] According to one embodiment, the processor is further configured to: fine-tune the intermediate bootstrapped trained object detection model using a target domain, wherein the target domain comprises images of industrial parts.

[0016] According to one embodiment, in the source domain comprises images of everyday objects.

[0017] According to one embodiment, the pre-trained object detection model comprises a backbone for extracting features of an image and a head for mapping the features for prediction.

[0018] According to one embodiment, in the image segmentation model is a Segment Anything Model (SAM).

[0019] According to one embodiment, the bootstrap training is unsupervised.

[0020] According to one embodiment, the industrial image is an engine and objects for detection are screws and / or gears.

[0021] According to one embodiment, the neural network model is a convolutional neural network (CNN).

[0022] According to one embodiment, a computer readable storage medium, characterized in that the storage medium stores at least one program code for execution by a processor to implement the training method described above.

[0023] According to one embodiment, a computer program product, the computer program product comprising computer instructions stored in a computer readable storage medium; a processor of a computer device reads the computer instructions from the computer readable storage medium, the processor executing the computer instructions, causing the computer device to implement the training method described above.

[0024] It should be noted that embodiments described in context of the method of training a large language model are analogously valid for the computer device and vice versa.BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The invention will be better understood with reference to the detailed description when considered in conjunction with the non-limiting examples and the accompanying drawings, in which:- FIGS. 1 A-1C show exemplary illustrations of various object detection models and their results.- FIG. 2 shows an exemplary method of training a neural network model for object detection of industrial images according to an embodiment.- FIG. 3A shows an exemplary illustration of different phases of the object detection model, according to an embodiment.- FIG. 3B shows an exemplary overview of the bootstrapping phase of the object detection model, according to an embodiment.- FIGS. 4 and 5 show exemplary graph results of various object detection models according to an embodiment.- FIG. 6 show exemplary chart results of various object detection models according to an embodiment.- FIG. 7 shows exemplary overview of Simple framework for Contrastive Learning of visual Representations (SimCLR) process according to an embodiment.- FIG. 8 shows a comparison of the sampling process in SimCLR process vs bootstrapping according to an embodiment.DETAILED DESCRIPTION

[0026] The following detailed description refers to the accompanying drawings that show, by way of illustration, specific details and embodiments in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure. Other embodiments may be utilized and structural, and logical changes may be made without departing from the scope of the disclosure. The various embodiments are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0027] Embodiments described in the context of one of the computer device or methods are analogously valid for the other computer device or methods. Similarly, embodiments described in the context of a computer device are analogously valid for a method, and vice-versa.

[0028] Features that are described in the context of an embodiment may correspondingly be applicable to the same or similar features in the other embodiments. Features that are described in the context of an embodiment may correspondingly be applicable to the other embodiments, even if not explicitly described in these other embodiments. Furthermore, additions and / or combinations and / or alternatives as described for a feature in the context of an embodiment may correspondingly be applicable to the same or similar feature in the other embodiments.

[0029] In the context of various embodiments, the articles “a”, “an” and “the” as used with regard to a feature or element include a reference to one or more of the features or elements.

[0030] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0031] In industrial automation, object detection (i.e. identification and localization of objects in an image) is a common computer vision task. From quality inspection to waste sorting and assembly / disassembly, having high-performance and reliable object detection models is crucial to many tasks. However, most neural network models readily available are trained on public datasets like Common Objects in Context (COCO), which only includes everyday objects (e.g. food, animals or vehicles). Thus, in order to yield decent results on industrial images, these neural network models need to be fine-tuned on the dataset of interest such as a dataset of industrial images. These datasets are typically much smaller than public datasets as annotation and finetuning is a costly process.

[0032] The data used for pretraining (e.g. everyday objects) is referred to as the source domain, and the dataset of interest (e.g. industrial images or parts) is called the target domain. A proprietary dataset of 150 high-resolution eaxle images (1944x1200), with 2 classes of objects to detect: screws and gears will be used as the target domain for the embodiments disclosed herein. Eaxle images are images of an industrial engine.

[0033] In various embodiments, the process of adapting a model to a different data distribution is known as domain adaptation and is usually performed in one of the following ways: zero shot learning, supervised domain adaptation and unsupervised domain adaptation.

[0034] FIGS. 1A-1C show exemplary illustrations of various object detection models and their results.

[0035] FIG. IB and 1C show the exale engine with screws and gears identified from the image.

[0036] In various embodiments, in a zero-shot approach, off-the-shelf pre-trained models are directly used to make predictions on the smaller dataset. Zero-shot learning (ZSL) is a machine learning scenario in which the model is trained to recognize and categorize objects or concepts without having seen any examples of those categories or concepts beforehand.

[0037] However, the distribution shift between the source and target domain should be relatively small for these approaches to work.

[0038] In various embodiments, in supervised domain adaptation, pre-trained models are fine-tuned on the smaller dataset. For example, the eAxle dataset would be used to fine-tune the model and adjust its learned features to better match the characteristics of those objects. Depending on the complexity of the task and the distribution shift between the two image domains, a large number of image-label pairs may be required.

[0039] In various embodiments, in unsupervised domain adaptation, it is assumed that no annotations are available for the target domain. While approaches used in this setting are much more diverse, one common technique is to mix images from the source domain (for which labels are available), with images from the target domain and use those hybrid images in order to make predictions.

[0040] In various embodiments, zero-shot has the advantage of being very fast to apply (as no further model training is required), but performance degrades quickly as the distribution shift increases between the source and target domain. With industrial images, those methods are not applicable as screws and gears are categories that are completely absent from most public datasets. In FIG.1 A, a state-of-the-art zero-shot object detection model is asked to detect screws in an eAxle image. As shown, the model as no prior knowledge of what a screw is and simply predicts the whole eAxle engine to be a screw. In FIG. IB, the exale engine with gears 102 identified from the image is shown. In FIG. 1C, the exale engine with screws 104 identified from the image is shown.

[0041] In various embodiments, fine-tuning is by far the most common setting. It can achieve decent results as long as enough data from the target domain is available. However, in practice, labeled target data is usually very scarce, leading to these methods underperforming.

[0042] In various embodiments, methods from unsupervised domain adaptation can adapt to slightly bigger distribution gaps between source and target compared to zero-shot, howeverthese approaches still struggle when the shift becomes too large. It makes them very hard to apply to the industrial domain.

[0043] FIG. 2 shows an exemplary method of training a neural network model for object detection of industrial images according to an embodiment.As shown in FIG. 2, there may be a method 200 of training a large language model, especially for an unseen task. The “unseen task” means that a model pre-trained on a given task is taken, usually classification of natural images (on ImageNet), and instead of directly fine-tuning it on a new task (e.g. screw detection on industrial images), first a “bootstrapping phase” is performed to adapt to it to the new task. In other words, “training for an unseen task” means that the model is trained to adapt it to a new task. After this, the model can be fine-tuned for this new task, but the training training significantly improves fine-tuning performances.

[0044] In the method 200, a first step 202 may include obtaining a pre-trained object detection model, wherein the pre-trained object detection model is pre-trained using a source domain. A second step 204 may include bootstrap training of the pre-trained object detection model to obtain an intermediate bootstrapped trained object detection model. A third step 206 may include sampling an industrial image from a dataset. A fourth step 208 may include segmenting the industrial image into semantic regions using an image segmentation model. A fifth step 210 may include sampling two regions of the semantic regions and sample two crops from each region of the two regions. A sixth step 212 may include training the pre-trained object detection model using the two regions and the two crops from each region to obtain the intermediate bootstrapped trained object detection model for object detection of industrial images. The intermediate bootstrapped trained object detection model may be used to train the neural network for object detection of industrial images.

[0045] It will be understood that the above operations described above relating to FIG. 2 are not limited to this particular order. Any suitable, modified order of operations may be used.

[0046] In various embodiments, a computer device for training a neural network model for object detection of industrial images may include a processor for performing computing tasks such as object detection pre-training, fine-tuning and any other relevant steps. The computer device may also include a memory. The memory may store at least one program code, the at least one program code loaded and executed by the processor to train the neural network.

[0047] In various embodiments, the computer device may obtain a pre-trained object detection model, wherein the pre-trained object detection model is pre-trained using a source domain.

[0048] In various embodiments, the computer device may bootstrap train of the pre-trained object detection model to obtain an intermediate bootstrapped trained object detection model by: sampling an industrial image from a dataset; segmenting the industrial image into semantic regions using an image segmentation model; sampling two regions of the semantic regions and sample two crops from each region of the two regions; and training the pre-trained object detection model using the two regions and the two crops from each region to obtain the intermediate bootstrapped trained object detection model for object detection of industrial images.

[0049] According to one embodiment, the processor is further configured to: train the pretrained object detection model using the two regions and the two crops from each region to obtain the intermediate bootstrapped trained object detection model by: training the pre-trained obj ect detection model that the two crops from a same region of the two regions are have similar features and crops from a different region of the two regions have different features.

[0050] According to one embodiment, the processor is further configured to: fine-tune the intermediate bootstrapped trained object detection model using a target domain, wherein the target domain comprises images of industrial parts.

[0051] According to one embodiment, in the source domain comprises images of everyday objects.

[0052] According to one embodiment, the pre-trained object detection model comprises a backbone for extracting features of an image and a head for mapping the features for prediction.

[0053] According to one embodiment, in the image segmentation model is a Segment Anything Model (SAM).

[0054] According to one embodiment, the bootstrap training is unsupervised.

[0055] According to one embodiment, the industrial image is an engine and objects for detection are screws and / or gears.

[0056] According to one embodiment, the neural network model is a convolutional neural network (CNN).

[0057] According to one embodiment, a computer program product, the computer program product comprising computer instructions stored in a computer readable storage medium; aprocessor of a computer device reads the computer instructions from the computer readable storage medium, the processor executing the computer instructions, causing the computer device to implement the training method described above.

[0058] FIG. 3A shows an exemplary illustration of different phases of the object detection model, according to an embodiment. FIG. 3B shows an exemplary overview of the bootstrapping phase of the object detection model, according to an embodiment.

[0059] In FIG. 3A, an intermediate training step of “Bootstrapping” 303 is added to the traditional object detection pipeline. Typical pipelines take an object detection model 301 pretrained 302 on a source domain (e.g. on everyday objects) and fine-tune 304 it on the target domain (e.g. industrial images). The bootstrap intermediate phase 303 aims at adapting the model 301 to industrial images in a more efficient and accurate manner.

[0060] Object detectors usually includes of 2 main components: a backbone, which extract features for the image, and a head, which maps those features to bounding boxes. The bootstrapping phase may be an unsupervised, and aims at making the backbone produce more relevant features for the head to make predictions. More specifically, the backbone may be trained to output similar features for the same semantic region of the eAxle.

[0061] In various embodiments, the input image 311 (i.e. industrial image) is segmented into semantic regions with a segmentation software such as SAM 312, and a plurality of random regions 313 for example 2 random regions are sampled. For each of the sampled region, two crops are sampled within the region and pass through the backbone 314 to get a feature vector 315. The neural network is then trained such that feature vectors from the same region have a high similarity (attract), whereas feature vectors from different regions are further apart (repel). The backbone thus implicitly learns to differentiate the different components of the eAxle. When the backbone is then plugged back in the object detection model, the object detection model reaches higher performances as the backbone has a prior understanding of what screws and gears are. This bootstrapping phase leads to faster and better convergence when reaching the fine-tuning stage.

[0062] FIGS. 4 and 5 show exemplary graph results of various object detection models according to an embodiment.

[0063] To evaluate the effectiveness of the method disclosed herein, a pretrained object detector such as DINO detector is fine-tuned on an eaxle dataset mentioned and used as a baseline. In other words, the baseline corresponds to only doing steps (1) Pre-training and (3)Fine-tuning. While the method disclosed does all 3 steps: (1) Pre-training, (2) Bootstrapping and (3) Fine-tuning.

[0064] As visible from FIGS. 4 and 5, bootstrapping enables to reach significantly better mAP, and lower loss compared to the baseline. This is a sign that the model with the additional bootstrap step generalizes better to the target distribution.

[0065] FIG. 4 shows the evolution of mean Average Precision (mAP) throughout finetuning when the bootstrapping was applied 402 vs without bootstrapping 404. FIG. 4 shows the global mAP, the mAP for large objects (gears) and the mAP for small objects (screws). All the mAPs increase significantly with bootstrapping.

[0066] FIG. 5 shows the evolution of validation loss throughout fine-tuning when the bootstrapping was applied 502 vs without bootstrapping 504. Both the loss for bounding boxes (i.e. localizing the objects in the image), and the loss for classification (i.e. telling apart screws and gears) is shown. Bootstrapping enables to reach lower loss and thus better generalization to the target domain.

[0067] The mAP at the end of training for both the baseline and bootstrapping method is shown in Table 1 and in FIG. 6.

[0068] FIG. 6 show exemplary chart results of various object detection models according to an embodiment.

[0069] FIG. 6 shows mAP at the end of fine-tuning for the baseline 604 and bootstrapping 602 method. FIG. 6 shows the global mAP (i.e. for both screws and gears, left), the mAP for only gears (middle), and the mAP only for screws (right).Table 1: mAP at the end of fine-tuning for the baseline and bootstrapping method.

[0070] FIG. 7 shows exemplary overview of Simple framework for Contrastive Learning of visual Representations (SimCLR) process according to an embodiment.

[0071] In SimCLR, two random images of the dataset are drawn, namely x; and Xj. Then, for each of them, two “augmentations” (i.e. cropping and color jittering) are sampled. The two crops are (x;1, x;2) and (xj1, Xj2). Thus, there are 2 crops for 2 random images. Finally, the neural network such as CNN 710 is trained to give a similar output for the crops coming from the same image, but a different output for the crops coming from two different images.

[0072] The intuition behind SimCLR is that two crops from the same image will likely have similar content (e.g. two views of the same dog), but if the dataset has enough variety, then 2 crops from 2 different images should represent completely different objects. Two random images are sampled, then 2 crops (called “augmentations” are sampled for each of them). The model is trained to give a similar output for the crops from the same image (“attract”), but different output for crops from different images (“repel”). The first image 702 has crops 706A and 706B and the second image 704 has crops 708A and 708B.

[0073] FIG. 8 shows a comparison of the sampling process in SimCLR process 804 vs bootstrapping 802 according to an embodiment.

[0074] However, when dealing with industrial data, there is the issue of only having a few images of the same object (e.g., eAxle), thus two crops from different images may (and most likely will) represent the same object. This prevents SimCLR to be used in industrial images. This problem is remedied by changing the sampling of the augmentations. Firstly, segment both input images using an existing segmentation model, SAM, which divides the image into semantic regions. Then, the two crops within the same region are sampled. This ensures that the crops from the same image represent the same object. Now, since industrial images are usually composed of many different parts, it becomes very unlikely that the crops sampled from different images will represent the same content. Once the crops are sampled, the training procedure is the same as SimCLR.

[0075] The methods described herein may be performed and the various processing or computation units and the devices and computing entities described herein may be implemented by one or more circuits. In an embodiment, a "circuit" may be understood as any kind of a logic implementing entity, which may be hardware, software, firmware, or any combination thereof. Thus, in an embodiment, a "circuit" may be a hard-wired logic circuit or a programmable logic circuit such as a programmable processor, e.g. a microprocessor. A "circuit" may also be software being implemented or executed by a processor, e.g. any kind of computer program, e.g. a computer program using a virtual machine code. Any other kind of implementation of the respective functions which are described herein may also be understood as a "circuit" in accordance with an alternative embodiment.

[0076] While the disclosure has been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of theinvention as defined by the appended claims. The scope of the invention is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced.

Claims

CLAIMS1. A method (200) of training a neural network model for object detection of industrial images (311) comprising: obtaining (202) a pre-trained object detection model, wherein the pre-trained object detection model is pre-trained using a source domain; bootstrap training (202) of the pre-trained object detection model to obtain an intermediate bootstrapped trained object detection model by: sampling (206) an industrial image (311) from a dataset; segmenting (208) the industrial image (311) into semantic regions using an image segmentation model; sampling (210) two regions of the semantic regions and sample two crops from each region of the two regions; and training (212) the pre-trained object detection model using the two regions and the two crops from each region to obtain the intermediate bootstrapped trained object detection model and using the intermediate bootstrapped trained object detection model to train the neural network for object detection of industrial images (311).

2. The method of claim 1, further comprising: training (212) the pre-trained object detection model using the two regions and the two crops from each region to obtain the intermediate bootstrapped trained object detection model by: training the pre-trained object detection model that the two crops from a same region of the two regions are have similar features and crops from a different region of the two regions have different features.

3. The method of claims 1 or 2, further comprising: fine-tuning (304) the intermediate bootstrapped trained object detection model using a target domain, wherein the target domain comprises images of industrial parts.

4. The method of any one of claims 1-3, wherein the source domain comprises images of everyday objects.

5. The method of any one of claims 1-4, wherein the pre-trained object detection model comprises a backbone (314) for extracting features of an image and a head for mapping the features for prediction.

6. The method of any one of claims 1-5, wherein the image segmentation model is a Segment Anything Model (SAM) (312).

7. The method of any one of claims 1-6, wherein the bootstrap training is unsupervised.

8. The method of any one of claims 1-7, wherein the industrial image (311) is an engine and objects for detection are screws and / or gears.

9. The method of any one of claims 1-8, wherein the neural network model is a convolutional neural network (CNN).

10. A computer device for training a large language model (LLM) for an unseen task comprising: a processor, a memory, the memory storing at least one program code, the at least one program code loaded and executed by the processor to: obtain (202) a pre-trained object detection model, wherein the pre-trained object detection model is pre-trained using a source domain; bootstrap train (204) of the pre-trained object detection model to obtain an intermediate bootstrapped trained object detection model by: sampling (206) an industrial image (311) from a dataset; segmenting (208) the industrial image (311) into semantic regions using an image segmentation model; sampling (210) two regions of the semantic regions and sample two crops from each region of the two regions; and training (212) the pre-trained object detection model using the two regions and the two crops from each region to obtain the intermediate bootstrapped trained objectdetection model and using the intermediate bootstrapped trained object detection model to train the neural network for object detection of industrial images (311).

11. The computer device of claim 10, wherein the processor is further configured to: train (212) the pre-trained object detection model using the two regions and the two crops from each region to obtain the intermediate bootstrapped trained object detection model by: training the pre-trained object detection model that the two crops from a same region of the two regions are have similar features and crops from a different region of the two regions have different features.

12. The computer device of claims 10 or 11, wherein the processor is further configured to: fine-tune (304) the intermediate bootstrapped trained object detection model using a target domain, wherein the target domain comprises images of industrial parts.

13. The computer device of any one of claims 10-12, wherein the source domain comprises images of everyday objects.

14. The computer device of any one of claims 10-13, wherein the pre-trained object detection model comprises a backbone (314) for extracting features of an image and a head for mapping the features for prediction.

15. The computer device of any one of claims 10-14, wherein the image segmentation model is a Segment Anything Model (SAM) (312).

16. The computer device of any one of claims 10-15, wherein the bootstrap training is unsupervised.

17. The computer device of any one of claims 10-16, wherein the industrial image (311) is an engine and objects for detection are screws and / or gears.

18. The computer device of any one of claims 10-17, wherein the neural network model is a convolutional neural network (CNN).

19. A computer readable storage medium, characterized in that the storage medium stores at least one program code for execution by a processor to implement the training method according to any of claims 1 to 9.

20. A computer program product comprising computer instructions stored in a computer readable storage medium; a processor of a computer device reads the computer instructions from the computer readable storage medium, the processor executing the computer instructions, causing the computer device to implement the training method according to any of claims 1 to 9.

Images