Transformer Object Detector Training With Self-Supervised Pseudo-Labels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing transformer-based object detection models like DETR are sample-inefficient and require extensive annotated data for training, with previous self-supervised methods focusing on backbone encoders and ignoring class-related information, leading to slow convergence and dependence on pre-trained backbones.
Innovation Solution
The SimDETR framework employs self-supervised pretraining with improved object proposals through clustering and iterative self-training, aligning pretraining with downstream tasks by using pseudo-labels and iteratively refining the detector, allowing both backbone and detection head to be trained simultaneously.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If transformer-based object detection models like DETR are used, then end-to-end trainable single-stage detection is achieved, but sample efficiency deteriorates and extensive annotated data is required
Solution Approach 1:
The patent applies self-supervised pretraining as a preliminary action before fine-tuning on annotated data. The model first learns from unlabelled images through self-supervised learning, acquiring general visual representations and object priors. This preliminary training reduces the amount of annotated data needed for subsequent supervised fine-tuning, directly addressing the sample inefficiency of transformer-based detectors while preserving their end-to-end trainable capability.
2Reliability
If self-supervised pretraining is applied to backbone encoders, then representation learning is improved, but class-related information is lost and convergence slows
Solution Approach 1:
The patent implements a feedback mechanism where the detector head provides class-related information back to the backbone encoder during self-supervised pretraining. Through the detection task itself, the model learns class-aware representations without requiring annotated labels. This feedback loop ensures that class-related information is preserved and leveraged during pretraining, preventing information loss while improving representation learning quality and accelerating convergence.
3Extent of automation
If DETR architecture is used, then hand-crafted components are bypassed, but training convergence becomes slow
Solution Approach 1:
The patent applies self-supervised pretraining as a preliminary action that accelerates subsequent supervised training convergence. By pretraining on unlabelled data, the model develops better initial representations and object priors, which serve as a strong foundation for the final detection task. This preliminary training reduces the time required for convergence during supervised fine-tuning, compensating for the slower convergence inherent in transformer-based architectures while maintaining the benefit of eliminating hand-crafted components.
Data Source
AI summary
A computer-implemented method for training a machine learning, ML, model to perform object detection, the method comprising: obtaining a first training dataset comprising a plurality of unlabelled images, each unlabelled image containing at least one object; analysing the first training dataset by using an object detector module of the ML model; forming a second training dataset using the unlabelled images of the first training dataset and their corresponding extracted bounding boxes and pseudo-labels; and training the object detector module, using the second training dataset, to output bounding boxes and pseudo-labels for input pseudo-labelled images.