Contrastive Fine-Tuning Zero-Shot Vision Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Standard fine-tuning procedures for large-scale models pretrained on image and text data, such as CLIP, improve in-distribution performance but come at the cost of robustness to distribution shifts.
Innovation Solution
The method involves casting downstream class labels as text prompts and continuing to optimize the contrastive loss between image embeddings and class-descriptive prompt embeddings, effectively mimicking the pretraining process through a process called contrastive fine-tuning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard supervised fine-tuning with crossentropy loss is applied to improve in-distribution classification performance, then classification accuracy on training distribution improves, but robustness to distribution shifts deteriorates
Solution Approach 1:
The patent changes the loss function parameter from crossentropy loss to contrastive loss, and modifies the training objective to optimize image-text similarity rather than direct classification. This parameter change in the optimization objective preserves robustness while maintaining accuracy improvements
Solution Approach 2:
The patent performs preliminary action by continuing to optimize the contrastive loss between image embeddings and text embeddings (mimicking pretraining) before or during fine-tuning, rather than directly applying supervised classification loss. This preliminary optimization preserves the robustness properties learned during pretraining
2Productivity
If the pre-trained model parameters are updated through supervised learning on labeled images, then in-distribution performance improves, but the model's generalization capability to unseen distributions deteriorates
Solution Approach 1:
Instead of directly applying supervised learning updates to improve in-distribution performance (the conventional approach), the patent inverts the approach by continuing to optimize the contrastive loss objective (similar to pretraining) rather than applying crossentropy loss. This inverted approach surprisingly achieves both improved in-distribution performance and preserved generalization capability
Data Source
AI summary
A method discloses receiving a plurality of input images, receiving text prompts, generating a visual matrix utilizing the images and an image encoder, generating a text matrix utilizing a text encoder, multiplying the text matrix and the visual matrix to generate an image-text similarity matrix that assigns a numerical value indicating similarities between each of encoded visual descriptors and each of the encoded images, wherein similarities are indicated by entries of the image-text similarity matrix having numerical values that determine a loss function associated with the image-text similarity matrix, identify a gradient of the loss function with respect to parameters associated with the image encoder and parameters associated with the text encoder, utilizing the gradient, update parameters associated with the image encoder or the text encoder, and outputting final updated parameters associated with either the text encoder or image encoder of the machine learning network.