Vision-Language Model Fine-Tuning for Multi-Label Chest X-Ray Classification

The patent segments the contrastive learning process into two distinct phases: (1) pretraining with web-scale image-text pairs to learn general visual-linguistic representations, and (2) task-specific fine-tuning with medical imaging data to adapt to multi-label pathology classification. This segmentation allows the model to maintain general capabilities while acquiring specialized multi-label classification skills without interference between learning objectives

The patent modifies key training parameters during fine-tuning, including adjusting the temperature scaling parameter (tau) to 0.07, modifying the loss function to incorporate multi-label specificity, and tuning learning rates differently for image and text encoders. These parameter changes enable the model to optimize for multi-label pathology detection while preserving the general representation learning from pretraining

The patent performs pretraining on large-scale web-scale image-text pairs before fine-tuning on smaller medical datasets. This preliminary action on abundant general data provides a strong foundation that reduces the amount of time-critical expert-labeled medical data needed for subsequent fine-tuning, thereby decreasing overall data acquisition time while maintaining or improving final accuracy

The model leverages its own pretrained representations to perform effective fine-tuning on limited medical data without requiring extensive external expert annotation. The self-supervised pretraining phase enables the model to serve itself by providing robust features that reduce dependency on large amounts of manually labeled medical imagery

The patent applies different augmentation strategies to different components: minimal or no spatial augmentations to medical images to preserve critical diagnostic information, while applying more aggressive augmentations to text data (such as synonym replacement, back-translation, and random insertion). This local differentiation maintains image fidelity while still providing robustness through text variations

The patent implements dynamic augmentation where the intensity and type of augmentations are adjusted based on the specific medical imaging modality, pathology type, and model training stage. For example, lighter augmentations are applied to early training stages or to images containing critical features, while more diverse augmentations are used when the model demonstrates sufficient robustness

The patent segments the labeling structure by separating the image-level multi-label classification from sentence-level pathology descriptions. This segmentation allows the contrastive loss to operate efficiently on paired image-text units while a separate mechanism handles the multi-label aspect through multiple positive text pairs per image, maintaining training efficiency while enabling multi-pathology detection

The patent enhances the contrastive loss function to serve multiple functions simultaneously: it performs the standard image-text matching while also learning to distinguish between multiple correct pathology labels for the same image. By incorporating multiple positive text pairs per image into the contrastive framework, the loss function becomes multi-functional, handling both single and multi-label cases without requiring separate training mechanisms