Vision-Language Model Debiasing With Additive Residuals
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional vision-language systems suffer from inaccuracies in representing diverse groups of people due to biases in training data, leading to biased image search results, and inefficiencies in rectifying these biases through costly and computationally expensive retraining.
Innovation Solution
A vision-language model debiasing system uses additive residuals to disentangle protected-attribute information from image encodings, generating debiased image encodings that reduce bias and improve accuracy while minimizing computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional vision-language systems use biased training data, then they can process image search queries, but they produce inaccurate and biased image search results
Solution Approach 1:
The patent extracts protected attribute information (race, gender, age) from image encodings using an additive residual learner. This extracted information is then removed from the image encoding to create a debiased encoding, effectively separating the harmful bias component from the useful visual representation component.
Solution Approach 2:
The patent introduces an intermediary debiasing module that sits between the image encoding and the search query comparison. This intermediary component processes the image encoding through the additive residual learner to produce a debiased encoding, which then participates in similarity calculations with text queries.
2Reliability
If conventional systems retrain vision-language models to reduce bias, then accuracy improves, but computational resources and time increase excessively
Solution Approach 1:
The patent segments the vision-language model into two independent parts: a pre-existing vision-language model for image encoding and a separate additive residual learner for bias removal. This segmentation allows the debiasing function to be added independently without requiring retraining of the entire model, thus reducing computational time and resources.
Solution Approach 2:
The additive residual learner is pre-trained offline to learn the mapping between biased and debiased image encodings. This preliminary training is performed once during model development, and the learned transformations can be applied efficiently during runtime image search operations without requiring continuous retraining.
3Reliability
If conventional systems retrain vision-language models to reduce bias, then accuracy improves, but computing resources increase excessively
Solution Approach 1:
The patent divides the computational workload into two segments: offline training of the additive residual learner and online application of the learned transformations. This segmentation reduces the computational resources required during actual image search operations, as only the lightweight additive residual learner needs to process images rather than the entire vision-language model.
Solution Approach 2:
The patent creates a copy of the biased vision-language model's image encoding pathway and applies the additive residual learner to this copy to generate debiased encodings. This copying approach allows the system to maintain the original model's capabilities while adding debiasing functionality without requiring complete retraining of the original model.
Data Source
AI summary
The present disclosure relates to systems, non-transitory computer-readable media, and methods for debiasing vision-language models utilizing additive residual learning. In particular, in one or more embodiments, the disclosed systems generate an encoded image representation of a digital image utilizing an image encoder of a vision-language neural network. Additionally, in some embodiments, the disclosed systems extract a protected attribute encoding from the encoded image representation of the digital image utilizing an additive residual learner. Upon extracting the protected attribute encoding, in some implementations, the disclosed systems determine a debiased image encoding for the digital image by combining the protected attribute encoding and the encoded image representation.