Diffusion Model Fairness via Distributional Alignment Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Text-to-image diffusion models often generate images with biases related to race, gender, and occupation, failing to represent minority groups fairly.
Innovation Solution
A method and system that optimize diffusion models by aligning generated images towards a target distribution using distributional alignment loss and adjusting the model through direct finetuning of the sampling process to minimize loss functions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If diffusion models are used for text-to-image generation, then image generation capability is improved, but bias in generated images increases
Solution Approach 1:
The patent introduces an intermediary debiasing module that sits between the diffusion model and the output. This module receives generated images, identifies biased attributes using pre-trained classifiers, and applies transformations to adjust the distribution of these attributes toward a target distribution, thereby reducing bias while preserving the core image generation capability
Solution Approach 2:
The patent changes the parameters of the generated images by adjusting the distribution of specific attributes (such as gender, race, occupation) through the debiasing module. By modifying the probability distribution of these attributes to match a target distribution, the system reduces bias without fundamentally altering the image generation process
2Reliability
If bias reduction techniques are applied to diffusion models, then fairness of generated images is improved, but image generation quality may deteriorate
Solution Approach 1:
The patent applies partial action by selectively debiasing only certain attributes (gender, race, occupation) rather than all aspects of the generated images. The debiasing module identifies specific biased attributes and adjusts only those, leaving other quality-determining aspects of the images unchanged, thus maintaining overall image quality while improving fairness
3Reliability
If distributional alignment loss is used to align generated images toward target distribution, then bias reduction is improved, but computational complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-training classifiers for attribute identification before the actual debiasing process. These pre-trained classifiers are ready to use when images need to be debiased, avoiding the need to train them from scratch each time. This preliminary preparation reduces the computational complexity of the main debiasing operation
Data Source
AI summary
The present invention provides solutions for reducing biases in text-to-image diffusion models, particularly biases related to gender, race, and their intersections in occupational prompts. The invention introduces a fairness framework based on distributional alignment, comprising two core technical solutions: (1) a distributional alignment loss that adjusts the output of the model toward user-defined target distributions, and (2) an adjusted direct finetuning (adjusted DFT) of the model's sampling process using an adjusted gradient to optimize losses based on generated images. These techniques reduce bias while supporting diverse perspectives on fairness, such as age-controlled debiasing across multiple concepts. The method's scalability allows for debiasing multiple prompts simultaneously, improving the inclusivity of diffusion model outputs across varied demographics.