Generative Image Model Training with Embedded Moderation Labels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current techniques for image moderation in generative AI models are ineffective in preventing the generation of undesirable content, such as nudity or violence, as users can evade moderation through prompt engineering or loss of quality due to filtered training data.
Innovation Solution
Training generative image models with annotated training data that includes both image captions and moderation labels, conditioning the model to generate images based on both inputs, ensuring compliance with desired content guidelines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If prompt processing is used for moderation, then moderation capability is improved, but users can evade moderation through prompt engineering
Solution Approach 1:
The system performs preliminary action by annotating training data with moderation labels before model training, embedding moderation constraints directly into the model's learned representations. This prevents evasion because the model internally understands moderation rules rather than relying on external prompt filtering that users can bypass through engineering.
Solution Approach 2:
The system introduces an intermediary layer of moderation labels that act as a mediator between the input prompt and image generation process. These labels serve as a bridge that translates user intent while enforcing moderation constraints, making it difficult for users to evade moderation through prompt engineering tricks.
2Reliability
If training data is filtered to remove undesired content, then moderation is improved, but image generation quality deteriorates
Solution Approach 1:
The system applies local quality by associating specific moderation labels with specific training images rather than uniformly filtering the entire training dataset. This allows the model to learn from all available images while locally applying moderation constraints only where needed, preserving overall image generation quality while achieving moderation goals.
Solution Approach 2:
The system changes parameters by adding moderation labels as an additional dimension to the training data rather than removing data points. This transforms the training process from data filtering to parameter-based constraint learning, allowing the model to maintain access to all training images while learning to respect moderation boundaries through the label parameters.
Data Source
AI summary
A method for training a generative image model includes receiving training data having training images and image captions and annotating the training data resulting in annotated training data having the training images, the image captions, and moderation labels, such that each training image is associated in the annotated training data with a corresponding image caption and at least one corresponding moderation label. The method further includes performing a training process to train a generative image model using the training images, the image captions, and the moderation labels, resulting in a trained model that generates images conditioned to both the image captions and the moderation labels.