Generative Data Augmentation for Protected Attribute Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face challenges in estimating protected attributes due to data constraints, legal and ethical risks, and over-training on specific attributes, leading to inadequate generalization and fairness issues.
Innovation Solution
A data augmentation method using generation models to generate images and captions, extracting and registering additional attributes from text, and creating augmented data sets for training, which includes attributes not directly present in the original data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a machine learning model is trained using only original attributes from the data set, then the training process is simple and fast, but the model performs over-training to specific attributes and cannot generalize well
Solution Approach 1:
The patent applies preliminary action by generating augmented data (images and captions) before the actual model training process. The system pre-generates additional attribute information through generation models, so that when training begins, the model already has access to diverse attributes without delay. This resolves the contradiction by preparing expanded training data in advance, enabling both comprehensive attribute coverage and efficient training execution.
Solution Approach 2:
The patent uses copying by creating synthetic representations of data through generation models. Instead of collecting more real-world data, the system generates copied versions (images and text captions) that mimic real data patterns but provide additional attribute information. This allows the model to learn from diverse attributes without requiring proportionally more original data, maintaining training efficiency while improving generalization.
2Reliability
If the data set is augmented by adding more attributes, then the model can estimate protected attributes better and generalize well, but the data processing complexity and time increase
Solution Approach 1:
The patent introduces generation models as intermediaries between the original data and the training process. These intermediary models automatically generate images and captions from original attributes, bridging the gap without requiring manual data collection or complex processing pipelines. This intermediary approach maintains reliability by ensuring attribute consistency while managing complexity through automated generation rather than manual intervention.
Solution Approach 2:
The system applies self-service by enabling the data augmentation process to be automatically performed by the generation models without extensive human intervention. The models self-generate augmented data based on original attributes, and the system automatically integrates this generated data into the training set. This self-service mechanism improves protected attribute estimation while keeping processing complexity manageable through automation rather than manual data curation.
3Adaptability or versatility
If generation models are used to create augmented data, then attribute diversity increases and over-training is reduced, but computational resources and processing time are consumed
Solution Approach 1:
The patent applies partial action by generating augmented data for only the necessary attributes required for protected attribute estimation, rather than exhaustively generating all possible attributes. The system selectively creates images and captions for attributes that directly contribute to improving model generalization and fairness. This partial approach achieves sufficient attribute diversity while avoiding the excessive computational cost of generating complete attribute sets for all data points.
Data Source
AI summary
A non-transitory computer-readable recording medium stores therein a data augmentation program that causes a computer to execute a process. The process includes acquiring an image by inputting, into a first generation model, first text including attribute values of a plurality of attributes included in first data among a plurality of pieces of data. The process includes acquiring second text by inputting the image into a second generation model. The process includes selecting an attribute value of other attribute different from the plurality of attributes from the second text. The process includes augmenting the plurality of pieces of data by adding the attribute value of the other attribute to the first data.