Image Processing Model Training With LLM-Targeted Synthetic Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Metric Learning (DML) models in image retrieval systems suffer from overfitting and limited generalization due to training on limited datasets, leading to reduced accuracy and susceptibility to adversarial attacks.
Innovation Solution
An improved training methodology using large language models (LLMs) to generate synthetic images by identifying under-represented domains and classes, enhancing the training set with these images, and employing a text-to-image model to create targeted synthetic data, along with outlier removal and diversity control to refine the training process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If DML models are trained on limited datasets, then training time and computational resources are reduced, but model generalization and robustness deteriorate
Solution Approach 1:
The patent applies synthetic data generation to create copies of training images through text-to-image models. Real images are converted to text descriptions, then synthesized back into images that augment the training set. This copying approach expands limited training data without requiring additional real-world image collection, thereby improving model generalization while maintaining training efficiency.
Solution Approach 2:
The patent performs preliminary analysis of the training dataset to identify under-represented domains and classes before generating synthetic images. By pre-analyzing data distribution and targeting specific gaps, the system prepares a focused synthetic data generation strategy that efficiently improves model robustness without unnecessary computational overhead.
2Reliability
If synthetic images are generated to enhance training data, then model generalization improves, but computational resources and training time increase
Solution Approach 1:
The patent applies local quality by generating synthetic images specifically for under-represented domains and classes rather than uniformly across all categories. The system identifies specific data gaps through text description analysis and targets synthetic generation only where needed, optimizing computational resources while effectively improving model generalization for critical areas.
Solution Approach 2:
The patent changes parameters by converting images to text descriptions and back to synthetic images through controlled text-to-image generation. This parameter transformation approach allows precise control over synthetic image characteristics, enabling efficient generation that targets specific data deficiencies without excessive computational resource consumption.
3Reliability
If the training set is expanded with synthetic images, then data balance and model robustness improve, but data processing complexity increases
Solution Approach 1:
The patent introduces text descriptions as an intermediary between real images and synthetic image generation. Real images are converted to text descriptions that serve as mediators, which then guide the text-to-image models to generate appropriate synthetic images. This intermediary approach simplifies the overall process by using natural language as a bridge, making the system more manageable despite increased data processing requirements.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method comprising generating image descriptions of images in an original training set of images; determining, using at least one LLM, at least one domain and/or class which is under-represented in the original training set; generating, using a second LLM and based on the determination of the at least one domain and/or class, at least one instruction for a third LLM to generate at least one text prompt; generating, using the third LLM and based on the at least one instruction, the at least one text prompt for a text-to-image model; generating, using the text-to-image model and based on the at least one text prompt, at least one synthetic image; and generating an enhanced training set of images for use in training an image processing machine learning, ML, model, the enhanced training set of images comprising the original training set of images and the at least one synthetic image.