Prompt Modification Distillation for Low-Latency Text-to-Vision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text-to-vision generation systems face challenges due to the mismatch between user prompts, which are often short and lacking detail, and the longer, more descriptive prompts used during training, leading to reduced performance and limited computational feasibility on user devices.
Innovation Solution
A two-stage approach involving a powerful large language model (LLM) to modify user prompts to align with training data and a smaller, distilled LLM for inference, along with vision scoring models to condition and filter datasets, ensuring improved generation quality and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If a large parameter count generative machine learning model is used to modify prompts, then the quality and alignment of modified prompts improve, but the computational requirements and latency increase
Solution Approach 1:
The patent applies preliminary action by pre-training a smaller model on a dataset generated by a larger model. The training dataset contains prompt-modification pairs created in advance by the larger model, allowing the smaller model to learn from this pre-computed knowledge during inference, thus avoiding real-time computation with the larger model.
Solution Approach 2:
The patent uses copying by creating a distilled version of the larger model's capabilities. A smaller model is trained to replicate the prompt modification behavior of the larger model using a dataset of examples, effectively copying the functional behavior while reducing computational overhead and enabling deployment on resource-constrained devices.
2Manufacturing precision
If a large parameter count generative machine learning model is used, then the generation quality improves, but the computational requirements exceed user device capabilities
Solution Approach 1:
The patent employs copying by training a smaller model to replicate the prompt modification capabilities of a larger model. The smaller model is distilled from the larger model's training data, enabling it to produce similar quality outputs with reduced computational requirements suitable for user devices.
Solution Approach 2:
The patent applies this principle by replacing the computationally expensive large model with a smaller, more efficient model for the actual inference process. The larger model is used only during the training data generation phase, while the smaller model handles the operational workload, effectively using a 'cheaper' computational resource for the ongoing task.
3Ease of operation
If short user prompts are used directly for text-to-vision generation, then the ease of operation improves, but the generation performance deteriorates due to mismatch with training data
Solution Approach 1:
The patent uses an intermediary approach by introducing a prompt modification model that translates simple user prompts into the more detailed format expected by the text-to-vision generation system. This intermediary layer bridges the gap between user convenience and system performance requirements without changing either the user interface or the core generation model.
Solution Approach 2:
The patent applies parameter changes by transforming the structure and content of user prompts through a learned modification process. The model adjusts prompt parameters such as length, detail level, and descriptive elements to match the distribution of training data, thereby improving generation performance while maintaining ease of use.
Data Source
AI summary
There is provided a method performed by one or more data processing apparatus. The method comprises obtaining a training prompt and a corresponding target modified prompt from a training dataset. The training dataset comprises one or more training prompt and target modified prompt pairs generated using a first generative machine learning model. The method further comprises processing, by a second generative machine learning model, the training prompt to generate an output modified prompt. The second generative machine learning model has a lower parameter count than the first generative machine learning model. The method further comprises updating the second generative machine learning model using a training objective based upon the output modified prompt and the target modified prompt.