Synthetic Video Data Generation via LLM Prompt Decomposition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current text-to-video models fail to synthesize visual processes involving drastic visual changes, often generating a single state without gradual evolution due to suboptimal language guidance and flawed temporal attention.
Innovation Solution
Decompose the text prompt into multiple descriptive sub-prompts using a Large Language Model (LLM) and generate text embeddings for each, then use a Video Diffusion Model (VDM) with temporal attention regularization to create coherent video data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If a single text prompt is used for video generation, then the process is simple and fast, but the visual transitions are inaccurate and lack gradual evolution
Solution Approach 1:
The patent applies segmentation by dividing a single text prompt into multiple temporal sub-prompts that represent different stages of a visual process. For example, a prompt like 'a boy is getting old' is segmented into 'a young boy', 'a middle-aged man', and 'an elderly man'. This segmentation enables the video generation model to produce accurate gradual transitions between visual states while maintaining manageable processing complexity through structured temporal decomposition.
2Reliability
If temporal transformer is added to capture temporal coherency, then video generation capability is improved, but the model still fails to synthesize drastic visual changes
Solution Approach 1:
The patent applies dynamics by making the text embedding dynamic across temporal steps rather than static. Instead of using the same text embedding for all frames, the system generates different text embeddings at different temporal steps corresponding to different visual states. This dynamic text embedding approach, combined with temporal transformer, enables the model to reliably maintain temporal coherency while accurately synthesizing drastic visual changes through temporally-varying language guidance.
3Productivity
If the same text embedding is replicated for all frames, then processing is efficient, but language guidance is suboptimal for temporal transformation
Solution Approach 1:
The patent applies dynamics by transitioning from static text embedding replication to dynamic text embedding generation across temporal steps. The system generates different text embeddings at different temporal steps using the temporal transformer, allowing language guidance to adapt to each visual state. This dynamic approach maintains reasonable processing efficiency while dramatically improving the quality of language guidance for temporal transformations.
4Stability of the object's composition
If conventional T2V models generate initial state only, then generation process is stable, but temporal transformation is not shown
Solution Approach 1:
The patent applies segmentation by dividing the video generation process into multiple temporal steps, each targeting a specific visual state. The temporal transformer coordinates these segmented steps to produce a sequence of frames showing gradual evolution from initial to final states. This segmentation approach maintains generation stability through structured temporal processing while achieving accurate temporal evolution by explicitly modeling intermediate states.
Data Source
AI summary
A method for generating synthetic video data form a text prompt, particularly for providing video data for training and/or testing and/or verifying and/or validating a machine learning model. The method includes: providing an input text prompt descriptive for the content of the video data to be generated; decomposing the provided text prompt into at least two text sub-prompts by a large language model; generating a text embedding for each of the at least two text sub-prompts; and generating synthetic video data by a Video Diffusion Model based on the generated text embeddings.