Video Generation Model Training With Simulated Labels and Domain Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI-based video generation models require large amounts of accurate labeled data, which is costly, time-consuming, and prone to inconsistency, making it difficult to train models effectively without sufficient data.
Innovation Solution
A method involving training a video generation model by leveraging a combination of virtual and real-world domain styles, using a simulator to generate initial label data, and employing a label interpolation model to enhance frame rate, allowing photorealistic video generation with a small amount of labeled data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual labeling is used to obtain accurate labeled data for training video generation models, then the accuracy and reliability of training data is improved, but the cost and time consumption increase significantly
Solution Approach 1:
The patent uses a simulator to generate virtual videos and automatically creates corresponding label data, copying the structure and content of real videos in a virtual environment. This eliminates manual labeling while providing sufficient training data for the video generation model
Solution Approach 2:
The system performs automatic labeling where the simulator generates both the video content and its corresponding labels autonomously. The labeling process serves itself without external human intervention, reducing time consumption while maintaining data accuracy
2Reliability
If a large amount of labeled data is collected to train video generation models, then the model training effectiveness is improved, but the cost and complexity of data collection increase
Solution Approach 1:
The patent introduces a simulator as an intermediary between real-world videos and the training data requirements. The simulator generates virtual videos that serve as sufficient training data, eliminating the need for complex data collection systems while maintaining model training effectiveness
Solution Approach 2:
Instead of collecting large amounts of real labeled data through complex systems, the patent copies video content and label structures in a virtual simulation environment, providing sufficient training data without the complexity of extensive data collection
3Quantity of substance
If multiple people perform manual labeling to obtain sufficient data, then the quantity of labeled data is improved, but labeling inconsistency issues arise
Solution Approach 1:
The simulator autonomously generates both video content and corresponding labels through automated processes. This self-service approach ensures consistent labeling without human intervention, eliminating inconsistency issues while providing sufficient data quantity
Solution Approach 2:
The system copies label data structures automatically from the simulated video content, ensuring consistent labeling across all generated videos. This automated copying process eliminates the variability introduced by multiple human labelers
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A video generation model training method includes obtaining first time-series label data and time-series images of a first domain style, training a first image generation model based on the first time-series label data and the time-series images of the first domain style, obtaining a plurality of label data sets and a plurality of images of a second domain style, training a second image generation model based on the plurality of label data sets and the plurality of images of the second domain style, training a first video generation model based on the first image generation model, the first time-series label data, and the time-series images of the first domain style, and generating a second video generation model associated with the second domain style based on the second image generation model and the first video generation model.