Autoregressive Video Generation With Progressive Long-Sequence Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing autoregressive generative language models struggle to generate long videos with consistent appearance and motion dynamics due to imbalanced loss during training and error accumulation during inference, limiting their ability to capture complex events over extended durations.
Innovation Solution
Implement a progressive multi-stage training process with loss re-weighting and inference strategies, including a video tokenizer and a decoder-only transformer, to mitigate imbalanced loss and error accumulation, enabling the generation of long videos by gradually increasing training video length and enhancing resolution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Duration of action of moving object
If autoregressive language models are used for video generation, then long video generation capability is improved, but imbalanced loss during training and error accumulation during inference occur
Solution Approach 1:
The patent segments the video generation task into multiple stages: first training on short video clips to establish basic capabilities, then progressively training on longer clips to extend temporal context. This segmentation allows the model to learn from manageable units before handling longer sequences, reducing the complexity of training and improving reliability.
Solution Approach 2:
The patent implements preliminary training on short video clips before transitioning to longer video generation. By pre-training on shorter sequences, the model establishes solid foundational representations and reduces initial errors before attempting to generate longer videos, thereby mitigating error accumulation during inference.
2Stability of the object's composition
If training video length is increased to generate long videos, then temporal coherence is improved, but training complexity and computational cost increase
Solution Approach 1:
The training process is segmented into multiple stages with progressively longer video clips. Instead of training directly on long videos which would be computationally intensive and unstable, the model first learns from short clips, then gradually increases clip length. This staged approach reduces training complexity at each stage while building temporal coherence incrementally.
Solution Approach 2:
The patent dynamically adjusts the training video length based on the current stage of training. The maximum video length for training is progressively increased across stages, allowing the model to adapt to longer temporal contexts as training progresses. This dynamic adjustment balances computational cost with improving temporal coherence.
3Reliability
If progressive multi-stage training is implemented, then error accumulation is reduced, but training time and computational resources increase
Solution Approach 1:
The training process is divided into multiple stages, each focusing on progressively longer video clips. By segmenting training into manageable stages with increasing complexity, the model reduces error accumulation at each stage while the cumulative effect achieves long-video generation capability. This segmentation makes the overall training process more efficient than a single-stage approach.
Solution Approach 2:
The model performs preliminary training on short video clips to establish solid foundational representations before transitioning to longer clips. This preliminary action reduces the difficulty of subsequent training stages and minimizes error accumulation, making the overall training process more time-efficient despite the multi-stage structure.
Data Source
AI summary
Implementations for autoregressively generating a video using a video generation model are provided. One aspect includes a method comprising: performing a progressive multi-stage training process comprising a first stage and a second stage, wherein: the first stage comprises training the video generation model to perform text-to-image generation; and the second stage comprises further training the video generation model using a training dataset comprising labeled video-text pairs, wherein further training the video generation model comprises: for each of the labeled video-text pair: generating at least one text token using a text tokenizer and a text annotation of the labeled video-text pair; generating a plurality of video tokens using a video tokenizer and a video of the labeled video-text pair; autoregressively generating frame tokens using the at least one text token; and training the video generation model using loss values calculated from the frame tokens and the video tokens.