Video Frame Synthesis With Shared Motion Latent Control

The video synthesis process is segmented into two distinct stages: training at low resolution and inference at high resolution. The model is trained on downsampled low-resolution videos to learn motion patterns, then during inference, it generates high-resolution frames by predicting frame differences. This segmentation allows the model to achieve high video quality without requiring substantial computational resources during training.

The patent transforms the problem from direct high-resolution video generation to a two-dimensional approach: first learning in the low-resolution domain during training, then projecting to the high-resolution domain during inference. By operating in the latent space of a pre-trained VAE and predicting frame differences rather than full frames, the model achieves high-resolution output with reduced memory requirements.

The patent applies preliminary action by pre-training a Variational Autoencoder (VAE) on large-scale video datasets before using it for the actual synthesis task. This pre-trained VAE provides a robust latent space representation that can be leveraged without requiring the main model to relearn basic video structures from scratch. Additionally, the motion embedding layer is pre-trained on large datasets to capture motion patterns, which are then reused during inference, reducing both training time and computational complexity.

Instead of training on full high-resolution video frames, the model works with compressed representations in the latent space of the pre-trained VAE. The motion embedding layer captures essential motion patterns that are then copied and applied across different video sequences. This copying approach allows the model to achieve high synthesis quality without requiring extensive training on large-scale high-resolution video datasets.

The patent operates in the latent space dimension rather than pixel space during training. By encoding videos through a pre-trained VAE and performing synthesis in this compressed latent representation, the model achieves high-resolution output capability while requiring minimal VRAM during training. The dimensionality reduction inherent in the VAE encoding allows the model to learn high-resolution video patterns without storing or processing large amounts of high-resolution data.

The synthesis process is segmented into learning motion patterns separately from learning appearance details. The motion embedding layer captures temporal dynamics at low resolution, while the frame prediction network generates high-resolution frame differences during inference. This segmentation allows the model to achieve high output quality without requiring high VRAM during training, as the computationally intensive high-resolution generation occurs only during inference on a single frame at a time.