Optical-Flow Guided Training for Stable AI Video Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative AI models suffer from temporal stability issues, leading to visual artifacts such as temporal flickering and content distortions due to powerful content generation abilities, making them unsuitable for real-world applications without computationally expensive components like transformer neural networks or 3D convolutions.
Innovation Solution
A spatiotemporal stability oriented training framework that introduces effective spatiotemporal losses to train generative AI models, using temporally registered video portions as batch training data, and employs an optimizer to update model parameters for both spatial and temporal consistency, without relying on expensive structures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If transformer neural networks or 3D convolutions are used to improve temporal stability, then temporal consistency is improved, but device complexity and computational cost increase
Solution Approach 1:
The patent applies preliminary action by extracting temporal information (optical flow and motion vectors) from video data before training the generative model. This preprocessed temporal information is then incorporated as additional input channels to guide the model's temporal consistency, avoiding the need for complex temporal processing structures like transformers or 3D convolutions during generation.
Solution Approach 2:
The patent introduces an intermediary approach by using optical flow and motion vector fields as mediators between input video data and the generative model. These intermediate representations capture temporal dynamics and are fed into the model as guidance, enabling temporal stability without requiring the model to inherently understand temporal relationships through complex architectures.
2Manufacturing precision
If generative AI models are trained with standard image data, then image quality is improved, but temporal consistency deteriorates
Solution Approach 1:
The patent merges multiple data sources by combining standard image data with extracted temporal information (optical flow and motion vectors) into a unified training input. This combination allows the model to simultaneously learn from both visual quality aspects and temporal dynamics, producing outputs that maintain both high image quality and temporal consistency.
Solution Approach 2:
The patent adds another dimension to the training data by incorporating temporal information as additional input channels alongside standard image data. This dimensional expansion allows the model to process both spatial and temporal information simultaneously, improving temporal consistency without sacrificing image quality.
3Reliability
If computationally intensive components are used, then temporal stability is improved, but processing speed decreases
Solution Approach 1:
The patent applies preliminary action by performing temporal information extraction (optical flow and motion vector calculation) during the offline training phase rather than during real-time generation. This shifts the computational burden to a preprocessing stage, allowing fast inference during actual video generation while still achieving temporal stability through the incorporated temporal guidance.
Data Source
AI summary
A method includes identifying at least one point in a set of image frames within a temporal window. The set of image frames within the temporal window forms video content. The method also includes extracting temporal information including movement of the at least one point through the set of image frames within the temporal window based on estimation of a local motion vector and/or a supervised optical flow represented in the set of image frames. The method further includes generating a video portion based on association of the temporal information with the set of image frames within the temporal window. In addition, the method includes inputting the video portion as at least part of batch training data for one or more generative machine learning models, where the one or more generative machine learning models that are configured by being trained with the video portion generate temporally stable video content.