Fake Optical Flow for Temporally Consistent Video Synthesis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for estimating optical flow using pre-trained models are inaccurate and fail to provide reliable temporal consistency, leading to misleading guidance and compromised content preservation in video synthesis, especially for hard categories.
Innovation Solution
A computer-implemented method for learning temporally consistent video synthesis using fake optical flow, which involves processing image-to-image translation across domains of source and target videos with unsupervised losses to construct bidirectional unsupervised temporal regularization, facilitating neural network training for accurate and consistent video synthesis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If pre-trained models are used to estimate optical flow, then the process is simplified, but the accuracy and reliability of temporal consistency deteriorates
Solution Approach 1:
The patent creates fake optical flow by copying and transforming existing video frames through geometric transformations and pixel manipulations. This synthetic optical flow serves as a substitute for expensive pre-trained model outputs, achieving both process simplicity and improved accuracy by eliminating reliance on external pre-trained models while maintaining computational efficiency.
2Use of energy by moving object
If pre-trained models are used for optical flow estimation, then computational resources are reduced, but temporal consistency regularization fails to preserve content
Solution Approach 1:
The patent introduces fake optical flow as an intermediary element that mediates between computational efficiency and temporal consistency. This synthetic optical flow acts as a bridge, providing the necessary motion guidance for temporal regularization without requiring expensive pre-trained models, thereby preserving content while reducing computational resource consumption.
3Measurement precision
If accurate optical flow estimation is achieved through pre-trained models, then temporal consistency improves, but the cost of managing and updating datasets increases
Solution Approach 1:
The patent enables the system to generate its own optical flow data internally through fake optical flow synthesis, eliminating the need for external pre-trained models and their associated datasets. This self-service approach allows the system to produce accurate optical flow estimates without the complexity of managing, updating, or maintaining external datasets, while still achieving reliable temporal consistency.
Data Source
AI summary
A system and method for learning temporally consistent video synthesis using fake optical flow that include receiving data associated with a source video and a target video. The system and method also include processing image-to-image translation across domains of the source video and the target video and processing a synthesized temporally consistent video based on the image-to-image translation. The system and method further include training a neural network with data that is based on synthesizing of the source video and the target video.