Multi-Scale Image Interpolation Model for Large Motion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image interpolation methods fail to effectively handle large scene motion in near-duplicate photographs, leading to poor generalization and increased training complexity, especially when dealing with scarce data for pre-training optical flow and depth networks.
Innovation Solution
A machine-learned image interpolation model with a multi-scale feature extractor sharing weights across scales and a novel scale-agnostic bi-directional motion estimator, optimized with the Gram matrix loss for improved realism and sharpness, and a unified single-network architecture trainable from regular frame triplets, simplifying the training process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing interpolation methods are used for large scene motion, then training complexity increases and generalization performance deteriorates, but if conventional methods are used, then training complexity remains manageable however generalization to large motion fails
Solution Approach 1:
The patent segments the feature extraction process into multiple scales (fine, medium, coarse) with shared weights, allowing the model to handle both small and large motion effectively. This segmentation enables the network to process different motion magnitudes at appropriate scales without increasing overall training complexity
Solution Approach 2:
The patent implements a universal feature extractor that serves multiple functions: it extracts features for both small and large motion scenarios, and the same extractor is used across all scales with shared weights. This multi-functionality improves generalization to large motion while avoiding the need for separate pre-trained networks
2Manufacturing precision
If additional pre-trained networks are added to handle large motion, then image quality improves, but device complexity and training difficulty increase
Solution Approach 1:
The patent merges the feature extraction, motion estimation, and image generation functions into a single unified network. This eliminates the need for separate pre-trained depth networks or optical flow networks, reducing overall network complexity while maintaining high image quality through integrated multi-scale processing
Solution Approach 2:
The single network performs multiple functions: it estimates motion, extracts multi-scale features, and generates interpolated images. The feature extractor is universally applied across all scales and motion magnitudes, replacing the need for multiple specialized pre-trained networks
3Reliability
If pre-trained optical flow and depth networks are used, then handling of large motion improves, but scarcity of pre-training data becomes a limiting factor
Solution Approach 1:
The patent makes the feature extractor self-sufficient by sharing weights across all scales and motion magnitudes. The network learns to handle both small and large motion from the same training data without requiring separate pre-training on scarce optical flow or depth data, eliminating the dependency on limited pre-training resources
4Adaptability or versatility
If interpolation is performed between near-duplicate photos with large motion, then engaging videos can be created, but existing methods fail to generalize
Solution Approach 1:
The patent segments feature extraction into multiple scales that can handle different motion magnitudes. The fine scale captures small details, medium scale handles moderate motion, and coarse scale manages large motion, allowing the system to generalize across the full range of motion in near-duplicate photos
Solution Approach 2:
The patent changes the parameter of scale in feature extraction to adapt to different motion magnitudes. By adjusting which scales are emphasized and sharing weights across scales, the model can generalize to various motion ranges without retraining, making it versatile for near-duplicate interpolation
Data Source
AI summary
Provided is a computer system that includes one or more processors and one or more non-transitory computer-readable media that collectively store a machine-learned image interpolation model. The machine-learned image interpolation model is configured to: extract, for each of multiple different scales, a respective set of feature values from each of a pair of input images; generate, for each of the multiple different scales, a respective flow estimate for each of the pair of input images that indicates a respective flow from the interpolation time to the respective capture time; warp, for each of the multiple different scales, the respective set of feature values for each of the pair of input images according to the respective flow estimate to generate respective warped sets of features; and generate a interpolated image based on the respective warped sets of features for the pair of input images and the multiple different scales.