Frame Interpolation Using Optical Flow and Game Engine Motion Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing frame interpolation methods in video games often result in undesirable effects such as flickering, dragging, and halo effects due to performance bottlenecks and inaccuracies in motion estimation, particularly when dealing with objects and special effects.
Innovation Solution
A method that generates interpolated frames using both optical flow and in-game motion estimation, with a weighting map to combine pixel locations from each estimator, ensuring accurate motion compensation and reducing visual quality degradation by identifying and handling potentially inaccurate pixels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If frame rate is reduced to lower processor loading, then processor load is reduced, but temporal up-sampling quality deteriorates due to missing frames
Solution Approach 1:
The patent divides the frame interpolation task into two separate motion estimation processes: optical flow-based motion estimation and game engine-based motion estimation. Each estimator handles different aspects of motion, allowing the system to process frames at lower rates while maintaining high interpolation quality by selectively combining results from both estimation methods.
Solution Approach 2:
The patent creates a composite motion estimation approach by combining optical flow motion vectors with game engine motion vectors. The system generates a first interpolated frame using optical flow and a second interpolated frame using game engine motion, then blends them using a weighting map to produce the final output, achieving superior quality compared to single-method approaches.
2Device complexity
If single motion estimator is used for frame interpolation, then device complexity is reduced, but motion estimation accuracy deteriorates causing visual artifacts
Solution Approach 1:
The patent introduces a weighting map as an intermediary mechanism that mediates between optical flow motion estimation and game engine motion estimation. The weighting map determines which pixels should use motion vectors from optical flow and which should use game engine motion vectors, enabling accurate handling of different scene elements (e.g., optical flow for natural scenes, game engine for UI elements and special effects).
Solution Approach 2:
The patent applies different motion estimation methods to different regions of the image based on local requirements. The weighting map identifies regions where optical flow provides accurate motion (typically natural scenes) and regions where game engine motion is more reliable (UI elements, special effects, occluded objects), applying the appropriate method locally rather than using a single global approach.
Data Source
AI summary
A frame interpolation method generates an interpolated frame that is temporally between a first frame and a second frame. A first and a second interpolated frames are generated using motion vectors from a first motion estimator and a second motion estimator, respectively. A weighting map is generated based on indications from the first motion estimator. First pixel locations and second pixel locations in the weighting map are assigned weight values of 1 and 0, respectively. A weighted combination is calculated using the weighting map to produce the interpolated frame output, which includes the first pixel locations from the first interpolated frame and the second pixel locations from the second interpolated frame. The first and the second motion estimators may be an optical flow estimator and the game engine renderer, respectively. Alternatively, the first and the second motion estimators may be the game engine renderer and the optical flow estimator, respectively.