Motion Vector Field Smoothing for Frame Rate Up Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Frame rate up conversion in video applications often results in a 'halo effect' due to erroneous motion estimation, particularly in occlusion regions, leading to undesirable interpolation around foreground objects.
Innovation Solution
A method that estimates and merges forward and backward motion vector fields using a recursive search algorithm, calculating a smoothness cost based on the sum of absolute differences and local motion vector smoothness metrics to generate a reliable and smoother motion vector field.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If motion estimation is performed for frame rate up conversion, then new frames can be generated for display, but erroneous motion vectors are assigned to occlusion regions causing halo effect
Solution Approach 1:
The patent segments the motion estimation process into multiple passes: a first pass that performs motion estimation without smoothness constraints to capture gross motion, and a second pass that refines motion vectors with smoothness constraints applied only to non-occluded regions. This segmentation allows the system to handle occlusion regions differently from regular regions, improving overall accuracy while maintaining productivity.
Solution Approach 2:
The patent applies different processing quality to different regions of the image. Non-occluded regions receive full smoothness constraint processing in the second pass, while occluded regions are identified and excluded from certain processing steps. This local quality approach ensures that regions with reliable motion information are refined while avoiding the introduction of errors in occluded regions, thereby reducing the halo effect.
2Stability of the object's composition
If smoothness constraints are applied to motion vector field, then spatial and temporal smoothness is achieved, but erroneous motion vectors in occlusion regions are reinforced
Solution Approach 1:
The patent implements a dynamic two-pass approach where the processing strategy changes based on the region type. In the first pass, all regions are processed to establish initial motion vectors. In the second pass, the system dynamically identifies occluded regions and applies smoothness constraints selectively only to non-occluded regions. This dynamic adaptation allows the system to maintain smoothness where appropriate while avoiding reliability degradation in occlusion regions.
Solution Approach 2:
The patent uses feedback from the first pass motion estimation results to guide the second pass processing. The initial motion vectors and their associated error metrics provide feedback that helps identify occluded regions. This feedback mechanism allows the system to adjust its processing strategy, applying smoothness constraints only where they will improve results rather than harm them, thus maintaining both smoothness and reliability.
Data Source
Figure 1~2
Figure 3~5
AI summary
A forward-backward motion estimation method in frame rate up conversion for video applications comprising the steps of; updating a forward motion vector field (MVF_fw) and a backward motion vector field (MVF_fw) using a recursive search method, merging updated vector fields (MVF_fw_upd, MVF_bw_upd) by selecting motion vector field which has minimum sum of absolute difference value and by adding a smoothness parameter, which is a vector field that depends on selected vector field. In this method, neighbour blocks of the block to be updated are used to calculate smoothness parameter. Thus, by this method a smoother motion vector field is generated.