Video Frame Feature Map Fusion Using Optical Flow Error Weighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Consecutive video frames often experience processing errors leading to jitter issues due to inaccuracies in image processing, particularly when large errors occur between adjacent frames.
Innovation Solution
The method involves converting a feature map of a previous video frame using optical flow generated from adjacent frames, determining a weight based on the optical flow error, and fusing the feature maps to create an updated feature map, which neutralizes positional deviations and improves accuracy by weighting the features accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If image processing is performed on each video frame independently, then processing speed and simplicity are improved, but processing accuracy deteriorates leading to jitter problems
Solution Approach 1:
The patent performs preliminary actions by generating optical flow between adjacent frames before processing the current frame. The optical flow information is used to guide the processing of the current frame, ensuring that motion compensation and alignment are done in advance, which prevents jitter while maintaining processing efficiency.
Solution Approach 2:
The patent implements feedback by using the optical flow error information from processed frames to adjust the processing of subsequent frames. The system continuously refines the optical flow estimation and uses this feedback to improve the accuracy of motion compensation, thereby reducing jitter without significantly increasing processing time.
2Measurement precision
If optical flow is used to convert feature maps between adjacent frames, then positional accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent applies local quality by performing optical flow-based feature map conversion selectively. Instead of processing all feature maps with the same level of detail, the system focuses computational resources on regions with significant motion or high importance, using the optical flow to identify these regions. This approach maintains high positional accuracy while reducing overall computational complexity.
3Measurement precision
If feature maps from multiple frames are fused, then processing accuracy is improved, but time consumption increases
Solution Approach 1:
The patent applies partial action by fusing feature maps from a limited number of adjacent frames rather than all available frames. The system selectively combines feature maps based on their relevance and quality, determined by optical flow error metrics. This approach achieves improved processing accuracy by incorporating sufficient temporal information while avoiding the excessive time consumption that would result from fusing too many frames.
Data Source
AI summary
A method and apparatus for processing a video frame are provided. The method may include: converting, using an optical flow generated based on a previous frame and a next frame of adjacent frames in a video, a feature map of the previous frame to obtain a converted feature map; determining, based on an error of the optical flow, a weight of the converted feature map, and obtaining a fused feature map based on a weighted result of a feature of the converted feature map and a feature of a feature map of the next frame; and updating the feature map of the next frame as the fused feature map.