Video Segmentation Motion Referencing to Eliminate Frame Flicker
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image segmentation models suffer from temporal inconsistencies, such as 'flickering' artifacts, where a pixel is incorrectly classified across consecutive frames, leading to degraded service quality and increased safety risks.
Innovation Solution
The use of a pseudo-optical flow operation, specifically a deformable convolution operation, is introduced to add temporal information and improve temporal consistency in image segmentation models, reducing computational demands compared to traditional optical flow methods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional optical flow operations (e.g., RAFT) are used to improve temporal consistency, then temporal consistency is improved, but computational demand increases significantly
Solution Approach 1:
The patent segments the optical flow computation into two parts: a coarse initial flow estimate using simple pixel difference, and a refined flow field using deformable convolution only where needed. This segmentation reduces overall computational demand while maintaining temporal consistency improvements.
Solution Approach 2:
The patent applies deformable convolution selectively in regions where motion occurs, rather than uniformly across the entire image. By identifying motion regions through initial flow estimation and applying refined convolution only there, computational resources are concentrated where they provide the most benefit to temporal consistency.
2Productivity
If image segmentation is performed on each frame independently, then processing speed is maintained, but temporal inconsistencies such as flickering artifacts occur
Solution Approach 1:
The patent performs preliminary action by computing an initial optical flow estimate between consecutive frames before performing segmentation. This preliminary flow information is then used to guide the deformable convolution process, allowing the segmentation to incorporate temporal context without significantly increasing processing time.
Solution Approach 2:
The patent introduces optical flow computation as an intermediary step between frame processing. This intermediary provides motion information that bridges temporal gaps, allowing segmentation results to be adjusted based on detected motion while maintaining processing efficiency through selective application of deformable convolution.
Data Source
AI summary
An example system include one or more memories and one or more processors coupled to the one or more memories. The one or more processors are configured to perform a first segmentation operation on a previous frame of image data to generate first segmentation data. The one or more processors are configured to perform a deformable convolution operation based on the first segmentation data to generate a deformable convolution output. The one or more processors are configured to perform a second segmentation operation on a current frame of image data to generate second segmentation data. The one or more processors are configured to combine the deformable convolution output with the second segmentation data to generate third segmentation data and control operation of a device based on the third segmentation data.