Text-Based Video Object Selection with Fusion Masks for Motion Blur
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional video editing systems face challenges in accurately selecting and tracking objects due to their spatio-temporal nature, particularly when objects warp, change scale, or have rapid motion blur, leading to inconsistent segmentation and tracking.
Innovation Solution
A video segmentation system that fuses image segmentation with frame referring segmentation using a neural network to generate precise masks for objects, even during rapid scene changes, by combining instance and image referring segmentations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional object segmentation is performed with a predetermined segmentation mode, then the segmentation process is simple, but the segmentation accuracy deteriorates during rapid scene changes and object motion
Solution Approach 1:
The patent divides the video processing into keyframe identification and non-keyframe processing. Keyframes are selected based on motion detection and scene change analysis, where full segmentation is performed. Non-keyframes use simplified tracking methods, reducing overall computational complexity while maintaining accuracy during critical moments.
Solution Approach 2:
The system dynamically adjusts segmentation intensity based on detected motion and scene changes. During rapid motion or scene transitions, the system increases segmentation frequency and detail. During stable periods, it reduces segmentation to maintain frames from previous analysis, adapting the processing level to current video conditions.
2Reliability
If tracking is performed on every frame to maintain object consistency, then object tracking accuracy is improved, but processing time and computational load increase
Solution Approach 1:
The system performs preliminary object detection and segmentation on keyframes before full video processing. By identifying objects and their properties in advance on selected keyframes, the system establishes tracking baselines that can be propagated through subsequent frames, reducing the need for repeated full segmentation while maintaining tracking consistency.
Solution Approach 2:
The system skips full segmentation processing on non-keyframes by using object tracking algorithms that leverage information from keyframes. Between keyframes, the system rapidly propagates object states using motion estimation and tracking, rushing through frames that don't require intensive processing while maintaining object consistency.
3Manufacturing precision
If segmentation is performed at high frequency to capture rapid motion, then motion blur and warping are reduced, but computational complexity increases
Solution Approach 1:
The system applies high-frequency segmentation only to regions and frames where motion is detected. By analyzing motion vectors and scene change indicators, the system concentrates computational resources on areas with rapid motion or scene transitions, applying full segmentation precision only where needed rather than uniformly across all frames.
Solution Approach 2:
The system changes segmentation parameters dynamically based on detected motion intensity. During periods of rapid motion, the system increases segmentation frequency and uses more sophisticated segmentation models. During stable periods, it reduces segmentation frequency and uses simpler models, adapting processing parameters to current video conditions to balance precision and complexity.
Data Source
AI summary
Embodiments are disclosed for receiving a user input and an input video comprising multiple frames. The method may include extracting a text feature from the user input. The method may further include extracting a plurality of image features from the frames. The method may further include identifying one or more keyframes from the frames that include the object. The method may further include clustering one or more groups of the one or more keyframes. The method may further include generating a plurality of segmentation masks for each group. The method may further include determining a set of reference masks corresponding to the user input and the object. The method may further include generating a set of fusion masks by combining the plurality of segmentation masks and the set of reference masks. The method may further include propagating the set of fusion masks and outputting a final set of masks.