Pose-Guided Video Segmentation Masks With ROI-Based Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional video segmentation systems suffer from inflexibility, inaccuracy, and inefficiency in generating segmentation masks for digital videos, particularly when high-quality masks with fine granularity are required, often leading to false positives and excessive computing resource consumption.
Innovation Solution
A joint-based segmentation system that globally tracks pose and locally segments objects using a pipeline model architecture, incorporating a pose tracking neural network and a segmentation neural network with a multi-layer perceptron mixer layer to efficiently generate accurate segmentation masks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional deep learning-based segmentation is performed using initial mask or scribble input, then segmentation can be performed with basic functionality, but the system lacks flexibility and accuracy in generating high-quality segmentation masks
Solution Approach 1:
The system divides the segmentation task into two distinct stages: global pose tracking to identify object locations and orientations, and local fine-grained segmentation to generate precise masks. This segmentation of the processing pipeline allows each stage to specialize, improving overall accuracy while maintaining flexibility through the modular architecture.
Solution Approach 2:
The patent introduces pose information (joint coordinates and bone lengths) as an intermediary element that bridges between the input video frame and the final segmentation mask. This intermediate representation provides structural guidance that enhances segmentation accuracy while allowing the system to adapt to different objects and scenarios.
2Extent of automation
If video instance segmentation techniques are employed to perform segmentation without user-provided guidance, then automation is improved, but segmentation quality deteriorates due to prioritization of detection and tracking over segmentation quality
Solution Approach 1:
The system separates automated pose tracking from automated segmentation into distinct modules. The global pose tracker automatically identifies object structure, while the local segmentation model automatically generates precise masks guided by the pose information. This modular automation maintains high segmentation quality without requiring user input.
Solution Approach 2:
The patent changes the evaluation parameters from average precision (which prioritizes detection) to metrics that emphasize segmentation quality such as mask intersection-over-union. This parameter change shifts the optimization focus toward generating higher quality masks while maintaining automation.
3Reliability
If conventional segmentation systems process entire frames to ensure accurate detection, then detection accuracy is improved, but computational resource consumption increases significantly
Solution Approach 1:
The system extracts and processes only the region of interest (ROI) containing the target object rather than processing the entire video frame. By extracting the bounding box defined by global pose tracking and applying the local segmentation model only to this smaller ROI, computational resources are significantly reduced while maintaining detection accuracy.
Solution Approach 2:
The processing pipeline is segmented into a global stage that operates on the full frame for detection and a local stage that operates on the extracted ROI for precise segmentation. This hierarchical segmentation of processing scope allows accurate detection while minimizing computational resource consumption in the resource-intensive segmentation stage.
4Measurement precision
If conventional systems use heavy segmentation models to achieve high accuracy, then segmentation quality is improved, but processing speed and efficiency deteriorate
Solution Approach 1:
The system segments the segmentation task into global pose estimation and local fine-grained mask generation. The local segmentation model only processes small ROI regions rather than full frames, and uses pose information to guide attention, achieving high accuracy with reduced computational load and improved processing speed.
Solution Approach 2:
The patent changes the input parameters to the segmentation model from raw pixel data of entire frames to pose-guided features extracted from small ROIs. This parameter transformation reduces the dimensional complexity of the input, allowing lighter models to achieve the same segmentation quality with faster processing.
Data Source
AI summary
The present disclosure relates to systems, methods, and non-transitory computer-readable media that generate joint-based segmentation masks for digital objects portrayed in digital videos. In particular, in one or more embodiments, the disclosed systems utilize a video masking model having a pose tracking neural network and a segmentation neural network to generate the joint-based segmentation masks. To illustrate, in some embodiments, the disclosed systems utilize the pose tracking neural network to identify a set of joints of the digital object across the frames of the digital video. The disclosed systems further utilize the segmentation neural network to generate joint-based segmentation masks for the video frames that portray the object using the identified joints. In some cases, the segmentation neural network includes a multi-layer perceptron mixer layer for mixing visual features propagated via convolutional layers.