Guided Video Object Segmentation for Real-Time Mobile Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing segmentation technologies for mobile devices face high computational demands, memory and power consumption, and complexity in maintaining temporal stability and object tracking across frames, making real-time salient and interactive object segmentation impractical.
Innovation Solution
A unified architecture for image and video segmentation using a single model that incorporates a guidance map, past frame information, and spatio-color mesh grid representation to efficiently segment objects in both salient and interactive modes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional neural networks are used for segmentation, then segmentation quality is improved, but computational demand and power consumption increase
Solution Approach 1:
The patent segments the segmentation task into multiple stages: first detecting salient objects and creating a guidance map, then using this guidance to selectively segment only relevant regions. This divides the computational workload and focuses processing on important areas rather than the entire image, reducing overall computational demand while maintaining quality.
Solution Approach 2:
The patent performs preliminary object detection and guidance map generation before the main segmentation process. By pre-identifying salient objects and creating a guidance map in advance, the system can then efficiently focus the segmentation model only on these pre-identified regions, reducing computational load during the main processing phase.
2Reliability
If separate segmentation models are used for images and videos, then task-specific performance is improved, but device complexity and memory requirements increase
Solution Approach 1:
The patent creates a unified segmentation model that handles multiple tasks (image segmentation, video segmentation, salient segmentation, interactive segmentation) through a single architecture. The model uses a guidance map mechanism that can operate on both images and videos, eliminating the need for separate specialized models and reducing overall system complexity.
Solution Approach 2:
The patent merges the functionality of multiple separate segmentation models into a single unified model. By combining image and video processing capabilities, as well as salient and interactive segmentation functions into one architecture, the system reduces memory requirements and device complexity while maintaining task-specific performance through the guidance map approach.
3Adaptability or versatility
If multiple segmentation models are maintained, then coverage of different segmentation types is improved, but memory consumption increases
Solution Approach 1:
The patent implements a single universal segmentation model that can perform multiple segmentation types (salient, interactive, image, video) by dynamically generating and applying guidance maps. This eliminates the need to store and manage multiple separate models, significantly reducing memory consumption while maintaining versatility across different segmentation tasks.
4Speed
If real-time segmentation is implemented, then responsiveness is improved, but computational intensity increases
Solution Approach 1:
The patent segments the computational process into efficient stages: rapid salient object detection, guidance map creation, and focused segmentation only on relevant regions. This segmentation of the processing pipeline enables real-time performance by avoiding unnecessary computational operations on the entire image and focusing resources only where needed.
Solution Approach 2:
The patent applies partial action by using the guidance map to select only the necessary regions for segmentation rather than processing the entire image. This selective approach reduces computational intensity while maintaining real-time responsiveness, as the model processes only the essential information needed for accurate segmentation.
Data Source
AI summary
A method and an electronic apparatus for performing unified segmentation of media content are provided. The method includes: determining a guidance map for an input frame based on a salient object from a past frame output mask and user-interacted objects in the media, operating in either salient mode or selective mode. The input frame of the media is cropped based on the guidance map and the salient ROIs of the salient object. A weighted grayscale image of the cropped frame is generated from the past frame output mask. A fused spatio-color mesh grid representation of the cropped frame in YUV format is determined. The cropped image frame, along with the weighted grayscale image and the fused spatio-color mesh grid representation, is input into a segmentation model. The segmentation model generates either a salient object segmentation or a user-interacted object segmentation for the media.