Video Segmentation Memory Fusion Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network-based video object segmentation models face challenges with long video sequences, limited processing performance, and difficulties in precise processing of moving, deforming, or occluded target objects in augmented reality environments.
Innovation Solution
A method involving the acquisition of video sequences, extraction of feature vectors, generation of hidden state information through fusion with stored memory information, and output of segmentation masks, utilizing an attention mechanism and dissimilarity-based storage to enhance processing efficiency and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If the length of input images increases when segmentation is performed consecutively, then more context information is available, but the neural network-based model has limited processing performance
Solution Approach 1:
The patent divides the long video sequence into multiple shorter sequences, each processed independently by the neural network model. This segmentation allows the model to maintain efficient processing performance while still accessing sufficient context information from previous frames through the memory module that stores and retrieves relevant historical data.
Solution Approach 2:
The patent introduces a memory module as an intermediary between the neural network model and the video sequence. This memory module stores historical frame information and makes it available to the model during processing, enabling the model to access context information without directly processing the entire long sequence at once.
2Loss of information
If the neural network-based model processes long video sequences, then more temporal context is captured, but processing accuracy decreases for moving and deforming objects
Solution Approach 1:
The patent implements a dynamic memory management system that adapts to the processing needs of different video sequences. The memory module selectively stores and retrieves historical frame information based on the current processing context, allowing the system to maintain high segmentation accuracy for dynamic objects while still capturing necessary temporal context.
Solution Approach 2:
The patent applies local quality by selectively accessing and processing only the most relevant historical frame information from memory rather than processing the entire long sequence uniformly. This allows the model to focus computational resources on the most pertinent temporal context for each specific segmentation task.
3Reliability
If consecutive segmentation is performed on long video sequences, then comprehensive coverage is achieved, but processing time increases
Solution Approach 1:
The patent segments the long video sequence into multiple shorter sequences that can be processed in parallel or with reduced computational overhead. Each segment is processed independently, and the memory module ensures that segmentation coverage is maintained by storing and retrieving relevant information across segment boundaries.
Solution Approach 2:
The patent performs preliminary actions by pre-processing and storing historical frame information in the memory module before the main segmentation processing begins. This preliminary storage of context information eliminates the need to re-process entire long sequences during segmentation, significantly reducing processing time while maintaining comprehensive coverage.
Data Source
AI summary
A method with video segmentation may include: acquiring, over time, a video sequence including a plurality of image frames, the plurality of image frames including a second image frame corresponding to a time t of the video sequence and a first image frame corresponding to a time t−1 before the time t; extracting a second feature vector from the second image frame; generating second hidden state information corresponding to the second image frame, based on first hidden state information corresponding to the first image frame and second fusion information in which the second feature vector is fused with information related to the second image frame stored in a memory; generating a second segmentation mask corresponding to the second image frame, based on an output vector corresponding to the second hidden state information; and outputting the second segmentation mask.