Memory-Based Video Object Segmentation with Static Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional memory-based video object segmentation systems face challenges in efficiently segmenting objects across multiple frames of a video while conserving computing resources, as they often require dynamic memory allocation and resizing of frames, leading to increased memory usage and loss of spatial information.
Innovation Solution
The proposed segmentation system employs a modified memory-based video object segmentation machine learning model that uses a static computational graph system and a simplified region-of-interest (ROI) tracking algorithm, allowing for efficient object segmentation in limited computing resource environments by processing only the ROI and using pre-allocated memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If dynamic memory allocation is used to store increasing numbers of previous frames, then object segmentation accuracy improves, but memory usage increases significantly
Solution Approach 1:
The patent transforms the memory management approach by changing the parameter of memory allocation from dynamic (increasing with each frame) to static (fixed pre-allocated size). The system pre-allocates a fixed number of memory slots for storing previous frames and uses a sliding window approach where old frames are overwritten when memory is full. This resolves the contradiction by maintaining segmentation accuracy through consistent memory access patterns while preventing unbounded memory growth.
2Quantity of substance
If frame resizing is performed to adapt to memory constraints, then memory usage is reduced, but spatial information is lost
Solution Approach 1:
The patent applies segmentation by dividing the video processing into independent frame units that are processed sequentially. Instead of resizing entire frames which loses spatial information, the system segments the temporal dimension into discrete frame slots in memory. Each frame maintains its original resolution and spatial integrity, while the segmentation of time into fixed memory slots controls memory usage without compromising spatial information.
3Reliability
If all previous frames are stored and processed, then segmentation reliability improves, but computational complexity increases
Solution Approach 1:
The patent implements periodic action through a cyclic memory buffer where frames are stored in a fixed-size buffer and automatically overwritten in a periodic fashion. Instead of continuously accumulating all previous frames, the system uses a sliding window of fixed size that periodically updates. This maintains segmentation reliability by always having access to a consistent number of recent frames while preventing computational complexity from increasing unboundedly.
Data Source
AI summary
Embodiments are disclosed for a method including obtaining a region of interest of a current frame of a video sequence depicting an object. The method may further include determining, by a mask propagation model, a likelihood of each pixel of the current frame being associated with the object in the region of interest of the current frame based on the region of interest of the current frame and a fixed number of previous frames of the video sequence including the object. The method may further include replacing a previous frame of the fixed number of previous frames with the current frame. The method may further include displaying the current frame of the video sequence including a masked object in the region of interest of the current frame based on the likelihood of one or more pixels of the current frame being associated with the object.