Neural Network Object Segmentation in Video Frames
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional object segmentation techniques face challenges such as the need for capturing static background images, inaccuracies due to noisy depth sensors, and inefficiencies in processing speed and accuracy, particularly in color-based and semantic segmentation methods.
Innovation Solution
An image-processing apparatus and method utilizing multiple neural networks, specifically a pre-trained convolutional neural network (CNN) and a second CNN, to segment objects in a sequence of color image frames, allowing for real-time segmentation of moving or deforming objects without requiring manual ground truth data for training, and improving processing speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional color-based segmentation removes a pre-determined static background image, then object segmentation can be achieved, but the system requires capturing a sequence of images when the object is not present and a static camera position, which reduces adaptability and increases complexity
Solution Approach 1:
The patent applies dynamics by transitioning from static background subtraction to dynamic foreground object detection using neural networks. The system processes video frames sequentially and identifies foreground objects based on temporal changes, allowing adaptation to moving cameras and dynamic scenes without requiring static background capture
Solution Approach 2:
The patent replaces the mechanical approach of capturing static background images with a neural network-based computational approach. Instead of using camera positioning and static scene capture, the system uses deep learning models to automatically detect and segment foreground objects from video sequences, eliminating the need for manual background preparation
2Productivity
If conventional semantic segmentation uses only color images, then processing speed improves, but segmentation accuracy deteriorates
Solution Approach 1:
The patent merges multiple neural network models with different strengths into a unified segmentation system. By combining foreground object detection networks with semantic segmentation networks, the system achieves both high processing speed and high segmentation accuracy, resolving the trade-off between speed and precision
3Productivity
If depth sensors are used for object segmentation, then processing speed improves, but measurement precision deteriorates due to noisy depth images with invalid depth values
Solution Approach 1:
The patent uses color image-based neural network segmentation as an intermediary approach between depth-based fast segmentation and color-based accurate segmentation. The neural networks process color images to achieve both high speed and high accuracy, serving as a mediator that eliminates the need for depth sensors and their associated noise problems
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
An image processing apparatus and method for object segmentation in color image frames, is provided. The image-processing apparatus generates, based on a first neural network model, a first foreground mask and a second foreground mask for an object-of-interest in a first color image frame and an upcoming color image frame, respectively, of the sequence of color image frames. The image-processing apparatus determines a third foreground mask based on interpolation of the first foreground mask and the second foreground mask and updates, by use of the second neural network model, the third foreground mask to a fourth foreground mask. The image processing apparatus segments the object-of-interest from at least the first color image frame, the upcoming color image frame, and the intermediate color image frame by use of the first foreground mask, the second foreground mask, and the fourth foreground mask, respectively.