Video Frame Segmentation via Reduced Resolution Neural Network
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video segmentation methods in video conferencing are computationally intensive, consuming high power and affecting battery life, as they require large neural networks that perform millions of arithmetic operations per second to accurately distinguish between foreground and background.
Innovation Solution
The use of a reduced resolution neural network that leverages previous frames' masks to segment video frames efficiently, with a downscaler reducing input frames to a smaller size and an upscaler restoring the segmentation mask to original resolution, allowing for real-time processing with significantly lower power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional CNNs are used for video segmentation, then segmentation accuracy is improved, but power consumption increases significantly
Solution Approach 1:
The patent divides the video segmentation task into two separate processing stages: (1) generating segmentation masks at reduced resolution using a compact neural network, and (2) upsampling these masks to full resolution. This segmentation of the processing task allows accurate segmentation to be achieved while using a much smaller network at the critical mask-generation stage, thereby reducing power consumption.
Solution Approach 2:
The patent introduces reduced-resolution segmentation masks as an intermediary representation. Instead of directly generating full-resolution masks using a large CNN, the system first creates masks at lower resolution using a compact network, then uses an upsampling operation to achieve the final full-resolution output. This intermediary approach enables accurate segmentation with significantly reduced computational requirements and power consumption.
2Measurement precision
If large neural networks are used for accurate segmentation, then segmentation quality is improved, but device battery life is reduced
Solution Approach 1:
The patent segments the computational workload by performing the intensive mask-generation operation at reduced resolution where full accuracy is less critical, then uses a lightweight upsampling operation to achieve the final high-resolution output. This allows the system to maintain segmentation quality while dramatically reducing the computational burden on the mobile device, thereby extending battery life.
Solution Approach 2:
The patent changes the resolution parameter during processing - performing neural network inference at reduced resolution to minimize computational requirements and power consumption, then restoring the full resolution through upsampling. This parameter change enables the use of a compact neural network while still producing full-resolution segmentation masks, thus preserving battery life.
3Measurement precision
If full resolution neural network processing is used, then segmentation detail accuracy is improved, but processing speed decreases
Solution Approach 1:
The patent segments the processing pipeline into a fast mask-generation stage at reduced resolution and a fast upsampling stage. By performing the computationally intensive neural network operations at lower resolution, the system achieves real-time processing speeds while maintaining the ability to produce detailed full-resolution segmentation masks through the subsequent upsampling operation.
Solution Approach 2:
The patent dynamically changes the resolution parameter based on the processing stage - using reduced resolution during neural network inference to maximize processing speed, then restoring full resolution in the upsampling stage to ensure segmentation detail accuracy. This parameter adaptation enables real-time processing without sacrificing final output quality.
Data Source
AI summary
An example apparatus for video frame segmentation includes a receiver to receive a current video frame to be segmented. The apparatus also includes a segmenting neural network to receive a previous mask including a segmentation mask corresponding to a previous frame and generate a segmentation mask for the current frame based on the previous mask and the video frame.