Neural Network Layer Segmentation for Streaming Video Mask Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video analysis applications using neural radiance field (NeRF) models are not well-suited for streaming scenarios due to the need for a large quantity of previously-acquired frames, which consumes time, storage resources, and computational resources, and are bandwidth and computationally intensive, leading to inefficiencies in static and dynamic separation mask generation.
Innovation Solution
The proposed solution involves a network management circuitry that separates the training efforts based on layer types, focusing on structure-based layers for motion and color-based layers, using a deformation network to encode motion information separately from structural data, and implementing a network activation schedule to alternate between motion and attribute network training, thereby reducing unnecessary calculations and bandwidth demands.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large quantity of previously-acquired frames are stored for NeRF training, then the quality of separation mask generation is improved, but storage resources and computational resources are consumed excessively
Solution Approach 1:
The patent extracts and processes only the most recent frame together with previously trained parameters, rather than requiring storage of a large bulk quantity of frames. This extraction approach allows the system to maintain high separation mask generation quality while significantly reducing storage resource consumption by processing minimal frame data in streaming scenarios.
Solution Approach 2:
The system performs preliminary training actions on frames as they are acquired, maintaining trained parameters for use in subsequent processing. This preliminary action approach eliminates the need to store large quantities of frames, as the training is done incrementally and results are preserved for future use.
2Measurement precision
If all layers are trained for every frame, then the accuracy of motion estimation is improved, but computational resources and time are consumed excessively
Solution Approach 1:
The patent segments the neural network into distinct layer types (structure-based layers and color-based layers) and applies different training strategies to each segment. Structure-based layers are trained selectively based on motion detection needs, while color-based layers are trained based on attribute changes. This segmentation maintains motion estimation accuracy while significantly improving processing efficiency by avoiding unnecessary full-network training for every frame.
Solution Approach 2:
The system applies partial training action by training only the necessary layers (structure-based or color-based) depending on the specific frame and detected changes, rather than training all layers for every frame. This partial action approach maintains sufficient motion estimation accuracy while reducing computational resource consumption and processing time.
3Reliability
If motion network and attribute network are trained simultaneously, then the completeness of learning is improved, but network bandwidth and computational resources are consumed excessively
Solution Approach 1:
The patent implements periodic action by alternating between training the motion network and training the attribute network in separate phases, rather than training them simultaneously for every frame. This periodic training approach ensures both networks receive adequate training attention while significantly reducing network bandwidth consumption and computational resource usage at any given time.
Solution Approach 2:
The system dynamically adjusts which network (motion or attribute) receives training based on the specific frame characteristics and processing stage. This dynamic approach allows the system to maintain learning completeness by training both networks over time, while optimizing resource usage by focusing training on one network at a time rather than simultaneously training both networks for every frame.
Data Source
AI summary
Systems, apparatus, articles of manufacture, and methods are disclosed to improve separation mask estimation. An example apparatus includes interface circuitry, instructions, and programmable circuitry to at least one of instantiate or execute the instructions to deactivate a motion network and activate an attribute network in response to a frame input of a stream to a neural network. The example apparatus also executes the instructions to train the attribute network based on the frame input, activate the motion network and deactivate the attribute network in response to a subsequent frame input of the stream to the neural network, train the motion network based on the subsequent frame, and determine pixels of the video stream that are moving.