Dual Neural Network Video Processing Architecture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning-based video processing (DLVP) requires significant computation resources and memory bandwidth, especially for high resolutions like 1080p or 4K, due to the complexity of generating content-dependent kernel weights and the need for high frame rates, leading to inefficiencies and potential quality drops when trying to simplify neural network topologies.
Innovation Solution
Decoupling the kernel weight prediction and filtering processes into two parallel workloads, with a weight prediction network and a filtering network operating on different hardware devices, utilizing auto-encoder structures tailored for specific AI hardware accelerators like GPUs and FPGAs, allowing for parallel execution and reduced computational complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single neural network is used to perform both kernel weight prediction and filtering operations, then device complexity is reduced, but computation resources and memory bandwidth requirements increase significantly
Solution Approach 1:
The patent divides the neural network into two separate networks: a first neural network for kernel weight prediction and a second neural network for filtering operations. This segmentation allows each network to be optimized for its specific function, reducing the computational burden on a single device while maintaining overall system efficiency.
Solution Approach 2:
The kernel weight prediction function is extracted from the filtering network and implemented as a separate first neural network. This extraction enables the filtering network to focus solely on convolution operations with pre-computed weights, significantly reducing its computational complexity and memory requirements.
2Productivity
If neural network topology is simplified to reduce computation resources, then processing efficiency improves, but visual quality deteriorates
Solution Approach 1:
By segmenting the processing into two independent networks, the patent allows the filtering network to use simpler, more efficient convolution operations while the weight prediction network handles the complex content-dependent weight generation. This maintains visual quality while improving overall processing efficiency.
Solution Approach 2:
The first neural network performs preliminary computation to predict kernel weights based on content characteristics before the second neural network applies filtering. This preliminary action separates the complex analysis task from the execution task, allowing optimization of both without compromising quality.
3Manufacturing precision
If higher input resolutions are processed to maintain visual quality, then output quality improves, but computation and memory requirements increase
Solution Approach 1:
The patent segments the processing workload such that the first neural network analyzes content characteristics at the required resolution to predict weights, while the second neural network performs filtering at the same resolution using the predicted weights. This segmentation enables efficient handling of high resolutions by distributing computational tasks.
Solution Approach 2:
The system changes the approach to handling high resolutions by pre-computing kernel weights based on content characteristics before filtering. This parameter change from direct high-resolution filtering to weight prediction followed by filtering reduces the computational burden while maintaining the ability to process high-resolution input.
Data Source
AI summary
This disclosure describes systems. methods. and devices related to deep learning-based video processing. A system may include a first neural network associated with generating kernel weights for the DL VP. the first neural network using a first hardware device: and a second neural network associated with filtering image pixels for the DLVP. the second neural network using a second hardware device, wherein the first neural network receives image data and generates the kernel weights based on the image data, and wherein the second neural network receives the image data and the kernel weights. and generates filtered image data based on the image data and the kernel weights.