Hardware Friendly Multi-Kernel Convolution Network for ISP
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image signal processing (ISP) designs, particularly for super resolution (SR), image noise reduction (NR), and temporal noise reduction (TNR), are computationally intensive and complex, making them difficult to implement in hardware due to inefficient hardware-aware network configurations and high receptive fields that are not supported by current hardware designs.
Innovation Solution
The introduction of a hardware-friendly multi-kernel convolution network (HFMCN) with a fundamental block called the hardware-friendly multi-kernel convolution block (HFMCB), which includes hardware-friendly operators to efficiently implement image restoration networks, allowing for diverse feature extraction with different receptive fields and improved parallel processing capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional CNN designs are used for image processing tasks, then processing accuracy is improved, but hardware implementation complexity increases
Solution Approach 1:
The network is divided into multiple independent kernel branches (first kernel, second kernel, third kernel, fourth kernel) that process different aspects of the input image simultaneously. Each kernel operates independently with its own convolution and activation layers, allowing parallel hardware implementation while maintaining comprehensive processing capability.
Solution Approach 2:
The HFMCN architecture serves multiple image processing functions (super resolution, noise reduction, temporal noise reduction) using a unified multi-kernel structure. The same network framework can be configured for different tasks by adjusting kernel parameters and combinations, eliminating the need for separate specialized networks for each function.
2Measurement precision
If multiple independent networks are used for different ISP tasks, then task-specific performance is improved, but computational cost increases
Solution Approach 1:
A single HFMCN network architecture is designed to perform multiple ISP tasks (super resolution, image noise reduction, temporal noise reduction) by configuring different kernel combinations and parameters. This eliminates the need to train and execute multiple separate networks, significantly reducing computational cost while maintaining task-specific performance through specialized kernel designs within the unified structure.
Solution Approach 2:
The patent combines multiple kernel branches (first, second, third, and fourth kernels) within a single network framework, where each kernel handles different processing objectives. The outputs of these kernels are integrated through addition operations, merging complementary processing results into a final output that achieves multiple tasks simultaneously with reduced overall computational cost.
3Measurement precision
If high receptive field networks are used for SR and NR, then processing performance is improved, but hardware support becomes limited
Solution Approach 1:
The high receptive field requirement is segmented across multiple independent kernel branches rather than requiring a single large receptive field network. Each kernel can have manageable receptive field size, and the combination of multiple kernels achieves the overall high receptive field effect, which is more compatible with current hardware architectures.
Solution Approach 2:
Different kernels are designed with different receptive field characteristics tailored to their specific processing functions. Some kernels may have larger receptive fields for capturing global context, while others have smaller receptive fields for local detail processing. This localized optimization of receptive field sizes allows hardware-friendly implementation while maintaining overall high processing performance.
Data Source
AI summary
A system and a method are disclosed for processing and combining feature maps using a hardware friendly multi-kernel convolution block (HFMCB). The method including splitting an input feature map into a plurality of feature maps, each of the plurality of feature maps having a reduced number of channels; processing each of the plurality of feature maps with a different series of kernels; and combining the processed plurality of feature maps.