Transformer-CNN Model Structure for Multi-Task Image Enhancement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer vision models, particularly convolutional neural networks (CNNs), struggle to effectively process low-level vision tasks due to their inability to attend to global information and require separate training for each image enhancement task, limiting their versatility.
Innovation Solution
A model structure combining a transformer module with multiple neural network layers is developed, allowing it to process different image enhancement tasks by integrating global information and adapting to specific tasks through a selection, segmentation, and recombination process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a convolutional neural network (CNN) is used to process low-level vision tasks, then feature extraction capability is improved, but the ability to attend to global information deteriorates
Solution Approach 1:
The patent segments the feature map into multiple feature blocks, which are then processed by transformer modules to capture global relationships while maintaining local feature extraction capabilities through the segmented structure
Solution Approach 2:
The patent introduces a transformer module as an intermediary between the CNN feature extraction stage and the final processing stage, allowing global information to be captured without compromising the local feature extraction capability of the CNN
2Reliability
If separate CNNs are trained for each image enhancement task, then task-specific performance is improved, but model complexity and training time increase
Solution Approach 1:
The patent creates a universal model structure that can handle multiple image enhancement tasks (denoising, defogging, deraining) through a single transformer module, eliminating the need for separate CNNs for each task while maintaining task-specific performance
Solution Approach 2:
The patent makes the model dynamic by allowing different task-specific configurations and adaptations of the transformer module for different enhancement tasks, enabling one base model to serve multiple functions through configurable parameters and task-specific training
Data Source
Figure 1(a)~1(b)
Figure 2
Figure 3
AI summary
Embodiments of this application disclose a model structure, a method for training a model, an image enhancement method, and a device, and may be applied to the computer vision field in the artificial intelligence field. The model structure includes: a selection module, a plurality of first neural network layers, a segmentation module, a transformer module, a recombination module, and a plurality of second neural network layers. The model overcomes a limitation that the transformer module can only be used to process a natural language task, and may be applied to a low-level vision task. The model includes the plurality of first/second neural network layers, and different first/second neural network layers correspond to different image enhancement tasks. Therefore, after being trained, the model can be used to process different image enhancement tasks. Compared with a manner in which most existing models that process the low-level vision task are based on a CNN (as a good feature extractor, the CNN has good performance in a high-level vision task, but can hardly pay attention to global information when processing the low-level vision task), the model in this application may pay attention to the global information by using the transformer module, to improve an image enhancement effect.