Transformer-CNN Image Enhancement Model for Multi-Task Vision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer vision models, particularly those based on convolutional neural networks (CNNs), struggle to effectively process low-level vision tasks due to their inability to attend to global information and require separate training for each image enhancement task, limiting their versatility.
Innovation Solution
A model structure combining a transformer module with multiple neural network layers is developed, allowing it to process different image enhancement tasks by integrating global information and adapting to specific tasks through a selection, segmentation, and recombination process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a CNN is used for image enhancement tasks, then feature extraction capability is improved, but the ability to attend to global information deteriorates
Solution Approach 1:
The model segments the feature map into multiple feature blocks and processes them through transformer modules independently, then recombines them. This segmentation allows the transformer to attend to global relationships while maintaining local feature extraction capabilities of CNNs.
Solution Approach 2:
The transformer module acts as an intermediary between the CNN feature extractor and the final output. It receives feature blocks from the CNN, processes global relationships, and passes transformed features back to the CNN structure, bridging the gap between local and global information processing.
2Reliability
If separate CNNs are trained for each image enhancement task, then task-specific performance is improved, but model complexity and training time increase
Solution Approach 1:
The model employs a unified architecture with multiple task-specific heads that share common backbone features. Different image enhancement tasks (denoising, deblurring, deraining) are handled by the same base model with task-adaptive modules, allowing one model to perform multiple functions without requiring separate trained models for each task.
Solution Approach 2:
The model merges multiple task-specific processing paths into a single unified framework. Instead of training separate CNNs for each task, the patent combines them into one model that processes all tasks through shared feature extraction followed by task-specific transformation modules.
3Device complexity
If a single model structure is used for multiple image enhancement tasks, then model simplicity is improved, but task-specific performance deteriorates
Solution Approach 1:
The model applies local quality adaptation through task-specific head modules that are tailored to each enhancement task's requirements. While the backbone structure remains simple and shared, the output layers and task-adaptive modules are optimized for specific tasks like denoising, deblurring, or deraining, ensuring high performance for each task.
Data Source
AI summary
Embodiments of this application disclose a model structure, a method for training a model, an image enhancement method, and a device, and may be applied to the computer vision field in the artificial intelligence field. The model structure includes: a selection module, a plurality of first neural network layers, a segmentation module, a transformer module, a recombination module, and a plurality of second neural network layers. The model overcomes a limitation that the transformer module can only be used to process a natural language task, and may be applied to a low-level vision task. The model includes the plurality of first/second neural network layers, and different first/second neural network layers correspond to different image enhancement tasks. Therefore, after being trained, the model can be used to process different image enhancement tasks.