Convolutional Transformer Layers for Global-Local Feature Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The transformer architecture in neural networks has too many parameters, leading to reduced local information acquisition capability, while convolutional neural networks lack global context information, resulting in diminished overall network performance.
Innovation Solution
A neural network system incorporating convolutional transformer layers with key, value, and attention embedding layers, utilizing key and value convolutional kernels of varying sizes to enhance feature extraction and improve multi-scale change resistance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If transformer architecture is used to improve global context information acquisition, then global context capability is improved, but the number of parameters increases excessively
Solution Approach 1:
The patent merges the advantages of both transformer and convolutional neural networks by integrating self-attention mechanisms with convolutional operations. The hybrid architecture combines global context acquisition from transformers with local feature extraction from CNNs, reducing parameter count while maintaining both global and local information processing capabilities.
Solution Approach 2:
The patent creates a multi-functional architecture that can perform both global context analysis (through self-attention mechanisms) and local feature extraction (through convolutional operations) within a unified framework. This universal structure eliminates the need for separate specialized models, reducing overall parameter requirements.
2Device complexity
If convolutional neural network is used to reduce parameters, then parameter count is reduced, but global context information is lost
Solution Approach 1:
The patent combines convolutional operations for local feature extraction with self-attention mechanisms for global context processing. This integration ensures that global context information is preserved while maintaining the parameter efficiency of convolutional networks through shared weights and localized operations.
3Measurement precision
If transformer architecture is used to improve feature extraction, then feature quality is improved, but local information acquisition capability is reduced
Solution Approach 1:
The patent applies local quality by using different operational mechanisms for different information scales: convolutional operations with localized receptive fields for local feature extraction, and self-attention mechanisms for global context. This differentiated approach ensures both local and global information are effectively captured without compromising either capability.
Data Source
AI summary
A neural network system and a signal processing method are provided. The neural network system includes at least one processing unit and a neural network module. The signal processing method includes: inputting a neural network input to the neural network module by the processing unit to generate an input at a previous layer of each convolutional transformer layer; performing pointwise convolution on the input by a key embedding layer based on key convolutional kernels to output a key tensor; performing convolution on the input by a value embedding layer based on value convolutional kernels to output a value tensor; performing a convolution on the cascading tensor of a first tensor and the key tensor by an attention embedding layer based on attention convolution kernels to output an attention tensor; and outputting an output tensor based on the attention tensor and the value tensor by an output module.