Spectral Transformer Classification With Global-Local Feature Splitting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer models face challenges in efficiently capturing both global and local information from large datasets, leading to increased computational complexity and loss of informational integrity, particularly in applications like medical imaging.
Innovation Solution
The integration of spectral transformations, specifically using the Hartley Transform with convolutional operators in initial layers, combined with deeper attention layers, reduces computational complexity while retaining informational integrity by capturing both global and local features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If transformer models use multiple attention layers to capture global and local information, then classification accuracy is improved, but computational complexity increases quadratically
Solution Approach 1:
The patent segments the embedding space into multiple distinct regions (e.g., global features region, local features region, texture features region) and assigns different attention mechanisms to process different segments. This allows the model to capture various types of information in parallel without requiring all tokens to attend to all other tokens, thus reducing quadratic complexity while maintaining comprehensive feature extraction capability.
Solution Approach 2:
The patent introduces a new dimension by adding spatial position embeddings and depth-wise separable convolutions to the transformer architecture. This transforms the problem from purely sequence-based attention to a multi-dimensional processing approach where spatial and channel dimensions are handled separately, reducing the computational burden while preserving the ability to capture global and local dependencies.
2Loss of information
If input data is split into tokens and processed through embedding layers, then feature representation is enhanced, but information loss occurs during tokenization
Solution Approach 1:
The patent merges multiple types of embeddings (patch embeddings, position embeddings, class embeddings) into a unified embedding space that preserves both local patch information and global positional context. By combining these different embedding types rather than processing them separately, the model maintains complete information from the original input while enabling sophisticated feature representation through the unified embedding structure.
3Loss of information
If the context window includes all tokens for attention processing, then global context is captured, but computational resources are excessively consumed
Solution Approach 1:
The patent applies different attention mechanisms to different regions of the embedding space based on their specific requirements. Global feature regions use simplified attention or pooling operations, while local feature regions use more sophisticated attention mechanisms. This localized approach ensures that global context is captured where necessary without applying computationally expensive operations uniformly across the entire input, thus reducing overall computational resource consumption.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method may embed the input dataset into a first embedding space. A method may input the first embedding space into a spectral module including a periodic information processor and an aperiodic information processor. A method may identify global features in the input dataset using the periodic information processor based on a first subset of the first embedding space. A method may identify first local features in the input dataset using the aperiodic information processor based on a second subset of the first embedding space, wherein the first subset and the second subset are different. A method may combine the global features and the first local features into a dataset of classified features of the input dataset.