Hybrid Sequence Modeling Architecture for Local and Global Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning models struggle to efficiently integrate local and global contexts in biological sequence modeling, particularly in fields like genomics, due to limitations in convolutional neural networks' receptive field and transformers' quadratic complexity, leading to challenges in capturing long-range interactions and associative recall tasks.
Innovation Solution
A machine learning architecture combining a projected gate convolution module with a state space module to process input data, enabling parallel processing of local and global features, using linear projections, RMS normalizations, and structured state space models to enhance adaptability and responsiveness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If transformers are used to model global pairwise relationships, then global context capture is improved, but computational complexity increases quadratically
Solution Approach 1:
The patent segments the sequence modeling task into local feature extraction using CNNs and global relationship modeling using transformers, processing different aspects of the sequence in parallel through separate architectural components rather than using a single monolithic model
Solution Approach 2:
The patent introduces a hierarchical structure that adds a global context dimension to the local feature representations, combining CNN extracted local features with transformer-generated global context through concatenation and projection layers, effectively moving from 1D sequence processing to multi-dimensional feature space
2Reliability
If CNNs are used to determine localized patterns, then local pattern recognition is improved, but receptive field is limited
Solution Approach 1:
The patent merges CNNs and transformers into a hybrid architecture where CNNs extract local patterns and transformers capture global relationships, combining the strengths of both architectures to achieve both local accuracy and global context understanding simultaneously
Solution Approach 2:
The patent performs preliminary local feature extraction using CNNs before feeding the representations to the transformer for global context modeling, preparing the data in advance to enable efficient hierarchical processing and reduce the computational burden on the transformer
3Length of stationary object
If multiple filters and dilated convolutions are used to expand receptive field, then long-range interaction capture is improved, but model complexity increases
Solution Approach 1:
The patent introduces transformer modules as intermediary components between the CNN local feature extraction layers and the final output, where the transformer processes the CNN representations to capture long-range dependencies without requiring the CNN itself to have an expanded receptive field through multiple dilated convolutions
Data Source
AI summary
Deep learning tools such as convolutional neural networks (CNNs) and transformers have spurred great advancements in computational biology. However, existing methods are constrained architecturally in context length, computational complexity, and model size. This application introduces a sub-quadratic architecture for modeling, which combines projected gated convolutions and structured state spaces to achieve local and global context with, for example, single-nucleotide resolution. These models outperform CNN-, GPT-, BERT-, and long convolution-based models in many tested genomics tasks without pre-training and with 4×-781× fewer parameters. In the proteomics domain, these models similarly outperform pretrained attention-based models, including ESM-1B and TAPE-BERT, on remote homology prediction without pre-training and while using 3,308×-23,636× fewer parameters.