Densely Connected Transformer Architecture for Linguistic Information Retention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing pre-training methods for Transformer-based NLP models, such as BERT, fail to fully utilize the connectivity between layers, leading to dilution of knowledge learned at lower layers in higher layers due to scarce connections, which affects the model's ability to retain and propagate linguistic information effectively.
Innovation Solution
A densely connected Transformer architecture is proposed where each layer takes inputs from and sends outputs to all preceding and subsequent layers, enabling maximum information sharing and feature reuse, and is pre-trained using a multi-scale discriminative objective that focuses on both local and global information through Ngram prediction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If Transformer network uses scarce connections between layers, then model complexity is reduced and training is easier, but knowledge learned at lower layers becomes diluted in higher layers
Solution Approach 1:
The patent merges the connectivity patterns of ResNet and Transformer by implementing dense connections where each layer receives inputs from all preceding layers and sends outputs to all subsequent layers. This combines the hierarchical feature extraction capability of Transformers with the strong connectivity benefits of ResNet, preventing knowledge dilution while maintaining computational efficiency through the shared embedding space.
Solution Approach 2:
The densely connected Transformer architecture makes each layer universally connected to every other layer, allowing any layer to access information from any other layer. This multi-functional connectivity enables lower layers to contribute to higher layers while avoiding the need for separate connection pathways, thus preventing information loss without significantly increasing overall model complexity.
2Use of energy by moving object
If Transformer network uses scarce connections between layers, then computational cost is reduced, but feature propagation effectiveness deteriorates
Solution Approach 1:
The patent merges ResNet's dense connection strategy with Transformer architecture, creating a system where computational resources are efficiently allocated. By sharing embedding spaces and using dense connections selectively, the model achieves effective feature propagation without proportionally increasing computational cost, as the dense connections leverage existing computational pathways rather than creating entirely new ones.
Solution Approach 2:
The densely connected architecture implements implicit deep supervision where each layer receives feedback from all subsequent layers through the dense connections. This feedback mechanism enhances feature propagation effectiveness by allowing higher layers to refine and reinforce features learned in lower layers, creating a mutually reinforcing computational process that improves efficiency.
3Ease of manufacture
If Transformer network uses scarce connections between layers, then model training is simpler, but linguistic information propagation is insufficient
Solution Approach 1:
The patent merges the training simplicity of Transformers with the reliable information propagation of ResNet by implementing dense connections that maintain the hierarchical structure while adding cross-layer connectivity. This combination allows linguistic information to propagate reliably through multiple pathways without significantly complicating the training process, as the dense connections operate within the existing Transformer optimization framework.
Solution Approach 2:
The densely connected architecture performs preliminary action by pre-establishing multiple information pathways during training, where lower layers are trained to provide robust linguistic information that is then distributed to higher layers through dense connections. This preliminary establishment of information pathways ensures reliable propagation without requiring complex training procedures, as the architecture is designed to leverage existing training data and objectives.
Data Source
AI summary
Embodiments described herein propose a densely connected Transformer architecture in which each Transformer layer takes advantages of all previous layers. Specifically, the input for each Transformer layer comes from the outputs of all its preceding layers; and the output information of each layer will be incorporated in all its subsequent layers. In this way, a L-layer Transformer network will have L(L+1)/2 connections. In this way, the dense connection allows the linguistic information learned by the lower layer to be directly propagated to all upper layers and encourages feature reuse throughout the network. Each layer is thus directly optimized from the loss function in the fashion of implicit deep supervision.