Cross-Attention Fusion for Text Classification Accuracy
Solution Overview
Problem
Existing natural language understanding (NLU) systems face challenges in optimally leveraging external features in conjunction with token features in deep neural net-based systems, which affects classification accuracy.
Innovation Solution
A method that involves obtaining dense representations of external features and text, along with a mask associating external features to tokens, and employing a cross-attention process to fuse this information, resulting in a joint representation that improves classification accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If external features are concatenated to contextual embeddings in deep neural networks, then the model can leverage project-specific external feature information, but the optimal leveraging of external features with token features remains unclear and classification accuracy is limited
Solution Approach 1:
The patent introduces an intermediary attention mechanism that mediates between external features and token embeddings. Instead of direct concatenation, the attention mechanism selectively integrates external features based on their relevance to each token, computed through attention scores. This intermediary layer enables optimal feature fusion by dynamically weighting external features according to contextual needs, thereby improving classification accuracy without requiring complex manual feature engineering
Solution Approach 2:
The patent transforms the fixed concatenation operation into a dynamic parameter-based integration using attention scores. The attention mechanism computes relevance parameters (attention scores) that dynamically adjust how external features are integrated with token embeddings. This parameter change allows the model to adaptively leverage external features based on input context, resolving the contradiction between simple concatenation and optimal feature leveraging
2Adaptability or versatility
If external features are integrated with token features, then the model can recognize entities not included in training examples, but the relative importance of each external feature with respect to token information is not properly calculated
Solution Approach 1:
The patent implements feedback through the attention mechanism, where the model computes attention scores that reflect the relevance of external features to each token. This feedback loop allows the model to dynamically adjust feature integration based on input context, ensuring that external features are weighted according to their actual importance for each specific token. This resolves the contradiction by providing precise, context-aware feature importance calculation that enhances entity recognition capability
Solution Approach 2:
The patent transforms static feature integration into a dynamic process through attention mechanisms. Instead of fixed weight integration, the attention scores dynamically determine the importance of each external feature relative to token information based on the specific input context. This dynamic adjustment enables the model to adaptively recognize entities not seen during training while maintaining precise measurement of feature importance through learned attention weights
Data Source
AI summary
There is provided a method that includes obtaining (a) a dense representation of external features, (b) a dense representation of text, and (c) a mask that associates the external features to tokens of the text, and employing a cross-attention process that utilizes the mask to perform an information fusion of the dense representation of the external features and the tokens of the text, thus yielding a joint representation of the external features and the tokens of the text. There is also provided a system that executes the method, and a storage device that includes instructions for controlling a processor to perform the method.