Path Dropout Training for Contextual Named Entity Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial intelligence-based chatbots struggle to accurately identify named entities due to insufficient attention on contextual information during training, leading to misclassification of entities.
Innovation Solution
Implementing path dropout techniques in machine-learning models, specifically by dropping out residual connections and diagonal entries in attention weight matrices, to enforce contextual learning and improve named entity recognition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If residual connections are used in the machine learning model to maintain training stability, then the model training is more stable, but the model fails to focus on contextual information leading to poor named entity recognition
Solution Approach 1:
The patent applies dropout technique to randomly discard (drop out) residual connections during training with a certain probability. This forces the model to rely on contextual information from self-attention mechanisms rather than shortcutting through residual connections. During inference, all connections are recovered and operate together, providing both regularization during training and full capacity during deployment.
2Productivity
If the model pays insufficient attention to contextual information during training, then the training process is simpler and faster, but the model misclassifies named entities
Solution Approach 1:
The patent pre-empts the problem of insufficient contextual attention by applying dropout to residual connections during training. This preliminary action forces the model to learn contextual information properly from the outset, preventing the development of poor attention patterns that would lead to misclassification.
Data Source
AI summary
Techniques are provided for improved training of a machine-learning model that includes multiple layers and is configured to process textual language input. The machine-learning model includes one or more blocks in which each block includes a multi-head self-attention network, a first connection for providing input to the multi-head self-attention network, and a second (residual) connection for providing the input to a normalization layer, bypassing the multi-head self-attention network. During training, the second connection is dropped out according to a dropout parameter. Additionally, or alternatively, an attention weight matrix is used for dropout by blocking diagonal entries in the attention weight matrix. As a result, the machine-learning model increasingly focuses on contextual information, which provides more accurate language processing results.