Transformer Model Entity Marker Consistency Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Pretrained language models are prone to making incorrect predictions when small perturbations, such as entity swapping, are applied to input entities, leading to performance degradation and failure to generalize across entities.
Innovation Solution
The introduction of entity markers and auxiliary loss functions, specifically consistency and semantics losses, within a transformer neural network model to learn entity-independent representations, allowing the model to generate robust and generalizable token embeddings that are less affected by entity swaps.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If pretrained language models are used for NLP tasks, then performance in a wide range of tasks is improved, but performance degrades when small perturbations such as entity swapping are applied to input entities
Solution Approach 1:
The patent introduces entity markers as intermediary tokens that replace actual named entities in the input text. These entity markers serve as a bridge between the original entity information and the transformer model processing, allowing the model to learn entity-independent representations. The entity markers enable the model to focus on contextual relationships rather than entity-specific details, thereby improving robustness to entity swapping perturbations while maintaining prediction accuracy.
Solution Approach 2:
The patent modifies the token type embedding parameter by introducing a new token type category for entity markers. This parameter change allows the transformer model to distinguish entity markers from regular tokens and apply appropriate processing. By changing the token type parameter, the model learns to generate embeddings that are invariant to entity identity, thus resolving the contradiction between maintaining accuracy and achieving robustness to entity perturbations.
2Loss of information
If entity-specific representations are learned, then model captures detailed entity information, but model fails to generalize across different entities of the same class
Solution Approach 1:
The patent extracts entity-specific information by replacing named entities with generic entity markers in the training process. This extraction approach separates the entity identity information from the contextual relationship information. By taking out the specific entity names and replacing them with standardized markers, the model learns to represent entities based on their contextual roles rather than their specific identities, enabling generalization across different entities of the same class while retaining essential entity information.
Solution Approach 2:
The entity markers serve as universal tokens that can represent any named entity regardless of its specific identity. This universal representation allows the model to handle different entities (persons, locations, organizations, etc.) using the same token type, thereby achieving multi-functionality. The model learns entity-independent representations that work across diverse entity types, improving generalization while maintaining the ability to capture detailed contextual information through the surrounding text.
Data Source
AI summary
A computer-implemented system and method and for learning an entity-independent representation are disclosed. The method may include: receiving an input text; identifying named entities in the input text; replacing the named entities in the input text with entity markers; parsing the input text into a plurality of tokens; generating a plurality of token embeddings based on the plurality of tokens; generating a plurality of positional embeddings based on the respective position of each of the plurality of tokens within the input text; generating a plurality of token type embeddings based on the plurality of tokens and the one or more named entities in the input text; and processing the plurality of token embeddings, the plurality of positional embeddings, and the plurality of token type embeddings using a transformer neural network model to generate a hidden state vector for each of the plurality of tokens in the input text.