Deep Reinforcement Learning Coreference Resolution Model
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep neural network models for coreference resolution in natural language processing rely on hand-crafted features and syntactic parsers, leading to errors and limited generalization, and current end-to-end models are trained with heuristic loss functions that do not directly optimize coreference evaluation metrics.
Innovation Solution
A deep reinforcement learning-based approach that uses a policy network to directly optimize coreference evaluation metrics by generating probability distributions over actions for linking mentions, updating a coreference graph, and computing rewards based on evaluation metrics, with entropy regularization to prevent premature convergence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If hand-crafted features and syntactic parsers are used in deep neural network models for coreference resolution, then the models can be trained with available data, but the models suffer from errors propagation and limited generalization
Solution Approach 1:
The patent extracts and removes the hand-crafted features and syntactic parsers from the coreference resolution system. By eliminating these fixed components, the model avoids errors propagation and limited generalization while focusing on learning from raw text data through reinforcement learning, thereby improving both reliability and adaptability
Solution Approach 2:
The system enables the model to automatically learn and optimize its own feature representations through reinforcement learning without relying on pre-defined hand-crafted features. The model serves itself by continuously improving its coreference resolution capability through reward-based learning from evaluation metrics
2Ease of manufacture
If heuristic loss functions are used to train end-to-end models, then the training process is simplified, but the models do not directly optimize coreference evaluation metrics
Solution Approach 1:
The patent implements a feedback mechanism where the model is trained using reinforcement learning with rewards computed from coreference evaluation metrics (MUC, B3, CEAF). This closed-loop feedback system allows the model to directly optimize these metrics by adjusting its policy based on the computed rewards, achieving precise metric optimization while maintaining training feasibility through automated reward computation
3Reliability
If deep reinforcement learning with entropy regularization is used, then exploration is encouraged and performance is improved, but the training complexity increases
Solution Approach 1:
The patent modifies the training objective by adding an entropy regularization term to the reward function. This parameter change encourages exploration by penalizing deterministic policies, allowing the model to discover better coreference resolution strategies while maintaining manageable training complexity through a single additional hyperparameter (entropy coefficient)
Solution Approach 2:
The entropy term acts as an intermediary mechanism that mediates between exploration and exploitation during training. It provides a balanced approach by encouraging diverse policy selection without completely overriding the reward signal, thus improving performance while keeping the training system relatively simple through a well-defined mathematical formulation
Data Source
AI summary
Described herein are embodiments for end-to-end reinforcement learning based coreference resolution models to directly optimize coreference evaluation metrics. Embodiments of a reinforced policy gradient model are disclosed to incorporate reward associated with a sequence of coreference linking actions. Furthermore, maximum entropy regularization may be used for adequate exploration to prevent a model embodiment from prematurely converging to a bad local optimum. Experiments on datasets compared with state-of-the-art methods verified the effectiveness of embodiments.