Named Entity Recognition Using ML-Guided Dictionary Updating
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing named entity recognition (NER) systems face challenges in maintaining up-to-date and accurate entity dictionaries due to the increasing number of entities and entity types, leading to inefficiencies and inaccuracies in identifying entities within large-scale datasets, particularly in bioinformatics and chem(o)informatics literature.
Innovation Solution
A combined NER system comprising an NER dictionary and a trained NER Machine Learning (ML) system, where entity results are analyzed and filtered to identify new entities and update dictionaries, using ML techniques such as neural networks and conditional random fields to enhance accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manually curated and annotated entity dictionaries are used for NER, then known entities can be reliably identified, but the system becomes time and resource intensive to maintain and cannot keep up with constantly increasing number of entities
Solution Approach 1:
The system enables automatic entity dictionary generation and updating through machine learning models that self-learn from corpus data, eliminating the need for manual curation and annotation while maintaining entity identification accuracy
Solution Approach 2:
Manual mechanical processes of dictionary curation and annotation are replaced with automated machine learning-based entity recognition systems that process and learn from large-scale corpus data
2Reliability
If manually curated entity dictionaries are used, then entity matching can be performed, but the system cannot identify new entities with different naming conventions
Solution Approach 1:
The entity recognition system transitions from static manually-curated dictionaries to dynamic machine learning models that continuously adapt and learn new entity types and naming conventions from incoming corpus data
Solution Approach 2:
The system changes from fixed entity lists to flexible probabilistic models that can recognize entities with varying parameters including different naming conventions, synonyms, and emerging entity types
3Adaptability or versatility
If the number of entities and entity types increases to cover more research domains, then entity recognition coverage improves, but manual curation becomes increasingly unmanageable and expensive
Solution Approach 1:
The complex manual mechanical process of dictionary curation is replaced with automated machine learning systems that scale efficiently to handle increasing numbers of entities and entity types across multiple research domains
Data Source
AI summary
Systems, methods and apparatus are provided for identifying entities in a corpus of text. The system comprising: a first named entity recognition (NER) system comprising one or more entity dictionaries, the first NER system configured to identify entities and/or entity types within a corpus of text based on the one or more entity dictionaries, a second NER system comprising an NER model configured for predicting entities and/or entity types within the corpus of text; and a comparison module configured for identifying entities based on comparing the entity results output from the first and second NER systems, where the identified entities are different to the entities identified by the first NER system. The system may further include an updating module.