Unsupervised Named Entity Resolution via Corpus Self-Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Named Entity Resolution (NER) methods in Natural Language Processing (NLP) require labeled training data, are not adaptable to new corpus additions, and necessitate expensive caching of disambiguation models, limiting their efficiency and scalability.
Innovation Solution
An unsupervised NER method that receives a named entity, searches for undisambiguated entities, identifies disambiguated entities and their aliases, trains a classifier using these aliases, and resolves the entity without labeled data, adapting to corpus changes and avoiding model caching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If supervised NER methods are used with labeled training data, then classification accuracy is improved, but data preparation cost and time increase
Solution Approach 1:
The system performs self-training by automatically generating training data from the corpus itself. The NER component identifies entities, the disambiguation component resolves them, and these results are fed back to train the classifier without requiring external labeled data, enabling the system to improve its own performance autonomously
Solution Approach 2:
The system performs preliminary entity resolution and disambiguation on the corpus to generate training examples before actual classification. This pre-processing creates a foundation of labeled data that accelerates subsequent classification operations without requiring manual annotation
2Measurement precision
If traditional NER methods are used, then model accuracy is maintained, but adaptability to new corpus additions deteriorates
Solution Approach 1:
The system implements dynamic adaptation by continuously training the classifier on newly added corpus portions. When the corpus is updated, the system automatically retrains the NER classifier with the new data, allowing the model to adapt its parameters and maintain accuracy on evolving data distributions
Solution Approach 2:
The system establishes a feedback loop where classification results and disambiguation outcomes are fed back into the training process. This continuous feedback mechanism allows the classifier to learn from its performance and automatically adjust to new patterns in the corpus
3Speed
If disambiguation models are cached for fast resolution, then query speed is improved, but memory requirements and cost increase
Solution Approach 1:
The system extracts only the essential classification functionality needed for entity resolution, separating it from comprehensive disambiguation models. By focusing on classification rather than full disambiguation, the system reduces model size and memory requirements while maintaining adequate query performance
Solution Approach 2:
Instead of caching large, expensive disambiguation models in memory, the system uses lightweight classifiers that can be rapidly trained and discarded. The system retrains classifiers on-demand rather than maintaining permanent cached models, reducing memory usage while maintaining responsiveness
Data Source
AI summary
A method for providing unsupervised entity resolution to a natural language processing system includes receiving a named entity for training from the natural language processing system, searching a corpus for a first undisambiguated named entity corresponding to the named entity, identifying a plurality of disambiguated named entities corresponding to the first undisambiguated named entity, identifying a plurality of aliases for each of the disambiguated named entities, training a classifier for the each of the disambiguated named entities utilizing the aliases identified for respective ones of the disambiguated named entities using the corpus, and resolving the named entity using the classifier, wherein resolving the named entity comprises selecting one of the disambiguated named entities from among the disambiguated named entities and returning, automatically, the selected disambiguated named entity to the natural language processing system.