Entity Recognition Using Out-of-Collection Probabilities
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current search engines fail to accurately disambiguate entities in large document sets due to limitations in contextual information and incomplete dictionaries, leading to incorrect entity classification.
Innovation Solution
A dictionary-based classifier is developed that uses a decision tree to estimate the probability of entities not included in the dictionary by partitioning entities by name and incorporating out-of-collection probabilities, improving the confidence in entity-document associations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a dictionary-based system is used to disambiguate entities, then entities in the dictionary can be accurately identified, but entities not in the dictionary cannot be recognized
Solution Approach 1:
The patent segments the entity recognition problem into two parts: dictionary-based exact matching for known entities, and probabilistic out-of-dictionary matching for unknown entities. This segmentation allows the system to handle both cases appropriately, resolving the contradiction between accuracy for known entities and coverage for unknown entities.
Solution Approach 2:
The patent introduces an intermediary probabilistic model that bridges the gap between the dictionary-based system and out-of-dictionary entities. This intermediary computes the probability that an entity name refers to an out-of-dictionary entity, allowing smooth transition between the two regimes and resolving the contradiction.
2Adaptability or versatility
If clustering-based techniques are used to disambiguate entities, then contextual information can be utilized, but incorrect clustering may occur when contextual information is insufficient or misleading
Solution Approach 1:
The patent incorporates feedback by computing the probability of out-of-dictionary entities based on the results of dictionary matching and contextual analysis. This feedback mechanism allows the system to adjust its disambiguation decisions, reducing the risk of incorrect clustering while still utilizing contextual information.
3Measurement precision
If context-based disambiguation is used, then entities can be distinguished in different contexts, but entities with the same name in different contexts may be incorrectly separated
Solution Approach 1:
The patent introduces dynamic probability computations that adapt to the specific context while maintaining entity identity information. The out-of-dictionary probability is computed dynamically based on contextual factors, allowing the system to distinguish entities in different contexts while preventing incorrect separation of the same entity across contexts.
Data Source
AI summary
A classifier that disambiguates among entities based on a dictionary, such as corpus of documents about those entities, is built by incorporating probabilities that an entity exists that is not in the dictionary. Given a document it is associated by the classifier with an entity. By incorporating out of collection probabilities into the classifier, a higher level of confidence in the match between an entity and a document is achieved.