NER Dictionary Generation from Structured Business Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Named Entity Recognition (NER) systems face challenges in achieving high precision when extracting business entities from unstructured texts, as references to entities are often incomplete or vary in form, leading to reduced recall and confidence in matches.
Innovation Solution
A computer-implemented method for generating a business-optimized NER dictionary by identifying structured data, generating unique variants of entity names, and associating them with underlying values, while using a language model corpus to remove common words/phrases, thereby maximizing precision in entity extraction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a comprehensive NER dictionary is created to capture all entity variations, then recall is improved, but precision deteriorates due to inclusion of common words and phrases
Solution Approach 1:
The patent segments the NER dictionary creation process into distinct phases: first generating variants from structured business data, then separately filtering out common words using a language model corpus. This segmentation allows independent optimization of recall (through comprehensive variant generation) and precision (through targeted filtering), resolving the contradiction between capturing all entity variations and excluding common words.
Solution Approach 2:
The patent introduces a language model corpus as an intermediary filtering mechanism between the generated NER dictionary and the final extraction process. This intermediary layer selectively removes common words and phrases while preserving business-specific entity variants, thereby maintaining high recall for relevant entities while improving precision by eliminating false positives from common vocabulary.
2Speed
If NER systems use simple matching algorithms, then processing speed is improved, but precision deteriorates when entities are referenced incompletely or with variations
Solution Approach 1:
The patent performs preliminary action by pre-generating all possible entity name variants from structured business data before the actual NER extraction process. This includes creating abbreviations, synonyms, and alternative formulations of business entities in advance. During extraction, the system can perform simple matching against this pre-generated variant dictionary, maintaining high processing speed while achieving high precision through comprehensive variant coverage.
Data Source
AI summary
The present disclosure involves systems, software, and computer implemented methods for generating NER dictionaries from structured business data. In one example, a method may include identifying a set of structured data for use in generating a NER dictionary, identifying a particular collection within the identified structured data to use in the dictionary generation, the particular identified collection including a plurality of values, identifying at least one variant of the value for each value in the particular identified collection, each of the at least one variants of the value including at least a portion of the underlying value, determining a set of unique variants from the identified at least one variants across each of the values in the particular identified subset, adding the determined set of unique variants to the generated dictionary, each unique variant added to the generated dictionary being associated with the value from which the unique variant was derived.