Machine Learning for Named Entity Recognition With Approximate Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing named entity recognition techniques struggle to accurately identify unknown words, particularly in specialized fields like biomedicine, due to limitations in dictionary coverage and reliance on exact string matching, which hinders the recognition of new or modified named entities.
Innovation Solution
A machine learning model that performs approximate string matching and combines token vectors with matching information to generate input data, using a multi-layer neural network to improve recognition accuracy for unknown named entities by leveraging both exact and approximate string matching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If exact string matching is used for named entity recognition, then recognition speed is improved, but recognition accuracy for unknown named entities deteriorates
Solution Approach 1:
The patent changes the matching parameter from exact string matching to approximate string matching based on edit distance. This allows the system to match named entities even when there are slight variations in spelling or formatting, thereby improving recognition accuracy for unknown named entities while maintaining reasonable processing speed through efficient distance calculation algorithms.
Solution Approach 2:
The patent segments the named entity recognition process into multiple stages: first performing approximate matching to identify candidate entities, then using edit distance calculations to refine matches. This segmentation allows the system to balance speed and accuracy by applying different matching strategies at different processing stages.
2Measurement precision
If dictionary coverage is expanded to include more named entities, then recognition accuracy is improved, but device complexity deteriorates
Solution Approach 1:
The patent introduces edit distance as an intermediary mechanism between the query and the dictionary. Instead of requiring exact matches or maintaining an exhaustive dictionary, the edit distance metric serves as a mediator that can identify named entities even with partial or approximate matches, reducing the need for comprehensive dictionary coverage while maintaining high recognition accuracy.
Solution Approach 2:
The patent changes the matching criterion from exact string equality to approximate matching based on edit distance thresholds. This parameter change allows the system to achieve high recognition accuracy with a more compact dictionary by accepting matches that fall within a predetermined distance threshold, thereby reducing dictionary complexity while maintaining or improving accuracy.
3Measurement precision
If approximate string matching is used for unknown named entities, then recognition accuracy is improved, but processing time deteriorates
Solution Approach 1:
The patent applies partial matching by performing approximate string matching only on candidate entities that meet certain preliminary criteria, rather than exhaustively comparing against all dictionary entries. This partial action approach maintains high recognition accuracy for unknown named entities while significantly reducing the overall processing time by limiting the scope of approximate matching operations.
4Adaptability or versatility
If character string editing is performed to generate candidate named entities, then recognition coverage is improved, but device complexity deteriorates
Solution Approach 1:
The patent implements a self-service mechanism where the system automatically generates candidate named entities by performing controlled character string editing operations (insertion, deletion, substitution) on existing dictionary entries. This self-service approach improves recognition coverage for unknown and variant named entities while managing processing complexity through automated generation rather than manual curation of extensive dictionaries.
Data Source
AI summary
A computer divides a character string included in text data into a plurality of tokens. The computer searches, by performing matching processing between a token string indicating a specific number of consecutive tokens among the plurality of tokens and dictionary information including a plurality of named entities, the plurality of named entities for a similar named entity whose similarity to the token string is equal to or more than a threshold. The computer converts matching information indicating a result of the matching processing between the token string and the similar named entity into first vector data. The computer generates input data by using a plurality of pieces of vector data converted from the plurality of tokens and the first vector data. The computer generates a named entity recognition model that detects a named entity by performing machine learning using the input data.