Named Entity Extraction Using Token Filtering and Phonetic Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently extracting artist names and song titles from decentralized and multilingual online sources, where information is often mixed with irrelevant data, and includes common words, aliases, abbreviations, and misspellings.
Innovation Solution
A method involving tokenization, comparison, filtering, and matching of text tokens using a rule parser and phonetic algorithms to identify named entities within a block of text, with a database storing artist and song names, aliases, and misspellings, and an inverted full text index for quick retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If automated information recovery is performed on decentralized social media sources, then the quantity of available information increases, but the difficulty of retrieving and analyzing relevant information increases
Solution Approach 1:
The system segments the text into tokens and divides the analysis process into distinct stages: tokenization, comparison with document tokens, filtering based on criteria, and matching sequences. This segmentation allows efficient processing of large volumes of decentralized social media data by breaking it down into manageable units and applying systematic analysis procedures.
Solution Approach 2:
The system introduces an intermediary indexing structure that maps document tokens to documents, enabling efficient retrieval and analysis of relevant information from decentralized sources. This intermediary layer facilitates the connection between raw social media text and structured information about artists and songs.
2Measurement precision
If the system compares every text token to every document token, then the completeness of named entity identification improves, but the processing time increases
Solution Approach 1:
The system performs preliminary tokenization of both the input text and the database documents before comparison. This preliminary processing organizes the data into token sequences, enabling more efficient matching and reducing the computational complexity of comparing text tokens with document tokens while maintaining completeness.
Solution Approach 2:
The system implements a filtering step that skips non-matching tokens and only performs detailed comparison when tokens match the filtering criteria. This allows the system to rapidly process large volumes of text by skipping irrelevant comparisons and focusing computational resources only on promising candidates.
3Measurement precision
If the system uses filtering criteria to eliminate common words, then the accuracy of named entity extraction improves, but the complexity of the filtering process increases
Solution Approach 1:
The system applies different filtering criteria to different contexts and positions of tokens within the text. Rather than using a uniform filtering approach, the system adapts its filtering behavior based on the local context, such as whether a token appears in a specific position or is surrounded by certain words, thereby improving accuracy while managing complexity through context-aware filtering.
Data Source
AI summary
A data processing method, program, and apparatus for identifying a document within a block of text. A block of text is tokenized into a plurality of text tokens according to at least one rule parser. Each of the plurality of text tokens is sequentially compared to a plurality of document tokens to determine if the text token matches one of the plurality of document tokens. The plurality of document tokens correspond to a plurality of documents which have been tokenized according to the one or more rule parsers. Each matched text token is filtered according to predetermined filtering criteria to generate one or more candidate text tokens. It is then determined whether sequence of candidate text tokens that occur in sequential order within the block of text match sequence of document tokens. If so, then it is determined that the document has been identified within the block of text. The document can correspond to an artist, a song names, and misspellings and aliases thereof.