Statistical and Named Entity Tokenization for NLU Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Natural language understanding (NLU) systems face challenges in token extraction, particularly when text inputs do not explicitly include named entities or when named entities are used in contexts irrelevant to the intended meaning, leading to inaccurate resolution of text meaning.
Innovation Solution
The method involves applying a combination of statistical tokenization models and named entity-based tokenization techniques within an NLU system, where a first statistical tokenization model is used initially, and a second named entity-based model is applied if the first model fails to extract necessary tokens, with a third model processing sentences lacking named entities to ensure accurate token extraction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If named entity-based tokenization is used, then token extraction accuracy is improved when named entities are present, but token extraction fails when named entities are absent or used in irrelevant contexts
Solution Approach 1:
The tokenization process is segmented into multiple independent stages: a first statistical tokenization model processes all sentences, and a second named entity-based tokenization model processes only sentences where the first model fails to extract needed tokens. This segmentation allows each model to specialize in its strength while collectively handling all sentence types.
Solution Approach 2:
The system dynamically selects which tokenization model to apply based on the sentence characteristics and extraction needs. The named entity-based model is activated only when necessary (when tokens are not extracted by the statistical model), making the system adaptive to different text patterns rather than using a fixed approach.
2Adaptability or versatility
If statistical tokenization model is used, then token extraction works for all sentence types, but extraction accuracy decreases for sentences with named entities used in irrelevant contexts
Solution Approach 1:
The tokenization function is segmented into two specialized models: a statistical tokenization model for general sentence processing and a named entity-based model for precise entity extraction. The system segments the processing workflow to apply the appropriate model based on needs.
Solution Approach 2:
The system uses feedback from the first statistical tokenization model to determine when to invoke the second named entity-based model. If needed tokens are not extracted by the statistical model, the feedback mechanism triggers the named entity-based model to provide additional or corrected extraction.
3Device complexity
If a single tokenization model is used, then system complexity is reduced, but token extraction accuracy varies across different sentence types
Solution Approach 1:
The tokenization system is segmented into multiple specialized models rather than using a single monolithic model. Each model handles specific aspects of token extraction, and the segmentation is managed through a coordinated processing framework that selects appropriate models based on sentence characteristics.
Solution Approach 2:
The tokenization system achieves multi-functionality by combining different models with complementary strengths. The statistical model handles general cases while the named entity-based model handles specific cases, making the overall system universally applicable to diverse sentence types without requiring each individual model to be perfectly accurate for all cases.
Data Source
AI summary
A method of processing text within a natural language understanding system can include applying a first tokenization technique to a sentence using a statistical tokenization model. A second tokenization technique using a named entity can be applied to the sentence when the first tokenization technique does not extract a needed token according to a class of the sentence. A token determined according to at least one of the tokenization techniques can be output.