Named Entity Extraction via Block Validation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing named entity recognition (NER) techniques face accuracy issues when extracting named entities from long documents, as the natural language processing models have an upper limit on the number of tokens that can be inputted at a time, leading to improper entity extraction and reduced accuracy.
Innovation Solution
An information processing apparatus that breaks down text into token strings, divides them into blocks within the processing capacity of a natural language processing model, estimates named entities for each block, determines the validity of these blocks, and outputs the characteristic named entities based on the estimation and determination results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the text is divided into multiple blocks to fit the model's input limit, then the model can process long documents, but the accuracy of extracting named entities representing document characteristics deteriorates
Solution Approach 1:
The patent divides the token string into multiple input blocks that can be processed by the natural language processing model. Each block is processed separately to extract named entities, and then the results are aggregated. This segmentation approach enables the model to handle long documents that exceed the token input limit while maintaining processing capability.
Solution Approach 2:
The patent performs preliminary actions by determining the validity of each input block before using it for named entity extraction. The system identifies whether each block contains the named entity representing document characteristics, and only uses valid blocks for final extraction. This preliminary validation ensures that the segmentation process does not compromise extraction accuracy.
2Productivity
If the token string is divided into multiple blocks, then the processing capacity of the natural language processing model is respected, but some extracted named entities become improper and reduce overall accuracy
Solution Approach 1:
The patent implements a feedback mechanism where the system determines whether each input block is valid for extracting named entities representing document characteristics. Based on the extraction results from each block, the system identifies valid blocks and uses this information to aggregate results accurately. This feedback loop ensures that improper extractions from individual blocks do not compromise the overall reliability of the extraction process.
Data Source
AI summary
The accuracy of extracting a named entity representing a characteristic of a document is improved. An information processing apparatus that extracts the named entity from document data by using a natural language processing model obtains data of text from the document data, generates a token string by processing to breakdown the text into a token unit, generates input blocks by dividing the token string into blocks in a unit that can be processed by the natural language processing model, estimates the named entity for each input block by inputting each of the input blocks to the natural language processing model, determines whether each of the input blocks is valid to be used to extract the named entity representing the characteristic of the document data based on an estimation result, and outputs the named entity representing the characteristic of the document data based on the estimation result and a determination result.