Token-Based Classification for PDF Field Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for extracting data from PDFs, such as Camelot, Tabula, and OCR tools, face challenges with token classification, leading to overlap and misclassification issues due to sentence-based classification and named entity recognition limitations, especially in complex document formats like conference papers where boundaries are unclear.
Innovation Solution
A processor-implemented method using a machine learning model, specifically Cased-Sci-Bert, for token-based classification that extracts text from PDFs, splits sentences into tokens, generates lists of tokens, tags, and confidence scores, and post-processes to accurately identify fields without relying on meta-information, effectively reducing overlap during parsing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If sentence-based classification or named entity recognition (NER) is used for field extraction, then the extraction process can be performed using existing tools and models, but field merging and misclassification occur due to unclear boundaries between fields
Solution Approach 1:
The patent segments the text into tokens (individual words or subwords) rather than processing at the sentence level. Each token is independently classified into a field category, allowing precise boundary detection between fields. This token-level segmentation resolves the field merging problem by assigning each token to a specific field based on its contextual position, eliminating the ambiguity inherent in sentence-based approaches.
Solution Approach 2:
The patent transitions from sentence-level classification to token-level classification, adding a finer granularity dimension to the extraction process. This dimensional change from coarser (sentence) to finer (token) resolution enables more precise field boundary detection and reduces misclassification, while still leveraging existing NER models through adapter layers or fine-tuning.
2Productivity
If NER models are used to extract entities, then author names and affiliations can be identified, but title content containing organization and location names is misclassified as author or affiliation information
Solution Approach 1:
The patent applies different classification behaviors to different positions and contexts within the text. Tokens in the title section are classified with awareness of their positional context, preventing organization and location names in titles from being misclassified as author or affiliation. The model learns position-specific classification patterns, assigning appropriate field labels based on local contextual cues rather than applying uniform NER rules throughout the document.
Solution Approach 2:
The system incorporates feedback mechanisms where the classification of each token influences subsequent classifications. By processing tokens sequentially and using the classification history and positional information, the model can correct potential misclassifications. When an organization name appears in the title context, the feedback from previous title token classifications helps maintain the correct field assignment, preventing erroneous classification as author or affiliation.
3Productivity
If existing extraction tools are used, then data can be extracted from PDFs quickly, but fields merge together and overlap occurs during the extraction process
Solution Approach 1:
The patent segments the extraction process into token-level operations, where each token is independently classified and assigned to a field. This fine-grained segmentation preserves field boundary information by clearly delineating which tokens belong to which fields, preventing the merging and overlapping problems that occur in coarser extraction approaches. The segmented token classifications can then be aggregated to reconstruct fields with precise boundaries.
Data Source
AI summary
A system and a method for token-based classification for reducing overlap in field extraction during parsing of a text is disclosed. The method includes extracting text from a resource. The method further includes splitting one or more sentences into a predetermined number of plurality of tokens. The method furthermore includes generating a plurality of lists using a machine learning model for identifying one or more fields in the text. The plurality of lists comprises at least a list of tokens, a list of tags and a list of confidence scores of tokens. The method furthermore includes post processing the plurality of lists for extracting one or more fields for parsing the text.