Scanned Document Separation Using OCR Token Context
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for automatically dividing scanned images of multiple documents en bloc suffer from inaccuracies due to not considering word order, dependence relationships, and contextual meaning, leading to incorrect document delimitation, especially when documents share similar names or organizations.
Innovation Solution
An image processing apparatus that utilizes a neural network model to determine document delimitation positions by generating vectors from tokenized text data of paired page images, considering semantic relationships and continuity, and employing token adjustment and fine-tuning to enhance accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If word frequency analysis is used for document delimitation, then the method is simple and fast, but the division accuracy is low because it does not consider word order, dependence relationships, or contextual meaning
Solution Approach 1:
The patent transforms the text representation from simple word frequency counts to contextualized token sequences with positional information. By changing the parameter of text representation to include word order and contextual relationships, the system achieves higher division accuracy while maintaining computational feasibility through efficient neural network processing
Solution Approach 2:
The patent introduces a neural network model as an intermediary between the input text data and the delimitation decision. This intermediary processes the complex relationships between tokens, word orders, and contextual meanings, transforming the raw text into accurate delimitation positions without requiring complex manual analysis
2Power
If simple text similarity calculation is used, then the computation is fast, but incorrect delimitation occurs when documents share the same template with different names
Solution Approach 1:
The patent applies local quality by focusing the neural network's attention on specific local contexts and token relationships rather than treating all text equally. The model identifies local patterns in word sequences and contextual dependencies that distinguish different documents even when they share similar templates, enabling accurate delimitation based on subtle local differences
Data Source
AI summary
An image processing apparatus divides scanned image data including page images obtained by scanning a plurality of documents for each page into image data of each document. The apparatus generates text data by performing character recognition processing for the plurality of page images, sequentially obtains a pair of page images in succession from the plurality of page images and then determines a document delimitation position based on text data of the two page images constituting the pair, and divides the scanned image data at the determined delimitation position. A vector corresponding to tokens is obtained by decomposing the text of each of the two page images constituting the pair is generated and input to a neural network model, the delimitation position is determined by using a score output from the neural network model and represents a possibility value that the two page images constituting the pair belong to different documents.