Document Segmentation Using Content Vectors and Nearest Labeled Pages
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems are inefficient and require subject matter expert intervention to identify the boundaries of documents in a set of documents, and existing systems do not provide sufficient accuracy for identifying the boundaries of the original documents without the help of subject matter experts.
Innovation Solution
A method and system using a machine-learning model to generate content vectors, determine nearest labeled pages based on vector distances, and apply dynamic programming to segment documents automatically.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional systems are used to identify document boundaries, then subject matter expert intervention is required, but this increases time consumption and operational complexity
Solution Approach 1:
The system performs self-service by automatically identifying document boundaries through machine learning models that analyze content vectors and determine segmentation options without requiring external subject matter expert intervention. The algorithm independently processes the set of documents, extracts content items, generates content vectors, and identifies document boundaries based on vector distances and segmentation options.
Solution Approach 2:
The patent replaces the mechanical system of manual expert review with an automated computational system. Instead of relying on subject matter experts to visually inspect and identify document boundaries, the system uses machine learning models to generate content vectors and algorithmically determine segmentation options, substituting human cognitive processes with automated computational analysis.
2Measurement precision
If manual expert review is used for document boundary identification, then accuracy can be achieved, but computational resources and time are excessively consumed
Solution Approach 1:
The system performs preliminary actions by pre-processing the documents to extract content items and generate content vectors before the actual boundary identification process. This preliminary extraction and vector generation enables the subsequent segmentation algorithm to work efficiently with structured data, reducing the time required for the final boundary identification compared to manual review processes.
Solution Approach 2:
The patent transforms the document data into a different parameter space by converting content into vectors. This parameter transformation allows the system to identify document boundaries through mathematical operations on vector distances and segmentation options, which is computationally more efficient than manual visual inspection while maintaining or improving accuracy.
3Productivity
If automated segmentation is implemented, then processing efficiency improves, but measurement precision of document boundaries may deteriorate
Solution Approach 1:
The system implements feedback mechanisms by using machine learning models that are trained to generate accurate content vectors and by employing algorithms that evaluate multiple segmentation options based on vector distances. The feedback loop involves the model learning from the document content and adjusting its segmentation decisions to maximize boundary identification accuracy while maintaining high processing efficiency through automated operations.
Solution Approach 2:
The patent moves the boundary identification problem into another dimension by transforming textual content into vector space. Instead of analyzing documents in their original textual form, the system operates in a vector dimension where mathematical distance calculations and segmentation algorithms can efficiently and accurately identify boundaries, combining high productivity with maintained precision.
4Measurement precision
If subject matter experts are involved in document boundary identification, then accuracy is maintained, but device complexity and operational overhead increase
Solution Approach 1:
The system achieves universality by creating a multi-functional automated platform that performs document extraction, content vector generation, and boundary identification all through integrated machine learning models and algorithms. This universal system replaces multiple specialized roles (including subject matter experts) with a single automated system that handles the entire document segmentation process, reducing system complexity despite the sophisticated processing involved.
Data Source
AI summary
In some examples, systems and methods for disaggregating a set of documents are provided. An example method includes receiving the set of documents. In some examples, the set of documents include a plurality of pages. In some examples, the method further includes extracting a plurality of content items from the plurality of pages and providing the plurality of extracted content items to a machine-learning model. In some examples, the machine-learning the model is trained to generate content vectors. In some examples, the method further includes receiving, from the machine learning model, a plurality of content vectors corresponding to the plurality of extracted content items, determining, for each page of the plurality of pages, a plurality of potential nearest labelled pages in one or more labelled documents and a plurality of vector distances from the plurality of potential nearest labelled pages, based on the plurality of content vectors, and determining a segmentation option based at least in part on the plurality of vector distances. In some examples, the segmentation option indicates that a group of pages in the plurality of pages belong to a specific document.