Document Graph Pre-Training for Low-Label Attribute Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training machine learning models to classify content in FAQ-type documents is challenging due to the high variability in document layouts, requiring large quantities of labeled data that are often not available, leading to inconsistent accuracy across different formats.
Innovation Solution
Perform unsupervised pre-training tasks on unlabeled document graphs by modifying the data through masking or swapping text content, followed by fine-tuning with labeled data to generate a machine learning model capable of identifying document attributes such as key-value pairs and tables.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If supervised training with labeled data is used to train machine learning models for document classification, then the model can learn to classify document content, but generating classification labels is time-intensive and large quantities of labeled data are required
Solution Approach 1:
The patent applies preliminary action by performing unsupervised pre-training on document structure and content representations before supervised fine-tuning. This pre-training phase prepares the model to better learn from subsequent labeled data, reducing the amount of time-intensive labeled data needed while maintaining classification accuracy.
Solution Approach 2:
The patent implements self-service through self-supervised learning tasks where the model generates its own training signals from unlabeled data. By performing tasks like masked language modeling and structure prediction on unlabeled documents, the model learns useful representations without requiring external label generation, thereby reducing the time burden of label creation.
2Measurement precision
If supervised training with labeled data is used to train machine learning models for document classification, then the model can learn to classify document content, but large quantities of training documents with corresponding classification labels are required
Solution Approach 1:
The patent implements self-service through self-supervised learning tasks where the model generates its own training signals from unlabeled data. By performing tasks like masked language modeling and structure prediction on unlabeled documents, the model learns useful representations without requiring external label generation, thereby reducing the time burden of label creation.
Solution Approach 2:
The patent applies preliminary action by performing unsupervised pre-training on document structure and content representations before supervised fine-tuning. This pre-training phase prepares the model to better learn from subsequent labeled data, reducing the amount of time-intensive labeled data needed while maintaining classification accuracy.
3Adaptability or versatility
If a machine learning model is trained on FAQ documents with varied layouts, then the model should handle different formats, but the high variability in document layouts makes it challenging to generate sufficient training data for each format
Solution Approach 1:
The patent applies universality by designing a unified model architecture and training approach that handles multiple document layouts simultaneously. The model learns general document structure representations and content relationships that transfer across different FAQ formats, eliminating the need for separate training data sets for each layout type while maintaining adaptability to various formats.
Data Source
AI summary
Techniques for multi-layer training of a machine learning model are disclosed. A system pre-trains a machine learning model on training data obtained from unlabeled document graph data by executing unsupervised pre-training tasks on the unlabeled document graph data to generate a labeled pre-training data set. The system modifies document graphs to change attributes of nodes in the document graphs. The system pre-trains the machine learning model with a data set including the modified document graphs and un-modified document graphs to generate prediction associated with the modifications to the document graphs. Subsequent to pre-training, the system fine-tunes the machine learning model with a set of labeled training data to generate predictions associated with a specific attribute of a document graph.