Semi-Supervised Document Tagging Using Embeddings and Graph Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing document tagging methods face challenges due to the lack of preexisting labels and the high cost and latency associated with using large language model (LLM) classifiers, particularly in zero-shot or few-shot learning scenarios, which lack domain-specific knowledge.
Innovation Solution
A semi-supervised approach combining a language model with a graph neural network, utilizing a mixed loss function that includes a soft silhouette score and cross-entropy score, to transform content portions into embeddings and generate category predictions without relying on expensive LLM API calls, leveraging user-defined and generative AI model-generated labels for improved accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If large language model (LLM) classifiers are used for document tagging in zero-shot or few-shot learning scenarios, then tagging capability is achieved without preexisting labels, but financial cost and latency increase significantly
Solution Approach 1:
The system segments the tagging task into two parts: using a pre-trained language model for embedding generation (which can be cached and reused) and using a separate classification model for category prediction. This segmentation allows the expensive LLM to be used only for embedding extraction rather than full classification, reducing latency and cost while maintaining adaptability to new domains.
Solution Approach 2:
The patent introduces an intermediary classification model that sits between the language model and the final tagging output. This intermediary model processes the embeddings and generates category predictions, acting as a mediator that reduces the direct dependency on expensive LLM API calls for the classification task, thereby reducing latency and cost.
2Adaptability or versatility
If large language model (LLM) classifiers are used for document tagging, then tagging capability is achieved without preexisting labels, but financial cost increases significantly
Solution Approach 1:
The system applies partial action by using the expensive LLM only for embedding extraction rather than full classification. The classification task is performed by a cheaper, lighter model that processes the embeddings, thereby reducing the overall financial cost while maintaining the adaptability to new domains through the embedding-based approach.
Solution Approach 2:
The patent creates a copying approach by generating embeddings that capture the semantic meaning of documents, which can then be reused and processed by cheaper classification models. Instead of calling the expensive LLM for every classification task, the system copies the essential information into embeddings that can be processed more economically.
3Adaptability or versatility
If large language model (LLM) classifiers are used for document tagging, then tagging capability is achieved without preexisting labels, but domain-specific knowledge is lacking
Solution Approach 1:
The system performs preliminary action by pre-training the language model on large amounts of general data to create robust embeddings that capture universal linguistic patterns. This preliminary training enables the model to handle new domains effectively without requiring retraining, as the embeddings capture fundamental language structures that can be adapted to any domain through the classification layer.
Solution Approach 2:
The patent applies parameter changes by adjusting the classification model's parameters based on domain-specific data while keeping the language model's embedding parameters fixed. This allows the system to adapt to domain-specific knowledge through the classification layer without retraining the entire LLM, thereby maintaining reliability in new domains while avoiding the cost of full retraining.
4Measurement precision
If user-labeled content portions are used for training, then tagging accuracy is improved, but the cost and complexity of procuring labeled data increases
Solution Approach 1:
The system implements self-service by using the model to generate its own training data through pseudo-labeling. The classification model predicts labels for unlabeled data, and these predictions are used to create training datasets, allowing the system to label itself without requiring manual annotation. This significantly reduces the complexity and cost of procuring labeled data while maintaining tagging accuracy through iterative improvement.
Solution Approach 2:
The patent incorporates feedback mechanisms where the model's performance on labeled data is used to refine its pseudo-labeling capabilities. The system continuously improves by using feedback from labeled examples to enhance the classification model, which then generates better pseudo-labels for unlabeled data, creating a feedback loop that improves accuracy while reducing the need for manual labeling.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method, computer program product, and computing system for transforming a plurality of content portions into a plurality of embeddings using a language model. A graph is generated with nodes representing respective embeddings and an edge between a pair of nodes representing a similarity distance between the respective embeddings that is less than or equal to a predefined threshold. A category prediction is generated for each content portion by processing the graph using a graph neural network. A loss function is determined using a plurality of predefined categories and the category predicted for each content portion. The language model and the graph neural network are finetuned for automatically tagging content portions with a category by maximizing the loss function.