Semi-Supervised Taxonomy Tagging With Graph Embedding Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of accurately tagging documents without sufficient pre-existing labels and the high cost and latency associated with using large language model (LLM) classifiers for document auto-tagging.
Innovation Solution
A semi-supervised approach using a combination of a language model and a graph neural network, leveraging a mixed loss function that includes a soft silhouette score and cross-entropy score, to generate category predictions for content portions, optimizing embeddings for better cluster separation and accuracy, without relying on expensive LLM API calls.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large language model (LLM) classifiers are used for document auto-tagging, then tagging accuracy is improved, but financial cost and latency increase
Solution Approach 1:
The patent creates a simplified copy of the LLM's embedding capabilities through a custom neural network model that reproduces the essential function of generating semantic embeddings without the computational overhead and cost of actual LLM API calls. This copy achieves sufficient accuracy for tagging while dramatically reducing latency and financial cost.
Solution Approach 2:
The patent replaces expensive, resource-intensive LLM API calls with a more economical custom neural network model that can be deployed locally. This cheaper model performs the same embedding generation function without the high financial cost and latency associated with LLM services, making the system more sustainable for production use.
2Measurement precision
If large language model (LLM) classifiers are used for document auto-tagging, then tagging accuracy is improved, but financial cost increases
Solution Approach 1:
The patent creates a simplified copy of the LLM's embedding capabilities through a custom neural network model that reproduces the essential function of generating semantic embeddings without the computational overhead and cost of actual LLM API calls. This copy achieves sufficient accuracy for tagging while dramatically reducing latency and financial cost.
Solution Approach 2:
The patent replaces expensive, resource-intensive LLM API calls with a more economical custom neural network model that can be deployed locally. This cheaper model performs the same embedding generation function without the high financial cost and latency associated with LLM services, making the system more sustainable for production use.
3Ease of manufacture
If few human-labeled content portions are used for training, then data collection cost is reduced, but model accuracy deteriorates
Solution Approach 1:
The patent enables the model to self-improve by using unsupervised learning techniques that allow it to automatically learn from unlabeled data. The model generates its own training data through embedding generation and clustering, eliminating the need for expensive human annotation while maintaining high accuracy through self-organized learning patterns.
Solution Approach 2:
The patent performs preliminary embedding generation and clustering on unlabeled data before final model training. This preliminary processing creates structured representations and temporary labels that guide subsequent supervised learning, allowing the model to achieve high accuracy even with minimal human-labeled examples by pre-organizing the data structure.
Data Source
AI summary
A method, computer program product, and computing system for transforming a plurality of content portions into a plurality of embeddings using a language model. A graph is generated with nodes representing respective embeddings and an edge between a pair of nodes representing a similarity distance between the respective embeddings that is less than or equal to a predefined threshold. A category prediction is generated for each content portion by processing the graph using a graph neural network. A loss function is determined using a plurality of predefined categories and the category predicted for each content portion. The language model and the graph neural network are finetuned for automatically tagging content portions with a category by maximizing the loss function.