Document Similarity Transfer Learning for Low-Data Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face challenges due to insufficient and biased learning data, leading to incomplete models, particularly in document classification tasks, where computational costs are high and determining document similarity is difficult, especially with short sentences or non-separable document sets.
Innovation Solution
A transfer learning method involving pre-training a similarity model to output document similarities, followed by fine-tuning with a new output function, using semantic similarities calculated through nonlinear functions to enhance document classification, especially for malicious comment detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If language modeling methodology is used for document classification, then the model can process sequential data, but the computational cost significantly increases in proportion to the number of tokens
Solution Approach 1:
The patent applies preliminary action by pre-training a similarity model on large-scale document data before fine-tuning for specific classification tasks. The pre-trained model learns general document similarity representations that can be reused across different classification problems, avoiding the need to process all training data from scratch and reducing computational costs for downstream tasks
2Adaptability or versatility
If multitask learning methodology is used, then a single model learns multiple classification challenges simultaneously, but the system requires document sets labeled as many as the number of tasks
Solution Approach 1:
The patent extracts the core similarity learning capability from the complex multitask framework. By focusing on learning document similarity as a standalone pre-training objective, the method separates the general representation learning from task-specific fine-tuning, eliminating the need for multiple labeled datasets while maintaining multi-task adaptability through transfer learning
3Reliability
If determination of whether two documents are consecutive documents is used, then the model can capture temporal relationships, but it cannot be readily applied when documents include short sentences or are not easy for sentence separation
Solution Approach 1:
The patent changes the fundamental parameter from sentence-level temporal relationships to document-level similarity. By operating at the document level rather than requiring sentence segmentation, the method maintains reliability in capturing relationships between documents while becoming applicable to short sentences and documents that are difficult to segment into sentences
4Reliability
If transfer learning is used to reuse layers from models with abundant data, then the model quality improves in fields with insufficient data, but the model may inherit biases from the source domain
Solution Approach 1:
The patent introduces document similarity as an intermediary representation layer between the pre-trained model and task-specific fine-tuning. This intermediary similarity space acts as a domain-agnostic representation that preserves the benefits of transfer learning while reducing the direct transfer of domain-specific biases, as similarity relationships are more universal than task-specific patterns
Data Source
AI summary
Disclosed is a method and system for transfer learning of a deep learning model based on a document similarity learning. A transfer learning method may include pre-training, by the at least one processor, a similarity model to output a similarity between documents, generating, by the at least one processor, a fine tuning model by replacing a first output function of the pre-trained similarity model with a second output function, and training, by the at least one processor, the fine tuning model to output a score for a document input to the fine tuning model.