Classification Model Training Method for Dictionary Extension
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for constructing or extending dictionaries in natural language processing require manual labor and resources, leading to high costs, incompleteness, and variability, especially with the emergence of new words and the need for frequent updates.
Innovation Solution
A classification model training method that optimizes the training set by automatically generating and iteratively refining a negative training set without manual labeling, improving the accuracy of the classification model and dictionary extension.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual labeling is used to construct training sets for dictionary construction, then the classification model can be trained, but the cost and time consumption increase significantly
Solution Approach 1:
The system uses the classification model to automatically identify and label potential negative samples from the corpus, which then serve as training data for improving the model itself. This self-service mechanism eliminates the need for external manual labeling while continuously improving classification accuracy through iterative refinement of the training set.
Solution Approach 2:
The method performs preliminary automatic identification of negative samples using initial classification rules or models before formal training begins. This preliminary action creates a foundation of labeled data that can be used to train the classification model without requiring manual intervention at the start of the process.
2Reliability
If manual labeling is performed to ensure training data quality, then classification accuracy improves, but the complexity and resource requirements increase
Solution Approach 1:
The classification model serves itself by automatically generating high-quality training data through its own predictions. The system identifies confident negative samples, uses them to retrain the model, and iteratively improves reliability without external human intervention or complex manual curation processes.
Solution Approach 2:
The system implements a feedback loop where the classification model's predictions are evaluated, confident negative samples are extracted, used to retrain the model, and the process repeats. This automated feedback mechanism ensures continuous improvement of dictionary reliability while maintaining simple system architecture.
3Adaptability or versatility
If a large corpus is processed to update the dictionary with new words, then the dictionary becomes more complete, but the processing time and computational resources increase
Solution Approach 1:
The method extracts only the confident negative samples from the large corpus based on the classification model's predictions, rather than processing the entire corpus. This extraction of essential training data maintains dictionary completeness while dramatically reducing processing time and computational resources required for updates.
Solution Approach 2:
The system performs partial processing by focusing only on the subset of corpus samples that are most valuable for training (the confident negative samples identified by the classifier). This partial action approach achieves the necessary dictionary updates without the excessive computational burden of processing the entire corpus.
Data Source
AI summary
A classification model training method includes obtaining a positive training set and a first negative training set, where the positive training set includes samples of a positive sample set in a corpus, where the first negative training set includes samples of an unlabeled sample set in the corpus, training, using the positive training set and the first negative training set, to obtain a first classification model, determining, using the first classification model, a pseudo negative sample in the first negative training set, removing the pseudo negative sample from the first negative training set, updating the first negative training set to a second negative training set, and training, using the positive training set and the second negative training set, to obtain a target classification model.