Text Classification Model Training via Cross-Language Translation Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Text classification models face challenges in performing effectively across languages due to a lack of labeled samples, particularly in the early stages of expanding a product's business into new markets, where manual labeling is time-consuming and labor-intensive.
Innovation Solution
A method that involves machine translation to acquire cross-language text samples, confidence-based filtering, and training a second text classification model with greater network depth to improve accuracy, using filtered samples from a first text classification model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual labeling is used to obtain labeled text samples for text classification in new languages, then the accuracy of text classification can be improved, but the time consumption and labor intensity increase significantly
Solution Approach 1:
The patent uses machine translation to copy text samples from a source language to a target language, creating synthetic labeled samples without manual intervention. The translation process automatically generates corresponding labeled data in the new language, replicating the labeling process through automated translation rather than human annotators.
Solution Approach 2:
The system performs self-service by using the trained text classification model from the source language to automatically filter and select high-quality translated samples. The model evaluates its own predictions on translated samples and retains only those with high confidence scores, enabling automatic quality control without external human input.
2Productivity
If machine translation is used to acquire cross-language text samples, then the productivity of text sample acquisition is improved, but the quality and reliability of samples may deteriorate due to translation errors
Solution Approach 1:
The patent applies partial action by selectively retaining only the high-confidence translated samples for training. Instead of using all translated samples regardless of quality, the system filters out low-confidence translations, using only the subset of samples that meet a predetermined confidence threshold to ensure adequate quality.
Solution Approach 2:
The system implements feedback by using the text classification model to evaluate the quality of translated samples. The model generates confidence scores for each translated sample, and this feedback information is used to filter and select high-quality samples, creating a closed-loop quality control mechanism.
3Speed
If a first text classification model with smaller network depth is used for filtering, then the training speed is improved, but the filtering accuracy may worsen
Solution Approach 1:
The patent applies preliminary action by using a smaller, faster model to perform the initial filtering of translated samples. This first model quickly identifies and retains high-confidence samples, preparing a filtered dataset that can then be used to train a more sophisticated second model without requiring the first model to achieve perfect filtering accuracy.
Solution Approach 2:
The patent segments the text classification task into two distinct stages: a first filtering stage using a smaller model for rapid sample selection, and a second training stage using a larger model for final classification. This segmentation allows each model to be optimized for its specific purpose, with the first model focusing on speed and the second on accuracy.
Data Source
AI summary
The disclosure provides a text classification model training method, a text classification method, an apparatus, an electronic device, and a computer-readable storage medium, and relates to artificial intelligence technology. The text classification model training method includes: performing machine translation on a plurality of first text samples in a first language to obtain a plurality of second text samples in a second language different from the first language; training a first text classification model for the second language based on a plurality of third text samples in the second language and corresponding class labels; performing confidence-based filtering on the plurality of second text samples by the trained first text classification model; and training a second text classification model for the second language based on the filtered second text samples.