Multilingual Text Classification via Selective Translation Pipeline
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer systems face inefficiencies when analyzing multilingual text streams due to the need for language-specific machine learning models and high-resource translation operations, especially when dealing with large volumes of text where only a small proportion is relevant.
Innovation Solution
Implementing a multi-stage classification technique that detects the language of text units, determines relevance scores using language-specific keyword sets, and translates only relevant text units to a base language for further analysis, thereby reducing computational resources and improving efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If translation operations are performed on all text units to enable base language classification, then classification accuracy is improved, but computational resource consumption increases significantly
Solution Approach 1:
The patent segments the text processing pipeline into multiple stages: language detection stage, keyword matching stage, and translation stage. Only text units that pass through all previous stages are translated, dividing the translation task into a selective subset rather than applying it universally to all text units.
Solution Approach 2:
The patent performs preliminary actions (language detection, keyword matching, relevance scoring) before translation to identify and filter text units that warrant translation. This preliminary filtering ensures that translation resources are allocated only to text units with potential relevance, avoiding unnecessary translations of irrelevant content.
2Measurement precision
If language-specific machine learning models are used for each language, then classification accuracy for that language is improved, but system complexity increases
Solution Approach 1:
The patent creates a universal base language classification model that can handle multiple languages through translation. Instead of maintaining separate models for each language, a single base language model performs classification for all languages after text units are translated to the base language, reducing model management complexity.
Solution Approach 2:
The patent introduces translation as an intermediary step between multilingual text units and the base language classification model. This intermediary allows the base language model to indirectly process and classify text in multiple languages without requiring separate models for each language.
3Reliability
If all text units are processed through translation and classification, then completeness of analysis is improved, but processing speed decreases
Solution Approach 1:
The patent extracts and removes text units that are determined to be irrelevant through language detection and keyword matching before they undergo translation and full classification processing. This extraction of irrelevant text units from the processing pipeline maintains completeness for relevant text while improving overall processing speed by eliminating unnecessary operations on irrelevant content.
4Loss of information
If translation is performed on every text unit, then no relevant information is lost, but computational overhead increases
Solution Approach 1:
The patent applies partial translation action by translating only the subset of text units that are deemed relevant based on language detection and keyword matching results. This partial action approach ensures that translation is applied sufficiently to capture all relevant information while avoiding the excessive translation of irrelevant text units, thereby reducing computational overhead.
Data Source
AI summary
Various examples are directed to systems and methods for identifying textual information regarding a first topic. A computer system may access a plurality of text units and detect that a first text unit of the plurality of text units is in a first language. The computer system may access a first language keyword set for the first topic, where the first language keyword set comprises a first plurality of keywords associated with the first language. The computer system may determine a first relevance score for the first text unit based at least in part on the first language keyword set. If the first relevance score is greater than a relevance score threshold, the computer system translates the first text unit to a base language and determines a text unit classification for the first text unit using a classification model trained with training data in the base language.