Synthetic Multilingual NER Data Generation via Rule-Based Translation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for creating training and test data for named entity recognition in multilingual AI products are costly and inefficient, particularly for under-resourced languages, as they often rely on third-party vendors or generic public datasets that lack diversity and specificity.
Innovation Solution
A method and system for extracting and labeling Named-Entity Recognition (NER) data in a target language using a hybrid approach that combines stochastic machine learning with rule-based classification, leveraging open-source translation and distributional embeddings to synthetically curate datasets, allowing for cost-effective and diverse entity recognition across languages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If third-party vendors are used to prepare labelled data, then data quality and diversity are improved, but cost and time consumption increase significantly
Solution Approach 1:
The patent creates synthetic training data by translating and transforming existing high-quality English training data into target languages using machine translation models. This copying approach generates diverse multilingual data without requiring manual annotation by third-party vendors, significantly reducing time and cost while maintaining data quality through automated translation pipelines
Solution Approach 2:
The system enables self-service data generation by automatically translating English training data into multiple target languages using machine translation models. This eliminates the need for manual intervention from third-party vendors for data preparation, allowing the system to serve itself in generating diverse, high-quality multilingual training datasets
2Reliability
If third-party vendors are used to prepare labelled data, then data quality is improved, but cost increases significantly
Solution Approach 1:
The patent creates synthetic training data by translating and transforming existing high-quality English training data into target languages using machine translation models. This copying approach generates diverse multilingual data without requiring manual annotation by third-party vendors, significantly reducing time and cost while maintaining data quality through automated translation pipelines
Solution Approach 2:
The system changes the language parameter of training data by automatically translating from English to multiple target languages. This parameter transformation enables the generation of diverse multilingual datasets at minimal cost, as the same English training data can be reused and transformed into multiple language versions through automated translation
3Quantity of substance
If publicly available data is used, then cost is reduced, but data diversity and specificity for under-resourced languages deteriorate
Solution Approach 1:
The patent creates synthetic training data by translating and transforming existing high-quality English training data into target languages using machine translation models. This copying approach generates diverse multilingual data without requiring manual annotation by third-party vendors, significantly reducing time and cost while maintaining data quality through automated translation pipelines
Solution Approach 2:
The system dynamically generates training data for under-resourced languages by translating and adapting English training data into target languages using machine translation models. This dynamic approach enables the system to adapt to different language requirements and generate diverse, domain-specific training data on-demand, improving data diversity for under-resourced languages while maintaining cost-effectiveness
4Quantity of substance
If generic public datasets are used, then cost is reduced, but specificity for domain-specific uses deteriorates
Solution Approach 1:
The patent creates domain-specific training data by translating and adapting English training data into target languages using machine translation models. This approach generates diverse multilingual data without requiring manual annotation by third-party vendors, significantly reducing time and cost while maintaining data quality through automated translation pipelines
Solution Approach 2:
The system dynamically generates training data for under-resourced languages by translating and adapting English training data into target languages using machine translation models. This dynamic approach enables the system to adapt to different language requirements and generate diverse, domain-specific training data on-demand, improving data diversity for under-resourced languages while maintaining cost-effectiveness
Data Source
AI summary
A method and system for extracting and labeling Named-Entity Recognition (NER) data in a target language for use in a multi-lingual software module has been developed. First, a textual sentence is translated to the target language using a translation module. A named entity is identified and extracted within the translated sentence. The named entity is identified by either: exact mapping; a semantically similar translated named entity that meets a predetermined minimum threshold of similarity; or utilizing a rule-based library for the target language. Once identified, the named entity is labeled with a pre-determined category and stored in a retrievable electronic database.