Parallel Corpus Augmentation via Named Entity Replacement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The creation and maintenance of large, high-quality parallel corpora for natural language processing tasks are hindered by the need for manual human effort, particularly for low-resource languages and specialized domains, leading to insufficient data for effective machine translation and other processing tasks.
Innovation Solution
An information processing apparatus that automatically generates a new parallel corpus by replacing named entities in an existing corpus with similar entities from dictionary data, using similarity thresholds and multilingual semantic models to enhance the corpus for machine translation and other tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual human effort is used to create and maintain large parallel corpora, then the quality and accuracy of the corpus can be maintained, but the productivity and time required for corpus creation are significantly reduced
Solution Approach 1:
The system creates augmented parallel corpora by copying and transforming existing high-quality parallel corpus data. It replaces named entities in source sentences with similar entities from dictionaries while maintaining the target sentence translations, thereby generating new training data without manual creation efforts.
Solution Approach 2:
The system performs self-service by automatically extracting named entities, finding similar entities using dictionaries and similarity calculations, and generating augmented corpus data without requiring continuous manual intervention. The process is automated through computer-executable instructions.
2Reliability
If manual human effort is used to create parallel corpora for low-resource languages and specialized domains, then the representation quality can be improved, but the time and resources required increase significantly
Solution Approach 1:
The system performs preliminary actions by pre-processing existing parallel corpora to identify and extract named entities, and pre-building dictionaries of similar entities. This preparation enables automatic augmentation of low-resource language corpora without time-consuming manual efforts during the actual corpus creation phase.
Solution Approach 2:
The system copies and adapts existing parallel corpus structures for low-resource languages by replacing named entities with similar ones from specialized dictionaries, thereby improving language representation without requiring manual creation of new corpus data from scratch.
3Quantity of substance
If the parallel corpus is augmented by replacing named entities with similar entities, then the quantity and diversity of training data are improved, but the risk of creating non-existent or inaccurate entities increases
Solution Approach 1:
The system uses dictionaries of named entities as intermediaries to bridge the gap between source sentences and similar entities. These dictionaries serve as reference sources that verify the existence and accuracy of candidate entities, reducing the risk of creating non-existent entities while enabling corpus augmentation.
Solution Approach 2:
The system replaces manual verification of entity accuracy with automated similarity calculations and dictionary lookups. By using computational methods to calculate similarity scores and verify entity existence through dictionaries, the system maintains entity accuracy while enabling large-scale corpus augmentation.
Data Source
AI summary
An information processing apparatus acquires a first parallel corpus in which a first sentence, which includes a first named entity in a first language, and a second sentence, which includes a second named entity in a second language corresponding to the first named entity, are associated, extracts a third named entity whose degree of similarity with the first named entity exceeds a threshold from first dictionary data including a plurality of named entities in the first language, specifies a fourth named entity corresponding to the third named entity using second dictionary data indicating correspondence between named entities in the first language and named entities in the second language, and generates a second parallel corpus by replacing the first named entity included in the first sentence with the third named entity and replacing the second named entity included in the second sentence with the fourth named entity.