Language Processing Apparatus for Accurate Cross-Lingual Text Association
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for generating parallel corpora from comparable corpora face challenges in accurately associating texts across languages due to differences in semantic content, even when sentences have high semantic similarity.
Innovation Solution
The proposed solution involves a language processing program that extracts named entities from texts in different languages, associates texts based on the similarity between these named entities and their alignment probabilities, and outputs association information to improve the accuracy of text alignment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If texts are associated based on semantic similarity between sentences, then the quantity of associated text pairs increases, but the accuracy of text alignment deteriorates due to differences in semantic content
Solution Approach 1:
The patent segments the text association process into two levels: sentence-level semantic similarity matching (for quantity) and named entity-level alignment probability matching (for accuracy). This segmentation allows the system to first identify potential text pairs efficiently, then filter and refine them using named entity correspondence, thereby resolving the contradiction between producing many associations and ensuring their accuracy.
Solution Approach 2:
Named entities serve as an intermediary mechanism between sentence-level semantic similarity and text-level alignment. The system uses named entity recognition and alignment probability as a mediating layer that bridges the gap between coarse-grained semantic similarity and fine-grained text equivalence, enabling both high productivity and high accuracy in text association.
2Manufacturing precision
If named entity extraction and alignment is performed to improve text association accuracy, then the quality of parallel corpus increases, but the processing complexity increases
Solution Approach 1:
The system performs preliminary named entity recognition and extraction from source texts before the association process. By pre-identifying and structuring named entities in advance, the system reduces the complexity of the subsequent alignment process, as entities are already segmented and ready for matching, thereby achieving high corpus quality without excessive processing complexity.
Solution Approach 2:
The named entity alignment process is designed to be self-service in nature, where the system automatically recognizes, extracts, and aligns named entities without requiring manual intervention. This automation maintains high processing quality while keeping the operational complexity manageable through algorithmic self-processing.
3Reliability
If alignment probability between named entities is calculated to improve association accuracy, then the reliability of text pairing increases, but the computational cost increases
Solution Approach 1:
The system applies alignment probability calculation selectively rather than uniformly across all text pairs. By focusing computational resources on local regions where named entities are present and potentially alignable, the system achieves high reliability for text pairing involving named entities while reducing overall computational cost by avoiding unnecessary calculations in regions without named entities.
Data Source
AI summary
A non-transitory computer-readable recording medium stores a language processing program for causing a computer to execute a process including: extracting, from a second text written in a second language, a second named entity corresponding to a first named entity contained in a first text written in a first language; associating the first text with the second text based on a similarity between the first named entity and the second named entity and an alignment probability between the first named entity and the second named entity; and outputting association information indicating a result of associating the first text with the second text.