Alignment Generation Device Using Encoded Sentence Codes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In statistical and neural machine translation, the generation of translation models from large bilingual corpora is inefficient due to the high computational cost and storage requirements of morphological analysis, leading to prolonged processing times and reduced learning efficiency, as well as impractical compression algorithms that do not align with term-based translation.
Innovation Solution
A method that encodes sentences in prescribed units, generating alignment information between codes in two languages, allowing for efficient alignment generation and translation without decompression or preprocessing, using normalized strings and codes to facilitate faster processing and improved translation accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If morphological analysis is performed on large bilingual corpora to generate translation models, then translation accuracy is improved, but computational cost and processing time increase significantly
Solution Approach 1:
The patent segments the bilingual corpus into smaller units (sentences, phrases, or terms) and processes them in parallel using multiple processors. This segmentation allows the system to maintain translation accuracy through thorough morphological analysis while reducing overall processing time by distributing the computational load across multiple processing units that work simultaneously on different segments of the corpus.
2Measurement precision
If morphological analysis is performed on large bilingual corpora to generate translation models, then translation accuracy is improved, but computational resources and storage requirements increase
Solution Approach 1:
The patent divides the large bilingual corpus into smaller segments that can be processed independently and in parallel. This segmentation reduces the memory footprint and computational resources required for each individual processing unit while maintaining the ability to perform comprehensive morphological analysis across the entire corpus through aggregated results from multiple processors.
Solution Approach 2:
The patent performs preliminary processing steps such as tokenization, lowercasing, and basic filtering on the corpus before the main morphological analysis. This preliminary action reduces the size and complexity of the data that requires intensive computational processing, thereby reducing overall computational cost and energy consumption while preserving the information needed for accurate translation model generation.
3Quantity of substance
If traditional compression algorithms are used on bilingual corpora, then storage space is reduced, but alignment with term-based translation is lost
Solution Approach 1:
The patent creates a compressed representation of the bilingual corpus that maintains the structural relationships between source and target terms. Instead of using traditional compression that loses semantic information, the system uses a format that preserves alignment data structures, allowing term-based translation to function correctly on the compressed data without requiring full decompression.
Solution Approach 2:
The patent changes the representation parameters of the bilingual corpus from a detailed text format to a compressed format that retains essential alignment information. This parameter change involves encoding the corpus in a way that reduces storage requirements while maintaining the critical term-to-term correspondence data needed for statistical and neural machine translation.
Data Source
AI summary
A computer generates a plurality of encoded sentences in the first language by encoding a plurality of sentences in the first language in prescribed units. Next, the computer generates a plurality of encoded sentences in the second language by encoding a plurality of sentences in the second language, each of which is associated with each of the plurality of sentences in the first language, in the prescribed units. The computer generates alignment information based on a code included in each of the plurality of encoded sentences in the first language and a code included in an encoded sentence in the second language, which is associated with each of the plurality of encoded sentences in the first language. The alignment information indicates an alignment between a plurality of codes in the first language and a plurality of codes in the second language.