Parallel Document Identification via Sentence Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for statistical machine translation lack sufficient large parallel corpora due to the inability to effectively identify parallel documents and fragments in multilingual collections, often resulting in false positives when distinguishing between parallel and comparable documents, as they rely on unreliable structural properties and word-level computations.
Innovation Solution
A system and method that translates source documents, queries a target library, aligns sentences, and determines parallel fragments by using a combination of engines such as the word translator engine, query engine, document selector engine, parallel document engine, and parallel fragment engine to identify and classify parallel documents and fragments independently of structural properties, thereby distinguishing between parallel and comparable documents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If word-level computation methods are used to identify parallel documents, then the process is simple and fast, but the accuracy deteriorates resulting in many false positives where comparable documents are erroneously classified as parallel translations
Solution Approach 1:
The patent segments documents into sentences and sentences into words, creating a hierarchical structure for analysis. This allows the system to first identify potential parallel documents at the sentence level using structural features, then verify at the word level using translation models, thereby maintaining speed while improving accuracy by processing in manageable chunks rather than treating entire documents as atomic units
Solution Approach 2:
The patent transitions from word-level computation to sentence-level computation, adding a new dimension of analysis. By operating at the sentence level rather than just the word level, the system can capture contextual relationships and structural patterns that word-level methods miss, enabling more accurate distinction between parallel documents and comparable documents while maintaining computational efficiency
2Reliability
If structural properties such as HTML structures and URL connections are used to identify parallel documents, then the identification process is reliable, but the method fails when web sites in different languages are organized differently or when connections are not maintained
Solution Approach 1:
The patent extracts structural features from HTML documents and URL patterns, separating these features from the textual content. By taking out structural properties as independent features for analysis, the system can identify parallel documents based on both structure and content, making it adaptable to different website organizations while maintaining reliability through multi-factor verification
Solution Approach 2:
The patent creates a universal identification system that can handle multiple types of websites and structures through a single framework. The system universally applies sentence-level alignment and translation models across different language pairs and website configurations, making it adaptable to various organizational structures without requiring language-specific or structure-specific specialized algorithms
3Quantity of substance
If comparable segments are included in the training data, then the amount of training data increases, but the quality deteriorates due to noisy translations that do not represent true parallel translations
Solution Approach 1:
The patent applies local quality by analyzing individual sentences and words within documents rather than treating entire documents uniformly. This allows the system to identify and retain only those local segments (sentences) that exhibit true parallel translation characteristics, while filtering out noisy comparable segments, thereby improving the overall quality of the training data while maintaining adequate quantity through selective retention
Data Source
AI summary
Systems, computer programs, and methods for identifying parallel documents and/or fragments in a bilingual collection are provided. The method for identifying parallel sub-sentential fragments in a bilingual collection comprises translating a source document from a bilingual collection. The method further includes querying a target library associated with the bilingual collection using the translated source document, and identifying one or more target documents based on the query. Subsequently, a source sentence associated with the source document is aligned to one or more target sentences associated with the one or more target documents. Finally, the method includes determining whether a source fragment associated with the source sentence comprises a parallel translation of a target fragment associated with the one or more target sentences.