Language Recognition Using Vocabulary List Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Natural language processing technologies face challenges in accurately recognizing languages that are similar, such as Cantonese and Mandarin, due to shared vocabularies, leading to errors in identifying the language used in information content.
Innovation Solution
A method that integrates vocabulary lists for similar languages into a comprehensive list, analyzing expressions to identify unique language-specific terms and determining language composition based on frequency of occurrence criteria, thereby distinguishing between languages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional language recognition methods are used that rely on shared vocabularies between similar languages, then the recognition process is simple, but the accuracy of language identification deteriorates due to high similarity and shared expressions between languages like Cantonese and Mandarin
Solution Approach 1:
The patent segments the vocabulary recognition process by creating separate vocabulary lists for each language (first language vocabulary list and second language vocabulary list) and processing them independently through different modules. This segmentation allows the system to handle similar languages by tracking their unique expressions separately, thereby improving language recognition accuracy without being confused by shared vocabularies
Solution Approach 2:
The patent introduces an intermediary expression tracking module that maintains separate vocabulary lists as intermediate representations for each language. These intermediary structures serve as mediators between the input information content and the final language recognition result, enabling accurate differentiation between similar languages by comparing against language-specific vocabulary lists
2Measurement precision
If separate vocabulary lists are created for each language to improve recognition accuracy, then language identification accuracy improves, but the complexity of managing and processing multiple vocabulary lists increases
Solution Approach 1:
The patent creates vocabulary lists that serve multiple functions: they act as reference databases for language identification, as tracking structures for expression frequency analysis, and as comparison bases for determining language similarity. This multi-functionality reduces the need for separate processing structures, thereby managing complexity while maintaining high recognition accuracy
Data Source
AI summary
A method is implemented at a computer to determine that certain information content is composed or compiled in a specific language selected among two or more similar languages. The computer integrates a first vocabulary list of a first language and a second vocabulary list of a second language into a comprehensive vocabulary list. The integrating includes analyzing the first vocabulary list in view of the second vocabulary list to identify a first vocabulary sub-list that is used in the first language, but not in the second language. The computer then identifies, in the information content, a plurality of expressions that are included in the comprehensive vocabulary list, and a subset of expressions that are included in the first vocabulary sub-list. Upon a determination that a total frequency of occurrence of the subset of expressions meets predetermined occurrence criteria, the computer determines that the information content is composed in the first language.