Language Identification for CJK Ideographs Using Contextual Character Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Information processing devices face challenges in accurately determining the language of special ideographs like CJK unified ideographs, which are commonly used in multiple languages such as Chinese, Japanese, and Korean, leading to inaccuracies in document processing.
Innovation Solution
The device acquires a first character commonly used in multiple languages and infers its language based on a second character unique to one of those languages, by analyzing the presence of unique characters in the surrounding string of characters, thereby enhancing language identification accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If language identification is based solely on code point analysis, then the process is simple and fast, but accuracy deteriorates for special ideographs used in multiple languages
Solution Approach 1:
The patent transitions from one-dimensional code point analysis to two-dimensional context analysis by examining surrounding characters. The language determination unit analyzes not only the target character's code point but also the code points of surrounding characters within a specified range, adding a contextual dimension to the identification process. This resolves the contradiction by maintaining simplicity through automated context analysis while significantly improving accuracy for special ideographs.
Solution Approach 2:
The patent introduces surrounding characters as intermediary elements that mediate between the ambiguous target character and the language identification process. By analyzing the code points of surrounding characters that have language-specific characteristics, the system uses these intermediaries to resolve the ambiguity of special ideographs that are common across multiple languages, thereby improving accuracy without requiring complex linguistic analysis.
2Measurement precision
If surrounding character analysis is performed to improve accuracy, then language identification accuracy improves, but processing time increases
Solution Approach 1:
The patent applies partial action by analyzing only a limited number of surrounding characters within a specified range rather than performing exhaustive analysis of the entire text. The language determination unit focuses on extracting and analyzing code points of surrounding characters within a defined window, which provides sufficient contextual information for accurate language identification while keeping processing time acceptable through selective rather than complete analysis.
3Adaptability or versatility
If only common characters are used for display, then compatibility across languages is improved, but language-specific rendering accuracy deteriorates
Solution Approach 1:
The patent applies local quality by determining the language specific to each character's context rather than applying a uniform language setting across the entire text. The language determination unit analyzes the code points of surrounding characters to identify the specific language context for each target character, enabling appropriate font selection and rendering for each character based on its local linguistic environment. This resolves the contradiction by maintaining cross-language compatibility through code point-based processing while achieving language-specific rendering accuracy through contextual analysis.
Data Source
AI summary
An information processing device is configured to identify language of character strings in a text. The information processing device includes a processor and a memory storing a program. The processor is configured to or the program, when executed by the processor, causes the processor to: identify a first character, which is used commonly in a plurality of languages, in an input text; infer to which one of the plurality of languages the first character belongs based on a second character used only in one of the plurality of languages in the input text; and infer that the first character belongs to a language that is the one of the plurality of languages to which the second character belongs, based on whether a string of characters immediately preceding or following the first character in the input text contains the second character.