Multilingual Text Tokenization via Character Attribute Intersection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current tokenization methods are inefficient and complex, especially when dealing with multilingual texts, as they rely on delimiters which require parsing for each character and are not adapted for intermixed languages, leading to increased storage and processing time.
Innovation Solution
A method that determines the attributes of characters and uses Boolean operations to group characters into tokens based on intersecting attributes, allowing for efficient tokenization without the need for delimiters and supporting multilingual texts by using a primitive data type and Boolean operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If delimiter-based tokenization is used, then basic language words can be identified, but storage and processing time increase due to parsing requirements for each delimiter
Solution Approach 1:
The patent changes the parameter representation from character codes to attribute values. Each character is represented by an attribute value indicating its class membership, allowing direct comparison without delimiter parsing. This parameter transformation enables O(1) token boundary detection compared to the O(n) delimiter scanning in traditional approaches.
Solution Approach 2:
The patent replaces the mechanical delimiter-matching process with a mathematical attribute comparison system. Instead of sequentially scanning for delimiter characters, the system uses attribute value intersection logic to determine token boundaries, substituting computational mechanics with mathematical operations.
2Measurement precision
If delimiter-based tokenization is used, then basic language words can be identified, but storage requirements increase due to delimiter parsing overhead
Solution Approach 1:
The patent transforms the storage representation from storing actual delimiter characters and their positions to storing compact attribute values that encode class membership information. This parameter change reduces the storage footprint while maintaining tokenization accuracy.
3Reliability
If traditional tokenization methods are used, then single-language text can be processed, but the system is not adapted for intermixed input text written in two or more languages
Solution Approach 1:
The patent creates a universal tokenization framework where characters from any language are represented by a common attribute system. The attribute values indicate class membership that works across multiple languages, allowing the same tokenization logic to handle both single-language and multilingual text effectively.
Solution Approach 2:
The patent changes the representation parameter from language-specific character properties to language-agnostic attribute values. This parameter transformation enables the tokenization system to handle intermixed languages by comparing attribute intersections rather than language-specific delimiters.
4Productivity
If delimiter-based tokenization is used, then text can be split into tokens, but the process becomes overly complicated with multiple parsing steps
Solution Approach 1:
The patent extracts the essential token boundary information into attribute values, separating the tokenization logic from the actual text content. This extraction simplifies the process by removing the need for complex delimiter parsing while retaining the core functionality of identifying token boundaries.
Solution Approach 2:
The patent simplifies the tokenization process by changing from multi-step delimiter parsing to a single attribute comparison step. The parameter transformation enables direct determination of token boundaries through attribute intersection logic, reducing process complexity.
Data Source
AI summary
Methods and systems for tokenizing multilingual textual documents are provided. A method implemented in a computer infrastructure, includes determining an attribute of a current character in input text, the attribute of the current character indicating one or more classes of characters the current character is assigned thereto. The method further includes determining one or more attributes of one or more next characters in the input text, the one or more attributes of the one or more next characters indicating the one or more classes the one or more next characters are assigned thereto. The method further includes constructing a token of the input text that includes the current character and the one or more next characters, the attribute of the current character and the one or more attributes of the one or more next characters intersecting with each other.