Morphology-Aware Tokenizer for Morpheme-Preserving BPE

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing tokenization methods, particularly Byte Pair Encoding (BPE), struggle to effectively handle non-concatenative morphological systems in languages like Arabic and Hebrew, and agglutinative languages such as Turkish and Korean, failing to preserve meaningful morpheme boundaries, leading to semantic ambiguity and inefficiency in Large Language Models (LLMs).

Innovation Solution

A morphologically aware tokenizer, MorphBPE, which integrates linguistic principles into the Byte Pair Encoding algorithm, ensuring that frequent symbol pair merges respect morpheme boundaries, and includes morphology-aware evaluation metrics like Morph.-Edit Distance Score and Morph.-Consistency F1-Score to assess tokenization quality.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If Byte Pair Encoding (BPE) is used for tokenization, then tokenization efficiency and vocabulary management are improved, but morpheme boundary preservation deteriorates in non-concatenative and agglutinative languages

Engineering Contradiction:

Improvetokenization efficiencyVSAvoidmorpheme boundary preservation

Core Design Contradiction:

ProductivityVSManufacturing precision

Solution Approach 1:

The patent applies segmentation by dividing words into morphemes based on linguistic knowledge before tokenization. Morpheme boundary information is used to guide the BPE merging process, ensuring that token boundaries align with meaningful linguistic units. This allows the system to maintain efficient tokenization while preserving morphological structure in non-concatenative and agglutinative languages.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent changes the parameters of the BPE algorithm by incorporating morpheme boundary constraints into the merging process. Instead of purely greedy merging based on frequency, the algorithm modifies its behavior to respect morphological boundaries, adjusting the merging strategy to maintain both efficiency and linguistic accuracy.

Inventive Principle:

Principle #35Parameter changes

2Manufacturing precision

If morphology-based segmentation is used, then morpheme boundary preservation is improved, but alignment with corpus-based learning patterns deteriorates

Engineering Contradiction:

Improvemorpheme boundary preservationVSAvoidalignment with corpus-based learning

Core Design Contradiction:

Manufacturing precisionVSAdaptability or versatility

Solution Approach 1:

The patent merges morphology-based segmentation with corpus-based statistical methods. By combining linguistic knowledge of morpheme boundaries with frequency-based merging from BPE, the system achieves both morphological accuracy and adaptability to corpus-based learning patterns. The merged approach leverages the strengths of both methods for improved tokenization quality.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The patent incorporates feedback mechanisms where morpheme boundary information from linguistic analysis feeds into the tokenization process, and tokenization results are evaluated against corpus data. This feedback loop allows the system to maintain morphological integrity while adapting to actual usage patterns found in corpora.

Inventive Principle:

Principle #23Feedback

Data Source

PatentEP4700644A1Morphologically aware tokenizer

Publication Date: 2026.02.25 HAMAD BIN KHALIFA UNIVERSITY

Figure 1
Figure 2
Figure 3(a)~3(b)

AI summary

The present disclosure provides for a morphologically aware tokenizer. According to one aspect of the present disclosure a morphologically aware tokenizer. According to a second aspect of the present disclosure a method of using a f morphologically aware tokenizer.