Language-Agnostic Word Representation with Hashed N-Gram Tokenization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional tokenizers for large language models require significant computational resources, are optimized for specific training sets, and poorly utilize vocabulary, leading to poor performance with different languages and increased memory and computational demands.
Innovation Solution
A method for deriving language agnostic representations through text splitting, tokenization, hashing, and aggregation, using n-grams and hashing algorithms to create static, efficient embeddings that capture contextual meaning and reduce redundancy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional tokenizers are trained on large datasets to improve representation quality, then the quality of text embeddings is improved, but computational resources and training time are significantly increased
Solution Approach 1:
The patent segments the text processing task into two independent stages: (1) a language-agnostic tokenization stage that creates character n-gram tokens without language-specific training, and (2) a language-specific embedding stage that processes only the tokenized sequences. This segmentation eliminates the need for computationally expensive end-to-end training while preserving embedding quality.
Solution Approach 2:
The patent performs preliminary tokenization using language-agnostic character n-grams before the main embedding process. By pre-processing text into standardized token sequences that capture subword patterns, the system prepares data in a form that requires minimal subsequent training, significantly reducing computational resources while maintaining representation quality.
2Measurement precision
If tokenizers are optimized for specific training sets to improve performance on those languages, then accuracy on training languages is improved, but performance on different languages significantly drops
Solution Approach 1:
The patent creates a universal, language-agnostic tokenization system based on character n-grams that can process any language without retraining. The same tokenizer and embedding model can handle multiple languages by simply changing the input text, achieving both high accuracy on training languages and excellent cross-language performance through its language-independent design.
Solution Approach 2:
Instead of training the tokenizer on language-specific data to adapt to each language, the patent inverts the approach by using language-agnostic character-level processing that naturally adapts to any language. This inversion eliminates the need for language-specific optimization while maintaining or improving cross-language performance.
3Adaptability or versatility
If conventional tokenizers create large vocabularies to capture diverse tokens, then tokenization coverage is improved, but memory and computational requirements increase significantly
Solution Approach 1:
The patent transitions from the conventional dimension of whole-word tokens to a different dimension by using character n-grams as basic units. This dimensional change allows the system to represent diverse vocabulary with a compact set of character-level tokens, dramatically reducing vocabulary size while maintaining or improving tokenization coverage through combinatorial n-gram formation.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer-implemented method for deriving a language agnostic representation for each word of a text, the method comprising: splitting the text into a plurality of words; tokenizing the plurality of words to obtain a plurality of tokens; calculating a token identification number for each token; hashing each token identification number to obtain a plurality of embedded tokens identification numbers; aggregating one or more embedded token identification numbers to obtain the language agnostic representation for each word. The invention also relates to a computer-implemented method for training a machine learning model, a computer-implemented method for generating text, a corresponding device or system and a corresponding computer program.