Language-Agnostic Word Embeddings Using Hashed Character N-Grams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional tokenizers require significant computational resources, are optimized for specific training sets, and poorly utilize vocabulary, leading to poor performance with different languages and increased memory and computational demands.
Innovation Solution
A method for deriving language agnostic representations through text splitting, tokenization, hashing, and aggregation, using n-grams and hashing algorithms to reduce vocabulary size and improve computational efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional tokenizers are trained on large training sets to improve representation quality, then the quality of text representation improves, but computational resources and training time increase significantly
Solution Approach 1:
The patent segments text into character n-grams (subword units) rather than treating entire words as tokens. This segmentation allows the model to build vocabulary from smaller, reusable character sequences, reducing the need for extensive training data while maintaining representation quality. The character-level segmentation enables efficient vocabulary construction without requiring large-scale training computations.
Solution Approach 2:
The patent performs preliminary action by pre-defining a fixed vocabulary of character n-grams before training the language model. This preliminary vocabulary construction eliminates the need for expensive tokenizer training on large datasets, as the vocabulary is created deterministically from character combinations. The embedding layer is then trained on this pre-defined vocabulary, significantly reducing computational resources while maintaining representation quality.
2Reliability
If tokenizers are optimized for specific training sets to improve performance on those languages, then performance on the training language improves, but performance on other languages deteriorates
Solution Approach 1:
The patent implements universality by using a character n-gram based vocabulary that is language-agnostic. Character sequences form the basis of tokens across all languages, allowing the same vocabulary and embedding system to handle multiple languages effectively. This universal approach enables the model to adapt to different languages without requiring language-specific tokenizer optimization, thereby improving both reliability on training languages and adaptability to new languages.
3Measurement precision
If vocabulary size is increased to capture more linguistic variations, then representation accuracy improves, but memory and computational resources increase
Solution Approach 1:
The patent applies local quality by using character n-grams of varying lengths (different local contexts) to represent tokens. Instead of using a single fixed token size, the model uses character sequences of different lengths (e.g., unigrams, bigrams, trigrams) to capture local linguistic patterns efficiently. This approach achieves high representation accuracy with a more compact vocabulary compared to whole-word tokenization, as common character sequences are reused across multiple words.
Data Source
AI summary
A computer-implemented method for deriving a language agnostic representation for each word of a text, the method comprising: splitting the text into a plurality of words; tokenizing the plurality of words to obtain a plurality of tokens; calculating a token identification number for each token; hashing each token identification number to obtain a plurality of embedded tokens identification numbers; aggregating one or more embedded token identification numbers to obtain the language agnostic representation for each word. The invention also relates to a computer-implemented method for training a machine learning model, a computer-implemented method for generating text, a corresponding device or system and a corresponding computer program.