Out-of-Vocabulary Word Encoding via Sub-word Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text analysis methods face challenges when encountering 'out-of-vocabulary' words, as they lack representation in the vocabulary set, leading to loss of information and reduced accuracy in tasks like text classification and translation.
Innovation Solution
A method that segments 'out-of-vocabulary' words into sub-words present in the vocabulary set, using criteria such as the number and frequency of sub-words to determine the optimal segmentation for encoding, allowing the use of pre-trained embedding vectors for improved representation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If a limited vocabulary set is used to reduce computational resources, then computational efficiency is improved, but the ability to represent out-of-vocabulary words deteriorates
Solution Approach 1:
The patent segments out-of-vocabulary words into sub-word units (e.g., prefixes, suffixes, root words) that can be individually encoded using embedding vectors. This allows the system to represent words not in the vocabulary set by combining representations of their constituent parts, thereby maintaining information representation capability while using a limited vocabulary set.
2Reliability
If a larger vocabulary set is used to improve coverage of all possible words, then the ability to represent any word is improved, but computational resources required deteriorate
Solution Approach 1:
Instead of storing embedding vectors for every possible word in a large vocabulary, the patent segments rare and out-of-vocabulary words into common sub-word units that are already represented in the vocabulary set. This approach maintains comprehensive word representation coverage while keeping the vocabulary set size manageable and computational resources efficient.
3Productivity
If out-of-vocabulary words are excluded from text analysis, then processing speed is improved, but accuracy of text analysis deteriorates
Solution Approach 1:
The patent enables processing of out-of-vocabulary words by segmenting them into sub-word units that can be quickly encoded using pre-computed embedding vectors. This maintains text processing speed while improving accuracy by ensuring that no words are excluded from analysis, allowing the system to capture meaningful information from all text including rare and specialized terminology.
Data Source
AI summary
A computer-implemented method of encoding a word for use in a method of text analysis comprises receiving input text to be analysed, the input text comprising a first word which is not represented in a vocabulary set stored on a storage. The vocabulary set comprises a plurality of words and an associated word embedding vector for each word in the set. The method comprises identifying the first word as a word which is not represented in the vocabulary set and determining one or more sub-words within the first word with which to encode the first word. Each of the one or more sub-words corresponds with a word represented in the vocabulary set and having an embedding vector in the vocabulary set. The method comprises determining an encoding for the first word based on the one or more sub-words.