Online Word Vector Dictionary Extension via Co-occurrence Projection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques for adding new words to word vector embeddings are computationally expensive and inefficient, requiring retraining of the entire embedding, which can alter existing word vectors and disrupt systems like neural networks in natural language processing.
Innovation Solution
The system employs an online dictionary extension method that estimates co-occurrence information for new words without altering the existing vector space, allowing new words to be incorporated into the word vector dictionary without retraining, using sparse co-occurrence information and projecting estimated information onto the existing dictionary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional retraining techniques are used to add new words to word vector embedding, then new words can be incorporated into the dictionary, but the entire embedding must be retrained which is computationally expensive and time-consuming
Solution Approach 1:
The patent segments the word vector embedding into two independent parts: the existing trained embedding and the new word vectors. Instead of retraining the entire embedding, the system only processes and trains the new words separately, then combines them with the original embedding. This segmentation allows incremental updates without full retraining, resolving the contradiction between adaptability and computational efficiency.
Solution Approach 2:
The patent performs preliminary actions by pre-computing and storing the original word vectors before adding new words. The system prepares the new word vectors independently using the same initialization process, then integrates them with the pre-existing embedding. This preliminary preparation enables efficient combination without requiring retraining of the entire system.
2Adaptability or versatility
If conventional retraining techniques are used to add new words, then the word vector dictionary can be updated, but previously computed word vectors are altered and systems utilizing them break
Solution Approach 1:
The patent segments the word vector space into original vectors and new vectors that remain independent. By training only the new words separately and combining their vectors with the original embedding, the system ensures that original word vectors remain unchanged. This segmentation preserves system reliability while enabling dictionary updates.
Solution Approach 2:
The patent creates new word vectors as copies following the same initialization process as the original embedding, rather than modifying existing vectors. The new vectors are generated independently using the same mathematical transformations and are then integrated with the original embedding, ensuring original vectors remain intact and systems continue to function reliably.
3Adaptability or versatility
If brute force retraining is used to incorporate novel words, then complete vocabulary update is achieved, but computational difficulty increases significantly
Solution Approach 1:
The patent divides the computational task into two separate segments: training the original embedding (already completed) and training only the new words. This segmentation reduces the computational complexity from O(N*M) where N is total words and M is training iterations, to approximately O(n*M) where n is only the number of new words, significantly reducing device complexity while maintaining full vocabulary coverage.
4Adaptability or versatility
If entire embedding is retrained to add new words, then new vocabulary is incorporated, but training time increases significantly
Solution Approach 1:
The patent performs preliminary actions by pre-training and freezing the original embedding before adding new words. The new words are trained independently using the same initialization process, then integrated with the pre-existing embedding. This preliminary preparation eliminates the need for time-consuming full retraining, reducing training time from hours to minutes while maintaining vocabulary flexibility.
Data Source
AI summary
Online dictionary extension of word vectors techniques and systems are described that are configured to provide online extension of existing word vector dictionaries and thus overcome the failures of conventional techniques. In one example, a dictionary extension system is employed by a computing system to extend a word vector dictionary to incorporate a new word in an online manner Co-occurrence information is estimated for the new word with respect to the words in the existing dictionary. This is done by estimating co-occurrence information with respect to a large word set based on the existing dictionary and sparse co-occurrence information for the new word. The estimated co-occurrence information is utilized to estimate a new word vector associated with the new word by projecting the estimated co-occurrence information onto the existing word vector dictionary. An extended dictionary is created incorporating the original dictionary and the estimated new word vector.