Dynamic Word Embedding Model Update via Hidden Layer Preservation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional word embedding models require significant processing resources and time to generate and update, making it inefficient to incorporate new data, especially in real-time or streaming data applications, as they need to be completely retrained with the entire database, leading to delays and outdated models.
Innovation Solution
The method involves dynamically updating a word embedding model by using a stored representation of a neural network's hidden layer, allowing incremental updates based on new data, reducing the need for full retraining and enabling near-real-time updates using unsupervised machine learning techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the word embedding model is completely retrained with the entire database to incorporate new data, then the model accuracy and up-to-date status are improved, but the processing time and computational resources are significantly increased
Solution Approach 1:
The patent segments the training process by dividing the database into original data and new data portions. Instead of retraining the entire model on all data, it applies incremental updates using only the new data portion, while preserving the pre-trained model weights from the original data. This segmentation allows the model to incorporate new information without the time cost of processing the entire database again.
Solution Approach 2:
The patent performs preliminary action by pre-training the word embedding model on the original database before new data arrives. This pre-trained model serves as a foundation that captures existing linguistic patterns and relationships. When new data is added, the model can leverage this pre-computed knowledge and only require incremental adjustments, rather than starting from scratch each time.
2Reliability
If the word embedding model is completely retrained with the entire database to incorporate new data, then the model remains up-to-date with current information, but the computational resources and processing power are significantly increased
Solution Approach 1:
The training process is segmented into an initial comprehensive training phase on the original database, followed by incremental update phases using only new data. This segmentation reduces the computational burden of each update operation, as the expensive full-model training is performed only once, while subsequent updates use efficient incremental learning algorithms that require minimal computational resources.
Solution Approach 2:
The patent discards the computationally expensive full retraining process and recovers the essential functionality through incremental updates. By preserving the pre-trained model weights and only updating them with new data gradients, the system recovers the model's up-to-date status without incurring the full computational cost of complete retraining, effectively discarding redundant computation.
3Adaptability or versatility
If the neural network is retrained with augmented data set, then the model incorporates all new data from the relational database, but the development time and model generation time are significantly increased
Solution Approach 1:
The data incorporation process is segmented into two phases: initial comprehensive training on the original database that establishes the base model, and subsequent incremental updates that process only new data additions. This segmentation ensures complete data incorporation over time while dramatically reducing the development time required for each update cycle, as the model adapts to new data without requiring a complete retraining cycle.
4Reliability
If the word embedding model is recreated from scratch to incorporate new data, then the model reflects all current database information, but the delays prevent real-time or near-real-time query capabilities
Solution Approach 1:
The model performs preliminary action by maintaining a pre-trained state on the original database that is ready for immediate querying. When new data arrives, the incremental update mechanism quickly adapts the pre-trained model, allowing the system to maintain continuous query responsiveness rather than entering a prolonged retraining state. This ensures real-time or near-real-time query capabilities are preserved while still incorporating fresh data.
Solution Approach 2:
The patent implements a dynamic model updating approach where the word embedding model transitions smoothly between states of original training and incremental updates. Rather than static periodic full retraining that causes delays, the model dynamically adapts to new data as it arrives, maintaining optimal query responsiveness while continuously reflecting current database information through on-demand incremental updates.
Data Source
AI summary
Methods, systems and computer program products for updating a word embedding model are provided. Aspects include receiving a first data set comprising a relational database having a plurality of words. Aspects also include generating a word embedding model comprising a plurality of word vectors by training a neural network using unsupervised machine learning based on the first data set. Each word vector of the plurality of word vector corresponds to a unique word of the plurality of words. Aspects also include storing the plurality of word vectors and a representation of a hidden layer of the neural network. Aspects also include receiving a second data set comprising data that has been added to the relational database. Aspects also include updating the word embedding model based on the second data set and the stored representation of the hidden layer of the neural network.