Dynamic Word Embeddings for Semantic Change Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional static word embeddings fail to detect changes in the meaning and usage of words over time due to their assumption that word meanings remain constant across a text corpus, which is not reflective of language evolution.
Innovation Solution
A dynamic word embedding model is developed using a diffusion process and variational inference to associate words with usage contexts over time, employing skip-gram models with word and context embedding vectors that evolve through a Kalman filter and Ornstein-Uhlenbeck process, allowing for the detection of semantic changes in word usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If static word embedding models are used, then computational simplicity is maintained, but the ability to detect semantic changes over time is lost
Solution Approach 1:
The patent transforms static word embeddings into dynamic models by introducing time-varying parameters. Word embeddings are now functions of time, allowing the model to capture evolving semantic meanings. This is achieved through dynamic Bayesian networks that model temporal dependencies in word usage patterns.
Solution Approach 2:
The patent segments the corpus into time-based windows or slices, creating multiple static embedding models for different time periods. This segmentation allows comparison of embeddings across time, enabling detection of semantic drift while maintaining the computational benefits of static models within each segment.
2Measurement precision
If dynamic models with multiple parameters are used, then semantic evolution detection is improved, but computational resources increase
Solution Approach 1:
The patent changes the parameters of word embeddings dynamically over time rather than keeping them fixed. By allowing embedding vectors to evolve as functions of time with controlled complexity, the model achieves better semantic tracking without proportionally increasing computational burden.
Solution Approach 2:
The patent applies dynamic modeling selectively to only those words showing semantic change, rather than uniformly applying complex dynamic models to all vocabulary. This partial application reduces overall computational energy consumption while maintaining detection precision for evolving terms.
3Measurement precision
If time-based segmentation of corpus is implemented, then semantic drift detection is enabled, but processing time increases
Solution Approach 1:
The patent divides the text corpus into temporal segments or windows, allowing independent processing of each time period. This segmentation enables parallel computation across time slices and facilitates efficient comparison of embeddings between periods, reducing overall processing time while maintaining detection accuracy.
Solution Approach 2:
The patent performs preliminary processing of the corpus to extract and organize time-stamped word contexts before embedding generation. This preprocessing creates ready-to-use time-structured data that speeds up subsequent embedding computation and comparison operations.
Data Source
AI summary
Systems, methods, and articles of manufacture to perform an operation comprising deriving, based on a corpus of electronic text, a machine learning data model that associates words with corresponding usage contexts over a window of time, according to a diffusion process, wherein the machine learning data model comprises a plurality of skip-gram models, wherein each skip-gram model comprises a word embedding vector and a context embedding vector for a respective time step associated with the respective skip-gram model, generating a smoothed model by applying a variational inference operation over the machine learning data model, and identifying, based on the smoothed model and the corpus of electronic text, a change in a semantic use of a word over at least a portion of the window of time.