Entity Embedding Generation via Vocabulary Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for generating entity embeddings in machine learning consume excessive computational resources and do not achieve optimal performance when applied to the entire vocabulary, while requiring more resources and yielding lower performance.
Innovation Solution
A computer-implemented method that selects a strategic subset of entity embeddings based on frequency or other criteria and applies modifications such as mean subtraction and principal component removal to produce improved embeddings with reduced computational consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If contextual word embeddings are generated for the entire vocabulary, then embedding performance is improved, but computational resources and memory consumption increase excessively
Solution Approach 1:
The patent segments the vocabulary into a subset of most frequent words (e.g., top 10k-100k words) and applies contextual embedding generation only to this subset. This segmentation allows the system to achieve high embedding performance for the most commonly used words while avoiding the excessive computational cost of processing the entire vocabulary, directly resolving the contradiction between performance and computational resource consumption.
Solution Approach 2:
The patent applies partial action by generating contextual embeddings only for a portion of the vocabulary (most frequent words) rather than the complete vocabulary. This partial approach yields sufficient performance improvement for practical applications while consuming significantly fewer computational resources, effectively balancing performance gains with resource consumption.
2Measurement precision
If contextual word embeddings are generated for the entire vocabulary, then embedding performance is improved, but memory consumption increases excessively
Solution Approach 1:
The patent segments the vocabulary into a manageable subset of most frequent words and generates contextual embeddings only for this segment. This reduces the memory footprint of embedding tables from storing all vocabulary embeddings to storing only the essential ones, directly addressing the memory consumption issue while maintaining performance for the most critical words.
Solution Approach 2:
The patent extracts and removes less frequent words from the vocabulary set, retaining only the most frequent words for contextual embedding generation. This extraction approach eliminates unnecessary memory consumption associated with storing embeddings for rare words while preserving the performance benefits for the most commonly used words.
3Measurement precision
If embedding modifications are applied to the entire vocabulary, then performance improvement is achieved, but computational consumption increases
Solution Approach 1:
The patent segments the vocabulary into a subset of most frequent words and applies embedding modifications (mean subtraction, principal component removal) only to this segment. This segmentation reduces the computational power required for modifications while maintaining performance improvements for the most commonly used words, resolving the contradiction between performance and computational consumption.
Solution Approach 2:
The patent applies partial modifications to a portion of the vocabulary rather than the entire vocabulary. This partial approach achieves sufficient performance improvement for practical purposes while consuming significantly less computational power, effectively balancing performance gains with power consumption.
Data Source
AI summary
Example aspects of the present disclosure are directed to systems and methods for generation of improved language embeddings (e.g., entity embeddings for natural language tokens) which provide improved model performance. In addition, the proposed techniques require less computational consumption relative to previous approaches.