Word Embedding Fusion With Word Scores for Low-Latency Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text classification methods, particularly those using context-independent word embedding models, struggle to achieve state-of-the-art performance while meeting low latency requirements, limiting their effectiveness in applications like chatbots.
Innovation Solution
A method that combines word embeddings with word scores, using a context-independent model like FastText, and applies a feed-forward layer with a softmax function for classification, incorporating term frequency, document frequency, and learned parameters to enhance text classification performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If contextual models are used to obtain word embeddings, then text classification accuracy is improved, but computational latency increases making it unsuitable for low latency applications
Solution Approach 1:
The patent combines context-independent word embeddings with word scores from a trained model to create a hybrid representation that achieves state-of-the-art text classification performance while maintaining low computational latency suitable for production applications
Solution Approach 2:
The patent creates a composite feature representation by merging word embeddings from a pre-trained context-independent model with word scores derived from training data, forming a new hybrid feature set that leverages both statistical patterns and contextual information
2Loss of time
If context-independent word embedding models are used, then computational latency is reduced, but text classification performance deteriorates compared to state-of-the-art methods
Solution Approach 1:
The patent merges context-independent word embeddings with word scores to create an enhanced representation that compensates for the limitations of context-independent models while preserving their low latency advantage
Solution Approach 2:
The patent transforms the feature representation by incorporating word scores as additional parameters alongside word embeddings, changing the parameter space to achieve better classification performance without increasing computational complexity
Data Source
AI summary
Techniques disclosed herein relate generally to text classification and include techniques for fusing word embeddings with word scores for text classification. In one particular aspect, a method for text classification is provided that includes obtaining an embedding vector for a textual unit, based on a plurality of word embedding vectors and a plurality of word scores. The plurality of word embedding vectors includes a corresponding word embedding vector for each of a plurality of words of the textual unit, and the plurality of word scores includes a corresponding word score for each of the plurality of words of the textual unit. The method also includes passing the embedding vector for the textual unit through at least one feed-forward layer to obtain a final layer output, and performing a classification on the final layer output.