Word Embedding Fusion With Word Scores for Low-Latency Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text classification methods face challenges in achieving competitive performance with state-of-the-art results while meeting low latency requirements, particularly in natural language processing tasks involving unstructured data.
Innovation Solution
The fusion of word embeddings with word scores using context-independent models, such as FastText, combined with feed-forward layers and techniques like softmax classification, enhances text classification performance by incorporating term frequency, document frequency, and learned parameters to generate composite embedding vectors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If contextual models are used to obtain word embeddings, then text classification accuracy is improved, but computational intensity increases and latency increases
Solution Approach 1:
The patent segments the text into individual words or subwords, obtaining embedding vectors for each segment independently using a pre-trained context-independent model. This avoids the computational burden of processing the entire text through a contextual model while preserving meaningful word-level representations that can be aggregated for classification.
Solution Approach 2:
The patent performs preliminary action by pre-training the word embedding model offline before deployment. The context-independent embeddings are pre-computed and stored, allowing the system to bypass the computationally intensive contextual modeling step during actual text classification operations, thus reducing latency and computational intensity while maintaining accuracy.
2Measurement precision
If contextual models are used to obtain word embeddings, then text classification accuracy is improved, but latency increases
Solution Approach 1:
The patent segments the text into individual words or subwords, obtaining embedding vectors for each segment independently using a pre-trained context-independent model. This avoids the computational burden of processing the entire text through a contextual model while preserving meaningful word-level representations that can be aggregated for classification.
Solution Approach 2:
The patent performs preliminary action by pre-training the word embedding model offline before deployment. The context-independent embeddings are pre-computed and stored, allowing the system to bypass the computationally intensive contextual modeling step during actual text classification operations, thus reducing latency and computational intensity while maintaining accuracy.
3Loss of time
If context-independent word embedding models are used, then latency is reduced, but text classification performance deteriorates
Solution Approach 1:
The patent merges multiple context-independent word embedding vectors corresponding to different words or subwords of a text into a single aggregated embedding vector. This combination captures the collective semantic information from multiple word representations, compensating for the lack of contextual information and improving text classification performance while maintaining low latency.
Solution Approach 2:
The patent uses pre-trained context-independent word embedding models to generate embedding vectors for individual words, effectively copying proven word representation knowledge into the text classification system. This allows the system to leverage existing high-quality word embeddings without incurring the computational cost of generating contextual embeddings from scratch.
4Measurement precision
If word embeddings are fused with word scores, then text classification accuracy is improved, but device complexity increases
Solution Approach 1:
The patent merges multiple context-independent word embedding vectors corresponding to different words or subwords of a text into a single aggregated embedding vector. This combination captures the collective semantic information from multiple word representations, compensating for the lack of contextual information and improving text classification performance while maintaining low latency.
Solution Approach 2:
The patent uses pre-trained context-independent word embedding models to generate embedding vectors for individual words, effectively copying proven word representation knowledge into the text classification system. This allows the system to leverage existing high-quality word embeddings without incurring the computational cost of generating contextual embeddings from scratch.
Data Source
AI summary
Techniques disclosed herein relate generally to text classification and include techniques for fusing word embeddings with word scores for text classification. In one particular aspect, a method for text classification is provided that includes obtaining an embedding vector for a textual unit, based on a plurality of word embedding vectors and a plurality of word scores. The plurality of word embedding vectors includes a corresponding word embedding vector for each of a plurality of words of the textual unit, and the plurality of word scores includes a corresponding word score for each of the plurality of words of the textual unit. The method also includes passing the embedding vector for the textual unit through at least one feed-forward layer to obtain a final layer output, and performing a classification on the final layer output.