Weighted Total Word Vector for Text Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text processing methods using recurrent neural networks with Conditional Random Field Classifiers face high-dimensional vector issues, leading to increased parameters and a need for word-dependent or context-dependent representations for effective classification.
Innovation Solution
A computer-implemented method using a model that forms a total word vector as a weighted combination of individual word vectors based on properties such as relative frequency, length, form, and syntactic category, with an attention function to optimize weighting, allowing for word-dependent and domain-dependent weighting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If concatenated word representations are used for text processing, then comprehensive text information is captured, but the dimensionality of vectors increases significantly
Solution Approach 1:
The patent combines multiple word vectors (contextual embedding, word embedding, and character embedding) into a single total word vector through weighted summation. This merging approach integrates comprehensive text information from different representation levels while maintaining a unified vector structure, resolving the contradiction between capturing complete text information and managing vector dimensionality.
Solution Approach 2:
The patent introduces learnable weight parameters (alpha, beta, gamma) that dynamically adjust the contribution of each word vector component. By changing these parameters during training, the model optimizes the balance between different embedding types, effectively managing the information-capture-to-dimensionality ratio adaptively for different text contexts.
2Measurement precision
If high-dimensional word vectors are used, then more text features are represented, but the number of parameters to be learned increases
Solution Approach 1:
The patent pre-computes multiple types of word vectors (contextual, word-level, and character-level embeddings) before the main classification task. These pre-computed vectors serve as foundational features that capture various aspects of text information in advance, reducing the need for the model to learn all features from scratch during training, thereby decreasing the effective number of parameters to be learned.
Solution Approach 2:
By merging multiple pre-computed word vectors into a total word vector through weighted summation, the patent creates a compact integrated representation that consolidates rich text features into a single vector. This reduces the parameter count compared to using multiple separate high-dimensional vectors, while preserving comprehensive text feature representation.
3Ease of operation
If standard word embeddings are used, then processing is simple, but word-dependent and context-dependent representations are lost
Solution Approach 1:
The patent applies different embedding strategies to different aspects of word representation: character embeddings capture local subword patterns and morphological information, word embeddings capture global lexical meaning, and contextual embeddings capture surrounding context. This local quality differentiation enables the model to capture word-dependent and context-dependent representations while maintaining processing efficiency through established embedding techniques.
Solution Approach 2:
The patent creates a composite word representation by combining multiple embedding types (character, word, and contextual embeddings) into a total word vector. This composite approach integrates the advantages of different embedding methods, achieving both word-dependent and context-dependent representation capabilities while building upon simple, well-established embedding techniques.
Data Source
AI summary
A computer-implemented method for processing text data including a multitude of text modules. In the method, a representation of the text is provided, and a model is used which predicts a classification for a respective text module of the text as a function of the representation of the text. The provision of the representation of the text includes the provision of a total word vector for a respective text module of the text. The total word vector is formed from at least two, preferably multiple word vectors, and a respective word vector being weighted as a function of properties of the respective text module.