Document Representation via Weighted Pseudo-Word Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional document classification methods, such as the bag-of-words (BOW) model, face challenges in efficiently handling new words not included in training data and result in large, resource-intensive document representations due to large vocabularies, especially in contexts like review data and social media posts.
Innovation Solution
The approach involves creating document representations based on weighted pseudo-words by processing text data to generate word-vectors using Word2vec, determining similarity scores, grouping words into clusters, and applying TF-IDF weights to reduce dimensionality and improve resource efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If the bag-of-words model is used to represent documents with all words as features, then the document representation captures all vocabulary information, but the resulting vectors become very large requiring significant computing resources
Solution Approach 1:
The patent merges multiple words that have similar meanings or usage patterns into single pseudo-words. Words are grouped based on co-occurrence statistics and semantic similarity, then represented as unified features in the document vector. This reduces the total number of features while preserving the semantic information that would have been distributed across multiple individual word features.
Solution Approach 2:
Each pseudo-word serves as a universal representation for multiple individual words that share similar characteristics. Instead of creating separate features for each synonym or semantically related word, a single pseudo-word feature captures the combined information, making the representation more efficient while maintaining comprehensive vocabulary coverage.
2Adaptability or versatility
If traditional bag-of-words representation is used, then all words in the vocabulary are accounted for, but new words not in training data cannot be efficiently handled
Solution Approach 1:
When a new word is encountered that wasn't in the training vocabulary, the system creates a pseudo-word representation by finding semantically similar words from the training data. It copies the feature representation of the most similar existing pseudo-word and uses it to represent the new word, allowing the system to handle unseen vocabulary while maintaining consistent representation quality.
Solution Approach 2:
The patent dynamically adjusts the parameter of word representation by transitioning from fixed vocabulary indices to flexible pseudo-word embeddings. This allows the system to adapt its feature space to accommodate new words by mapping them to existing pseudo-word representations based on semantic similarity, rather than requiring pre-defined vocabulary slots.
3Loss of information
If word vectors are created for every word in large vocabularies, then comprehensive text representation is achieved, but memory requirements and processing load increase significantly
Solution Approach 1:
The patent extracts only the essential semantic information from individual words by grouping them into pseudo-words. Instead of maintaining separate vector representations for every word in the vocabulary, it extracts the common semantic patterns and represents them once per pseudo-word, significantly reducing the total memory footprint while retaining the essential meaning of the text.
Solution Approach 2:
The vocabulary is segmented into meaningful groups of semantically related words, where each group is represented by a single pseudo-word. This segmentation reduces the computational burden by processing and storing one representative vector per group rather than one vector per individual word, while still capturing the diversity of the original vocabulary.
Data Source
AI summary
Methods, systems, and computer-readable storage media for providing weighted vector representations of documents, with actions including receiving text data, the text data including a plurality of documents, each document including a plurality of words, processing the text data to provide a plurality of word-vectors, each word-vector being based on a respective word of the plurality of words, determining a plurality of similarity scores based on the plurality of word-vectors, each similarity score representing a degree of similarity between word-vectors, grouping words of the plurality of words into clusters based on the plurality of similarity scores, each cluster including two or more words of the plurality of words, and providing a document representation for each document in the plurality of documents, each document representation including a feature vector, each feature corresponding to a cluster.