Commodity Short Text Core Word Extraction via Vector Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods, such as the Bag of Words model, are ineffective in accurately extracting core words from short texts due to their sparse features and unclear themes, which are common in commodity data from e-commerce websites.
Innovation Solution
A method involving word segmentation, document vector computation using tools like word2vec, clustering, and weight determination through chi-square values and inverse file frequency, combined with local and document-level weights, to identify core words in commodity short texts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If Bag of Words model is used to extract core words from short texts, then the extraction process is simple, but the accuracy is low due to sparse features and unclear themes
Solution Approach 1:
The patent segments the text processing into multiple stages: word segmentation, document vector computation, clustering, and weight calculation. Each stage processes specific aspects of the data independently before integrating results, allowing complex analysis to be broken down into manageable steps that improve accuracy without excessive complexity
Solution Approach 2:
The patent transforms short texts from discrete word sequences into document vectors in a continuous vector space. This dimensional transformation enables the use of clustering algorithms and weight calculations that operate on vector representations, providing a new dimension for analysis that improves extraction accuracy by capturing semantic relationships and contextual information
2Measurement precision
If manual labeling is used to mine core words, then the accuracy is high, but the method cannot adapt to massive commodity data
Solution Approach 1:
The patent implements an automated system that performs word segmentation, document vector computation, clustering, and weight calculation without requiring manual intervention. The system processes commodity data independently through algorithmic operations, enabling it to handle massive datasets that would be impractical for manual labeling while maintaining high accuracy through computational methods
Solution Approach 2:
The patent replaces manual labeling with automated computational processes. Instead of human annotators systematically reviewing and marking core words, the system uses algorithms to automatically identify and extract core words through vector space analysis and clustering, dramatically increasing processing capacity while maintaining accuracy through rigorous computational methods
Data Source
AI summary
A method and a device for extracting a core word of a commodity short text, is provided. The method of extracting a core word of a commodity short text includes: obtaining commodity short texts in a data set; performing word segmenting on each commodity short text; obtaining a document vector of the commodity short text according to context information of word segments of the commodity short text; clustering the commodity short texts in the data set according to document vectors; determining a cluster level weight of each word segment of the commodity short text in a category to which the commodity short text belongs; and determining a core word of the commodity short text according to the cluster level weight of each word segment.