Document Vectorization Without Stop Word Dictionaries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The Bag-of-Words (BoW) method requires the selection and dictionary of stop words for vectorizing documents, which is inefficient and unnecessary for capturing document characteristics.
Innovation Solution
A data processing apparatus and method that selects words based on part-of-speech types, weights their occurrences or frequencies, and generates vectors without the need for stop word selection or dictionaries, allowing for efficient document representation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the Bag-of-Words method is used for document vectorization, then document data can be expressed by vectors for similarity measurement, but the method requires manual selection of stop words and dictionaries which increases complexity and work effort
Solution Approach 1:
The system automatically determines stop words and weightings based on the training data itself, without requiring external dictionaries or manual configuration. The learning unit analyzes the training documents and autonomously identifies which words to exclude and how to weight remaining words, making the system self-sufficient and eliminating manual setup complexity
Solution Approach 2:
The patent changes the parameter of word weighting from fixed manual assignment to dynamic calculation based on part-of-speech types and occurrence frequencies. By automatically adjusting weighting parameters based on linguistic characteristics and data distribution, the system adapts to different document corpora without manual reconfiguration
2Ease of operation
If all words are used as vector elements, then no stop word selection is needed, but the vector dimensionality increases and computational efficiency decreases
Solution Approach 1:
The patent applies different quality treatments to different parts of the word set by categorizing words into part-of-speech types (nouns, verbs, adjectives, adverbs). Each category receives appropriate weighting based on its linguistic function and relevance to document characteristics, rather than treating all words uniformly. This selective differentiation optimizes vector quality without requiring manual stop word removal
3Measurement precision
If manual stop word dictionaries are created, then document characteristics can be captured more accurately, but the time and effort required for dictionary creation and maintenance increases
Solution Approach 1:
The system performs preliminary analysis of the training data to automatically identify stop words and establish weighting schemes before actual document vectorization begins. By conducting this preparatory work automatically on the training corpus, the system eliminates the need for manual dictionary creation while ensuring the stop words are appropriate for the specific domain and language patterns in the data
Solution Approach 2:
The learning unit uses feedback from training data analysis to automatically refine stop word selection and word weighting. The system evaluates which words provide meaningful discrimination between documents and adjusts parameters accordingly, creating an adaptive process that improves accuracy without manual intervention
Data Source
AI summary
A data processing apparatus includes a word selection unit configured to select words corresponding to a part-of-speech type targeted for selection from words disassembled from document data, and a generation unit configured to find, for each of the words selected by the word selection unit, a value obtained by weighting the number of occurrences of each of the words or frequency of occurrences of each of the words according to the part-of-speech type of each of the words to generate a vector with a plurality of the values that are found as values of elements.