Webpage Classification Using Word2vec Vector Similarity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current webpage classification methods face inefficiencies, particularly with Chinese webpages, due to high processing complexity and low classification efficiency caused by large amounts of short sentences or words, which complicates data processing and reduces accuracy.
Innovation Solution
The method employs a word-to-vector tool like word2vec to convert corpus words into vectors, allowing for vector calculations that enhance comparison and similarity analysis, and uses a scoring model to determine webpage categories by calculating vector similarities and applying weight coefficients to improve classification accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional classification algorithms are used with large quantities of short sentences or words as characteristic data, then comprehensive webpage analysis is achieved, but data processing complexity increases and classification efficiency decreases
Solution Approach 1:
The patent transforms the parameter representation of webpage characteristic data by converting words and short sentences into vector forms with specific dimensions (e.g., 300-dimensional vectors). This parameter transformation enables efficient vector operations while preserving semantic information, thereby improving classification efficiency without sacrificing accuracy.
Solution Approach 2:
The patent replaces traditional text-based classification mechanisms with vector-based computational mechanisms. By substituting string matching and text processing with vector arithmetic operations (addition, subtraction, similarity calculation), the system achieves faster processing while maintaining comprehensive analysis capabilities.
2Adaptability or versatility
If Chinese words are processed in characteristic data, then Chinese webpage classification is achieved, but processing complexity increases due to language-specific challenges
Solution Approach 1:
The patent applies parameter transformation by converting Chinese words into numerical vectors that capture semantic relationships. This transformation eliminates the need for complex Chinese-specific text processing while preserving language nuances, thereby reducing processing complexity while maintaining adaptability to Chinese webpages.
Solution Approach 2:
The patent introduces vectors as an intermediary representation between Chinese text and classification algorithms. These vectors serve as a bridge that translates linguistic complexity into computationally efficient numerical forms, enabling Chinese webpage processing without directly handling language-specific complexities.
3Reliability
If massive webpages are parsed to extract characteristic data from URLs and headers, then training data is obtained, but data processing time and computational resources increase
Solution Approach 1:
The patent extracts only the most essential characteristic data from webpages—specifically titles and keywords—rather than processing entire page contents. This selective extraction maintains training data quality by focusing on the most informative elements while dramatically reducing data processing time and computational resource requirements.
Solution Approach 2:
The patent performs preliminary vectorization of characteristic data during the data preparation phase, converting text into vector representations before the actual classification training begins. This preliminary action reduces the computational burden during subsequent processing stages by pre-computing the complex text-to-vector transformations.
Data Source
AI summary
A webpage classification method and apparatus, a computing device and a machine readable storage medium are disclosed. Each corpus word in a corpus is converted into a vector by using a word-to-vector tool word2vec, and therefore a processing process such as comparison between corpus words or similarity analysis is converted into vector calculation, so as to more conveniently implement computer automation, thereby improving webpage classification efficiency. Moreover, corresponding corpus words are screened according to preset classification seed words, and a corpus word unrelated to a webpage type may be removed, thereby improving webpage classification accuracy.