Web Domain Classification via Progressive Crawling Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current URL classification methods for web domains are inefficient and impractical due to the vast number of web domains and pages, leading to inaccurate categorization and high resource consumption, especially in security applications where real-time filtering is required.
Innovation Solution
The implementation of optimized web domains classification based on progressive crawling with clustering, which involves crawling and categorizing web domains at different levels, using a cloud service to focus on popular domains, and employing machine learning techniques to classify web pages into category clusters, thereby improving the efficiency and accuracy of URL categorization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional web crawling methods are used to classify all web domains, then comprehensive coverage is achieved, but resource consumption becomes excessively high and processing time increases significantly
Solution Approach 1:
The patent segments the web domain classification task by dividing domains into clusters based on their structural characteristics (such as domain name patterns, TLDs, and hierarchical relationships). This segmentation allows the system to process clusters rather than individual domains, reducing the computational burden while maintaining comprehensive coverage through systematic cluster-by-cluster processing.
Solution Approach 2:
The patent performs preliminary classification actions by pre-processing domain names to identify patterns, extract features, and organize them into preliminary clusters before full classification. This preliminary action reduces the complexity of subsequent classification tasks and enables more efficient resource utilization by focusing computational efforts on the most challenging domains first.
2Quantity of substance
If traditional web crawling methods are used to classify all web domains, then comprehensive coverage is achieved, but processing time becomes excessively long
Solution Approach 1:
The patent segments domains into clusters based on structural patterns and hierarchical relationships, enabling parallel processing of multiple clusters simultaneously. This segmentation strategy reduces processing time by allowing the system to handle multiple domains or domain groups concurrently rather than sequentially.
Solution Approach 2:
The patent performs preliminary organization of domains into clusters based on name patterns and structural features before executing the full classification process. This preliminary action pre-sorts and pre-organizes the data, significantly reducing the time required for subsequent classification operations.
3Device complexity
If traditional classification methods are used, then simplicity of implementation is maintained, but classification accuracy deteriorates due to the vast number of domains
Solution Approach 1:
The patent segments domains into meaningful clusters based on structural patterns and hierarchical relationships, which improves classification accuracy by processing smaller, more manageable groups rather than handling the entire vast domain space at once. This segmentation enables the system to maintain higher precision in categorization while keeping implementation relatively simple through automated pattern recognition.
Solution Approach 2:
The patent changes the parameters used for classification by focusing on structural characteristics of domain names (such as patterns, TLDs, and hierarchical elements) rather than relying solely on traditional content-based methods. This parameter change enables more accurate classification without significantly increasing implementation complexity, as the structural analysis can be performed through straightforward string processing algorithms.
Data Source
AI summary
Techniques for optimized web domains classification based on progressive crawling with clustering are disclosed. In some embodiments, optimized web domains classification based on progressive crawling with clustering includes crawling a domain (e.g., a web site domain) to collect data for a subset of pages (e.g., web pages) of a corpus of content associated with the domain; classifying each of the crawled pages into one or more category clusters, in which the category clusters represent a content categorization of the corpus of content associated with the domain (e.g., a URL content categorization for the domain, host of that domain, and/or directory of that domain); and determining which of the one or more category clusters to publish for the domain.