Configurable Web Crawler Throttling and Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing search engines are too aggressive in crawling web content, leading to server overload and inefficiencies in crawling specific areas or types of pages, and lack the ability to configure crawl rules on a per-domain, per-page, and per-element basis, resulting in irrelevant search results and resource wastage.
Innovation Solution
A configurable web crawler system that allows users to set thread throttling rules, domain restrictions, page blacklists, element filtering, and crawling rules, enabling fine-grained control over the crawling process, including rule inheritance, to focus on relevant content and manage crawl speed and parallelism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a search engine crawler aggressively crawls web content to improve search result coverage, then the quantity of indexed content increases, but server overload occurs and crawling efficiency decreases
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting crawler behavior based on domain-specific rules. Different domains can have different crawl rates, depth limits, and resource consumption parameters configured to prevent server overload while maintaining comprehensive indexing. The system modifies crawling parameters adaptively rather than using a fixed aggressive crawl strategy across all domains.
Solution Approach 2:
The patent implements local quality by allowing different crawling strategies for different domains or URL patterns. Specific domains can be configured with customized crawl rules that reflect their unique characteristics and server capacity requirements. This enables the crawler to be aggressive where appropriate and conservative where needed, preventing universal server overload while maximizing useful content indexing.
2Quantity of substance
If a crawler follows all hyperlinks to ensure comprehensive indexing, then search result coverage improves, but irrelevant content is indexed reducing result quality
Solution Approach 1:
The patent applies local quality by implementing element-level filtering rules that differentiate between relevant and irrelevant content within pages. Different sections of a page can be selectively crawled or excluded based on configured rules. This allows the crawler to comprehensively index relevant content while filtering out navigation menus, footers, and other irrelevant elements that would otherwise pollute the index.
Solution Approach 2:
The patent uses extraction by removing irrelevant content from the crawling process through configured exclusion rules. Specific URL patterns, page types, or content elements can be extracted from the crawl queue or excluded from indexing. This separates relevant content that should be indexed from irrelevant content that should be ignored, improving search result quality without sacrificing comprehensive coverage of relevant material.
3Ease of operation
If a crawler operates without configuration options to simplify the system, then ease of operation improves, but adaptability to different crawling needs decreases
Solution Approach 1:
The patent applies segmentation by dividing the configuration system into modular, manageable components. Configuration options are organized into distinct categories such as domain rules, URL patterns, element filters, and resource limits. Each segment can be independently configured and applied. This modular approach maintains ease of operation through organized interfaces while providing comprehensive adaptability through granular control options.
4Productivity
If a crawler uses multiple threads to increase crawling speed, then productivity improves, but server load increases causing instability
Solution Approach 1:
The patent applies parameter changes by dynamically controlling thread configuration based on domain-specific rules. Different domains can have different maximum thread counts, request rates, and concurrency limits configured. The system adjusts threading parameters adaptively rather than using a fixed high-thread count approach, maintaining high productivity where server capacity allows while preventing instability on resource-constrained servers.
Solution Approach 2:
The patent implements dynamics by making the crawler's parallelism and resource consumption adaptive rather than static. Thread pools and concurrency levels can be dynamically adjusted based on configured domain rules and observed server responses. This enables the system to maximize crawling speed when servers can handle the load while automatically reducing activity to maintain stability when servers are overwhelmed.
Data Source
AI summary
A configurable web crawler allows a user configure a web crawl by specifying one or more of thread throttling rules, domain restriction rules, and crawling rules. The configurable web crawler crawls the web beginning with a seed uniform resource locator and according to the crawl configuration.