Web Crawler Score-Based Link Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional web crawlers face inefficiencies in expanding their collection range and adapting to changes in site configurations, requiring manual administrator intervention and resource-intensive link configuration management, which complicates the setting of collection rules and leads to decreased processing efficiency.
Innovation Solution
A method that calculates a score for link-destination addresses based on their relevance to the collection rule, allowing for automatic inclusion or exclusion from the collection target, even if outside the initial rule specifications, and adjusts scores based on site characteristics and time limits to reflect changing relevance and resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If the collection rule is set in detail to exclude unnecessary files, then the collection precision is improved, but the device complexity increases
Solution Approach 1:
The web crawler automatically determines collection targets by calculating scores based on link path information and comparing them against threshold values, eliminating the need for administrators to manually configure complex URL patterns. The system serves itself by autonomously deciding which pages to collect based on relevance scoring rather than relying on pre-set exclusion rules.
Solution Approach 2:
The invention changes the parameter from static URL pattern matching to dynamic score-based selection. Instead of using complex URL patterns that require detailed configuration, the system uses score thresholds that can be easily adjusted. The score is calculated based on link path information and can be modified by changing threshold values rather than rewriting complex collection rules.
2Adaptability or versatility
If the collection rule is changed manually to include new pages, then the adaptability is improved, but the loss of time increases
Solution Approach 1:
The system automatically adapts to site changes by continuously calculating scores for link destinations based on current link path information. When new pages are added or site structures change, the crawler automatically evaluates their relevance scores and includes them in collection targets without requiring manual rule updates, thus eliminating maintenance time.
Solution Approach 2:
The invention implements a feedback mechanism where the crawler continuously monitors link path information and adjusts collection decisions based on actual site structure. The score calculation uses feedback from link analysis to dynamically determine collection targets, allowing the system to adapt to changes without manual intervention.
3Productivity
If the web crawler collects all linked pages without score filtering, then the productivity is improved, but the loss of energy increases
Solution Approach 1:
The system changes the collection criterion from binary (collect or not collect based on URL pattern) to continuous (score-based filtering). By using score thresholds, the system can efficiently process pages and stop collecting when scores fall below the threshold, optimizing resource consumption while maintaining productivity through parallel processing of high-score pages.
Solution Approach 2:
Instead of collecting all linked pages (excessive action), the system collects only pages with scores above the threshold (partial action). This selective collection approach reduces processing resource consumption while maintaining sufficient coverage of important pages, achieving energy efficiency without significantly impacting productivity.
Data Source
AI summary
The present invention provides an information collection apparatus, an information collection method, and a program capable of collecting information from information resources on a network effectively as well as a search engine that searches the information resources collected. An information collection apparatus of the present invention that collects information from information resources on a network includes an extraction unit that acquires data from an information resource via the network to extract a link-destination address included in the data, a calculation unit that calculates, by comparing each link-destination address with a collection rule describing a set of addresses qualified for a collection target, a score for each link-destination address that reflects a distance from the set to a link-destination information resource indicated by the link-destination address, and a judgment unit that judges whether the link-destination information resource is to be included in the collection target or not in accordance with the score calculated for the link-destination information resource.