Web Crawler Triggering via Registry Change Logs

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for providing domain information to registrars and search engines are inefficient, as they rely on outdated data and do not allow for timely crawling of domains with changes, leading to delayed and incomplete search results.

Innovation Solution

A method that involves receiving and processing domain information from registrars, storing it in a registry zone file, and forming a list of registry data to trigger crawling within minutes of domain changes, using rapid zone update files and threshold-based domain resolution requests to prioritize and initiate crawls based on near real-time data.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of time

If traditional web crawling methods are used to crawl all domains in a zone file, then comprehensive domain coverage is achieved, but time delay and resource waste occur due to crawling domains that have not changed

Engineering Contradiction:

Improvetime delay in crawlingVSAvoidcrawling efficiency

Core Design Contradiction:

Loss of timeVSProductivity

Solution Approach 1:

The patent segments the zone file into a complete zone file and a change log, separating domains that have changed from those that have not. The crawler then processes only the changed domains from the change log rather than the entire zone file, reducing crawling time and resources while maintaining comprehensive coverage over time.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The registry performs preliminary action by maintaining a change log that records all domain changes before they are crawled. This change log is updated in advance as domains are modified, so when crawling is needed, the crawler can immediately access the pre-compiled list of changed domains without scanning the entire zone file.

Inventive Principle:

Principle #10Preliminary action

2Loss of time

If domain information is updated in real-time, then search result timeliness is improved, but system complexity increases due to continuous monitoring and data processing requirements

Engineering Contradiction:

Improvetimeliness of search resultsVSAvoidsystem complexity

Core Design Contradiction:

Loss of timeVSDevice complexity

Solution Approach 1:

The patent introduces a change log as an intermediary component between the domain registration system and the crawler. The change log acts as a mediator that receives domain change information from registrars and translates it into a crawlable format, simplifying the overall system architecture while enabling real-time updates.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system implements feedback mechanisms where the registry continuously monitors domain changes and feeds this information back to the crawler through the change log. This feedback loop ensures that the crawler receives updated domain information automatically, maintaining real-time search results without requiring complex continuous monitoring systems.

Inventive Principle:

Principle #23Feedback

3Quantity of substance

If the entire zone file is crawled periodically, then complete domain information is obtained, but resource consumption and processing time increase significantly

Engineering Contradiction:

Improveamount of domain data processedVSAvoidprocessing energy consumption

Core Design Contradiction:

Quantity of substanceVSUse of energy by moving object

Solution Approach 1:

The patent extracts only the necessary information (changed domains) from the complete zone file by using the change log. Instead of processing the entire zone file, the system extracts and processes only the subset of domains that have changed, significantly reducing processing energy consumption while maintaining data completeness for crawled domains.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The system applies partial action by crawling only the changed domains identified in the change log rather than the entire zone file. This partial crawling approach processes a subset of domains (exactly those that changed) which is both sufficient for updating search results and optimal for reducing resource consumption.

Inventive Principle:

Principle #16Partial or excessive action

4Loss of time

If domain changes are monitored continuously, then up-to-date search results are achieved, but data processing load and infrastructure requirements increase

Engineering Contradiction:

Improveupdate speed of domain dataVSAvoiddata processing infrastructure

Core Design Contradiction:

Loss of timeVSDevice complexity

Solution Approach 1:

The change log system implements self-service by automatically capturing and storing domain change information as it occurs. The registry's existing domain change detection mechanisms automatically populate the change log without requiring additional complex monitoring infrastructure, enabling continuous updates while keeping the system simple.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS8812479B2Method and system for triggering web crawling based on registry data

Publication Date: 2014.08.19 VERISIGN INC

AI summary

A method of triggering crawling of a domain includes receiving information related to a domain from a registrar and processing the information related to the domain. The method also includes storing the processed information in a registry zone file and forming a list of registry data based on the processed information. The list of registry data comprises a subset of the registry zone file. The method farther includes crawling one or more of the domains in the list of registry data.