Web Mirror Detection via IP and Link Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Search engines face inefficiencies due to multiple hostnames pointing to the same content, leading to redundant crawling and indexing efforts, which waste bandwidth and affect search result rankings.
Innovation Solution
A method to identify hostname or subtree mirrors by obtaining and processing multiple signals such as link structure, content similarity, IP address, ownership, and redirect information to determine if hostnames or subtrees refer to the same content, allowing search engines to crawl and index only unique content.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If search engines crawl and index pages from multiple hostnames that refer to the same content, then search engines can discover and index all accessible content, but this wastes crawl bandwidth and index space while putting twice the crawl load on the website
Solution Approach 1:
The patent merges multiple hostnames that resolve to the same IP address into a single canonical hostname. The system detects mirrors by comparing hostnames against a list of known mirrors (e.g., www. and non-www variants) and consolidates their crawl and indexing operations into a single entry, thereby eliminating redundant crawling while maintaining complete content discovery.
Solution Approach 2:
The patent introduces an intermediary mechanism that resolves hostnames to IP addresses and uses this resolution information as a key to identify mirrors. The system uses the IP address as an intermediary identifier to group together hostnames that point to the same server, enabling automatic detection and consolidation of mirror sites without requiring direct comparison of all content.
2Reliability
If search engines treat multiple hostnames as separate entities for ranking, then each hostname can be independently ranked, but the ranking of each hostname is reduced to about half of what it would be if hostnames were ranked together
Solution Approach 1:
The patent merges the ranking calculations for multiple hostnames into a single unified ranking. By consolidating link counts and authority signals across mirror hostnames, the system ensures that the canonical hostname receives the full ranking weight that would otherwise be split across multiple separate entries, thereby maintaining accurate and high-quality search results.
3Device complexity
If search engines use existing ranking techniques without mirror detection, then the system remains simple to implement, but crawl bandwidth is wasted and index space is consumed on duplicate content
Solution Approach 1:
The patent performs preliminary hostname resolution and mirror detection before the actual crawling and indexing processes begin. By pre-identifying which hostnames are mirrors of each other and selecting a single canonical hostname for crawling, the system avoids wasting crawl bandwidth on duplicate content while maintaining relatively simple implementation through the use of straightforward hostname-to-IP resolution and mirror list comparison.
Data Source
AI summary
A system obtains multiple signals associated with first and second hostnames or subtrees. The system uses the multiple signals to determine whether the first hostname or subtree is a mirror of the second hostname or subtree. The multiple signals may include, for example, a same link structure and/or sitemap for the first and second hostnames or subtrees, duplicate content associated with the first and second hostnames or subtrees, a same Internet Protocol (IP) address or subnet for the first and second hostnames or subtrees, a same owner for the first and second hostnames or subtrees, nearly duplicate content associated with the first and second hostnames or subtrees, similarity between the hostnames of the first and second hostnames or subtrees; and/or an indication that one hostname or subtree of the first and second hostnames or subtrees redirects to the other hostname or subtree.