Spam URL Filtering Using Subdomain Randomness Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing spam burst attack techniques overwhelm computing and network resources by fetching and analyzing large numbers of spam URLs, degrading performance and user experience, and manual or machine learning-based methods are inefficient and costly.
Innovation Solution
A statistical model determines domain trustworthiness based on URL and subdomain analysis, using a neural network to block spam subdomains with randomly generated prefixes, isolating them in a sandbox environment for further processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all discovered URLs are fetched and analyzed by the web crawler, then comprehensive indexing is achieved, but computing and network resources are overwhelmed by spam URLs
Solution Approach 1:
The patent applies statistical models and neural networks to analyze URL patterns and domain characteristics before the web crawler fetches content. By pre-identifying spam URLs through subdomain randomness analysis and domain reputation scoring, the system filters out malicious content in advance, preventing resource waste on spam while maintaining comprehensive indexing of legitimate content.
Solution Approach 2:
The patent extracts and analyzes specific features from URLs such as subdomain prefixes, domain registration data, and hyperlink patterns to identify spam characteristics. By separating spam detection as an independent preprocessing step, the system removes harmful content from the crawling pipeline before resources are consumed, achieving both reliability and efficiency.
2Reliability
If manual or machine learning-based spam detection methods are used, then spam filtering capability is improved, but operational cost and complexity increase
Solution Approach 1:
The patent implements self-updating statistical models that automatically learn from newly discovered spam patterns and domain behaviors. The system continuously refines its detection algorithms by analyzing crawled data, eliminating the need for manual model retraining and reducing operational complexity while maintaining high detection accuracy through adaptive learning.
Solution Approach 2:
The patent incorporates feedback loops where detection results from crawled content are fed back into the statistical models to improve future spam identification. The system uses real-world spam examples encountered during crawling to continuously enhance its detection capabilities, creating a self-improving system that reduces complexity over time through automated learning rather than manual intervention.
Data Source
AI summary
Embodiments of the technology described programmatically decrease the number of spam Uniform Resource locators (URLs) that are accessed from untrusted domains when the subdomain prefix is above a threshold probability of having been randomly generated. In this regard, prior to adding a discovered set of URLs to a crawl queue of a web crawler, the URLs are filtered into URLs from trusted domains and untrusted domains determined by a statistical model. The trusted domain URLs are added to the crawl queue, and the remaining URLs are sandboxed to filter out spam URLs. The subdomain prefixes of the sandboxed URLs are applied to a neural network to determine the probability that the subdomain prefixes are randomly generated. When a subdomain prefix is above a threshold probability of having been randomly generated, the subdomain is determined to be a spam subdomain and can be blocked.