URL Redirect Analysis System for Spam Identification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current link-following and security systems fail to efficiently and reliably analyze URLs in messages due to the complexity introduced by URL redirects, URL shortening services, and DOM manipulation, leading to difficulties in identifying and blocking spam messages.
Innovation Solution
A method and system that automatically obtain web page content by analyzing incoming messages for URLs, initiating a URL redirect analysis process to follow and identify types of redirects, and applying specific processing procedures to traverse redirects, thereby accessing the end destination web page content.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional URL analysis methods are used to activate and analyze web page content, then spam identification can be performed, but the presence of nested redirects makes it impossible to efficiently access the actual URL content in a reasonable amount of time
Solution Approach 1:
The system performs preliminary actions by pre-fetching and caching web page content and redirects before they are needed for spam analysis. When a URL is encountered in a message, the system checks if the content is already cached, avoiding the need to traverse nested redirects in real-time and enabling rapid spam identification.
Solution Approach 2:
The system prepares for potential redirect scenarios by implementing a caching mechanism that stores web page content and redirect information in advance. This cushioning approach ensures that even if nested redirects are encountered, the system has pre-retrieved content available, preventing timeouts and ensuring reliable spam analysis.
2Object-affected harmful factors
If URL shortening services are used to mask actual spam URLs, then spam can be hidden from analysis, but this complicates the URL analysis process and prevents efficient content retrieval
Solution Approach 1:
The system introduces an intermediary caching layer between the URL analysis process and the web page content retrieval. When analyzing a URL, the system first checks the cache for pre-fetched content, avoiding the need to actively traverse redirect chains and resolve URL shortening services in real-time, thereby simplifying the analysis complexity while maintaining the ability to detect masked spam.
3Reliability
If multiple types of redirects are employed by spammers to frustrate analysis, then URL content can be protected from access, but this requires handling numerous different redirect types and methods
Solution Approach 1:
The caching system implements a universal approach that handles multiple redirect types (HTTP redirects, HTML Meta redirects, JavaScript redirects, DOM manipulation, tracking bugs) through a single mechanism. By pre-fetching and caching content regardless of the redirect type encountered, the system provides reliable content access without requiring complex, specialized handling for each redirect variant.
Data Source
AI summary
A method and apparatus for automatically obtaining web page content in the presence of redirects whereby an incoming message is received and analyzed to determine if the message contains any URLs. Any detected URLs are then extracted and activated to analyze the contents of the web page linked to by the URL. The HTTP response headers and content sent from a web page server in response to the browser HTTP requests to activate the URL link are analyzed to determine if the response includes a redirect to a new, or destination, URL, and associated web page, i.e., to determine if the detected URLs result in redirects. If the HTTP response indicates a redirect, a URL redirect analysis process is initiated that includes a set of redirect processing procedures that are selectively applied depending on the type of redirect encountered, and each redirect is automatically followed. For chains of redirects, the process is recursive, i.e., is repeated automatically for each redirect, from the beginning, and as if the new (destination) URL is itself an original URL.