Encrypted Interstitial Web Security for Scraper Threat Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Web applications face challenges in managing sophisticated automation attacks, such as those from scrapers, which result in fraud and disrupt user experiences, while existing security measures like CAPTCHA are easily bypassed and create user friction.
Innovation Solution
Implementing encrypted interstitial techniques that intercept webpage requests, generate encrypted webpages, and serve interstitial pages with instrumentation code to collect telemetry data. This allows for threat analysis to determine if the request is legitimate, enabling decryption of the webpage only if access is granted.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional CAPTCHA challenges are used to block scraper bots, then security against unwanted automation traffic is improved, but user friction increases and scrapers can easily bypass the challenge
Solution Approach 1:
The patent extracts the security verification process from the main user flow by implementing an interstitial page that loads separately. The verification logic is removed from the critical resource loading path, allowing legitimate users to access content without friction while scrapers are identified and blocked through background telemetry analysis of the interstitial page behavior
Solution Approach 2:
The interstitial page serves as an intermediary element between the user agent and the protected resource. It collects telemetry data about the user agent's behavior and environment without blocking the main resource loading, enabling security verification to occur in parallel rather than sequentially, thus eliminating user friction while maintaining security effectiveness
2Reliability
If an interstitial page is used to verify user agents before accessing protected resources, then security against scrapers is improved, but resource loading time increases due to additional network requests
Solution Approach 1:
The interstitial page is loaded preliminarily as part of the normal resource loading process, allowing telemetry data collection to begin before security verification is complete. This preliminary action enables the security check to occur in the background while the user experience progresses, eliminating perceived waiting time while maintaining security effectiveness
Solution Approach 2:
The patent maintains continuity of useful action by allowing the main resource loading to proceed uninterrupted while the interstitial page simultaneously collects telemetry data for security verification. The security verification continues in parallel with resource loading rather than blocking it, ensuring that legitimate users experience no delay while security checks are performed
3Reliability
If all sensitive resources are placed behind authentication flow requiring login, then security is improved, but accessibility is reduced for applications that do not involve user accounts
Solution Approach 1:
The patent applies local quality by implementing security verification only for specific protected resources rather than requiring authentication for all content. The interstitial page selectively applies security checks to sensitive resources while leaving other content accessible, allowing applications without user accounts to remain fully accessible while still protecting specific valuable resources from scraper bots
Data Source
AI summary
A method, non-transitory computer readable medium, device and system intercepts a request for a webpage including a protected resource. The request is sent from a client device to a server device. One or more link tags or other mechanisms corresponding to one or more sub-resources included in the webpage are generated. The webpage is encrypted. An interstitial page is served to the client device that includes an encrypted portion that includes the encrypted webpage, an unencrypted portion that includes the link tags, and instrumentation code that collects telemetry data when executed at the client device. The telemetry data is received from the client device and a threat analysis is performed on the telemetry data. A decryption key is transmitted to the client device in response to determining that the request is allowed, based on the performed threat analysis. The decryption key allows the client device to decrypt the encrypted webpage.