Autonomous Program Detection for Scraping Request Blocking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Servers are often overloaded by requests from both legitimate and malicious actors, including autonomous programs that scrape content, leading to service disruptions and resource insufficiencies.
Innovation Solution
A classification scheme is established based on client behavior patterns, distinguishing between legitimate users and autonomous programs by analyzing request characteristics such as burst size, total requests, and content types, allowing for the detection and blocking of autonomous program activity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the server handles all requests from clients, then legitimate user access is maintained, but the server becomes overloaded by malicious scraping requests
Solution Approach 1:
The system performs preliminary classification of client behavior patterns before processing requests. By analyzing request frequency, content types, and burst sizes in advance, the system identifies autonomous programs and blocks them proactively, preventing server overload before it occurs. This allows the server to maintain resource sufficiency while continuing to serve legitimate users.
Solution Approach 2:
The classification scheme acts as an intermediary between the client and the server. It analyzes request patterns and makes decisions about whether to allow or block requests, filtering out malicious scraping traffic before it reaches the server. This mediator approach protects server resources while maintaining service availability for legitimate users.
2Object-affected harmful factors
If the server blocks requests from autonomous programs, then scraping attacks are prevented, but legitimate user access may be interrupted
Solution Approach 1:
The system applies different treatment to different clients based on their local behavior patterns. By analyzing individual client characteristics such as request frequency, content type diversity, and burst patterns, the system makes localized decisions to block only those exhibiting autonomous program behavior while allowing legitimate users to access the service normally.
Solution Approach 2:
The system monitors changes in request parameters over time to distinguish between legitimate users and autonomous programs. By analyzing variations in request frequency, content types, and temporal patterns, the system can dynamically adjust its blocking decisions, ensuring that legitimate users are not mistakenly blocked while effectively identifying and blocking scraping operations.
Data Source
AI summary
Systems and methods for scraping detection include a device which receives a plurality of requests from a client to extract data from a resource. The device may classify activity of the client as activity of an autonomous program based at least on a number of the plurality of requests, and one of i) one or more content types of the requests, or ii) a frequency in which the requests are received. The device may block, responsive to classification of the activity, a subsequent request from the client to extract data from the resource.