Web Scraping Latency Reduction via Resource Path Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Web scraping technologies face challenges in efficiently retrieving data from web pages without being blocked by web servers, as they drain server resources and trigger defensive measures, making it difficult to provide users with additional information without affecting user experience.
Innovation Solution
A method and system for near real-time web scraping that differentiates between textual and non-textual resources, requesting textual resources over a faster network path and non-textual resources over a slower path to minimize latency, allowing for the generation of a modified web page with augmented content while avoiding detection by web servers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If web scraping is performed using standard methods, then data can be retrieved from web pages, but web servers will detect and block the scraping activity
Solution Approach 1:
The patent segments the web page resources into two categories: textual resources (HTML, CSS, JavaScript) and non-textual resources (images, videos, audio). By separating the retrieval paths for different resource types, the system can optimize for scraping speed while maintaining natural browsing patterns that avoid detection
Solution Approach 2:
The patent introduces an intermediary system that sits between the user agent and the web server, managing resource requests through different network paths. This intermediary layer can manipulate request timing and patterns to appear more like legitimate user behavior, reducing the likelihood of server-side blocking
2Productivity
If all resources are requested over the fastest network path, then scraping speed is maximized, but latency increases and user experience is impacted
Solution Approach 1:
The patent applies different quality levels of network paths to different resource types based on their specific needs. Textual resources, which are critical for scraping and have smaller sizes, are retrieved over low-latency paths. Non-textual resources, which are less critical for data extraction and larger in size, are retrieved over higher-latency paths, optimizing the overall scraping efficiency while managing latency
3Adaptability or versatility
If web scraping is performed, then additional information can be provided to users, but server resources are drained
Solution Approach 1:
The patent implements partial action by selectively scraping only the textual resources (HTML, CSS, JavaScript) that are necessary for providing augmented information to users, while deliberately excluding non-textual resources (images, videos, audio). This partial scraping approach reduces server resource consumption while still achieving the goal of enhancing user experience with additional information
Data Source
AI summary
A system and method perform near real time scraping of a web page to generate a modified web page. The method includes receiving a uniform resource locator (URL) request directed at a web server; detecting in the URL request a textual resource and a non-textual resource; generating a request for the textual resource over a first network path including a first determined latency; generating a request for the non-textual resource over a second network path including a second determined latency, which is greater than the first determined latency; receiving the textual resource and the non-textual resource; scraping data from the textual resource; and generating a modified web page based on the scraped data, the textual resource, and the non-textual resource.