Markup Obfuscation to Thwart Automated Scrapers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current techniques for preventing automated traffic from scraping or crawling websites are limited, leading to increased costs and resource strain for websites, as malicious bots can fetch large batches of data quickly, compromising sensitive information intended for human users.
Innovation Solution
Implementing a system that dynamically modifies and obfuscates markup documents, making it difficult for automated-agents to parse by transforming reference markers and elements, while ensuring human users can access information as needed without interference.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If signature detection and user blocking techniques are used to stop automated traffic, then some malicious bots can be blocked, but these techniques are insufficient to prevent sophisticated scrapers from accessing data at high velocity
Solution Approach 1:
The patent applies dynamics by making the markup document dynamic and randomized. Instead of static markup that bots can easily parse, the system generates different randomized markup documents for each request, transforming reference markers and elements dynamically. This resolves the contradiction by maintaining reliable protection against automated scraping (bots cannot parse randomized markup) while preserving productivity for human users (who receive properly rendered content through browser interpretation).
Solution Approach 2:
The patent applies parameter changes by transforming markup parameters such as reference markers, class names, and element attributes through randomization functions. The system changes the structural parameters of the markup document while maintaining the visual presentation, making it unreadable for bots but functional for human users. This resolves the contradiction by altering markup parameters to prevent automated access while preserving user experience.
2Ease of operation
If markup documents are left unmodified to ensure fast access for human users, then user experience is maintained, but automated agents can easily scrape and crawl data at high velocity
Solution Approach 1:
The patent applies segmentation by separating the markup document into two functional layers: a randomized structural layer that protects against scraping, and a presentation layer that ensures proper rendering for human users. The system segments the protection mechanism from the content delivery, allowing each to function independently. This resolves the contradiction by segmenting the markup into protective randomized elements and user-friendly display elements.
Solution Approach 2:
The patent applies the intermediary principle by introducing a markup transformation layer between the data source and the user interface. This intermediary system randomizes the markup structure while maintaining the underlying data integrity and visual presentation. It acts as a mediator that prevents direct access for bots while ensuring proper display for human users, resolving the contradiction between protection and ease of operation.
3Reliability
If traditional blocking methods are used to prevent bot access, then some malicious traffic is stopped, but these methods cause increased costs and resource strain for the website
Solution Approach 1:
The patent applies self-service by making the markup document itself the protection mechanism rather than relying on external blocking systems. The randomized markup structure inherently prevents bot access without requiring additional server resources for detection, analysis, or blocking operations. The system protects itself through the design of the markup rather than through active defense mechanisms, resolving the contradiction between reliable blocking and resource consumption.
Data Source
AI summary
Network resources are subject to automated downloading of information by automated-agents that cause business- or user-specific data to be compiled en masse. In order to control resources and information available to automated-agents, markup documents or elements of markup documents are obfuscated such that a client must reconfigure the obfuscated information in order to display the requested content contained in the document. The obfuscation is applied such that the content will render and display to a human user in the same manner as if the element was untransformed; however, the markup will appear syntactically different to an automated-agent. Thus, in response to a user request, a server can transform elements of a structured document and provide the transformed document to the client for reconfiguring and rendering.