Network Page Obfuscation for Automated Data Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Web scraping technologies consume resources and impose costs on web site owners by extracting data without control over usage, as the robots exclusion standard is advisory and not enforceable, allowing malicious scrapers to ignore it.
Innovation Solution
Implementing network page obfuscation techniques that randomly alter the document structure of web pages to inhibit automated data extraction, while maintaining the visual appearance for users, and encouraging a service-based architecture with API access to monitor and control content usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If web scraping is allowed to operate freely, then data extraction is easy and automated, but resource consumption increases and website owners lose control over content usage
Solution Approach 1:
The patent applies dynamics by making the document structure non-static and unpredictable. The website owner can dynamically change the structure of network pages (e.g., altering HTML tags, attribute names, or element positions) in response to detected scraping activities, thereby preventing automated extraction while maintaining legitimate access.
Solution Approach 2:
The system implements feedback mechanisms where the website owner receives notifications when scraping is detected and can respond by modifying the page structure. This feedback loop allows the owner to control resource consumption by adjusting the scraping behavior based on real-time detection and response.
2Reliability
If the robots exclusion standard is enforced strictly, then automated scraping is prevented, but the standard becomes advisory and unenforceable against malicious scrapers
Solution Approach 1:
The patent applies preliminary anti-action by proactively detecting scraping activities and preemptively modifying the document structure before malicious scrapers can extract data successfully. This prevents harmful factors by establishing a defensive mechanism that anticipates and counteracts scraping attempts.
Solution Approach 2:
The system provides feedback to website owners about scraping detection, enabling them to take controlled actions to prevent malicious scraping. This feedback mechanism transforms the advisory robots exclusion standard into an enforceable system where owners can actively respond to scraping threats.
3Object-affected harmful factors
If network page structure is made complex and changing frequently to prevent scraping, then automated data extraction is inhibited, but the visual appearance and user experience may be affected
Solution Approach 1:
The patent applies local quality by making structural changes only in specific areas of the document that do not affect the visual appearance. The owner can modify HTML tags, attributes, or element positions in ways that are invisible to users but confusing to scrapers, thereby preventing extraction while maintaining user experience.
Solution Approach 2:
The system can change the 'visual characteristics' of the document structure in ways that are imperceptible to users, similar to color changes that are invisible to the human eye. The structure changes are designed to be semantically equivalent but syntactically different, preserving appearance while preventing scraping.
Data Source
AI summary
Disclosed are various embodiments for inhibiting or preventing automated data extraction from network pages. A source for a network page having a document structure is obtained. An obfuscated network page is generated from the network page by altering the document structure to inhibit automated extraction of data. The obfuscated network page is configured to have a visual appearance that is the same as that of the network page when rendered by a client for display. The obfuscated network page is sent to the client in response to a request from the client for the network page.