Distributed Mobile Web Scraping for Credential Security and Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data scraping techniques are inflexible, insecure, and prone to latency issues due to server dependence, which can be hindered by enhanced security features and server downtime.
Innovation Solution
Distributed data scraping using mobile devices that collect and upload data under user-specified conditions, eliminating the need for server storage of user credentials and allowing direct download, thereby enhancing security and reducing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data scraping is performed from a server, then centralized data collection is achieved, but security is compromised due to server storage of user credentials and latency issues occur due to server dependence
Solution Approach 1:
The patent segments the data scraping function from the central server and distributes it to multiple mobile devices. Each mobile device independently performs scraping operations locally, eliminating the need for the server to store user credentials or coordinate scraping activities. This segmentation resolves the security issue by removing credential storage from the server while maintaining system functionality through distributed execution.
2Ease of operation
If data scraping is performed from a server, then centralized control is maintained, but user convenience deteriorates due to server downtime interruptions and latency
Solution Approach 1:
The patent enables mobile devices to autonomously perform data scraping operations without requiring server coordination or authentication. The mobile devices self-manage the entire scraping process including authentication with target websites, data collection, and local storage. This self-service capability eliminates service availability dependencies on the central server, allowing users to access their data even when the server is down.
3Adaptability or versatility
If conventional data scraping is used, then server-based processing is simplified, but adaptability to enhanced security features deteriorates when target sites prevent server login
Solution Approach 1:
The patent inverts the traditional scraping architecture by making the mobile device (the endpoint) the originator of scraping requests rather than the server. This inversion allows the scraping operations to originate from registered mobile devices that have legitimate access to target websites, thereby bypassing server login restrictions imposed by target sites. The mobile devices maintain productivity by independently executing scraping tasks without being blocked by server-based authentication barriers.
Data Source
AI summary
Methods, systems and computer program products implementing data aggregation using distributed Web scraping are disclosed. A mobile device can scrape one or more target sites to collect data from accounts of a particular user. The scraping can occur under scraping conditions as specified by the user. The scraping conditions can include conditions based on time, power, bandwidth, usage, or any combination of the above. The scraping conditions can ensure that the scraping occurs at time that is most convenient to the user, e.g., when sufficient bandwidth is available to the mobile device or the mobile device is not performing other tasks. The mobile device can upload the scraped data to a data aggregation server under submission conditions as specified by the user. The data aggregation server can aggregate the scraped data, enrich the aggregated data, and provide the enriched data to the user through Web access.