Cloud Scraping System Using Pre-Scraped Big Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cloud scraping technologies face inefficiencies due to increased throughput with multiple scraping requests and types of targets, as they perform scraping for each request individually, leading to slower response times and reduced server operation efficiency.
Innovation Solution
A cloud scraping system and method that pre-scrapes information in a big data storage for quick responses to user requests, utilizing a scraping server to handle dynamic information and employing single or multi-processing servers based on external institution policies, with an error analysis and update mechanism for user terminals.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If cloud scraping performs scraping for each scraping request individually, then it can handle various types of scraping requests, but the throughput decreases as the number of scraping requests or types of scraping target external institutions increases
Solution Approach 1:
The patent pre-scrapes information from external institutions and stores it in a big data storage system before receiving user requests. This preliminary action allows the system to quickly retrieve pre-scraped data when requests arrive, avoiding the need to perform scraping operations in real-time for each request, thereby maintaining high throughput while handling diverse scraping targets
Solution Approach 2:
The patent divides the scraping system into multiple scraping servers that can operate independently and concurrently. Each scraping server can handle different external institutions or types of requests, allowing the system to maintain adaptability across various scraping targets while improving overall throughput through parallel processing
2Reliability
If cloud scraping performs scraping for each scraping request, then it ensures data freshness, but the response time increases with more scraping requests
Solution Approach 1:
The system performs scraping operations in advance and stores the scraped data in big data storage. When users submit requests, the system retrieves pre-scraped data immediately, significantly reducing response time while maintaining acceptable data freshness for static information that does not change frequently
Solution Approach 2:
The patent implements a dynamic scraping strategy where the system determines whether to use pre-scraped data or perform real-time scraping based on the type of information requested. For dynamic information that changes frequently, the system performs real-time scraping to ensure data freshness, while for static information, it uses pre-scraped data to minimize response time
Data Source
AI summary
A cloud scraping system using pre-scraped big data includes an information providing server which, when receiving a scraping request from a user terminal, provides the user terminal with response information to the received scraping request, and a big data storage which stores pre-scraped information, wherein when the scraping request is about static information, the information providing server acquires the response information using the pre-scraped information. According to the above cloud scraping system using pre-scraped big data, there is an advantage that it is possible to quickly respond to a scraping request from the user terminal afterwards by pre-scraping and storing static information in the big data storage. Additionally, it is possible to improve the scraping server operation efficiency by making a proper use of a single or multi-processing scraping server based on policy information of a scraping target external institution.