Distributed Scraping Service Architecture

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional scraping services require frequent updates and redistributions when target web sites change, leading to inconvenience for users and inefficiencies in managing authentication methods and system requirements.

Innovation Solution

A distributed method for providing a scraping service using a client terminal and remote server, where scraping rule information defines tasks that can include local and remote commands, allowing for efficient data collection and minimization of application updates by executing tasks asynchronously and controlling command execution remotely.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a separate scraping module is configured for each target web site to directly cope with changes, then the ability to handle target site changes is improved, but the application requires frequent updates and redistribution causing user inconvenience

Engineering Contradiction:

Improveability to handle target site changesVSAvoiduser convenience

Core Design Contradiction:

Adaptability or versatilityVSEase of operation

Solution Approach 1:

The scraping system is divided into independent scraping modules, each responsible for specific target websites. These modules are managed separately and can be updated independently through a scraping rule server, allowing the main application to remain stable while individual scraping functionalities are updated as needed.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

A scraping rule server is introduced as an intermediary between the scraping modules and the main application. This server manages scraping rules and coordinates updates, allowing modules to be updated without requiring application redistribution, thus resolving the contradiction between adaptability and ease of operation.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If the scraping module is updated continuously to monitor changes in target web sites, then the responsiveness to target site changes is improved, but the complexity of managing and updating multiple scraping modules increases

Engineering Contradiction:

Improveresponsiveness to target site changesVSAvoidcomplexity of managing scraping modules

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The scraping rule server provides universal functionality for managing all scraping modules. It handles rule distribution, update coordination, and monitoring for multiple target websites through a single centralized system, reducing the overall complexity despite the number of scraping modules.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

Scraping rules are prepared and validated in advance on the scraping rule server before being deployed to scraping modules. This preliminary action ensures that modules are updated with pre-tested rules, reducing the complexity of real-time management and monitoring.

Inventive Principle:

Principle #10Preliminary action

3Adaptability or versatility

If the application is reinstalled every time to use the service, then the service can be updated to reflect target site changes, but the user convenience and service continuity are degraded

Engineering Contradiction:

Improveservice update capabilityVSAvoidservice interruption time

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The scraping service maintains continuous operation through the scraping rule server architecture. When scraping rules need updates, the server distributes new rules to running modules without requiring application reinstallation, ensuring service continuity and eliminating user-facing interruptions.

Inventive Principle:

Principle #20Continuity of useful action

Solution Approach 2:

The scraping modules automatically receive and apply updated scraping rules from the scraping rule server without user intervention. This self-service mechanism allows the system to adapt to target site changes while maintaining service continuity, as users do not need to reinstall or manually update the application.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS11768891B2Method for providing scraping-based service and application for executing the same

Publication Date: 2023.09.26 KAKAOBANK CORP

AI summary

The disclosure relates to a method for providing a scraping-based service. The method for providing a scraping-based service includes: requesting and receiving at least one of scraping rule information defining a series of tasks for obtaining external data; collecting the external data by executing tasks according to the received rule information; and providing service information to a user using the collected external data. According to the disclosure, by providing a scraping service through a client terminal and a remote server in a distributed manner, it is possible to increase efficiency of a scraping task and actively cope with changes in a scraping target.