Script Engine Data Scraping Across Operating Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current screen scraping technologies require separate development of scraping modules for each operating system, leading to increased costs and efforts due to the need for conforming to different security policies and operating systems, resulting in performance degradation and inefficiencies, especially in client-side screen scraping for financial institutions.
Innovation Solution
A data scraping system with a script engine that stores environment information, allowing scraping to be processed from various operating systems in a single module, generates communication sessions, and transmits authentication values based on the target institution's methods, enabling operation across multiple platforms without the need for separate module development.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If separate scraping modules are developed for each operating system, then compatibility with different security policies is improved, but device complexity and development costs increase
Solution Approach 1:
The patent implements a universal scraping module that can operate across multiple operating systems (Windows, Linux, macOS, mobile OS) without requiring separate module development. The module uses platform-independent script languages and standardized communication protocols to achieve multi-functionality, eliminating the need to maintain separate scraping modules for each OS while still complying with different security policies through configurable authentication methods.
Solution Approach 2:
The patent introduces an intermediary layer consisting of a configuration file and script engine that mediates between the scraping module and different operating systems. The configuration file stores OS-specific parameters and authentication methods, allowing the single scraping module to adapt to different environments without code modification. This intermediary approach enables the module to handle various security policies while maintaining a unified codebase.
2Reliability
If separate scraping modules are developed for each operating system, then security policy compliance is improved, but development time and costs increase
Solution Approach 1:
The scraping module is designed with universal functionality that works across all major operating systems including Windows, Linux, macOS, and mobile OS. By using platform-independent scripting and standardized communication protocols, the module achieves security policy compliance on all platforms without requiring separate development cycles, thereby reducing development time while maintaining reliability.
Solution Approach 2:
The patent uses parameter-based configuration to adapt to different operating systems and security policies. Instead of developing separate modules, the system changes runtime parameters (such as authentication methods, protocol types, and system-specific paths) based on the detected operating system. This allows a single module to comply with various security policies by dynamically adjusting its behavior rather than requiring separate development for each policy.
3Reliability
If sequential screen scraping is performed, then service stability is improved, but productivity decreases
Solution Approach 1:
The patent implements a dynamic scraping architecture that can adapt its execution mode based on requirements. The system supports both sequential execution (for stability-critical operations) and parallel execution (for performance-critical operations) through configurable task management. The script engine dynamically adjusts the scraping strategy based on the target website characteristics, authentication requirements, and system resources, thereby achieving both stability and productivity.
Solution Approach 2:
The scraping process is segmented into independent, modular tasks that can be executed sequentially or in parallel. Each scraping task is divided into discrete steps (authentication, data collection, data extraction) that can be independently managed. This segmentation allows the system to maintain service stability through controlled sequential execution when needed, while enabling parallel processing of multiple independent tasks to improve overall productivity.
Data Source
AI summary
A data scraping system includes: a management module having a service program that calls inquiry or execution requiring scraping to receive user information for scraping and allocate user information and a task using an internal scheduling algorithm; and a scraping module connected to a plurality of different institutions, allowing communication, based on the user information and the task allocated by the management module, to scrape data corresponding to the called inquiry or execution from the plurality of institutions. The scraping module includes a meta database to designate a data item to be scraped, and is configured to extract data to scrape based on the meta database from a data set provided by the institution. According to the data scraping system, it is possible to enable data scraping in various platforms such as mobile computing devices, and collect multiple scraping information in one device without constraints of a security module.