Scalable Data Extractor for Legacy Cloud Migration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data transfer from legacy software systems to cloud computing systems is challenging due to reliability issues, scalability problems, and the need for dynamic resource management, especially when dealing with large data volumes and intermittent connectivity across different infrastructure providers.
Innovation Solution
A data extraction system that extracts data in a scalable and fault-tolerant manner by querying the source system on a per-page basis, retrying failed queries, and using message-oriented middleware for coordination, allowing for horizontal scalability and statelessness, which enables efficient data transfer even in constrained environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If bulk extraction of data through web services is used to transfer data from legacy software systems, then data transfer capability is improved, but system reliability deteriorates due to time-outs and errors
Solution Approach 1:
The patent divides the bulk data extraction process into smaller page-level extraction units. Each page is extracted independently through web service calls, allowing the system to handle large data volumes while maintaining reliability. If a page extraction fails, only that specific page needs to be retried rather than re-extracting all data.
Solution Approach 2:
The system performs preliminary actions by establishing retry mechanisms and error handling protocols before bulk extraction begins. The patent implements a queue-based approach where extraction requests are prepared and managed in advance, with automatic retry logic configured to handle potential time-outs and service unavailability.
2Productivity
If computing resources are scaled up for data extraction processes, then extraction performance is improved, but resource management complexity increases due to dynamic scaling requirements
Solution Approach 1:
The patent implements dynamic resource allocation where extraction workers are spawned and terminated based on queue depth and system load. The system automatically scales computing resources up during extraction operations and down during idle periods, maintaining high performance while simplifying resource management through automated orchestration.
Solution Approach 2:
The extraction system performs self-service by automatically managing its own resource allocation. Workers monitor queue depth and autonomously initiate extraction tasks or terminate when complete, without requiring external resource management intervention. This self-managing approach reduces operational complexity while maintaining extraction performance.
3Productivity
If data extraction is performed in bursts to complete transfers quickly, then productivity is improved, but system stability deteriorates due to high I/O load
Solution Approach 1:
The patent implements periodic action by processing data in structured batches or pages with controlled frequency. Rather than continuous burst extraction, the system processes a defined number of pages, pauses to allow system stabilization, then resumes extraction. This periodic approach maintains transfer productivity while preventing sustained high I/O load that would destabilize the system.
Data Source
AI summary
Methods, systems, and computer-readable storage media for determining, by an extraction worker of a data extraction system, a number of pages based on data that is to be extracted from the source system, for each page, by a respective extraction worker: querying the source system, and storing a query result in a respective page file within a staging data file system of the data extraction system, for each extraction worker that queries the source system, verifying a status of the extraction worker after expiration of a predetermined delay time based on a timestamp of the page that the extraction worker is handling, and determining that a page status for each page is completed, and in response, posting an extraction completed message to inform the consuming system that the data has been extracted from the source system.