External Stage Replication for Cloud Data Lake Fail-Over
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems fail to seamlessly replicate external stages across different deployments of a cloud data lake, leading to manual and error-prone processes during fail-over, which disrupt data loading and continuity in case of outages.
Innovation Solution
Implement a modified storage integration that can point to multiple storage locations, allowing seamless replication of external stages across deployments by using a storage integration with defined active and base file paths, ensuring data loading processes can be automatically continued in case of fail-over.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If external stages are replicated across deployments manually, then data loading continuity can be maintained during fail-over, but the process becomes error-prone and requires manual scripting
Solution Approach 1:
The patent implements automatic copying of external stage definitions from a source deployment to a target deployment using storage integrations. The external stage configuration including source storage location, target storage location, and credentials are replicated automatically through the data replication service, eliminating manual copying efforts and reducing errors while maintaining data loading continuity during fail-over scenarios
Solution Approach 2:
The system enables self-service replication by allowing the data replication service to automatically manage the replication of external stages without requiring user intervention. The service autonomously copies storage integration definitions, manages credential security, and configures external stages in the target deployment, freeing users from manual scripting and configuration tasks
2Adaptability or versatility
If storage integration points to multiple storage locations, then seamless fail-over between deployments is enabled, but the storage integration complexity increases
Solution Approach 1:
The storage integration is designed with multi-functionality to serve multiple storage locations across different deployments. A single storage integration object can reference multiple source and target storage locations, enabling the same integration to facilitate data replication to different cloud storage accounts and regions, thereby providing seamless fail-over capability while managing complexity through a unified interface
Solution Approach 2:
The data replication service acts as an intermediary that manages the complexity of multi-location storage integrations. It handles the configuration, credential management, and coordination between multiple storage locations, abstracting the complexity from the user while enabling adaptable fail-over capabilities through standardized external stage definitions
3Device complexity
If external stages are not replicated automatically, then system simplicity is maintained, but manual intervention is required during fail-over which disrupts data loading
Solution Approach 1:
The system performs preliminary action by automatically replicating external stage definitions to the target deployment before a fail-over event occurs. The data replication service proactively copies storage integration configurations and external stage metadata, ensuring that the target deployment is pre-configured and ready to immediately resume data loading operations without disruption, thereby maintaining productivity while managing complexity through automated processes
Data Source
AI summary
Described herein are techniques for replicating external stages between deployments of e.g., a cloud data lake using a modified storage integration. The modified storage integration may be defined with a set of storage locations. The storage integration comprises a base URL for each of the set of storage locations and each storage location identifies a remote deployment where a cloud platform is hosted and a geographic region of the remote deployment. An external stage object may be bound to the storage integration, where the external stage object facilitates a data loading operation that is currently in progress on the first storage location. In response to detecting an outage at the first storage location, the data loading operation that was in progress on the first storage location may be continued at the second storage location using the storage integration.