Resilient Data Transfer Mechanism for Bulk to Streaming Transition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cloud computing environments, data migrations from one system to another, especially large volumes of data, face challenges in ensuring no data loss during state transitions from bulk data transfer to continuous streaming, particularly when the initial data transfer exceeds the default data retention limit.
Innovation Solution
The implementation of a resilient data transfer mechanism that utilizes an intermediate persistent storage for bulk and streaming traffic, allowing data to be stored temporarily if the initial transfer exceeds the retention window, ensuring data integrity and decoupling dependency on bulk data transfer limits, with features like versioning and dynamic storage allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If bulk data transfer is performed without intermediate storage, then system resource costs are reduced, but data loss occurs when transfer exceeds retention window
Solution Approach 1:
The patent introduces an intermediate persistent storage table as a mediator between the bulk data transfer process and the streaming data transfer process. This intermediate storage temporarily holds data that arrives during the bulk transfer window, allowing the system to maintain data integrity without requiring the bulk transfer to complete within the default retention window. The intermediate storage acts as a buffer that decouples the timing constraints of bulk transfer from streaming transfer requirements.
2Reliability
If data is stored temporarily in intermediate storage, then data loss is prevented during state transitions, but system resource costs increase
Solution Approach 1:
The patent segments the data transfer process into distinct phases: bulk transfer phase, transition phase, and streaming transfer phase. The intermediate persistent storage is specifically activated during the transition phase to handle data that arrives during state changes. This segmentation allows temporary storage to be used only when necessary, rather than continuously, thereby reducing overall resource consumption while maintaining data integrity during critical transition periods.
Solution Approach 2:
The intermediate persistent storage is designed to be temporary and disposable. Data is stored in the intermediate storage only during the bulk-to-streaming transition period, and once the transition is complete and data is successfully transferred to the streaming process, the intermediate storage is cleared or discarded. This approach minimizes long-term resource occupation while providing protection during the critical transition window.
3Productivity
If bulk data transfer completes within default retention window, then no intermediate storage is needed, but data loss occurs when transfer exceeds the window
Solution Approach 1:
The patent implements preliminary actions by establishing the intermediate persistent storage structure before the bulk-to-streaming transition begins. This pre-prepared infrastructure ensures that when data arrives during the bulk transfer process, especially near or beyond the retention window, there is already a designated storage location ready to capture and preserve that data. This preliminary preparation eliminates the risk of data loss without requiring the bulk transfer to complete within strict time constraints.
Data Source
AI summary
An indication to migrate requested data objects stored in a source database environment to a destination database environment is received. Some of data objects have many-to-one relationships with other data objects in the source database environment. At least one snapshot file generated by the source database environment is transferred to a destination database environment in bulk transfer mode. Subsequent incoming data received during bulk transfer mode, after the indication is stored in a temporary table. Upon completion of migration of the requested data objects the system transitions from bulk transfer mode to streaming mode. The subsequent incoming data from the temporary table is transferred to the destination database environment in response to the transition to streaming mode. Additional data received after the temporary table is empty is transferred from the source database environment to the destination database environment without use of the temporary table.