Universal Cloud Data Migration Framework via Direct Streaming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data migration tools between cloud storage systems face challenges in performance, flexibility, scalability, automation, and data verification, particularly when dealing with large data volumes and varying network conditions, leading to inefficiencies and repetitive development needs for each cloud storage system combination.
Innovation Solution
A universal data migration framework that generates descriptive information for files, transmits migration requests to servers, and retrieves files from the source to write directly to the destination, using a client-server architecture with Thrift RPC communication, allowing for scalable and flexible data migration across different cloud storage systems without intermediate disk storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is written into a disk during migration, then data can be stored temporarily, but unnecessary disk I/O is generated and migration efficiency is reduced
Solution Approach 1:
The patent extracts the intermediate disk storage step from the migration process. Instead of writing data to local disks as an intermediate step, the system directly streams data from the source cloud storage system to the destination cloud storage system through a coordinated pull-pull mechanism, eliminating unnecessary disk I/O operations while maintaining data integrity
Solution Approach 2:
The patent introduces a migration management system as an intermediary that coordinates between source and destination systems. This mediator manages the migration tasks, handles authentication, and orchestrates the direct data transfer without requiring intermediate local storage, thus improving efficiency while reducing energy loss from disk operations
2Adaptability or versatility
If migration tools are developed for every cloud storage system combination, then specific migration needs are met, but development work becomes repetitive and intensive
Solution Approach 1:
The patent implements a universal migration management system that can handle multiple cloud storage systems through a standardized interface. The system uses configuration files to define source and destination systems, allowing the same core migration engine to work across different cloud providers without requiring separate tool development for each combination, thus reducing development workload while maintaining broad compatibility
Solution Approach 2:
The patent uses parameter-based configuration to adapt the migration system to different cloud storage systems. By changing parameters in configuration files (such as authentication credentials, endpoint URLs, and system-specific settings) rather than modifying the core code, the system can support multiple cloud providers, reducing development complexity while maintaining versatility
3Productivity
If migration speed is increased to handle large data volumes, then migration time is reduced, but system performance and network stability may be affected
Solution Approach 1:
The patent implements dynamic speed adjustment in the migration process. The migration management system monitors network conditions and system performance during migration, automatically adjusting the transfer speed to optimize both efficiency and stability. This dynamic control allows the system to maintain high speeds when conditions permit while reducing speed when network or system stability is at risk
Solution Approach 2:
The patent incorporates feedback mechanisms where the migration management system continuously monitors the migration progress, network conditions, and system performance metrics. Based on this feedback, the system automatically adjusts migration parameters such as parallel transfer threads and data chunk sizes to maintain optimal speed while ensuring network stability and system reliability
4Ease of operation
If data migration is performed interactively over a network, then flexibility is maintained, but uncertainties arise and automation becomes difficult
Solution Approach 1:
The patent performs preliminary actions by pre-configuring migration tasks with all necessary parameters including authentication credentials, source and destination definitions, and migration policies. The system prepares migration plans in advance and automatically executes them without requiring interactive input during the actual data transfer, thus achieving high automation while maintaining flexibility through pre-configurable options
Solution Approach 2:
The migration management system provides self-service capabilities by automatically handling authentication, error recovery, and migration orchestration. The system can independently manage the entire migration process from initiation to completion, reducing the need for manual intervention while maintaining flexibility through configurable migration policies and parameters that adapt to different scenarios
Data Source
AI summary
The objective of the present invention is to provide a method, apparatus, system, computing device and computer-readable medium for data migration between cloud storage systems. Here, a user equipment generates, according to a list of to-be-migrated files, descriptive information of each file in the list; and transmits one or more data migration requests to available server(s), respectively, such that the corresponding server performs data migration between different cloud storage systems, wherein each data migration request includes indication information of a source end and a destination end of the data migration and descriptive information of one or more to-be-migrated files. The present invention may satisfy various performance requirements on performance, flexibility, scalability, automation, data verify and framework universality, etc.