Cloud Data Migration via Virtual Compute Instance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face inefficiencies in migrating and consolidating data across cloud buckets, resulting in high egress costs, network bandwidth consumption, and latency due to the need for multiple data transfers and staging storage requirements.
Innovation Solution
The method involves generating a virtual compute instance, or container, on a cloud platform to identify and copy data segments from a source cloud bucket to a destination cloud bucket, while generating metadata for the destination and storing it locally, thereby reducing the need for data recall to on-premises systems and minimizing network traffic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If data is migrated from on-premises active tier to cloud object storage, then storage cost is reduced and long-term retention is improved, but egress costs and network bandwidth consumption increase during consolidation operations
Solution Approach 1:
The system creates a virtual compute instance that copies data segments from source cloud bucket to destination cloud bucket using metadata identification. This copying approach avoids the need to recall data to on-premises systems and re-upload, eliminating egress costs while maintaining the cloud-to-cloud migration benefit
Solution Approach 2:
A virtual compute instance acts as an intermediary between source and destination cloud buckets. This intermediary performs the data copying operation directly in the cloud environment, serving as a mediator that eliminates the need for data to leave the cloud ecosystem, thus avoiding egress fees while still enabling consolidation
2Productivity
If data is recalled from cloud bucket to on-premises system for migration, then data consolidation is achieved, but network latency and bandwidth consumption increase
Solution Approach 1:
The system performs preliminary actions by creating a virtual compute instance in the cloud environment before any data movement occurs. This instance is pre-configured with metadata from the source bucket, enabling it to directly identify and copy data segments without needing to recall data to on-premises systems first, thus eliminating network latency
Solution Approach 2:
The solution shifts the migration operation from the on-premises dimension to the cloud dimension by executing the copy operation within the cloud environment itself. This dimensional change allows data to remain in the cloud throughout the migration process, accessed and manipulated through cloud-based metadata rather than requiring physical recall to on-premises systems
3Productivity
If conventional migration methods are used, then data can be moved between cloud buckets, but staging storage requirements increase device complexity
Solution Approach 1:
The system extracts only the essential metadata from the source cloud bucket and provides it to the virtual compute instance. This extraction approach eliminates the need for staging storage of actual data segments, as the virtual instance can directly identify and copy data using metadata alone, significantly reducing device complexity
Solution Approach 2:
The virtual compute instance performs self-service by autonomously identifying data segments to migrate using the provided metadata, copying them directly from source to destination, and generating new metadata for the destination bucket. This self-service capability eliminates the need for external staging storage infrastructure, simplifying the overall system
Data Source
AI summary
A method of migrating or consolidating cloud data includes generating a container on a cloud platform and receiving, at the container, source metadata identifying a set of data to be migrated from a source cloud bucket associated with a source data domain to a destination cloud bucket associated with a destination data domain. The method further includes copying, by the container, set of data from the source cloud bucket to the destination cloud bucket based on the source metadata and generating, by the container, destination metadata for the set of data as stored at the destination cloud bucket.