Cloud Migration Bulk Upload Resumability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cloud migration systems face performance issues when migrating large amounts of data, as they require multiple API calls for each object, leading to slowed performance for other organizations and necessitate re-uploading entire sets of content if any item fails, which is inefficient and resource-intensive.
Innovation Solution
A cloud-based migration system exposes a source-independent API for uploading data as a single entity, using a migration package with a manifest and resumability processing to manage and resume migrations, allowing incremental updates and reducing the need for repeated uploads by storing data in a cloud-based storage system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple API calls are made for each object to be migrated, then migration flexibility is maintained, but system performance deteriorates and other organizations experience slowed service
Solution Approach 1:
The patent combines multiple individual API calls into a single bulk upload operation. Instead of calling the migration API separately for each object, the system accepts a single request containing multiple objects (files, folders, databases) and processes them collectively, thereby maintaining migration flexibility while dramatically improving system performance and reducing impact on other organizations.
Solution Approach 2:
The patent segments the migration process into distinct phases: content generation on the source system, single bulk upload to the migration system, and subsequent processing. This segmentation allows the system to maintain flexibility in what can be migrated while optimizing the transfer mechanism itself.
2Ease of operation
If data is loaded directly into the migration system, then migration simplicity is maintained, but resource consumption increases and failed migrations require complete re-upload
Solution Approach 1:
The patent implements preliminary action by generating a complete migration package (including manifest, content, and metadata) on the source system before upload. This package is self-contained and can be processed incrementally, eliminating the need for complete re-upload in case of failure and reducing resource consumption during migration operations.
Solution Approach 2:
The patent enables selective reprocessing of only failed objects within the migration package rather than discarding and re-uploading the entire dataset. The manifest allows the system to track which objects have been successfully processed and which require retry, thereby recovering partial progress and reducing resource waste.
3Reliability
If large data migration calls are made, then migration completeness is improved, but data ingestion rate is limited and other organizations experience performance degradation
Solution Approach 1:
The patent implements dynamic processing where the migration system can adjust its behavior based on workload conditions. The system processes migration packages at optimal speeds while maintaining fairness across multiple organizations, allowing large data migrations to complete reliably without permanently degrading service for others through rate limiting or throttling mechanisms.
Data Source
AI summary
A cloud-based migration system exposes a source-independent application programming interface for receiving data to be migrated. The data is uploaded and stored as a single entity in a cloud-based storage system. A migration system then accesses the migration package and begins migrating the data to its destination, from the cloud-based storage system.