Object Copy Driver Module for Low-Bandwidth Volume Migration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional volume migration techniques incur high costs and inefficiencies due to multiple read and write operations when transferring data from a source cluster to a destination cluster, particularly when dealing with objects stored in a capacity tier, which consumes significant network bandwidth and resources.
Innovation Solution
A cost-optimized volume migration technique using backend object copy operations that transfers metadata instead of actual data, employing an object copy driver module and work queue to manage the migration process, thereby reducing the number of read and write operations and network usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional volume migration techniques are used to transfer data from source cluster to destination cluster, then the volume migration can be completed, but multiple read and write operations are incurred which consume significant network bandwidth and resources
Solution Approach 1:
The patent extracts only the necessary metadata (object identifiers, block mappings) from the source object store and transfers it to the destination cluster, rather than transferring the entire actual data. This extraction approach allows the migration to proceed with minimal network bandwidth consumption while maintaining the ability to reconstruct the volume at the destination.
Solution Approach 2:
The patent creates a copy of the metadata structure at the destination cluster that mirrors the source volume's organization. By copying only the metadata layer (object identifiers, block mappings) rather than the full data, the system achieves migration capability with significantly reduced resource consumption.
2Productivity
If objects stored in capacity tier are migrated using conventional techniques, then the migration can be completed, but the number of read and write operations increases significantly
Solution Approach 1:
The patent extracts object identifiers and block mapping information from objects in the capacity tier without performing full data reads. This selective extraction of metadata enables the migration process to bypass the need for numerous read operations on actual data blocks, thereby reducing operation count while maintaining migration capability.
Solution Approach 2:
The patent performs preliminary evaluation of the source object store to identify objects stored in the capacity tier before initiating the migration. This preliminary action allows the system to prepare an optimized migration plan that targets only the necessary metadata for capacity tier objects, reducing the overall number of operations required during the actual migration execution.
3Loss of energy
If metadata transfer is used instead of actual data transfer, then network utilization is reduced, but backend copy operations and work queue management are required
Solution Approach 1:
The patent introduces an object copy driver module as an intermediary component that manages the migration process. This driver module coordinates between the source object store, the work queue, and the destination object store, handling the complexity of metadata transfer and backend copy operations while presenting a simplified migration interface to the user.
Solution Approach 2:
The patent implements a work queue mechanism that automatically manages the migration tasks without requiring continuous external intervention. The work queue self-manages the coordination of metadata transfer, tracks migration progress, and handles the reconstruction of volume structures at the destination, thereby reducing operational complexity despite the sophisticated underlying processes.
Data Source
AI summary
Techniques are provided for migrating a volume utilizing an object copy work queue and an object copy driver module. Data of the volume is stored within objects stored across a storage tier and capacity tier of a source object store. As part of migrating the volume to a destination object store, the objects are migrated to the destination cluster. Directly copying the objects involves multiple read operations to the source object store and a write operation at the destination object store. The techniques provided herein improve the efficiency of the migration by initially sending metadata from the source object store to the destination object store for performing backend object copy operations to migrate the volume. This results in fewer operations and less network usage, thus improving the efficiency and cost of migrating the volume.