Object Storage Data Migration Without Service Disruption
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data migration methods in object storage services disrupt normal customer services, incur additional storage costs, and require complex operations with high skill requirements, as they involve changing bucket identifiers and APIs, and do not allow seamless migration without affecting the customer's service.
Innovation Solution
A data migration method that keeps the bucket identifier and API unchanged, allowing metadata to be migrated first, followed by other data, enabling simultaneous online service provision from both storage systems during migration, and directing data access requests to the new storage system for processing, thus minimizing service disruption and costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If cross-region replication technology is used to back up data across regions, then data redundancy is improved, but storage costs increase and data access complexity increases
Solution Approach 1:
The patent uses copying technology to create a copy of the source bucket in the target region. The copy bucket contains the same data as the source bucket, providing data redundancy and backup capabilities. This allows the system to maintain data reliability while using standard storage pricing for the copy bucket, rather than expensive cross-region replication features.
2Reliability
If cross-region replication technology is used, then data backup capability is improved, but operation complexity increases
Solution Approach 1:
The system creates a copy bucket that mirrors the source bucket structure and data. This copying approach simplifies operations because the copy bucket can be accessed directly using standard OSS APIs, eliminating the need for complex cross-region replication configuration and management.
3Adaptability or versatility
If data is migrated between buckets in different regions, then system replacement is enabled, but service disruption occurs
Solution Approach 1:
The system performs preliminary actions by creating the copy bucket before the source bucket is replaced. This allows the new storage system to be fully prepared and tested before going live, ensuring that data migration can occur without service disruption. The copy bucket is created with the same structure and permissions, enabling seamless switching.
Solution Approach 2:
By creating a complete copy of the source bucket including all data, metadata, and access permissions, the system enables a seamless replacement process. The copy bucket can immediately take over when the source bucket is replaced, maintaining service continuity without requiring data transfer during the switch.
4Adaptability or versatility
If bucket identifier and API are changed during migration, then new storage system integration is achieved, but customer service is affected
Solution Approach 1:
The copy bucket is created with universal compatibility, supporting the same OSS APIs and access methods as the source bucket. This allows customers to access their data using the same code and interfaces without modification, eliminating service impact while enabling integration with the new storage system.
Solution Approach 2:
The system copies not only data but also bucket metadata, access permissions, and configuration settings. This ensures the copy bucket is fully compatible with existing customer applications and APIs, allowing seamless transition without requiring code changes or service disruption.
5Manufacturing precision
If historical data is fully migrated before new data migration, then data consistency is improved, but migration time increases
Solution Approach 1:
The system performs preliminary migration of the source bucket data to the copy bucket before the source bucket is replaced. This preliminary action ensures that historical data is migrated and consistent before new data generation begins, while the copy bucket structure is already in place to receive new data immediately.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A data migration method and an apparatus are provided, and relate to the field of storage technologies. This can resolve a problem that a customer's normal service is affected during data migration. The method is as follows: sending, by a first storage system, a location update request to a location server, where the location update request is used to indicate the location server to update location information of a first bucket from being located in a second storage system to being located in the first storage system (402); migrating, by the first storage system, data (405) in a first bucket from the second storage system; receiving, by the first storage system, a data access request, where the data access request is used to access the data (406) in the first bucket; and determining, by the first storage system based on a type of the data access request and a migration status of the data, that the first storage system or the second storage system processes the data access request (407). The foregoing method is used for an intra-bucket data migration between storage systems.