Object storage system-based data processing method and apparatus, device and medium

Abstract

Provided are an object storage system-based data processing method and apparatus, a device and a medium. The method comprises: during the process of migrating data of a source object storage system to a target object storage system, intercepting a first request and forwarding same, wherein, on the basis of instruction of first configuration information, the first request of a write data type is forwarded to the source object storage system; when it is determined that there is no second request, adjusting the first configuration information to second configuration information; on the basis of instruction of the second configuration information, forwarding the first request of the write data type to the target object storage system; and comparing the data of the source object storage system and data of the target object storage system to identify differences, and performing data synchronization. Thus, data consistency can be ensured without interrupting related services of the source object storage system, thus implementing smooth migration, and making data migration processes imperceptible to users.

Description

Methods, apparatus, devices, and media for data processing based on object storage systems

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411706804.3, filed on November 26, 2024, entitled “Method, Apparatus, Device and Medium for Data Processing Based on Object Storage System”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer technology, and in particular to methods and apparatus for data processing, electronic devices, and storage media. Background Technology

[0004] An object storage system is a data storage system that stores data as objects rather than in file hierarchies or blocks. It is widely used for large-scale data storage needs, such as cloud storage, backup, archiving, and big data analytics.

[0005] Object storage systems often have data migration requirements. For example, in the initial stage of a business, an existing object storage system may be used directly for the purpose of rapid deployment. However, as the business grows, the original object storage system architecture becomes unsuitable, and a new object storage system is redesigned, requiring the data from the original object storage system to be migrated to the new object storage system.

[0006] In related technologies, when migrating data from an old object storage system to a new one, all services of the old object storage system are typically stopped to ensure data consistency and stability during the migration. However, for certain critical business operations, service interruption is unacceptable, as it not only affects the normal operation of front-end services but also leads to a poor user experience. Summary of the Invention

[0007] In view of the above problems, methods, apparatus, devices, and media for data processing based on object storage systems are proposed to overcome or at least partially solve the above problems, including:

[0008] A data processing method based on an object storage system, the method comprising:

[0009] In response to a request to migrate data, migrate data from the source object storage system to the target object storage system;

[0010] During the process of migrating data from the source object storage system to the target object storage system, the first request is intercepted; the first request is a business request initiated by the client to the source object storage system.

[0011] Obtaining first configuration information and forwarding the first request according to the first configuration information includes: if the type of the first request is write data, forwarding the first request to the source object storage system according to the instruction of the first configuration information;

[0012] During the forwarding of the first request based on the first configuration information, the source object storage system is queried to determine if a second request exists; wherein, the second request is a business request received by the source object storage system from the client that has not been intercepted.

[0013] If there is no second request, obtain the second configuration information and adjust the first configuration information to the second configuration information;

[0014] Forwarding the first request according to the second configuration information includes: when the type of the first request is write data, forwarding the first request to the target object storage system according to the instruction of the second configuration information;

[0015] Compare the differences between the data stored in the source object storage system and the target object storage system, determine the difference data, and synchronize the data between the source object storage system and the target object storage system based on the difference data;

[0016] Feedback on the results of the data migration.

[0017] In some embodiments of this application, forwarding the first request based on the first configuration information further includes:

[0018] If the type of the first request is read data, determine whether the source object storage system has target data corresponding to the first request according to the instructions of the first configuration;

[0019] If the target data exists, the first request will be forwarded to the source object storage system;

[0020] If the target data does not exist, the first request will be forwarded to the target object storage system;

[0021] Forwarding the first request based on the second configuration information also includes:

[0022] If the type of the first request is read data, determine whether the target object storage system has target data corresponding to the first request according to the instructions of the second configuration;

[0023] If the target data exists, the first request will be forwarded to the target object storage system;

[0024] If the target data does not exist, the first request will be forwarded to the source object storage system.

[0025] In some embodiments of this application, forwarding the first request based on the first configuration information further includes:

[0026] If the type of the first request is data upload in fragments, confirm whether the source object storage system has fragmented data corresponding to the first request according to the instructions of the first configuration;

[0027] If the source object storage system has sharded data corresponding to the first request, then the first request will be forwarded to the source object storage system.

[0028] If the source object storage system does not have the sharded data corresponding to the first request, the first request will be forwarded to the target object storage system.

[0029] Forwarding the first request based on the second configuration information also includes:

[0030] If the first request is of type fragmented data upload, the system confirms whether the target object storage system has fragmented data corresponding to the first request according to the instructions of the second configuration.

[0031] If the target object storage system has sharded data corresponding to the first request, then the first request will be forwarded to the target object storage system.

[0032] If the target object storage system does not have the sharded data corresponding to the first request, the first request will be forwarded to the source object storage system.

[0033] In some embodiments of this application, forwarding the first request based on the first configuration information further includes:

[0034] If the type of the first request is to check if the container exists or to obtain the data list corresponding to the container, the first request is forwarded to the source object storage system according to the instructions of the first configuration.

[0035] Forwarding the first request based on the second configuration information also includes:

[0036] If the type of the first request is to check if the container exists or to obtain the data list corresponding to the container, the first request is forwarded to the target object storage system according to the instructions of the second configuration.

[0037] In some embodiments of this application, forwarding the first request based on the first configuration information further includes:

[0038] If the type of the first request is to operate on a container, the first request is forwarded to the target object storage system according to the instructions of the first configuration;

[0039] If the type of the first request is to delete data, the first request is forwarded to the source object storage system and the target object storage system according to the instructions of the first configuration;

[0040] Forwarding the second request based on the second configuration information also includes:

[0041] If the type of the second request is to operate on a container, the second request is forwarded to the target object storage system according to the instructions of the second configuration;

[0042] If the type of the second request is to delete data, the second request is forwarded to the source object storage system and the target object storage system according to the instructions of the second configuration.

[0043] In some embodiments of this application, after synchronizing data between the source object storage system and the target object storage system based on the difference data, the method further includes:

[0044] Determine if a first request of type read data exists;

[0045] If there is no first request of type read data, then obtain the third configuration information and adjust the second configuration information to the third configuration information;

[0046] Based on the instructions of the third configuration information, all first requests are forwarded to the target object storage system.

[0047] In some embodiments of this application, the differences between the data stored in the source object storage system and the target object storage system are compared to determine the difference data, including:

[0048] Iterate through each first piece of data in the source object storage system and each second piece of data stored in the target object storage system;

[0049] If the target object storage system does not have second data corresponding to the first data, then the first data is determined to be the difference data;

[0050] If the target object storage system has second data corresponding to the first data, then verify whether the first data and the second data are consistent;

[0051] If the first data and the second data are inconsistent, the first data will be identified as the difference data.

[0052] If the second data is data that has been deleted from the source object storage system, then the second data is identified as the difference data.

[0053] An apparatus for data processing based on an object storage system, the apparatus comprising:

[0054] The data migration module is used to migrate data from the source object storage system to the target object storage system in response to data migration requests.

[0055] The first request interception module is used to intercept the first request during the process of migrating data from the source object storage system to the target object storage system; wherein, the first request is a business request initiated by the client against the source object storage system;

[0056] The first forwarding module is used to obtain first configuration information and forward the first request according to the first configuration information, including: when the type of the first request is write data, forwarding the first request to the source object storage system according to the instruction of the first configuration information;

[0057] The query module is used to query whether a second request exists through the source object storage system during the forwarding of the first request based on the first configuration information; wherein the second request is a business request received by the source object storage system from the client that has not been intercepted.

[0058] The first configuration adjustment module is used to obtain the second configuration information if there is no second request, and adjust the first configuration information to the second configuration information.

[0059] The second forwarding module is used to forward the first request according to the second configuration information, including: when the type of the first request is write data, forwarding the first request to the target object storage system according to the instruction of the second configuration information;

[0060] The data synchronization module is used to compare the differences between the data stored in the source object storage system and the target object storage system, determine the difference data, and synchronize the data between the source object storage system and the target object storage system based on the difference data.

[0061] The results feedback module is used to provide feedback on the results of the data migration.

[0062] An electronic device includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the above-described data processing method.

[0063] A computer-readable storage medium is characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the above-described data processing method.

[0064] The embodiments of this application have the following advantages: In the process of migrating data from the source object storage system to the target object storage system, the embodiments of this application intercept and forward the first request, and forward the first request of type write data to the source object storage system according to the instruction of the first configuration information. When it is confirmed that there is no second request, the first configuration information is adjusted to the second configuration information, and the first request of type write data is forwarded to the source object storage system according to the instruction of the second configuration information. The data differences between the source server and the destination server are compared and data synchronization is performed. This achieves the goal of ensuring data consistency without interrupting the relevant business of the source server, realizing a smooth migration, and making the data migration process imperceptible to the user. Attached Figure Description

[0065] To more clearly illustrate the technical solution of this application, the drawings used in the description of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 is a flowchart of the steps of a data processing method based on an object storage system provided in some embodiments of this application;

[0067] Figure 2 is a schematic diagram of a scanning tool provided in some embodiments of this application;

[0068] Figure 3 is a schematic diagram of a synchronization tool provided by some embodiments of this application;

[0069] Figure 4 is an initial flow diagram of a service request provided by some embodiments of this application;

[0070] Figure 5 is a schematic diagram of the flow of service requests after configuration modification provided by some embodiments of this application;

[0071] Figure 6 is a schematic diagram of the flow of service requests after another modified configuration provided by some embodiments of this application;

[0072] Figure 7 is a schematic diagram of the flow of service requests after another modified configuration provided by some embodiments of this application;

[0073] Figure 8 is a schematic diagram illustrating the flow of service requests after data migration, provided by some embodiments of this application;

[0074] Figure 9 is a structural block diagram of a data processing apparatus based on an object storage system provided in some embodiments of this application. Detailed Implementation

[0075] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0076] With the development of cloud computing, object storage, as a native cloud storage solution, is being used more and more widely, and almost all cloud-based activities rely on the support of object storage.

[0077] Consider the following scenario: In the initial stage of business development, an existing object storage system was used directly for rapid deployment. However, as the business scaled up, the original object storage system architecture became unsuitable, and a new object storage system was redesigned. This necessitates migrating data from the original system to the new one—a typical object storage migration scenario. In fact, during the use of object storage, as data volume increases and technology advances, or due to considerations such as cost and data security, enterprises may need to migrate data from one object storage system to another to improve performance, reduce costs, or meet other business needs. In such cases, data migration from the object storage system is a crucial step.

[0078] For data migration between different object storage systems, many migration tools are provided in related technologies, and some public cloud vendors also provide online migration services.

[0079] The migration tools or services mentioned above are generally business-driven and follow the migration process below:

[0080] 1. The target for migration is a bucket (a container for storing objects).

[0081] 2. Cease operations targeting storage buckets.

[0082] 3. Traverse the above buckets to obtain a list of all object names.

[0083] 4. Traverse the above list, read data from the source object storage system, write it to the destination object storage system, and perform data verification.

[0084] 5. Re-enable the service using the new access address.

[0085] For some business operations, stopping is unacceptable, and coordinating the migration is extremely complicated and time-consuming. Furthermore, for scenarios involving write operations, failing to stop the business will lead to severe data inconsistencies. For example, if a business continues to write data to the source object storage system after migration, its read / write operations will be unable to access that data when switched to the destination object storage system.

[0086] To address the shortcomings of the aforementioned related technologies, this application proposes a method, apparatus, device, and medium for data processing based on an object storage system. During the process of fully migrating data from a source object storage system to a target object storage system, client requests to the source object storage system are intercepted, and write requests are forwarded to the source object storage system. Once it is confirmed that no client requests to the source object storage system exist, the write requests are forwarded to the target object storage system. Then, data synchronization (i.e., incremental migration) is performed on the differences between the source and target object storage systems, thereby achieving smooth data migration and making the data migration process imperceptible to the user.

[0087] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0088] Referring to Figure 1, a flowchart of the steps of a data processing method based on an object storage system provided by some embodiments of this application is shown, which may specifically include the following steps:

[0089] Step 101: In response to the request to migrate data, migrate the data from the source object storage system to the target object storage system;

[0090] In step 101, upon receiving a request to migrate data, the data migration process begins, transferring all data from the source object storage system to the target object storage system. During this process, the services of the source object storage system are not interrupted.

[0091] In some embodiments of this application, a migration proxy system is also designed, which achieves smooth data migration through three core components, as follows:

[0092] 1) Scanning Tool: The scanning tool is mainly used for scanning data. During the first scan, the scanning tool performs a full scan of the data to be migrated from the source object storage system and records the scan results in the migration agent system. In subsequent scans, as shown in Figure 2, the scanning tool performs a full scan from both the source object storage system and the migration agent system, compares and records the scan results, for example, recording the differences.

[0093] In addition, scanning tools can be designed to be reentrant, meaning that the scanning tool can be safely invoked by multiple tasks or threads at the same time to ensure the stability of data migration.

[0094] 2) Synchronization tool: As shown in Figure 3, the synchronization tool is mainly used to perform a full scan from the migration agent system, and synchronize the data of the source object storage system from the source gateway (i.e., the gateway of the source object storage system) to the target object storage system through the destination gateway (i.e., the gateway of the target object storage system) according to the information recorded by the scanning tool, and record the synchronization results.

[0095] Specifically, the scanning tool can be viewed as an out-of-band operation log recording system, while the synchronization tool synchronizes according to the operation log. In this way, the scanning tool and the synchronization tool can realize the business logic of full data migration and incremental migration.

[0096] 3) Migration Proxy Gateway (or Migration Proxy Service): By setting the resolution of bucket domain names in DNS (Domain Name System), it intercepts business requests from clients to the source object storage system and then forwards different types of business requests to the source object storage system and / or the target object storage system. In other words, the migration proxy gateway is mainly used to control the flow of business requests.

[0097] The migration proxy gateway can also achieve fine-grained parsing of business requests, parsing each business request into a specific object storage S3 (Simple Storage Service, a communication protocol for object storage) request, rather than a simple coarse-grained read and write request.

[0098] In some examples, the data migration request can be a migration request for all objects within a bucket in the source object storage system (hereinafter referred to as the source bucket). Combined with the migration proxy system described above, the implementation process for a full migration of data in the source bucket is as follows:

[0099] 1. Identify the source buckets that need to be migrated. First, synchronize the user information (including access credentials) of the source buckets to the target object storage system. Then, synchronize the bucket information to the target object storage system (including creating buckets, configuring bucket permission policies, etc.).

[0100] 2. Import the information of the source bucket and the destination bucket (i.e., the corresponding bucket in the target object storage system) into the migration agent system. This mainly includes the bucket's domain (to distinguish between the source object storage system and the target object storage system), access IP (Internet Protocol) address, and access credentials.

[0101] 3. Configure the proxy record, which includes the bucket name, source domain, and destination domain to be synchronized. Set the initial proxy rule for the migration proxy gateway, named all_to_src, meaning all requests are forwarded to the source object storage system. The flow of business requests at this point is shown in Figure 4.

[0102] 4. Use a scanning tool to perform a scan, and then use a synchronization tool to perform a full data migration.

[0103] Step 102: During the process of migrating data from the source object storage system to the target object storage system, intercept the first request; wherein, the first request is a business request initiated by the client against the source object storage system;

[0104] In step 102, certain types of business requests can be processed directly in the target object storage system without affecting the consistency of existing business processes and data, such as reading data and operations on buckets. Therefore, the objects providing services can be gradually transitioned from the source object storage system to the target object storage system. Based on this, it is necessary to first uniformly intercept business requests initiated by clients targeting the source object storage system, and then forward them to the source object storage system and / or the target object storage system according to different types of business requests, thereby controlling the flow of business requests.

[0105] In practice, the first request can be intercepted by modifying the DNS configuration. For example, the DNS resolution configuration of the bucket domain can be modified to change the destination address of the client's business request from the source gateway IP address to the IP address of the proxy server (such as the migration proxy system mentioned above). Since the DNS settings change do not take effect immediately on the client side, some business requests will still flow directly to the source object storage system through the client. These uninterrupted business requests are the ones that failed to be intercepted.

[0106] Step 103, obtaining first configuration information and forwarding the first request according to the first configuration information, including: if the type of the first request is write data, forwarding the first request to the source object storage system according to the instruction of the first configuration information;

[0107] In step 103, the first configuration information is configuration information indicating the rules for controlling the flow of business requests, i.e., forwarding rules. The first configuration information defines, through different configuration items, the forwarding of different types of first requests to the source object storage system and / or the target object storage system.

[0108] Since write data requests will affect the data consistency between the source and target object storage systems after data migration, and in order not to affect the original business, write data requests will still be forwarded to the source object storage system.

[0109] In a specific implementation, the first configuration information can be the proxy rule of the proxy server, which can be named write_to_src, meaning that the data request is forwarded to the source.

[0110] In some examples, after changing the initial proxy rule of the migration proxy gateway from all_to_src to write_to_src, the flow of business requests is as shown in Figure 5. At this time, there are business requests (reading objects, writing objects, etc.) initiated directly by the client to the source object server, as well as business requests forwarded by the migration proxy gateway (whose destination address is the IP address of the source gateway and / or the destination gateway).

[0111] Step 104: During the forwarding of the first request according to the first configuration information, the source object storage system is queried to see if a second request exists; wherein, the second request is a business request received by the source object storage system from the client that has not been intercepted.

[0112] In step 104, the second request is a business request initiated directly by the client to the source object storage system during the data migration process without interception or forwarding. Because there is a time lag between the interception of the first request and its effectiveness, some business requests may still not be intercepted. For example, after modifying the DNS configuration, some clients may not be able to immediately perform domain name resolution according to the modified DNS configuration due to caching or other reasons. These clients will still directly initiate business requests to access the source storage system without interception or forwarding.

[0113] In practice, the existence of a second request can be confirmed by querying the access logs of the source system.

[0114] In some examples, the migration proxy gateway can set the UserAgent in the request header to "s3-sync-proxy" when forwarding the first request, thus declaring that the business request is forwarded by the migration proxy gateway. When querying the access logs of the source system, the UserAgent information can be used to distinguish which requests were intercepted and forwarded as the first request and which were not intercepted as the second request.

[0115] Step 105: If there is no second request, obtain the second configuration information and adjust the first configuration information to the second configuration information;

[0116] In step 105, similar to the first configuration information, the second configuration information is also used to define the forwarding rules for the first request, but the configuration items in the second configuration information and the first configuration information are not entirely the same. When no second request is found, it can be assumed that all client business requests to the source object storage system have been intercepted, and the forwarding rules for the first request need to be adjusted.

[0117] In practical applications, if no new second request is found within a preset time period, it can also be assumed that the second request does not exist, and all client business requests to the source object storage system have been intercepted. For example, during data migration, if a second request is found at time t1, but no new second request is found after time t2, it can also be assumed that the second request does not exist.

[0118] Step 106, forwarding the first request according to the second configuration information, includes: if the type of the first request is write data, forwarding the first request to the target object storage system according to the instruction of the second configuration information;

[0119] In step 106, since there is no second request, the write data service can be completely taken over by the target object storage system. At this time, business requests of type write data will no longer affect the data consistency after data migration.

[0120] Based on this, the first configuration information is adjusted to the second configuration information, and the first request of type write data is forwarded to the target object storage system through the second configuration information.

[0121] In a specific implementation, the second configuration information can be the proxy rule of the proxy server. This proxy rule can be named write_to_dest, which means that write data requests are forwarded to the target object storage system.

[0122] In some examples, after changing the proxy rule `write_to_src` to `write_to_dest` in the migration proxy gateway, the flow of business requests is as shown in Figure 6. At this point, only some first requests (such as read data requests) are forwarded to the source gateway, while all first requests of type write data are forwarded to the destination gateway.

[0123] In some embodiments of this application, forwarding the first request based on the first configuration information further includes:

[0124] If the type of the first request is read data, determine whether the source object storage system has target data corresponding to the first request according to the instructions of the first configuration;

[0125] If the target data exists, the first request will be forwarded to the source object storage system;

[0126] If the target data does not exist, the first request will be forwarded to the target object storage system;

[0127] Forwarding the first request based on the second configuration information also includes:

[0128] If the type of the first request is read data, determine whether the target object storage system has target data corresponding to the first request according to the instructions of the second configuration;

[0129] If the target data exists, the first request will be forwarded to the target object storage system;

[0130] If the target data does not exist, the first request will be forwarded to the source object storage system.

[0131] In some embodiments of this application, during the forwarding of the first request based on the first configuration information, data migration is performed simultaneously, and the source object storage system remains the primary service provider. Therefore, when the type of the first request is read data, the target data is queried first from the source object storage system. If the target data exists, the first request is forwarded to the source object storage system; if it does not exist, the first request is forwarded to the target object storage system.

[0132] Conversely, during the forwarding of the first request based on the second configuration information, the target object storage system is the primary service provider. Furthermore, the target object storage system has already begun receiving write data requests from clients, and some data may only exist within the target object storage system. Therefore, the target data is queried first from the target object storage system. If the target data exists, the first request is forwarded to the target object storage system; otherwise, the first request is forwarded to the source object storage system.

[0133] In some embodiments of this application, forwarding the first request based on the first configuration information further includes:

[0134] If the type of the first request is data upload in fragments, confirm whether the source object storage system has fragmented data corresponding to the first request according to the instructions of the first configuration;

[0135] If the source object storage system has sharded data corresponding to the first request, then the first request will be forwarded to the source object storage system.

[0136] If the source object storage system does not have the sharded data corresponding to the first request, the first request will be forwarded to the target object storage system.

[0137] Forwarding the first request based on the second configuration information also includes:

[0138] If the first request is of type fragmented data upload, the system confirms whether the target object storage system has fragmented data corresponding to the first request according to the instructions of the second configuration.

[0139] If the target object storage system has sharded data corresponding to the first request, then the first request will be forwarded to the target object storage system.

[0140] If the target object storage system does not have the sharded data corresponding to the first request, the first request will be forwarded to the source object storage system.

[0141] In some embodiments of this application, during the forwarding of the first request according to the first configuration information, if the source object storage system is still the main service provider, then the system will first query whether there is shard data corresponding to the first request; in contrast, during the forwarding of the first request according to the second configuration information, if the target object storage system is the main service provider, then the system will first query whether there is shard data corresponding to the first request.

[0142] Then, for data uploaded in chunks, data integrity needs to be ensured. Therefore, in the first configuration information, if the source object storage system already contains chunked data corresponding to the first request (e.g., other chunked data that has already been successfully uploaded), the business request needs to be forwarded to the source object storage system so that the chunked data can be merged into complete data; if it does not exist, then it is the first chunked data uploaded, and can be forwarded to the target object storage system. The second configuration information is similar to the first configuration information and will not be described in detail here.

[0143] In practical applications, the type of the first request can be determined by checking whether the business request contains an uploadId. Here, uploadId is the sequence number of the data chunk when uploading data in chunks.

[0144] In some embodiments of this application, forwarding the first request based on the first configuration information further includes:

[0145] If the type of the first request is to check if the container exists or to obtain the data list corresponding to the container, the first request is forwarded to the source object storage system according to the instructions of the first configuration.

[0146] Forwarding the first request based on the second configuration information also includes:

[0147] If the type of the first request is to check if the container exists or to obtain the data list corresponding to the container, the first request is forwarded to the target object storage system according to the instructions of the second configuration.

[0148] In some embodiments of this application, during the forwarding of the first request according to the first configuration information, the source object storage server, as the main service provider, forwards the first request of type "check if the container exists" or "obtain the data list corresponding to the container" to the source object storage server.

[0149] In contrast, during the forwarding of the first request based on the second configuration information, the target object storage server, as the primary service provider, forwards the first request, which is of type "check if the container exists" or "get the data list corresponding to the container", to the target object storage server.

[0150] In a specific implementation, if the first request contains a HeadBucket directive, it can be confirmed that the type of the first request is to check if the container exists; if the first request contains a ListObjects directive, it can be confirmed that the type of the first request is to retrieve the data list corresponding to the container.

[0151] In some embodiments of this application, forwarding the first request based on the first configuration information further includes:

[0152] If the type of the first request is to operate on a container, the first request is forwarded to the target object storage system according to the instructions of the first configuration;

[0153] If the type of the first request is to delete data, the first request is forwarded to the source object storage system and the target object storage system according to the instructions of the first configuration;

[0154] Forwarding the second request based on the second configuration information also includes:

[0155] If the type of the second request is to operate on a container, the second request is forwarded to the target object storage system according to the instructions of the second configuration;

[0156] If the type of the second request is to delete data, the second request is forwarded to the source object storage system and the target object storage system according to the instructions of the second configuration.

[0157] In some embodiments of this application, since the first request of type is to operate on the container, it will not affect the data consistency after data migration. Therefore, the configuration items of the forwarding rule for the first request of type is to operate on the container in the first configuration information and the second configuration information can be the same, which is to forward the first request to the target object storage system.

[0158] For the first request, which is of type "delete data," bidirectional forwarding is performed between the source and target object storage systems, ensuring data consistency by deleting data simultaneously on both sides. Therefore, the configuration items for the forwarding rule of type "delete data" in both the first and second configuration information can be the same, both setting the forwarding to both the source and target object storage systems.

[0159] In some examples, the first configuration information for the migration proxy gateway settings, namely the configuration item for the proxy rule `write_to_src`, can be as follows:

[0160] 1) All bucket operations are forwarded to the destination gateway.

[0161] 2) HeadBucket / ListObjects are forwarded to the source gateway.

[0162] 3) The newly created object (i.e., the written data) is forwarded to the source gateway.

[0163] 4) Read objects are first forwarded to the source gateway, and if they do not exist, they are forwarded to the destination gateway.

[0164] 5) For requests with an uploadId, the origin gateway is checked first to see if it exists. If it exists, the request is forwarded to the origin gateway; otherwise, it is forwarded to the destination gateway.

[0165] 6) Delete operations are forwarded bidirectionally.

[0166] The second configuration information for the migration proxy gateway, namely the configuration item for the proxy rule `write_to_dest`, can be as follows:

[0167] 1) All bucket operations are forwarded to the destination gateway.

[0168] 2) HeadBucket / ListObjects are forwarded to the destination gateway.

[0169] 3) Forward the newly created object to the destination gateway.

[0170] 4) Read objects are forwarded to the destination gateway first, and if they do not exist, they are forwarded to the source gateway.

[0171] 5) For requests with an uploadId, the destination gateway is first checked to see if it exists. If it exists, the request is forwarded to the destination gateway; otherwise, it is forwarded to the source gateway.

[0172] 6) Delete operations are forwarded bidirectionally.

[0173] Step 107: Compare the differences between the data stored in the source object storage system and the target object storage system, determine the difference data, and synchronize the data between the source object storage system and the target object storage system based on the difference data;

[0174] In step 107, after adjusting the first configuration information to the second configuration information and forwarding the first request according to the second configuration information, since no data has been written to the original object storage system, the difference data in the source object storage system and the target object storage system can be synchronized, i.e., incremental migration can be performed.

[0175] In some embodiments of this application, the differences between the data stored in the source object storage system and the target object storage system are compared to determine the difference data, including:

[0176] Iterate through each first piece of data in the source object storage system and each second piece of data stored in the target object storage system;

[0177] If the target object storage system does not have second data corresponding to the first data, then the first data is determined to be the difference data;

[0178] If the target object storage system has second data corresponding to the first data, then verify whether the first data and the second data are consistent;

[0179] If the first data and the second data are inconsistent, the first data will be identified as the difference data.

[0180] If the second data is data that has been deleted from the source object storage system, then the second data is identified as the difference data.

[0181] In some embodiments of this application, each first data in the source object storage system, if the migration has been completed, will have a corresponding and consistent second data in the target object storage system. However, since the source object storage system will still receive business requests to write data during the process of forwarding the first request according to the first configuration information, the data in the source object storage system may be added, modified or deleted. Therefore, it is necessary to traverse the data in the source object storage system and the target object storage system respectively to identify the difference data.

[0182] If the target object storage system does not have second data corresponding to the first data, then the first data is determined as newly added data and as differential data.

[0183] If the target object storage system has second data corresponding to the first data, then further verify whether the first data and the second data are consistent;

[0184] If the first data and the second data are inconsistent, the first data will be identified as the modified data and thus as the difference data.

[0185] If the second data is data that has been deleted from the source object storage system, then the second data is identified as data that was not deleted in time and is identified as differential data.

[0186] In some examples, incremental migration can be achieved using the scanning and synchronization tools within the migration agent system described above. Details are as follows:

[0187] The scanning tool performs a full scan simultaneously from both the source and migration agent systems, comparing the results. If an object exists in the migration agent system but not in the source object store, it needs to be deleted. If an object exists in the source object store but not in the migration agent system, it indicates that the object is newly added. If the object's modification time, size, and data validation are inconsistent, it means the object has been modified. All these discrepancies are recorded in the migration agent system.

[0188] Then, use a synchronization tool to perform the migration. It should be noted that if write requests are still being forwarded to the source gateway during this process, the scan synchronization needs to be repeated until no write requests are being forwarded to the source gateway.

[0189] In practical applications, after incremental migration is complete, the DNS configuration of the bucket domain name can be changed from the IP address of the migration proxy gateway to the IP address of the destination gateway. This stops intercepting the first request and migrates all services from the source object storage system to the target object storage system. At this point, the flow of business requests is shown in Figure 7. Some clients can directly access the target object storage system, but because the DNS configuration takes time to take effect, some clients' business requests are still sent to the proxy gateway.

[0190] In some embodiments of this application, after synchronizing data between the source object storage system and the target object storage system based on the difference data, the method further includes:

[0191] Determine if a first request of type read data exists;

[0192] If there is no first request of type read data, then obtain the third configuration information and adjust the second configuration information to the third configuration information;

[0193] Based on the instructions of the third configuration information, all first requests are forwarded to the target object storage system.

[0194] In some embodiments of this application, if there is no first request of type read data, it indicates that data does not need to be read from the source object storage system, and the target object storage system can independently and completely provide services. Therefore, the second configuration information is adjusted to the third configuration information, and all first requests are forwarded to the target object storage system according to the instructions of the third configuration information.

[0195] The function of the third configuration information is similar to that of the first and second configuration information, and will not be repeated here. The difference is that the third configuration information only instructs that all first requests be forwarded to the target object storage system, that is, all services are taken over by the target object storage system.

[0196] In a specific implementation, the third configuration information can be the proxy rule of the proxy server. This proxy rule can be named all_to_dest, which means that all business requests are forwarded to the target object storage system.

[0197] In practical applications, once the IP address of the migration proxy gateway is changed to the IP address of the destination gateway, and it is confirmed that no more business requests are passing through the migration proxy gateway, the function of forwarding requests by the migration proxy gateway can be stopped, thus achieving complete data and business migration. At this point, the flow of business requests is as shown in Figure 8, with only business requests initiated by the client to the destination gateway.

[0198] Step 108: Feedback on the results of the data migration.

[0199] In step 108, after the full migration and incremental migration of data are completed, the results of the data migration are fed back to the party that requested the data migration.

[0200] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0201] The embodiments of this application have the following advantages: In the process of migrating data from the source object storage system to the target object storage system, the embodiments of this application intercept and forward the first request, and forward the first request of type write data to the source object storage system according to the instruction of the first configuration information. When it is confirmed that there is no second request, the first configuration information is adjusted to the second configuration information, and the first request of type write data is forwarded to the source object storage system according to the instruction of the second configuration information. The data differences between the source server and the destination server are compared and data synchronization is performed. This achieves the goal of ensuring data consistency without interrupting the relevant business of the source server, realizing a smooth migration, and making the data migration process imperceptible to the user.

[0202] Referring to Figure 9, a schematic diagram of the structure of a data processing apparatus based on an object storage system provided by some embodiments of this application is shown, which may specifically include the following modules:

[0203] The data migration module 901 is used to migrate data from the source object storage system to the target object storage system in response to a request to migrate data.

[0204] The first request interception module 902 is used to intercept the first request during the process of migrating data from the source object storage system to the target object storage system; wherein, the first request is a business request initiated by the client for the source object storage system;

[0205] The first forwarding module 903 is used to obtain first configuration information and forward the first request according to the first configuration information, including: when the type of the first request is write data, forwarding the first request to the source object storage system according to the instruction of the first configuration information;

[0206] The query module 904 is used to query whether a second request exists through the source object storage system during the process of forwarding the first request according to the first configuration information; wherein the second request is a business request received by the source object storage system from the client that has not been intercepted.

[0207] The first configuration adjustment module 905 is used to obtain the second configuration information if there is no second request, and adjust the first configuration information to the second configuration information.

[0208] The second forwarding module 906 is used to forward the first request according to the second configuration information, including: when the type of the first request is write data, forwarding the first request to the target object storage system according to the instruction of the second configuration information;

[0209] The data synchronization module 907 is used to compare the differences between the data stored in the source object storage system and the target object storage system, determine the difference data, and synchronize the data between the source object storage system and the target object storage system based on the difference data.

[0210] The result feedback module 908 is used to provide feedback on the results of the migration data.

[0211] In some embodiments of this application, the first forwarding module 903 is further configured to:

[0212] The target data determination submodule is used to determine whether the source object storage system has target data corresponding to the first request, based on the indication of the first configuration, when the type of the first request is read data.

[0213] The first submodule of source forwarding is used to forward the first request to the source object storage system if the target data exists.

[0214] The first submodule for forwarding is used to forward the first request to the target object storage system if the target data does not exist.

[0215] The second forwarding module 906 is also used for:

[0216] If the type of the first request is read data, determine whether the target object storage system has target data corresponding to the first request according to the instructions of the second configuration;

[0217] If the target data exists, the first request will be forwarded to the target object storage system;

[0218] If the target data does not exist, the first request will be forwarded to the source object storage system.

[0219] In some embodiments of this application, the first forwarding module 903 is further configured to:

[0220] If the type of the first request is data upload in fragments, confirm whether the source object storage system has fragmented data corresponding to the first request according to the instructions of the first configuration;

[0221] If the source object storage system has sharded data corresponding to the first request, then the first request will be forwarded to the source object storage system.

[0222] If the source object storage system does not have the sharded data corresponding to the first request, the first request will be forwarded to the target object storage system.

[0223] The second forwarding module 906 is also used for:

[0224] If the first request is of type fragmented data upload, the system confirms whether the target object storage system has fragmented data corresponding to the first request according to the instructions of the second configuration.

[0225] If the target object storage system has sharded data corresponding to the first request, then the first request will be forwarded to the target object storage system.

[0226] If the target object storage system does not have the sharded data corresponding to the first request, the first request will be forwarded to the source object storage system.

[0227] In some embodiments of this application, the first forwarding module 903 is further configured to:

[0228] If the type of the first request is to check if the container exists or to obtain the data list corresponding to the container, the first request is forwarded to the source object storage system according to the instructions of the first configuration.

[0229] The second forwarding module 906 is also used for:

[0230] If the type of the first request is to check if the container exists or to obtain the data list corresponding to the container, the first request is forwarded to the target object storage system according to the instructions of the second configuration.

[0231] In some embodiments of this application, the first forwarding module 903 is further configured to:

[0232] If the type of the first request is to operate on a container, the first request is forwarded to the target object storage system according to the instructions of the first configuration;

[0233] If the type of the first request is to delete data, the first request is forwarded to the source object storage system and the target object storage system according to the instructions of the first configuration;

[0234] The second forwarding module 906 is also used for:

[0235] If the type of the second request is to operate on a container, the second request is forwarded to the target object storage system according to the instructions of the second configuration;

[0236] If the type of the second request is to delete data, the second request is forwarded to the source object storage system and the target object storage system according to the instructions of the second configuration.

[0237] In some embodiments of this application, the apparatus further includes:

[0238] The first judgment module is used to determine whether there is a first request of type read data;

[0239] The second configuration information adjustment module is used to obtain the third configuration information and adjust the second configuration information to the third configuration information if there is no first request of type read data.

[0240] Based on the instructions of the third configuration information, all first requests are forwarded to the target object storage system.

[0241] In some embodiments of this application, the data synchronization module 907 includes:

[0242] The traversal submodule is used to traverse each first piece of data in the source object storage system and each second piece of data stored in the target object storage system, respectively.

[0243] The first difference data determination submodule is used to determine the first data as difference data if the target object storage system does not have second data corresponding to the first data;

[0244] The verification submodule is used to verify whether the first data and the second data are consistent if the target object storage system has second data corresponding to the first data.

[0245] The second difference data determination submodule is used to determine the first data as the difference data if the first data and the second data are inconsistent.

[0246] The third difference data determination submodule is used to determine the second data as difference data if the second data is data that has been deleted from the source object storage system.

[0247] Some embodiments of this application also provide an electronic device that may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the above-described data processing method based on an object storage system.

[0248] Some embodiments of this application also provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the above-described data processing method based on an object storage system.

[0249] Some embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the above-described data processing method based on an object storage system.

[0250] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0251] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0252] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0253] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0254] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0255] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0256] These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable terminal equipment, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0257] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0258] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the aforementioned element.

[0259] The above provides a detailed description of the data processing methods, apparatus, devices, and media based on object storage systems. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method based on an object storage system, characterized in that, The method includes: In response to a request to migrate data, migrate data from the source object storage system to the target object storage system; During the process of migrating data from the source object storage system to the target object storage system, the first request is intercepted; wherein, the first request is a business request initiated by the client for the source object storage system; Obtaining first configuration information and forwarding the first request according to the first configuration information includes: when the type of the first request is write data, forwarding the first request to the source object storage system according to the indication of the first configuration information; During the forwarding of the first request according to the first configuration information, the source object storage system is queried to determine whether a second request exists; wherein, the second request is a business request received by the source object storage system from the client that has not been intercepted; If the second request does not exist, obtain the second configuration information and adjust the first configuration information to the second configuration information; Forwarding the first request according to the second configuration information includes: when the type of the first request is write data, forwarding the first request to the target object storage system according to the instruction of the second configuration information; The differences between the data stored in the source object storage system and the target object storage system are compared to determine the difference data, and the data of the source object storage system and the target object storage system are synchronized according to the difference data; Feedback on the results of the data migration.

2. The method according to claim 1, characterized in that, The step of forwarding the first request according to the first configuration information further includes: If the type of the first request is read data, determine whether the source object storage system has target data corresponding to the first request according to the indication of the first configuration; If the target data exists, the first request is forwarded to the source object storage system; If the target data does not exist, the first request is forwarded to the target object storage system; The step of forwarding the first request according to the second configuration information further includes: If the type of the first request is read data, determine whether the target object storage system has target data corresponding to the first request according to the instructions of the second configuration; If the target data exists, the first request is forwarded to the target object storage system; If the target data does not exist, the first request is forwarded to the source object storage system.

3. The method according to claim 1, characterized in that, The step of forwarding the first request according to the first configuration information further includes: If the first request is of type fragmented data upload, confirm whether the source object storage system has fragmented data corresponding to the first request according to the indication of the first configuration; If the source object storage system has sharded data corresponding to the first request, then the first request is forwarded to the source object storage system; If the source object storage system does not have sharded data corresponding to the first request, then the first request is forwarded to the target object storage system; The step of forwarding the first request according to the second configuration information further includes: If the first request is of type fragmented data upload, the system confirms whether the target object storage system has fragmented data corresponding to the first request according to the instructions of the second configuration. If the target object storage system has sharded data corresponding to the first request, then the first request is forwarded to the target object storage system; If the target object storage system does not have sharded data corresponding to the first request, then the first request is forwarded to the source object storage system.

4. The method according to claim 1, characterized in that, The step of forwarding the first request according to the first configuration information further includes: If the type of the first request is to check if the container exists or to obtain the data list corresponding to the container, the first request is forwarded to the source object storage system according to the instructions of the first configuration. The step of forwarding the first request according to the second configuration information further includes: If the type of the first request is to check if the container exists or to obtain the data list corresponding to the container, the first request is forwarded to the target object storage system according to the instructions of the second configuration.

5. The method according to any one of claims 1-4, characterized in that, The step of forwarding the first request according to the first configuration information further includes: If the type of the first request is to operate on a container, the first request is forwarded to the target object storage system according to the instructions of the first configuration; If the type of the first request is data deletion, the first request is forwarded to the source object storage system and the target object storage system according to the instructions of the first configuration; The step of forwarding the second request according to the second configuration information further includes: If the type of the second request is to operate on a container, the second request is forwarded to the target object storage system according to the instructions of the second configuration; If the type of the second request is data deletion, the second request is forwarded to the source object storage system and the target object storage system according to the instructions of the second configuration.

6. The method according to any one of claims 1-4, characterized in that, After synchronizing the source object storage system and the target object storage system based on the difference data, the method further includes: Determine if the first request of type read data exists; If there is no first request of type read data, then obtain the third configuration information and adjust the second configuration information to the third configuration information; According to the instructions of the third configuration information, all the first requests are forwarded to the target object storage system.

7. The method according to claim 1, characterized in that, The step of comparing the differences between the data stored in the source object storage system and the target object storage system to determine the difference data includes: Each first data in the source object storage system and each second data stored in the target object storage system are traversed separately. If the target object storage system does not have the second data corresponding to the first data, then the first data is determined to be the difference data; If the target object storage system has second data corresponding to the first data, then verify whether the first data and the second data are consistent; If the first data and the second data are inconsistent, then the first data is determined to be the difference data; If the second data is data that has been deleted from the source object storage system, then the second data is identified as the difference data.

8. A data processing apparatus based on an object storage system, characterized in that, The device includes: The data migration module is used to migrate data from the source object storage system to the target object storage system in response to data migration requests. The first request interception module is used to intercept a first request during the process of migrating data from the source object storage system to the target object storage system; wherein, the first request is a business request initiated by the client for the source object storage system; The first forwarding module is used to obtain first configuration information and forward the first request according to the first configuration information, including: when the type of the first request is write data, forwarding the first request to the source object storage system according to the indication of the first configuration information; The query module is used to query whether a second request exists through the source object storage system during the process of forwarding the first request according to the first configuration information; wherein the second request is a business request received by the source object storage system from the client that has not been intercepted; The first configuration adjustment module is used to obtain second configuration information if the second request does not exist, and adjust the first configuration information to the second configuration information. The second forwarding module is used to forward the first request according to the second configuration information, including: when the type of the first request is write data, forwarding the first request to the target object storage system according to the instruction of the second configuration information; The data synchronization module is used to compare the differences between the data stored in the source object storage system and the target object storage system, determine the difference data, and synchronize the data between the source object storage system and the target object storage system based on the difference data. The results feedback module is used to provide feedback on the results of the data migration.

9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the data processing method as described in any one of claims 1 to 7.

Images