Data recovery method and apparatus, electronic device, and readable medium
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- CHINA TELECOM CLOUD TECH CO LTD
- Filing Date
- 2025-11-20
- Publication Date
- 2026-06-11
Smart Images
Figure CN2025136446_11062026_PF_FP_ABST
Abstract
Description
Data recovery methods, devices, electronic equipment and readable media
[0001] Cross-references to related applications
[0002] This application claims priority to Chinese Patent Application No. 2024117768182, filed on December 5, 2024, entitled “Data Recovery Method, Apparatus, Electronic Device and Readable Medium”, the entire contents of which are incorporated herein by reference. Technical Field
[0003] This application relates to the field of distributed storage technology, and in particular to a data recovery method, a data recovery device, an electronic device, and a computer-readable medium. Background Technology
[0004] With the continuous advancement of network technology, users have increasingly higher requirements for data storage and management. Traditional network storage systems have gradually become unable to meet user needs. In this environment, distributed storage systems have emerged, which distribute data across multiple independent devices.
[0005] Meanwhile, to ensure the reliability of data in a distributed storage system, redundant data is generally used. This redundancy is mainly achieved through replication redundancy and error correction redundancy. When a disk in a distributed storage system fails, the placement group (PG) to be recovered can be written into the recovery queue, and then the data can be recovered sequentially according to the recovery queue. However, if disks fail consecutively, the placement group may suffer further disk damage before the data recovery is completed, resulting in the data being unrecoverable. Summary of the Invention
[0006] This application provides a data recovery method, apparatus, electronic device, and computer-readable storage medium to solve the problem that data in a storage group cannot be reliably recovered when a disk suffers continuous damage.
[0007] This application discloses a data recovery method, including:
[0008] After a disk failure occurs in the distributed cluster storage pool, determine the number of missing disks in each placement group;
[0009] The recovery priority of each placement group is determined based on the number of missing disks in each placement group, wherein the placement group with more missing disks has a higher recovery priority.
[0010] Each placement group is written into the recovery queue in descending order of recovery priority; the placement group with higher recovery priority is placed before the placement group with lower recovery priority.
[0011] The data of each placement group in the recovery queue is restored sequentially from front to back.
[0012] In one embodiment, determining the recovery priority of each placement group based on the number of missing disks in each placement group includes:
[0013] The number of live replicas for each placement group is determined based on the number of missing disks in each placement group.
[0014] The recovery priority of each placement group is determined based on the number of surviving copies of each placement group, wherein the placement group with fewer surviving copies has a higher recovery priority.
[0015] In one embodiment, determining the recovery priority of each placement group based on the number of missing disks in each placement group includes:
[0016] Obtain the number of redundant blocks in the erasure coding corresponding to the distributed cluster storage pool;
[0017] The recovery priority of each placement group is determined based on the number of missing disks and the number of redundant blocks in each placement group.
[0018] In one embodiment, determining the recovery priority of each placement group based on the number of missing disks and the number of redundant blocks in each placement group includes:
[0019] Obtain the difference between the number of missing disks and the number of redundant blocks in each of the placement groups;
[0020] When the difference is not greater than 0, it is determined that among the placement groups with a difference not greater than 0, the placement group with more missing disks has a higher recovery priority.
[0021] When the difference is greater than 0, it is determined that the placement group with the difference greater than 0 cannot perform data recovery.
[0022] In one embodiment, the method further includes:
[0023] Check if there is any data currently being restored;
[0024] If so, stop the data being restored and restore the placement group with the highest restoration priority.
[0025] In one embodiment, stopping the data being recovered and restoring the placement group with the highest recovery priority includes:
[0026] Check whether the recovery priority of the data being recovered is the same as the highest recovery priority;
[0027] If they are the same, the placement group with the highest recovery priority will be placed after the data being recovered. After the data being recovered is completed, the placement group with the highest recovery priority will be recovered.
[0028] If not, the data being restored will be stopped, and the placement group with the highest restoration priority will be restored.
[0029] In one embodiment, the method further includes:
[0030] Scan the recovery queue at preset intervals;
[0031] Detect whether the placement groups in the recovery queue are arranged in descending order of recovery priority;
[0032] If not, the placement groups in the recovery queue are rearranged in descending order of recovery priority.
[0033] This application also discloses a data recovery device, including:
[0034] The first determination module is used to determine the number of missing disks in each placement group after a disk failure occurs in the distributed cluster storage pool.
[0035] The second determining module is used to determine the recovery priority of each placement group based on the number of missing disks in each placement group, wherein the placement group with more missing disks has a higher recovery priority.
[0036] The writing module is used to write each of the placement groups into the recovery queue in descending order of recovery priority, wherein the placement groups with higher recovery priority are placed before the placement groups with lower recovery priority.
[0037] The recovery module is used to restore the data of each placement group in the recovery queue in a sequential order from front to back.
[0038] This application also discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;
[0039] The memory is used to store computer programs;
[0040] When the processor executes a program stored in the memory, it implements the method described in the embodiments of this application.
[0041] This application also discloses one or more computer-readable media storing instructions that, when executed by one or more processors, cause the processors to perform the methods described in this application.
[0042] The embodiments of this application have the following advantages:
[0043] After a series of disk failures in the storage pool, the severity of each placement group can be determined based on the number of missing disks in each data group placement group. The more missing disks, the higher the severity. The recovery priority of each placement group is determined based on the severity of each placement group, and the higher the severity, the higher the recovery priority. In this way, placement groups with higher severity can be recovered first. Even if disk failures continue, it will not affect the data recovery of placement groups, thus improving the reliability of data recovery. Attached Figure Description
[0044] Figure 1 is a flowchart of a data recovery method provided in an embodiment of this application;
[0045] Figure 2 is a flowchart of a data recovery method provided in an embodiment of this application.
[0046] Figure 3 is a schematic diagram of the placement group recovery appendix provided in the embodiment of this application;
[0047] Figure 4 is a schematic diagram of the second appendix for the restoration of placement groups provided in the embodiments of this application;
[0048] Figure 5 is a schematic diagram of the placement group recovery appendix provided in the embodiments of this application;
[0049] Figure 6 is a schematic diagram of the placement group recovery appendix provided in the embodiments of this application;
[0050] Figure 7 is a structural block diagram of a data recovery device provided in an embodiment of this application;
[0051] Figure 8 is a block diagram of an electronic device provided in an embodiment of this application;
[0052] Figure 9 is a schematic diagram of a computer-readable medium provided in an embodiment of this application. Specific Implementation
[0053] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0054] Ceph, a well-known distributed storage system, employs two data redundancy methods to achieve cluster data reliability. One is replication, where each data object in the storage pool is replicated and distributed across different disks (referred to as OSDs in the cluster). If a disk fails, all data stored on that disk is lost, but because other disks have identical copies of the data, no actual data loss occurs. The other method is erasure coding. In a storage pool using erasure coding, each data object is divided into k data blocks. These k blocks are used as input to a deterministic algorithm to generate m redundant blocks. These k+m blocks are then distributed across different disks. If a disk fails, all data stored on that disk is lost, but because other disks have blocks calculated by the algorithm, and the number of simultaneously damaged blocks does not exceed m, the storage cluster can recover the lost blocks using the same algorithm (this is based on the mathematical guarantee of the algorithm), thus preventing actual data loss.
[0055] In addition to implementing two data redundancy methods, the distributed storage system Ceph also implements a self-healing mechanism. When disk failure leads to the complete loss of data on the disk, the storage cluster will detect the situation and automatically recover the missing data. Specifically, the data stored on the disk is actually divided into groups, with each data object uniquely corresponding to a data group called a placement group. In replica mode, each placement group has a unique ID, and the placement group entities will exist on different OSDs and be completely identical. In erasure coding mode, placement groups with the same ID will have k+m blocks. After a disk failure, the storage cluster will use data placement groups on other disks to recover the missing data. The process differs slightly between replica mode and erasure coding mode. In replica mode, since all placement groups are identical, the storage pool can directly copy data from other replica placement groups to recover the lost placement groups. In erasure coding mode, the storage pool needs to obtain k blocks of the placement group as input based on an algorithm to recalculate the lost placement group data. The final result is that the entire data group with missing data will be restored to the currently working disk. The whole process is automatic by Ceph and cannot be manually intervened.
[0056] In real-world production environments, disk failures can occur consecutively. For example, in a storage pool with 3-replica redundancy, two disks might fail consecutively, or in a storage pool with k+m of 4+2, two disks might fail consecutively. In such cases, although the data remains reliable and undamaged, if another disk fails before the already lost data has been fully recovered, some data will be permanently lost and unrecoverable. Ceph's implementation treats data placement groups with different levels of damage equally for recovery. However, due to recovery speed limitations, not all placement groups can be recovered simultaneously. Therefore, Ceph uses a recovery queue for sorting. This default recovery method has a hidden danger: data with fewer missing placement groups may be recovered first, while data with more missing placement groups may suffer further disk damage and be lost before data recovery is complete.
[0057] To address the aforementioned technical problems, this application proposes a data recovery method, apparatus, electronic device, and readable medium.
[0058] First, let's introduce the technical terminology used in this application:
[0059] Distributed storage: A storage method that distributes data across multiple nodes.
[0060] Placement group (PG): A group of data stored on disk in the Ceph distributed storage cluster. All data objects in the cluster are uniquely associated with a placement group.
[0061] Erasure coding: a coding technique for data redundancy and fault tolerance. It divides data into multiple segments and generates some redundant segments, so that even if some of these segments are damaged or lost, the data can still be recovered from the redundant segments.
[0062] Referring to Figure 1, a flowchart of the steps of a data recovery method provided in an embodiment of this application is shown, which may specifically include the following steps S101-S104:
[0063] S101. After a disk failure occurs in the distributed cluster storage pool, determine the number of missing disks in each placement group.
[0064] After a disk failure occurs in the distributed cluster storage pool, all placement groups in the storage pool are immediately checked to determine the number of missing disks in each placement group.
[0065] S102. Determine the recovery priority of each placement group based on the number of missing disks in each placement group, wherein the placement group with more missing disks has a higher recovery priority.
[0066] Specifically, since distributed storage systems have two redundancy methods, it is necessary to analyze the redundancy methods used by the storage pool to determine the priority level for recovery.
[0067] When the redundancy mode is replica mode, the recovery priority of each placement group is determined based on the number of missing disks in each placement group, including the following sub-steps A1-A2:
[0068] A1. Determine the number of live replicas for each placement group based on the number of missing disks in each placement group.
[0069] A2. Determine the recovery priority of each placement group based on the number of surviving copies in each placement group. Placement groups with fewer surviving copies have a higher recovery priority.
[0070] For storage pools with replica redundancy, the number of replicas is calculated and the recovery priority of the placement groups is graded according to the number of missing replicas.
[0071] For example, if the number of replicas in the redundancy mode is 3, then the highest level is when only 1 replica survives, followed by the next highest level when only 2 replicas survive. In other words, the highest level is when only 2 replicas are missing, followed by the next highest level.
[0072] This is because if the placement group with 2 live copies is restored first, while the placement group with 1 live copy is placed in the recovery queue, if another disk failure occurs during the restoration process, the placement group with 1 live copy may lose its remaining copy and become unrecoverable. Therefore, during restoration, the placement group with 1 live copy can be restored first, while the placement group with 2 live copies is placed in the recovery queue. In this way, even if another disk failure occurs during the restoration process, the placement group with 2 live copies may lose another copy, reducing its live copy count to 1. However, even with its live copy count reduced to 1, data recovery can still be achieved without data loss.
[0073] When the redundancy mode is erasure coding, the recovery priority of each placement group is determined based on the number of missing disks in each placement group, including the following sub-steps B1-B2:
[0074] B1. Obtain the number of redundant blocks in the erasure coding corresponding to the distributed cluster storage pool.
[0075] For erasure coding, you can first obtain the number of redundant blocks in the erasure coding of the distributed cluster storage pool.
[0076] The number of redundant blocks in erasure coding can be obtained using the following method:
[0077] If the total number of data blocks n and the number of original data blocks k are known, the number of redundant blocks can be calculated using the formula m = nk. For example, if there are a total of 8 data blocks, of which 6 are original data blocks, then the number of redundant blocks m = 8 - 6 = 2.
[0078] In storage systems, erasure coding configuration parameters are typically stored in configuration files. These parameters can be obtained by viewing the storage system's management interface or by directly accessing the storage device's configuration file. Different storage systems may have different configuration file formats and locations. For example, in Ceph storage systems, the configuration file stores detailed information about the erasure-coded pool, including parameters of the coding scheme.
[0079] Many storage systems provide application programming interfaces (APIs) to query erasure coding information. For example, the OpenStack Swift storage system's API can be used to query erasure coding configurations in object storage. By sending a specific request to the Swift API, you can obtain the erasure coding parameters used by a bucket or object, including the number of redundant blocks.
[0080] Some programming languages provide dedicated libraries for handling storage and erasure coding. For example, in Python, there are third-party libraries that can interact with storage systems, such as the boto3 library for interacting with S3 storage services. If the S3 bucket uses erasure coding, the boto3 library can be used to query the relevant erasure coding parameters to obtain the number of redundant blocks.
[0081] B2. Determine the recovery priority of each placement group based on the number of missing disks and redundant blocks in each placement group.
[0082] Specifically, based on the number of missing disks and redundant blocks in each placement group, the recovery priority of each placement group is determined, including the following sub-steps B11-B13:
[0083] B11. Obtain the difference between the number of missing disks and the number of redundant blocks for each placement group.
[0084] The more disks missing, the more difficult and dangerous the recovery becomes; the fewer disks missing, the easier and less dangerous the recovery becomes. Therefore, after obtaining the number of redundant blocks in erasure coding, the difference between the number of missing disks and the number of redundant blocks in each placement group can be obtained.
[0085] B12. When the difference is not greater than 0, among the placement groups with a difference not greater than 0, the placement group with more missing disks corresponds to a higher recovery priority.
[0086] B13. When the difference is greater than 0, it is determined that placement groups with a difference greater than 0 cannot perform data recovery.
[0087] When the difference is greater than 0, it indicates that the number of missing disks is too large to recover the placement group. In this case, the placement group does not need to be written into the recovery queue. When the difference is not greater than 0, it indicates that the number of missing disks is still large enough to recover the placement group. In this case, the placement group can be written into the recovery queue.
[0088] In this application, the more missing disks a placement group has, the higher its recovery priority. Taking erasure coding 4+2 as an example, data recovery can be achieved as long as the number of missing disks is greater than or equal to 4. If the number of missing disks is 4, the calculated difference is 2; if the number of missing disks is 2, the calculated difference is 0; if the number of missing disks is 1, the calculated difference is -1. When the difference is 2, it indicates that data recovery cannot be performed, and the data is not written to the recovery queue. When the difference is 2, if another disk fails, the data cannot be recovered. However, when the difference is -1, if another disk fails, the data can still be recovered. Therefore, in this application, the recovery priority of placement groups with a difference of 2 is set to the highest, and they are placed first in the recovery queue to prioritize the recovery of this placement group.
[0089] In this application, since Ceph's recovery queue does not perform reliability grading on the placement groups to be recovered, this application uses an algorithm to perform reliability statistics and grading of placement groups (here, reliability represents the recovery priority in the above embodiments; the lower the reliability, the higher the recovery priority, and the higher the reliability, the lower the recovery priority). This application first obtains the erasure coding ratio k and m of the storage pool, then counts all placement groups in the entire storage pool, and grades them according to the number of missing placement group replicas. The placement group missing m blocks has the highest recovery priority grading level, the placement group missing m-1 blocks has the next highest recovery priority grading level, and so on. The placement group with the highest recovery priority is set to the highest recovery priority (255), and the running resources of the placement group currently being recovered are immediately preempted, and recovery is performed immediately. The placement group with the second highest recovery priority is set to the second highest recovery priority (254), but the running resources of the placement group currently being recovered are not preempted, but the recovery data is waited for.
[0090] S103. Write each placement group into the recovery queue in descending order of recovery priority.
[0091] The placement group with the highest recovery priority is placed before the placement group with the lowest recovery priority; that is, the placement group with the highest recovery priority is first in the recovery queue.
[0092] The recovery queue in this application is a queue that stores tasks or data items waiting for recovery operations. During recovery, the tasks are recovered sequentially from front to back in the recovery queue.
[0093] It is worth noting that if multiple placement groups have the same recovery priority, the placement groups with the same recovery priority will be arranged in the recovery queue according to the order of detection time.
[0094] S104. Restore the data of each placement group in the recovery queue in order from front to back.
[0095] In this application, after a series of disk failures occur in the storage pool, the degree of danger of each placement group can be determined based on the number of missing disks in each data group placement group, and the recovery priority of each placement group can be determined based on the degree of danger of each placement group. The higher the degree of danger, the higher the recovery priority. In this way, placement groups with a high degree of danger can be recovered first. Even if disk failures continue, it will not affect the data recovery of placement groups with a large number of missing disks, thus improving the reliability of data recovery.
[0096] In one embodiment, as shown in FIG2, the above method further includes the following sub-steps S105-S107:
[0097] S105, Interval preset time interval scan recovery queue.
[0098] The preset interval scan means that the system will check the recovery queue at a pre-set time interval.
[0099] For example, in software systems, timers can be used to implement interval-based scanning. Many high-level programming languages (such as Java) provide timer-related classes (e.g., java.util.Timer and java.util.TimerTask). Scanning of the recovery queue can be achieved by setting the timer interval and writing code to scan the recovery queue in the run method of the timer task.
[0100] In some system-level recovery operations, the operating system's task scheduler can be used to implement interval scanning. By configuring task scheduling policies, the system can start processes that scan the recovery queue at specific time intervals. For example, in Linux systems, the cron task scheduling tool can be used to set execution rules for scheduled tasks to periodically execute scripts that scan the recovery queue.
[0101] S106. Check whether the placement groups in the recovery queue are arranged in descending order of recovery priority.
[0102] The recovery queue is checked at pre-set time intervals to check whether the placement groups in the recovery queue are arranged in descending order of recovery priority.
[0103] S107. If not, then rearrange the placement groups in the recovery queue in descending order of recovery priority.
[0104] If it is detected that the placement groups in the recovery queue are not arranged in descending order of recovery priority, that is, the order is disordered, the recovery queue will be reordered so that placement groups with higher recovery priority are placed before placement groups with lower recovery priority, thereby allowing placement groups with higher recovery priority to recover data first.
[0105] In this application, the entire recovery queue will also be scanned periodically to check whether the placement groups in the current queue are arranged according to their hierarchical level. If there are any changes, the placement groups will be re-ranked immediately to ensure that placement groups with higher hierarchical levels can achieve faster recovery.
[0106] After the group is classified, the recovery queue is scanned and maintained regularly to ensure the effectiveness of the classified recovery.
[0107] In this application, after scanning the placement group with the highest recovery priority, it will also check whether there is data being recovered; if so, it will stop the data being recovered and restore the placement group with the highest recovery priority.
[0108] Specifically: Check if the recovery priority of the data being recovered is the same as the highest recovery priority; if so, place the highest recovery priority placement group after the data being recovered, wait for the data being recovered to be completed, and then restore the highest recovery priority placement group; if not, stop the data being recovered and restore the highest recovery priority placement group.
[0109] The system checks if the recovery priority of the data being recovered is the highest. If it is, it does not preempt the current recovery resources, but waits until the recovery is complete before attempting to recover the placement group with the highest recovery priority. If the system detects that the recovery priority of the data being recovered is not the highest, it directly preempts the current recovery resources and recovers the placement group with the highest recovery priority.
[0110] This application, combined with Ceph's data recovery mechanism, intervenes in the queue priority of placement groups during automatic data recovery when a disk failure occurs in the storage pool. It categorizes placement groups by recovery priority, ensuring that more critical data receives priority for recovery, thereby reducing the risk to the storage pool. The following detailed examples illustrate the solution in this application:
[0111] Taking erasure coding 4+2 as an example, when a disk in the storage pool fails, as shown in Figure 3 by default, all the placement groups involved will have 1 missing block. Ceph's recovery mechanism will automatically start, put the placement groups involved into a recovery queue, and then perform data recovery from the head of the recovery queue. At the same time, the number of placement groups performing data recovery is 1.
[0112] As shown in Figure 4, if another disk in the storage pool fails, the number of missing blocks in some placement groups reaches 2. At this time, the order of the recovery queue does not change, and the recovery continues according to the previous recovery queue. That is, the difference between the number of missing disks in all placement groups and the number of redundancy codes 2 is not greater than 0.
[0113] If the priority of each placement group in the recovery queue is not changed in Figure 4, when the situation shown in Figure 5 occurs, and the storage pool experiences another disk failure while the missing data in Figure 4 has not been fully recovered, the number of missing blocks in the placement group exceeds 2, resulting in permanent data loss. That is, in the case of Figure 5, the difference between the number of missing disks in the placement group and the number of redundant codes 2 is greater than 0.
[0114] Using the method of this application, the recovery queues in Figure 4 will be reordered. The recovery priority of the placement group with 2 missing blocks is higher than that of the placement group with 1 missing block. The placement group with 2 missing blocks is written at the beginning of the recovery queue, and the placement group with 1 missing block is written after the placement group with 2 missing blocks in the recovery queue. This is to prioritize the recovery of data in the placement group with 2 missing blocks when recovering data, as shown in Figure 6.
[0115] In Figures 3-6, the recovering characterization is in progress. - The "wait" signifies waiting for a response.
[0116] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.
[0117] Referring to Figure 7, a structural block diagram of a data recovery device provided in an embodiment of this application is shown, which may specifically include the following modules:
[0118] The first determining module 11 is used to determine the number of missing disks in each placement group after a disk failure occurs in the distributed cluster storage pool.
[0119] During data recovery, after a disk failure occurs in the distributed cluster storage pool, the first determination module 11 will immediately check all placement groups in the storage pool to determine the number of missing disks in each placement group.
[0120] The second determining module 12 is used to determine the recovery priority of each placement group based on the number of missing disks in each placement group, wherein the placement group with more missing disks has a higher recovery priority.
[0121] The second determining module 12 determines the recovery priority of each placement group based on the number of missing disks in each placement group determined by the first determining module 11. In this application, the placement group with more missing disks has a higher recovery priority.
[0122] The writing module 13 is used to write each of the placement groups into the recovery queue in descending order of recovery priority; the placement groups with higher recovery priority are placed before the placement groups with lower recovery priority.
[0123] The writing module 13 writes the data into the recovery queue in descending order of recovery priority, according to the recovery priority of each placement group determined by the second determining module 12.
[0124] The recovery module 14 is used to restore the data of each placement group in the recovery queue in a sequential order from front to back.
[0125] After the writing module 13 writes each placement group in the recovery queue in descending order of recovery priority, the recovery module 14 will restore each placement group in the recovery queue in descending order of recovery priority.
[0126] In one embodiment, the second determining module is specifically used for:
[0127] The number of live replicas for each placement group is determined based on the number of missing disks in each placement group.
[0128] The recovery priority of each placement group is determined based on the number of surviving copies of each placement group, wherein the placement group with fewer surviving copies has a higher recovery priority.
[0129] In one embodiment, the second determining module is specifically used for:
[0130] Obtain the number of redundant blocks in the erasure coding corresponding to the distributed cluster storage pool;
[0131] The recovery priority of each placement group is determined based on the number of missing disks and the number of redundant blocks in each placement group.
[0132] In one embodiment, in determining the recovery priority of each placement group based on the number of missing disks and the number of redundant blocks in each placement group, the second determining module is specifically used for:
[0133] Obtain the difference between the number of missing disks and the number of redundant blocks in each of the placement groups;
[0134] When the difference is not greater than 0, it is determined that among the placement groups with a difference not greater than 0, the placement group with more missing disks has a higher recovery priority.
[0135] When the difference is greater than 0, it is determined that the placement group with the difference greater than 0 cannot perform data recovery.
[0136] In one embodiment, the apparatus further includes:
[0137] The first detection module is used to detect whether there is currently any data being restored.
[0138] The stop module is used to stop the data being restored if the first detection module detects that there is data being restored, and to restore the placement group with the highest restoration priority.
[0139] In one embodiment, the stopping module is specifically used for:
[0140] Check whether the recovery priority of the data being recovered is the same as the highest recovery priority;
[0141] If they are the same, the placement group with the highest recovery priority will be placed after the data being recovered. After the data being recovered is completed, the placement group with the highest recovery priority will be recovered.
[0142] If not, the data recovery process will be stopped, and the placement group with the highest recovery priority will be restored.
[0143] In one embodiment, the apparatus further includes:
[0144] The scanning module is used to scan the recovery queue at preset time intervals;
[0145] The second detection module is used to detect whether each of the placement groups in the recovery queue is arranged from high to low according to the recovery priority;
[0146] The sorting module is configured to, if the detection module detects that the placement groups in the recovery queue are not arranged in descending order of recovery priority, rearrange the placement groups in the recovery queue in descending order of recovery priority.
[0147] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.
[0148] In addition, this application embodiment also provides an electronic device, as shown in FIG8, including a processor 1301, a communication interface 1302, a memory 1303, and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304.
[0149] Memory 1303 is used to store computer programs;
[0150] When processor 1301 executes a program stored in memory 1303, it performs the following steps:
[0151] After a disk failure occurs in the distributed cluster storage pool, determine the number of missing disks in each placement group;
[0152] The recovery priority of each placement group is determined based on the number of missing disks in each placement group, wherein the placement group with more missing disks has a higher recovery priority.
[0153] Each placement group is written into the recovery queue in descending order of recovery priority; the placement group with higher recovery priority is placed before the placement group with lower recovery priority.
[0154] The data of each placement group in the recovery queue is restored sequentially from front to back.
[0155] In one embodiment, determining the recovery priority of each placement group based on the number of missing disks in each placement group includes:
[0156] The number of live replicas for each placement group is determined based on the number of missing disks in each placement group.
[0157] The recovery priority of each placement group is determined based on the number of surviving copies of each placement group, wherein the placement group with fewer surviving copies has a higher recovery priority.
[0158] In one embodiment, determining the recovery priority of each placement group based on the number of missing disks in each placement group includes:
[0159] Obtain the number of redundant blocks in the erasure coding corresponding to the distributed cluster storage pool;
[0160] The recovery priority of each placement group is determined based on the number of missing disks and the number of redundant blocks in each placement group.
[0161] In one embodiment, determining the recovery priority of each placement group based on the number of missing disks and the number of redundant blocks in each placement group includes:
[0162] Obtain the difference between the number of missing disks and the number of redundant blocks in each of the placement groups;
[0163] When the difference is not greater than 0, it is determined that among the placement groups with a difference not greater than 0, the placement group with more missing disks has a higher recovery priority.
[0164] When the difference is greater than 0, it is determined that the placement group with the difference greater than 0 cannot perform data recovery.
[0165] In one embodiment, it also includes:
[0166] Check if there is any data currently being restored;
[0167] If so, stop the data being restored and restore the placement group with the highest restoration priority.
[0168] In one embodiment, stopping the data being recovered and restoring the placement group with the highest recovery priority includes:
[0169] Check whether the recovery priority of the data being recovered is the same as the highest recovery priority;
[0170] If they are the same, the placement group with the highest recovery priority will be placed after the data being recovered. After the data being recovered is completed, the placement group with the highest recovery priority will be recovered.
[0171] If not, the data recovery process will be stopped, and the placement group with the highest recovery priority will be restored.
[0172] In one embodiment, it also includes:
[0173] Scan the recovery queue at preset intervals;
[0174] Detect whether the placement groups in the recovery queue are arranged in descending order of recovery priority;
[0175] If not, the placement groups in the recovery queue are rearranged in descending order of recovery priority.
[0176] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.
[0177] The communication interface is used for communication between the aforementioned terminal and other devices.
[0178] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0179] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0180] As shown in Figure 9, in another embodiment provided in this application, a computer-readable storage medium 1401 is also provided, which stores instructions that, when run on a computer, cause the computer to execute the data recovery method described in the above embodiments.
[0181] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the data recovery method described in the above embodiments.
[0182] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).
[0183] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0184] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0185] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.
Claims
1. A data recovery method, characterized in that, include: After a disk failure occurs in the distributed cluster storage pool, determine the number of missing disks in each placement group; The recovery priority of each placement group is determined based on the number of missing disks in each placement group, wherein the placement group with more missing disks has a higher recovery priority. Each placement group is written into the recovery queue in descending order of recovery priority; the placement group with higher recovery priority is placed before the placement group with lower recovery priority. The data of each placement group in the recovery queue is restored sequentially from front to back.
2. The method according to claim 1, characterized in that, The step of determining the recovery priority of each placement group based on the number of missing disks in each placement group includes: The number of live replicas for each placement group is determined based on the number of missing disks in each placement group. The recovery priority of each placement group is determined based on the number of surviving copies of each placement group, wherein the placement group with fewer surviving copies has a higher recovery priority.
3. The method according to claim 1, characterized in that, The step of determining the recovery priority of each placement group based on the number of missing disks in each placement group includes: Obtain the number of redundant blocks in the erasure coding corresponding to the distributed cluster storage pool; The recovery priority of each placement group is determined based on the number of missing disks and the number of redundant blocks in each placement group.
4. The method according to claim 3, characterized in that, The step of determining the recovery priority of each placement group based on the number of missing disks and the number of redundant blocks in each placement group includes: Obtain the difference between the number of missing disks and the number of redundant blocks in each of the placement groups; When the difference is not greater than 0, it is determined that among the placement groups with a difference not greater than 0, the placement group with more missing disks has a higher recovery priority. When the difference is greater than 0, it is determined that the placement group with the difference greater than 0 cannot perform data recovery.
5. The method according to claim 1, characterized in that, The method further includes: Check if there is any data currently being restored; If so, stop the data being restored and restore the placement group with the highest restoration priority.
6. The method according to claim 5, characterized in that, The step of stopping the data being restored and restoring the placement group with the highest restoration priority includes: Check whether the recovery priority of the data being recovered is the same as the highest recovery priority; If they are the same, the placement group with the highest recovery priority will be placed after the data being recovered. After the data being recovered is completed, the placement group with the highest recovery priority will be recovered. If not, the data being restored will be stopped, and the placement group with the highest restoration priority will be restored.
7. The method according to claim 1, characterized in that, The method further includes: The recovery queue is scanned at preset intervals; Detect whether the placement groups in the recovery queue are arranged in descending order of recovery priority; If not, the placement groups in the recovery queue are rearranged in descending order of recovery priority.
8. A data recovery device, characterized in that, include: The first determination module is used to determine the number of missing disks in each placement group after a disk failure occurs in the distributed cluster storage pool. The second determining module is used to determine the recovery priority of each placement group based on the number of missing disks in each placement group, wherein the placement group with more missing disks has a higher recovery priority. The writing module is used to write each of the placement groups into the recovery queue in descending order of recovery priority, wherein the placement groups with higher recovery priority are placed before the placement groups with lower recovery priority. The recovery module is used to restore the data of each placement group in the recovery queue in a sequential order from front to back.
9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method as described in any one of claims 1-7.
10. A computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method as described in any one of claims 1-7.