Fault processing method and computing device

Abstract

A fault processing method and a computing device. The method is applied to a distributed storage system, the distributed storage system comprises at least one storage node, the storage node comprises at least one disk, and the disk comprises one or more actuator arms. The method comprises: acquiring fault information; determining a fault type on the basis of the fault information, the fault type comprising disk fault and actuator arm fault; and on the basis of the fault type, reconstructing at least one faulty logical unit in a faulty disk.

Description

Troubleshooting methods and computing equipment

[0001] Cross-references to related applications

[0002] This application is based on Chinese Patent Application No. 202510124185.5, filed on January 26, 2025, entitled "A Fault Handling Method and Computing Device", and claims priority to that Chinese Patent Application, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer technology, and in particular to a fault handling method and a computing device. Background Technology

[0004] In distributed storage systems, erasure coding (EC) algorithms / techniques are typically used to ensure the reliability of disk data and node data in order to improve the reliability of the storage system.

[0005] Erasure coding is a coding technique characterized by high fault tolerance and high storage efficiency; it is also a data fragment redundancy protection mechanism. In a distributed storage system employing erasure coding, data is stored using EC (Extended Check Set) mechanisms. When writing data, a data segment is divided into multiple data fragments, and multiple check fragments are generated based on these data fragments (the multiple data fragments and check fragments corresponding to a single data segment together constitute an EC). These data fragments and check fragments are then distributed and stored in logical units within storage nodes. When a fragment (data fragment or logical fragment) is lost due to a logical unit failure, the lost fragment can be reconstructed, ensuring data reliability.

[0006] In current technology, the common approach to handling disk failures is as follows: when a logical unit in a distributed storage system fails, the data in the failed logical unit is reconstructed, and the corresponding failed disk is replaced. For a single-arm disk, it presents a single logical unit to the upper-level operating system; when this logical unit fails, the corresponding single-arm disk fails. Therefore, reconstructing the data in the failed logical unit and replacing the single-arm disk is sufficient. However, for multi-arm disks, which present at least two logical units to the upper-level operating system, when some logical units in a multi-arm disk fail, reconstructing the data in the failed logical units and replacing the multi-arm disk will result in the loss of data in the remaining logical units of the multi-arm disk.

[0007] Therefore, current disk failure handling methods cannot guarantee the reliability of data in distributed storage systems, and may lead to data loss or even EC degradation. Summary of the Invention

[0008] The fault handling method and computing device provided in this application embodiment are applicable to distributed storage systems including multi-arm disks, and can ensure the reliability of data in distributed storage systems and avoid data loss or even EC degradation.

[0009] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0010] Firstly, embodiments of this application provide a fault handling method applied to a distributed storage system. The distributed storage system includes at least one storage node, each storage node including at least one disk, and each disk including one or more read / write arms. The method includes: acquiring fault information; determining the fault type based on the fault information, where the fault type includes disk failure and read / write arm failure; and reconstructing at least one faulty logical unit in the faulty disk based on the fault type. Embodiments of this application can effectively distinguish the fault type of a faulty disk, that is, it can distinguish whether a multi-reader disk is experiencing a complete disk failure or a partial read / write arm failure, thereby determining the scope of data reconstruction, avoiding data loss or even EC (Extended EC) degradation, and thus ensuring the reliability of the distributed storage system.

[0011] In one possible implementation, when the fault type is a disk fault, the data stored in all logical units of the faulty disk is reconstructed.

[0012] In one possible implementation, when the fault type is disk failure, after reconstructing the data stored in all logical units of the failed disk, a disk replacement flag corresponding to the failed disk is set as the first disk replacement flag, which is used to indicate the replacement of the failed disk. In response to the first disk replacement flag being set, a disk replacement operation is performed on the failed disk. The distributed storage system automatically performs the disk replacement operation on the failed disk in response to the first disk replacement flag being set.

[0013] In one possible implementation, when the fault type is a magnetic arm fault, the data stored in the fault logic unit in the faulty disk is reconstructed.

[0014] In one possible implementation, when the fault type is a read / write head failure, after reconstructing the data stored in the faulty logical unit of the faulty disk, the disk replacement flag corresponding to the faulty disk is set to a second disk replacement flag, which indicates that the faulty disk should not be replaced; and / or the health status corresponding to the faulty disk is set to a sub-healthy state, where the health status indicates the current state of the disk. Setting the disk replacement flag corresponding to the faulty disk to the second disk replacement flag indicates that the faulty disk should not be replaced, thus avoiding data loss or even EC degradation; setting the health status corresponding to the faulty disk to a sub-healthy state indicates the current state of the faulty disk, meaning that the disk still has some non-fatal problems, but does not need to be replaced.

[0015] In one possible implementation, after reconstructing at least one faulty logical unit in the faulty disk based on the fault type, the distributed storage system is expanded when the unusable storage capacity exceeds a first preset storage capacity. When the unusable storage capacity exceeds the first preset storage capacity, it indicates that the available storage capacity in the distributed storage system is insufficient, leading to a significant performance degradation or even a performance bottleneck that fails to meet user requirements. Therefore, the distributed storage system is expanded to satisfy user needs (performance requirements, storage capacity requirements, etc.).

[0016] In one possible implementation, based on fault information, it is determined whether the proportion of the faulty logical unit in the faulty disk among all logical units on the faulty disk is not less than a preset proportion; if yes, the fault type is disk fault; if no, the fault type is read / write head fault. When the proportion of the faulty logical unit in the faulty disk among all corresponding logical units is not less than the preset proportion, it means that the proportion of the faulty read / write head among all read / write heads is not less than the preset proportion, so the fault type of the disk is disk fault; otherwise, the fault type is read / write head fault.

[0017] In one possible implementation, the system iterates through the identifiers of faulty logical units included in the fault information; based on the identifiers of the faulty logical units and the stored annotation information, it identifies the corresponding faulty disk; based on the identifiers and annotation information of the faulty disk, it identifies all logical units belonging to the faulty logical unit; and based on the fault information, it determines whether the proportion of faulty logical units in the faulty disk among all logical units in the faulty disk is not less than a preset proportion. Through the stored annotation information, the faulty disk and all logical units within it can be quickly located, thereby accurately determining whether the proportion of faulty logical units in the faulty disk among all logical units in the faulty disk is not less than a preset proportion.

[0018] In one possible implementation, the distributed storage system rebalancing its storage capacity to ensure that the storage capacity usage of each storage node and each disk is balanced, that is, to ensure that the utilization rate of all storage nodes or disks is similar, avoiding overload of some storage nodes or disks. This optimizes performance, improves resource utilization, and enhances the stability and reliability of the distributed storage system.

[0019] Secondly, embodiments of this application provide a fault handling device applied to a distributed storage system. The distributed storage system includes at least one storage node, each storage node includes at least one disk, and each disk includes one or more read / write arms. The fault handling device includes: an information acquisition module for acquiring fault information; a type determination module for determining the fault type based on the fault information, whereby the fault type includes disk failure and read / write arm failure; and a data reconstruction module for reconstructing at least one faulty logical unit in the faulty disk based on the fault type. Embodiments of this application can effectively distinguish the fault type of a faulty disk, that is, it can distinguish whether a multi-reader disk is experiencing a complete disk failure or a partial read / write arm failure, thereby determining the scope of data reconstruction, avoiding data loss or even EC (Extended EC) degradation, and thus ensuring the reliability of the distributed storage system.

[0020] In one possible implementation, the data reconstruction module includes a faulty disk reconstruction module and a faulty read / write head reconstruction module. The faulty disk reconstruction module is used to reconstruct the data stored in all logical units of the faulty disk based on the fault type being disk fault; the faulty disk reconstruction module is used to reconstruct the data stored in the faulty logical units within the faulty disk based on the fault type being disk fault.

[0021] Thirdly, embodiments of this application provide a distributed storage system, which includes at least one storage node, each storage node including at least one disk, and each disk including one or more magnetic arms; the distributed storage system is used to implement the fault handling method in the first aspect and its various possible implementations described above.

[0022] Fourthly, embodiments of this application provide a computing device, including: a processor and a memory; the processor runs storage system software, which is used to manage and maintain a distributed storage system; the memory stores computer program instructions; and the processor executes the computer program instructions stored in the memory to implement the fault handling method in the first aspect and its various possible implementations described above.

[0023] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions that, when executed on a computing device, cause the computing device to implement the fault handling method described in the first aspect and its various possible implementations.

[0024] Sixthly, embodiments of this application provide a computer program product including computer program instructions that, when executed on a computing device, cause the computing device to implement the fault handling method described in the first aspect and its various possible implementations. Attached Figure Description

[0025] Figure 1 is an example diagram of a data storage scenario of a distributed storage system provided in an embodiment of this application;

[0026] Figure 2 is an example diagram of a data storage scenario for another distributed storage system provided in an embodiment of this application;

[0027] Figure 3 is a schematic diagram of the composition of a data storage system architecture provided in an embodiment of this application;

[0028] Figure 4A is a schematic flowchart of a fault handling method provided in an embodiment of this application;

[0029] Figure 4B is a flowchart illustrating another fault handling method provided in an embodiment of this application;

[0030] Figure 5 is a structural example diagram of a distributed storage system provided in an embodiment of this application;

[0031] Figure 6 is a flowchart illustrating a fault handling method corresponding to a disk failure provided in an embodiment of this application;

[0032] Figure 7 is a flowchart illustrating a fault handling method corresponding to a magnetic arm fault provided in an embodiment of this application;

[0033] Figure 8 is an application example diagram of a fault handling method provided in an embodiment of this application;

[0034] Figure 9 is a flowchart illustrating another fault handling method provided in an embodiment of this application;

[0035] Figure 10 is a flowchart illustrating another fault handling method provided in an embodiment of this application;

[0036] Figure 11 is a flowchart illustrating another fault handling method provided in an embodiment of this application;

[0037] Figure 12 is an application example diagram of another fault handling method provided in the embodiments of this application;

[0038] Figure 13 is a flowchart illustrating another fault handling method provided in an embodiment of this application. Detailed Implementation

[0039] The terms "first," "second," and "third," etc., used in this application specification, claims, and drawings are used to distinguish different objects, not to limit a specific order.

[0040] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0041] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the related technologies is given first:

[0042] With the rapid development of technologies such as High-Performance Computing (HPC), Big Data, and Artificial Intelligence (AI), massive amounts of data are impacting various industries at an unprecedented growth rate. To effectively manage and maintain this data, users have increasingly higher demands for data storage capacity and data processing efficiency. Against this backdrop, disk technology is evolving towards larger capacity and higher performance. Multi-arm disks, in particular, can balance disk capacity and performance, offering data read / write efficiency many times higher than traditional single-arm disks. Therefore, multi-arm disks are increasingly widely used in distributed storage systems.

[0043] In a hard disk drive (HDD), the actuator arm is a mechanical component that connects the read / write head to the drive motor, and is responsible for precisely positioning the read / write head onto the tracks on the disk surface (platter surface).

[0044] A single-actuator disk (HDD) is a hard disk drive that uses a single actuator (or a single read / write head) to read and write data. In other words, a single-actuator disk uses one actuator to serve all tracks on all platters. Specifically, a single-actuator disk contains one actuator, one or more platters, and one or more read / write heads. Data can be stored on both the top and bottom surfaces of a platter, and correspondingly, one head can be placed on each surface for reading and writing. A platter can be divided into multiple sectors. By moving the single actuator, the head can be positioned on the target sector (the sector where data needs to be written or read) on the platter, and then the head can read or write data from the target sector.

[0045] A multi-actuator disk (MAD) is a hardware drive that uses multiple actuators (or read / write heads) to read and write data. In other words, a MAD has multiple actuators that each serve all tracks on multiple disk surfaces. Specifically, an MAD has multiple actuators, multiple platters, and multiple read / write heads. The disks within a MAD can be divided into multiple groups, with each actuator responsible for reading and writing data to one group of platters.

[0046] As can be seen from the above, since multi-arm disks have multiple arms performing data read and write concurrently, while single-arm disks only have one arm performing data read and write, the read and write efficiency of multi-arm disks is much higher than that of single-arm disks.

[0047] Furthermore, it's understandable that since a single-arm disk uses only one arm for reading and writing data, it generally presents a single logical unit to the operating system (OS). This logical unit can also be called a logical volume / disk (LUN / Disk), meaning the ratio of single-arm disk to logical unit (LUN / Disk) is 1:1. Conversely, a multi-arm disk uses multiple arms (e.g., two) for reading and writing data, and these arms can operate simultaneously. Therefore, a multi-arm disk generally presents multiple logical units (LUN / Disk) to the operating system, meaning the ratio of multi-arm disk to logical unit (LUN / Disk) is 1:E (where E is an integer greater than 1). This means that the number of logical units a multi-arm disk presents to the operating system corresponds to the number of arms it has. For example, a multi-arm disk with two arms presents two logical units to the operating system (E=2); a multi-arm disk with three arms presents three logical units (E=3).

[0048] A distributed storage system is a computer network that distributes data across multiple storage nodes. In distributed storage systems, storage nodes are typically built on general-purpose servers, and erasure coding algorithms / techniques are generally used to ensure the reliability of disk data and node data in order to improve the reliability of the distributed storage system.

[0049] Erasure coding (EC) is a coding technique characterized by high fault tolerance and high storage efficiency. It's also a data fragmentation redundancy protection mechanism that achieves data redundancy protection by calculating parity fragments. Specifically, when a distributed storage system writes data, it typically divides the data into N data fragments. Based on these N data fragments, the EC algorithm calculates M parity fragments. These N data fragments and M parity fragments are then distributed and stored in logical units. If a fragment (data fragment or parity fragment) is lost due to a logical unit failure, the original data (i.e., the lost fragment) can be recovered using any N fragments from the N+M fragments. Therefore, if the number of lost fragments in a distributed storage system is less than or equal to M, the lost fragment can be recovered from the remaining fragments. Here, N is a positive integer and M is a positive integer.

[0050] In practical applications, users can configure different EC redundancy ratios based on the number of storage nodes. When the number of storage nodes is ≥ (N+M), an N+M redundancy configuration is adopted. With an N+M redundancy ratio, N+M shards (i.e., N data shards and M parity shards) are stored on logical units on N different storage nodes. Therefore, an N+M redundancy ratio can tolerate the failure of M logical units or M storage nodes without data loss. When ((N+M) / M) ≤ the number of storage nodes < (N+M), a redundancy ratio of N+M:1 or N+M:2 can be used. With an N+M:1 or N+M:2 redundancy ratio, N+M shards are distributed across all storage nodes, with each storage node storing at least one shard. It should be understood that an N+M:1 redundancy ratio can tolerate the failure of M logical units or 1 storage node without data loss, while an N+M:2 redundancy ratio can tolerate the failure of M logical units or 2 storage nodes without data loss. Similarly, the N+M:F redundancy ratio allows for the failure of M logical units or F storage nodes without data loss. F is an integer greater than 0.

[0051] Furthermore, compared to replication technology, EC technology has a higher capacity utilization rate. For example, when EC uses N+2:1 redundancy, its capacity utilization rate is about 66% when n is 1, while when using dual replication (i.e., each piece of data is stored twice), its capacity utilization rate is 50%.

[0052] To make it easier to understand, the following examples, using Figures 1 and 2, illustrate data storage scenarios in distributed storage systems.

[0053] The following section, using Figure 1 as an example, illustrates the scenario where the number of storage nodes in a distributed storage system is greater than or equal to (N+M) when N=4, M=2, and the number of storage nodes is 6.

[0054] As shown in Figure 1(a), each storage node includes four logical units. When writing data, the data is divided into four data shards, and two parity shards are obtained based on these four data shards using the EC algorithm. Since the number of shards equals the number of storage nodes, an N+M (4+2) redundancy ratio is adopted, meaning that one shard can be stored on each storage node, i.e., a logical unit is selected on each storage node to store the shard. For example, as shown in Figure 1, the four data shards are stored on storage nodes 2, 3, 4, and 6, respectively, and the two parity shards are stored on storage nodes 1 and 5, respectively. This ensures that no data is lost when two single-arm disks or two storage nodes fail simultaneously in the distributed storage system.

[0055] Furthermore, when a logical unit 101 of the storage data shard in storage node 2 (as shown in Figure 1(a)) fails, resulting in the loss of the stored data shard, the current disk failure handling method is shown in Figure 1(b). Based on a reconstruction strategy (generally divided into delayed reconstruction and timely reconstruction), the data shard stored in logical unit 101 of storage node 2 is reconstructed using the remaining shards and stored in other logical units within storage node 2. The disk corresponding to the failed logical unit is replaced, thereby ensuring the availability and data reliability of the distributed storage system.

[0056] The following section, referring to Figure 2, takes N=4, M=2, and the number of storage nodes as an example to illustrate the scenario where the number of storage nodes in a distributed storage system is greater than or equal to ((N+M) / M) and less than (N+M).

[0057] As shown in Figure 2(a), each storage node includes 4 logical units. When writing data, the data is divided into 4 data shards, and 2 parity shards are obtained based on these 4 data shards using the EC algorithm. Since the number of shards is greater than the number of storage nodes, and the number of storage nodes is greater than (N+M) / M = 3, an N+M:1 (4+2:1) redundancy ratio is adopted. In this case, two logical units can be selected on each storage node to store the shards, meaning each storage node stores two shards. For example: one data shard is stored in one logical unit on storage node 1, two data shards are stored in two logical units on storage node 2 respectively, and one data shard is stored in one logical unit on storage node 3; the two parity shards are stored in one logical unit on storage node 1 and one logical unit on storage node 3 respectively. This ensures that data is not lost when two logical units or one storage node fail simultaneously in the distributed storage system.

[0058] Furthermore, when a logical unit 201 of the storage data shard in storage node 1 (as shown in Figure 2(a)) fails, resulting in the loss of the stored data shard, the current disk failure handling method is shown in Figure 2(b). According to the reconstruction strategy, the data shard stored in the failed logical unit 201 in storage node 1 is reconstructed using the remaining shards and stored in other logical units in storage node 1; the disk corresponding to the failed logical unit is replaced, thereby ensuring the availability and data reliability of the distributed storage system.

[0059] In related technologies, disk failures are handled by reconstructing the data stored in the failed logical units and replacing the disk corresponding to the failed logical unit. However, when only some of the logical units in a multi-arm disk fail, current technologies reconstruct the data stored in the failed logical units, but do not reconstruct the data in the non-failed logical units corresponding to that multi-arm disk. In this case, replacing the multi-arm disk will result in the loss of data in the remaining non-failed logical units. Furthermore, the remaining non-failed logical units may store data fragments or parity fragments. Replacing the multi-arm disk will lead to the loss of data fragments or parity fragments in the EC (Erasure Coding Degradation), thus causing EC degradation.

[0060] This application provides a fault handling method, including: acquiring fault information, the fault information including the identifier of a faulty logical unit; determining the fault type based on the fault information, the fault type including disk failure and read / write head failure; and reconstructing at least one faulty logical unit in the faulty disk based on the fault type. This application can effectively distinguish the fault type of a faulty disk, that is, it can distinguish whether a multi-reader disk is experiencing a complete disk failure or a partial read / write head failure, thereby determining the scope of data reconstruction, avoiding data loss or even EC (Extreme Read / Write) degradation, and thus ensuring the reliability of the distributed storage system.

[0061] Example 1:

[0062] The following section, with reference to Figures 3-7, details a fault handling method provided in an embodiment of this application.

[0063] First, referring to Figure 3, the system architecture of a fault handling method provided in the embodiments of this application will be described.

[0064] As shown in Figure 3, the data storage system architecture includes: computing device 310, network equipment 320, and distributed storage system 330.

[0065] The computing device 310, also known as a client device, can be a tablet, laptop, desktop server, etc. The computing device 310 includes a client 311, meaning that storage client software is installed on the computing device 310. The client 311 is responsible for reading and writing data. For example, when writing data, the client 311 on the computing device 310 can divide the data into N data fragments, and then obtain M parity fragments based on the N data fragments using the EC algorithm. Subsequently, the client 311 on the computing device 310 stores the N data fragments and M parity fragments into the distributed storage system 330 through the network device 320.

[0066] Specifically, when the client 311 on the computing device 310 needs to write data, the client 311 on the computing device 310 can obtain the initialized disk information, that is, obtain the disk type and the logical unit corresponding to each disk; select candidate logical units according to the disk type; at the same time, the client 311 divides the data into N data fragments, and then obtains M parity fragments based on the N data fragments through the EC algorithm, that is, the client 311 obtains N data fragments and M parity fragments; according to the EC redundancy ratio, the target logical unit is selected from the candidate logical units, and the N data fragments and M parity fragments are written into the corresponding target logical units through the network device 320, thereby realizing that the client 311 writes data into the distributed storage system 330.

[0067] For example, as shown in Figure 3, taking M=4 and N=2 as an example, when the client 311 on the computing device 310 needs to write data, the data is divided into 4 data fragments, and based on the 4 data fragments, 2 parity fragments are obtained through the EC algorithm, resulting in 4 data fragments and 2 parity fragments. According to the EC redundancy ratio (N+M:1), the target logical unit is determined, and the 4 data fragments and 2 parity fragments are written into the corresponding target logical units, thereby enabling the client 311 to write data into the distributed storage system 330.

[0068] The network device 320 is connected to the computing device 310 and the distributed storage system 330. Communication (i.e., data exchange) between the computing device 310 and the distributed storage system 330 is achieved through the network device 320. Furthermore, each storage node in the distributed storage system 330 (i.e., storage node 331, storage node 332, and storage node 333) is connected to the network device 320, enabling communication (i.e., data exchange) between the storage nodes in the distributed storage system 330.

[0069] Network device 320 refers to various hardware and software components used to build, manage, and maintain computer networks. It plays a crucial role in data communication, ensuring reliable data transmission between different nodes in the network. For example, network device 320 can be a switch, router, etc.

[0070] The distributed storage system 330 includes multiple storage nodes, as shown in Figure 3. The distributed storage system includes three storage nodes: storage node 331, storage node 332, and storage node 333. In one possible implementation, each storage node can be one or more servers, such as blade servers, high-density servers, rack servers, or cabinet servers. Figure 3 illustrates an example where each storage node is a computing device. Each storage node in the distributed storage system 330 includes one or more multi-arm disks, each multi-arm disk corresponding to multiple logical units (LUNs / Disks). As shown in Figure 3, each node in the distributed storage system 330 also includes two dual-arm disks, each dual-arm disk corresponding to two logical units.

[0071] Furthermore, the servers corresponding to each storage node in the distributed storage system 330 constitute a server cluster for the distributed storage system. Storage system software is deployed on the server cluster; that is, all servers in the server cluster have storage system software deployed on or running. The storage system software is used to manage and maintain the distributed storage system, for example, to handle faults in the distributed storage system (data reconstruction). It can be understood that the distributed storage system 330 manages and maintains itself through the storage system software deployed on its corresponding server cluster.

[0072] In one possible implementation, the distributed storage system 330 may also include a coordination node specifically for managing and maintaining the distributed storage system, the server to which the coordination node belongs also having storage system software deployed.

[0073] It should be noted that the distributed storage system 330 shown in Figure 3 is merely illustrative and does not constitute a limitation thereof. For example, the distributed storage system 330 may also include 2, 4, 5, or other storage nodes; each storage node may include multi-arm disks and single-arm disks; multi-arm disks may also include disks with 3 or 4 arms, etc., and this application embodiment does not impose specific limitations. Furthermore, the system architecture shown in Figure 3 is also merely illustrative and does not constitute a limitation thereof. For example, the data storage system framework may include more or fewer devices or modules than those shown in Figure 3, and this application embodiment does not impose specific limitations.

[0074] The following describes in detail, with reference to Figures 4A-7, a fault handling method provided by an embodiment of this application. This method is applied to a distributed storage system including a multi-arm disk, and the computing device has written fragments (N data fragments and M parity fragments) into the distributed storage system.

[0075] It should be noted that the fault handling method can be executed by the computing device (running storage system software) to which any storage node belongs in the distributed storage system; it can also be executed by the computing device (running storage system software) to which all / some storage nodes belong in the distributed storage system; in addition, it can also be executed by the coordination node (running storage system software) in the distributed storage system that is specifically used to manage and maintain the distributed storage system.

[0076] As shown in Figure 4A, a fault handling method provided in this application embodiment includes the following steps:

[0077] S410, Obtain fault information.

[0078] The fault information indicates a failure in a logical unit that stores shards (data shards and / or parity shards). Specifically, the fault information includes the identifier of the faulty logical unit.

[0079] In one possible implementation, the storage system software deployed in the distributed storage system periodically performs health checks to monitor the status of logical units in each storage node, including checking data consistency / integrity. For example, the distributed storage system checks data integrity by comparing redundant data (N data shards and M parity shards), thereby detecting the status of logical units in each storage node. For instance, when the distributed storage system compares redundant data and finds that one parity shard is missing, it determines that the logical unit storing that parity shard is faulty and reports a fault information carrying the identifier of the faulty logical unit.

[0080] In another possible implementation, storage nodes in the distributed storage system maintain a heartbeat connection with the system via a heartbeat mechanism. Specifically, multiple storage nodes periodically send heartbeat messages to the storage system software running in the server cluster, allowing the software to ascertain its current operational status and ensuring a valid connection between the storage nodes and the server cluster. For example, storage nodes may report fault information via heartbeat messages, including the identifier of the faulty logical unit and possibly details of the failure.

[0081] It should be noted that, in addition to the two methods mentioned above for obtaining fault information reported by storage nodes, fault information can also be obtained through performance monitoring, i.e., the distributed storage system continuously monitors the performance indicators of logical units, etc. This application embodiment does not make specific limitations.

[0082] S420. Based on the fault information, determine the fault type.

[0083] The types of failures include: disk failure and read / write head failure.

[0084] Disk failure is defined as a failure where the proportion of faulty logical units corresponding to a disk across all logical units on that disk is not less than a preset proportion. Specifically, for the same disk, if the ratio of faulty logical units to all logical units is not less than a preset proportion, the disk is considered a disk failure. The preset proportion is (0, 1). For a single-arm disk, since it has only one arm (corresponding to only one logical unit), a failure in the logical unit of that single-arm disk indicates a failure in only that arm. In other words, the faulty logical unit of the single-arm disk corresponds to only one logical unit across all logical units. If the proportion of the element is 100% (1), then the proportion must not be less than the preset proportion, so the fault type of the single-arm disk is disk fault; for multi-arm disks, since multi-arm disks include multiple arms, that is, multi-arm disks correspond to multiple logical units, for example, assuming the preset proportion is 70% (0.7), for multi-arm disks including 5 arms, when there are 4 fault logical units in the multi-arm disk, the proportion of fault logical units in the 5 logical units corresponding to the multi-arm disk is 80%, which is not less than the preset proportion of 70%, so the fault type of the multi-arm disk is disk fault.

[0085] Furthermore, since the preset percentage is a positive number not greater than 1, when all logical units corresponding to the disk fail, the fault type of the disk must be disk failure.

[0086] A read / write arm failure is a type of failure where the proportion of the faulty logical unit corresponding to the disk to all logical units is less than a preset proportion. In other words, for the same disk, when the ratio of the faulty logical unit to all logical units is less than the preset proportion, the disk's failure type is a read / write arm failure. Specifically, for a single-arm disk, there is no arm failure because a single-arm disk corresponds to only one logical unit. Therefore, when one logical unit fails, all logical units of the single-arm disk fail (faulty logical unit / all logical units = 100%), meaning the failure type of a single-arm disk is disk failure. For a multi-arm disk, which includes multiple arms, meaning it corresponds to multiple logical units, the failure type of the multi-arm disk is arm failure when the proportion of faulty logical units in all logical units is less than a preset proportion. For example, assuming the preset proportion is 70% (0.7), for a multi-arm disk with 5 arms, when there are 3 faulty logical units in the multi-arm disk, the proportion of faulty logical units in the 5 logical units corresponding to the multi-arm disk is 60%, which is less than the preset proportion of 70%. Therefore, the failure type of the multi-arm disk is arm failure.

[0087] In one possible implementation, the preset percentage is 1 (i.e., 100%). When all logical units corresponding to the disk fail, the disk failure type is disk failure; when some logical units corresponding to the disk fail, the disk failure type is read / write head failure.

[0088] In one possible implementation, the fault type of the faulty disk is determined based on fault information and stored labeling information.

[0089] The annotation information is used to indicate all logical units belonging to the same disk. For example, the annotation information includes the correspondence between disks and all their corresponding logical units. Furthermore, the annotation information is obtained and stored during the initialization phase of the distributed storage system.

[0090] In one possible implementation, the annotation information is obtained in the following way:

[0091] When each storage node of a distributed storage system is powered on, the system initializes the disks on each node and obtains their information, including logical unit identifiers, disk identifiers (e.g., disk serial number, globally unique name), and disk type. Based on these identifiers, all logical units belonging to the same disk are labeled, and this labeling information is stored within the distributed storage system itself, such as in a specific database.

[0092] The identifier of a logical unit is used to uniquely identify the logical unit. For example, the identifier of a logical unit is the Logical Unit Number (LUN). A LUN is a unique identifier for a logical unit / logical volume, representing a storage resource that can be recognized and used by the operating system.

[0093] The disk identifier is used to uniquely identify the disk. For example, the disk identifier is the disk serial number (SN), which is a unique identifier assigned to each physical disk by the manufacturer and is usually recorded in the disk's firmware; the disk identifier can also be a globally unique name (WWN), which is a globally unique identifier.

[0094] The disk types include single-arm disks and multi-arm disks. Further, disk types can be subdivided into single-arm disks, dual-arm disks, triple-arm disks, etc.

[0095] Specifically, when each storage node of the distributed storage system is powered on, the disks of each storage node will perform initialization operations. During the initialization process, the disks will report disk-related information (such as disk identifier, disk type, etc.) to the upper-layer operating system. The operating system of each storage node (corresponding server) can generate the logical unit (LUN / Disk) corresponding to the disk. Since each storage node belongs to the same distributed storage system, each storage node can obtain or synchronize the logical unit information of other storage nodes to obtain the disk information of each storage node.

[0096] In one possible implementation, the distributed storage system (i.e., the server corresponding to each storage node) can also call each disk interface to obtain or read information about all disks in the distributed storage system (i.e., the disk information of each storage node).

[0097] In one possible implementation, the disk information is structured as "logic unit identifier - corresponding disk identifier - corresponding disk type", for example: identifier of logical unit a1 - identifier of disk A to which logical unit a1 belongs - disk type of disk A.

[0098] It should be noted that the disk type can be obtained directly during the initialization of the distributed storage system, or it can be determined by combining the logical unit identifier and the disk identifier during the initialization process. For example, if the obtained disk information includes "LUN1-SN30", "LUN2-SN30", "LUN3-SN31", and "LUN4-SN30", then based on the disk information, the disk with the identifier SN30 contains three logical units, namely logical units identified as LUN1, LUN2, and LUN4. Therefore, the disk type of the disk with the identifier SN30 is determined to be a multi-arm disk (or a three-arm disk). The disk with the identifier SN31 contains one logical unit, namely logical unit identified as LUN3. Therefore, the disk type of the disk with the identifier SN31 is determined to be a single-arm disk.

[0099] Specifically, after obtaining the disk information of each storage node, based on the disk identifier in the disk information, all logical units belonging to the same disk are marked, that is, the identifiers of all logical units belonging to the same disk are recorded, thereby obtaining the marking information.

[0100] In one possible implementation, after obtaining the disk information of each storage node, the disk identifiers and corresponding disk types in the disk information are traversed / viewed. If the disk type is a multi-arm disk, then based on the disk identifier corresponding to the multi-arm disk, all logical units belonging to the same multi-arm disk are marked, that is, the identifiers of all logical units belonging to the same multi-arm disk are recorded, thereby obtaining the marking information. If the disk type is a single-arm disk, it is not necessary to mark the logical units belonging to the single-arm disk again, because a single-arm disk only includes one logical unit. Therefore, the obtained disk information has clearly indicated the single-arm disk and its corresponding logical unit, and there is no need to repeat the marking, which simplifies the operation process and avoids unnecessary resource waste during the initialization of the distributed storage system.

[0101] To make it easier to understand, the following example, with reference to Figure 5, illustrates how to obtain annotation information.

[0102] As shown in Figure 5, the disk information obtained by the distributed storage system is as follows: "Disk1-WWN / SN1-Multi-arm disk", "Disk2-WWN / SN1-Multi-arm disk", "Disk3-WWN / SN2-Multi-arm disk", "Disk4-WWN / SN2-Multi-arm disk", "Disk5-WWN / SN3-Multi-arm disk", and "Disk6-WWN / SN3-Multi-arm disk".

[0103] Based on the disk identifier in the disk information, all logical units belonging to the same disk are labeled. Specifically, based on the disk identifier "WWN / SN1", the logical units "Disk1" and "Disk2" corresponding to "WWN / SN1" are identified and labeled; based on the disk identifier "WWN / SN2", the logical units "Disk3" and "Disk4" corresponding to "WWN / SN2" are identified and labeled; based on the disk identifier "WWN / SN3", the logical units "Disk5" and "Disk6" corresponding to "WWN / SN3" are identified and labeled. The final labeling information includes: WWN / SN1->(Disk1, Disk2), WWN / SN2->(Disk3, Disk4), WWN / SN3->(Disk5, Disk6), used to indicate all logical units belonging to the same disk.

[0104] Specifically, based on fault information and stored annotation information, the fault type of the disk is determined, including: traversing the identifiers of the faulty logical units included in the fault information; identifying the corresponding faulty disk based on the identifiers of the faulty logical units and the stored annotation information; identifying all logical units belonging to the faulty disk based on the identifiers and annotation information of the faulty disk; determining whether the proportion of the faulty logical units in the faulty disk to all logical units in the faulty disk is not less than a preset proportion based on the fault information; if the proportion of the faulty logical units in the faulty disk to all corresponding logical units is not less than the preset proportion, the fault type of the faulty disk is a disk fault; if the proportion of the faulty logical units in the faulty disk to all corresponding logical units is less than the preset proportion, the fault type of the faulty disk is a read / write head fault.

[0105] That is, for the identifier of the faulty logical unit being traversed, the identifier of the disk to which the faulty logical unit belongs is determined, and the disk indicated by the identifier of the disk to which it belongs is the faulty disk; based on the identifier of the faulty disk, all logical units belonging to the faulty disk are obtained from the stored label information; based on the fault information, it is determined whether the proportion of the faulty logical units belonging to the faulty disk to all logical units corresponding to the faulty disk is not less than a preset proportion.

[0106] For example, with a preset ratio of 1 and the labeling information as: SN40->(LUN1, LUN2, LUN4) and SN41->(LUN3), assume the fault information includes: LUN2 and LUN4. Iterate through the identifiers of the fault logic units included in the fault information. For LUN2, based on LUN2, the disk identifier SN40 corresponding to LUN2 is determined, that is, the disk indicated by SN,30 is the faulty disk; based on SN40, the identifiers LUN1, LUN2, and LUN4 of all logical units corresponding to SN40 are obtained from the labeling information, and based on the fault information, it is determined that among all logical units of the disk belonging to SN40, there are logical units with identifiers LUN2 and LUN4 that are faulty, that is, two logical units are faulty. Then the proportion of the faulty logical units of the disk with identifier SN40 in the corresponding total logical units is 2 / 3, which is not greater than the preset proportion of 1. Therefore, the fault type of the disk with identifier SN40 is the read / write head fault, and the read / write head corresponding to the logical units with identifiers LUN2 and LUN4 in the disk with identifier SN40 is faulty. The judgment process for LUN4 is the same as that for LUN2, and the result is that the fault type of the disk with identifier SN40 is the read / write head fault.

[0107] S430. Based on the fault type, reconstruct at least one fault logic unit in the faulty disk.

[0108] Specifically, when the fault type is disk fault, all logical units in the faulty disk are reconstructed; when the fault type is read / write head fault, all faulty logical units in the faulty disk are reconstructed.

[0109] For ease of understanding, the specific implementation of a fault handling method provided in this application embodiment will be described in detail below with reference to Figure 4B.

[0110] As shown in Figure 4B, a fault handling method provided in this application embodiment includes the following steps:

[0111] S401, Obtain fault information.

[0112] S402. Determine the fault type based on the fault information.

[0113] It should be noted that S401 and S402 are the same as S410 and S420 above. Therefore, for details on the specific implementation of S401 and S402, please refer to S410 and S420 above, and will not be repeated here.

[0114] When the fault type is disk fault, S403 is executed.

[0115] When the fault type is magnetic arm fault, proceed to S404.

[0116] S403. Reconstruct the data stored in all logical units of the faulty disk.

[0117] When the fault type of the faulty disk is disk fault, it means that the proportion of the faulty logical unit in the faulty disk is not less than the preset proportion of all logical units. Therefore, the data of the faulty disk is reconstructed based on the remaining fragments (the remaining data fragments and parity fragments), that is, the data (data fragments or parity fragments) stored in all logical units of the faulty disk are reconstructed.

[0118] In one possible implementation, the EC algorithm is used to recalculate the fragments stored in all logical units of the failed disk based on the remaining data fragments and check fragments, and then copy them to the free logical units of the remaining unfailed disks.

[0119] In one possible implementation, when reconstructing the data stored in all logical units of the failed disk, the distributed storage system can be configured with reconstruction settings. These settings include reconstruction speed and rate limiting capabilities. For example, the distributed storage system can be configured to support high-speed reconstruction; rate limiting can be enabled with specific conditions.

[0120] Specifically, the reconstruction speed of the distributed storage system is set, meaning the reconstruction of data shards is completed according to a certain speed requirement. For example, the reconstruction speed of the distributed storage system is set to high speed, meaning the distributed storage system supports high-speed reconstruction. In this case, the distributed storage system chooses to complete the reconstruction as quickly as possible, utilizing all available resources to accelerate the data (shard) reconstruction process.

[0121] Specifically, the distributed storage system is configured to support rate limiting, and rate limiting conditions are set to prevent the reconstruction process from excessively impacting other services of the distributed storage system. For example, rate limiting conditions include: maximum bandwidth, maximum input / output (I / O) count, etc. Setting the maximum bandwidth sets the maximum bandwidth allowed during the reconstruction process to prevent excessive network resource consumption; setting the maximum I / O count sets the maximum number of I / O operations allowed per second or per minute during the reconstruction process to avoid putting excessive pressure on the distributed storage system.

[0122] In one possible implementation, when the fault type is a disk fault, after reconstructing the data stored in all logical units of the faulty disk, the method further includes: setting the disk swap flag corresponding to the faulty disk as the first disk swap flag; and performing a disk swap operation on the faulty disk in response to the disk swap flag corresponding to the faulty disk being set as the first disk swap flag.

[0123] To facilitate understanding, the following section, with reference to Figure 6, details the troubleshooting process for a disk with a fault type of disk failure. As shown in Figure 6, the steps include:

[0124] S601. Reconstruct the data stored in all logical units of the faulty disk.

[0125] S602. Set the disk replacement flag corresponding to the faulty disk as the first disk replacement flag.

[0126] The disk replacement flag corresponding to the faulty disk indicates whether the faulty disk needs to be replaced after the reconstruction of lost / faulty fragments (data) in the faulty disk. When the disk replacement flag corresponding to the faulty disk is the first disk replacement flag, it means that the faulty disk needs to be replaced after the reconstruction of lost / faulty fragments (data) in the faulty disk; when the disk replacement flag corresponding to the faulty disk is the second disk replacement flag, it means that the faulty disk does not need to be replaced after the reconstruction of faulty / lost fragments (data) in the faulty disk.

[0127] In one possible implementation, each disk in the distributed storage system has a one-to-one disk swap flag, and the disks and swap flags are stored in the database in a one-to-one correspondence.

[0128] For example, the disk replacement flag corresponding to the faulty disk is set to "Y", where "Y" is the first disk replacement flag.

[0129] S603. In response to the faulty disk's corresponding disk swap flag being the first disk swap flag, perform a disk swap operation on the faulty disk.

[0130] In one possible implementation, the distributed storage system automatically performs a disk replacement operation on the failed disk in response to the disk replacement flag corresponding to the failed disk being designated as the first disk replacement flag. For example, in response to the disk replacement flag corresponding to the failed disk being designated as the first disk replacement flag, the distributed storage system automatically selects a suitable new disk from the backup disks according to preset rules and replaces the failed disk, thereby completing the disk replacement operation on the failed disk.

[0131] Typically, distributed storage systems pre-configure a certain number of backup disks so that they can be immediately put into use when needed. Furthermore, suitable new disks are generally those with the same structure as the failed disk from among the backup disks.

[0132] In another possible implementation, the distributed storage system responds to the disk replacement identifier corresponding to the faulty disk as the first disk replacement identifier, triggering a disk replacement command; so as to facilitate the disk replacement operation on the faulty disk based on the disk replacement command.

[0133] The disk replacement command is used to prompt / guide the user to replace the faulty disk.

[0134] Specifically, based on the disk replacement command, the user removes the faulty disk from the distributed storage system and inserts a replacement disk with the same structure as the faulty disk, thereby completing the disk replacement operation for the faulty disk.

[0135] In one possible implementation, the distributed storage system can display disk replacement instructions through the servers corresponding to each storage node to prompt / guide the user to replace the faulty disk; in another possible implementation, the distributed storage system responds to the disk replacement identifier corresponding to the faulty disk being designated as the first disk replacement identifier, triggering the indicator light of the faulty disk to flash at a preset frequency to trigger the disk replacement instruction, thereby prompting / guiding the user to replace the faulty disk.

[0136] The above, in conjunction with Figure 6, details the fault handling method for a disk with a fault type of disk failure. The following, in conjunction with Figure 4B, introduces a fault handling method provided by an embodiment of this application.

[0137] S404. Reconstruct the data stored in the faulty logical unit in the faulty disk.

[0138] When the fault type of the faulty disk is a read / write head failure, it means that the proportion of the faulty logical unit in the faulty disk (faulty multi-reader disk) is less than the preset proportion among all the corresponding logical units. Therefore, the data stored in the faulty logical unit (data shards or check shards) is reconstructed based on the remaining shards (the remaining data shards and check shards), that is, the faulty read / write head data is reconstructed.

[0139] In one possible implementation, the EC algorithm is used to recalculate the fragments stored in the faulty logical units of the faulty disk based on the remaining data fragments and check fragments, and then copy them to the logical units in the remaining unfaulty space.

[0140] In one possible implementation, when reconstructing data stored in faulty logical units on a faulty disk, it is supported to set a reconstruction configuration for the distributed storage system.

[0141] In one possible implementation, when the fault type is a magnetic arm fault, after reconstructing the data stored in the faulty logical unit in the faulty disk, the following steps are also included: setting the disk replacement flag corresponding to the faulty disk as the second disk replacement flag, and / or setting the health status corresponding to the faulty disk as a sub-healthy state; and expanding the distributed storage system based on the unusable storage capacity in the distributed storage system.

[0142] To facilitate understanding, the troubleshooting method for a disk with a read / write head failure will be explained in detail below with reference to Figure 7. As shown in Figure 7, the steps include:

[0143] S701. Reconstruct the data stored in the faulty logical unit in the faulty disk.

[0144] S702. Set the disk replacement flag corresponding to the faulty disk as the second disk replacement flag.

[0145] Specifically, the disk replacement flag corresponding to the faulty disk is set as the second disk replacement flag, which indicates that after the lost / faulty fragments (data) in the faulty disk are reconstructed, it is not necessary to replace the faulty disk.

[0146] For example, the disk replacement flag corresponding to the faulty disk is set to "N", where "N" is the second disk replacement flag.

[0147] S703. Set the health status of the faulty disk to sub-healthy status.

[0148] The disk health status indicates the current state of the disk. Specifically, the disk health status includes: normal status, sub-healthy status, and faulty status.

[0149] A healthy state indicates that the disk is functioning well, with all performance parameters within preset ranges and no errors or anomalies detected. A failed state indicates that the disk is no longer functioning properly and can no longer provide reliable service. A sub-healthy or degraded state is a state between healthy and failed, indicating that the disk has some non-fatal problems.

[0150] It should be noted that in this embodiment of the application, the health status of the disk is not considered as normal or faulty, but only as sub-healthy.

[0151] In one possible implementation, the disk's health status is either empty or sub-healthy, or it is either sub-healthy or not sub-healthy.

[0152] In one possible implementation, each disk in the distributed storage system has a one-to-one corresponding health status, and the disks and health statuses are stored in a database in a one-to-one correspondence. That is, the disk, the disk swap flag, and the health status are stored in the database in a corresponding manner.

[0153] To make it easier to understand, the following examples, with reference to Figure 8, illustrate the handling methods for disk failures and read / write head failures.

[0154] As shown in Figures 8(a) and (b), the distributed storage system includes three storage nodes: storage node 1, storage node 2, and storage node 3. Each storage node includes two dual-arm disks, and each dual-arm disk corresponds to four logical units. Specifically, four (N=4) data fragments are stored in logical unit 821 of dual-arm disk 820 in storage node 1, logical units 841 and 842 of dual-arm disk 840 in storage node 2, and logical unit 851 of dual-arm disk 850 in storage node 3; two (M=2) parity fragments are stored in logical unit 822 of dual-arm disk 820 in storage node 1 and logical unit 862 of dual-arm disk 860 in storage node 3.

[0155] As shown in Figure 8(a), taking the dual-arm disk 820 as the faulty disk, and the fault type of the dual-arm disk 820 as disk failure, that is, both the logical units 821 and 822 corresponding to the dual-arm disk 820 are faulty. Since the data (fragments) stored in logical units 821 and 822 are lost, the data fragments in logical unit 821 and the parity fragment in logical unit 822 are reconstructed based on the remaining data fragments and parity fragments (i.e., 3 data fragments and 1 parity fragment), and these are saved to logical unit 812 of dual-arm disk 810 and logical unit 832 of dual-arm disk 830, respectively. After data reconstruction, the disk swap flag of dual-arm disk 820 is set to the first disk swap flag (i.e., "Y"). In response to the disk swap flag of dual-arm disk 820 being set to the first disk swap flag, a disk swap operation is performed on dual-arm disk 820.

[0156] As shown in Figure 8(b), taking the dual-arm disk 820 as the faulty disk, and the fault type of the dual-arm disk 820 as an arm fault, i.e., the fault of the corresponding logical unit 822 of the dual-arm disk 820, as an example. If the parity fragment stored in the logical unit 822 is lost, the parity fragment in the logical unit 822 is reconstructed based on the remaining data fragments and parity fragments (i.e., 4 data fragments and 1 parity fragment), and it is saved to the logical unit 812 of the dual-arm disk 810. After the data is reconstructed, the disk swap flag of the dual-arm disk 820 is set to the second disk swap flag (i.e., "N"), and the health status of the dual-arm disk 820 is set to a sub-healthy state.

[0157] S704. Determine whether the unusable storage capacity in the distributed storage system is greater than the first preset storage capacity.

[0158] When a faulty disk (a multi-arm disk) experiences an arm failure, meaning only some logical units within the disk fail, but another logical unit remains functional, only the data stored in the faulty logical unit is reconstructed. The faulty disk is not replaced. Because this doesn't replace the faulty multi-arm disk, the faulty logical units in the distributed storage system remain unusable. Therefore, the distributed storage system must use its remaining storage capacity (storage resources) to handle all tasks. The larger the total storage capacity of the faulty logical units in the distributed storage system—meaning a larger unusable storage capacity—the smaller the usable storage capacity. This results in more tasks (IO operations) per unit of storage capacity, leading to performance degradation and potentially causing overall performance bottlenecks due to overload.

[0159] When the unusable storage capacity in a distributed storage system exceeds the first preset storage capacity, it indicates that the unusable storage capacity exceeds the threshold, resulting in insufficient usable storage capacity and a significant drop in the system's performance, potentially creating a performance bottleneck and failing to meet user needs. Conversely, when the unusable storage capacity is no greater than the first preset storage capacity, it indicates that the unusable storage capacity does not exceed the threshold, and the usable storage capacity meets user needs.

[0160] In one possible implementation, the system iterates through the faulty disks in the distributed storage system that are in a sub-healthy state, and determines whether the total storage capacity of the faulty logical units in these sub-healthy disks is greater than a first preset storage capacity. Here, the total storage capacity of the faulty logical units refers to the unusable storage capacity.

[0161] Specifically, the system iterates through the faulty disks in the distributed storage system that are in a sub-healthy state, obtains the total storage capacity of the faulty logical units in the sub-healthy faulty disks, and determines whether the total storage capacity of the faulty logical units is greater than the first preset storage capacity.

[0162] In another possible implementation, the system iterates through the faulty disks in the distributed storage system whose disk replacement flag is the second disk replacement flag, and determines whether the total storage capacity of the faulty logical units in these faulty disks is greater than a first preset storage capacity. Here, the total storage capacity of the faulty logical units refers to the unusable storage capacity.

[0163] Specifically, the system iterates through the faulty disks in the distributed storage system marked with the second disk replacement identifier, obtains the total storage capacity of the faulty logical units in the faulty disks marked with the second disk replacement identifier, and determines whether the total storage capacity of the faulty logical units is greater than the first preset storage capacity.

[0164] In one possible implementation, the first preset storage capacity is obtained based on a first preset ratio and the total storage capacity of the distributed storage system. For example, the first preset storage capacity is 5% of the total storage capacity of the distributed storage system, where "5%" is the first preset ratio.

[0165] If the unusable storage capacity in the distributed storage system is no greater than the first preset storage capacity, the fault handling method provided in this application embodiment can continue.

[0166] If the unusable storage capacity in the distributed storage system exceeds the first preset storage capacity, then proceed to step S705.

[0167] S705, Expand the capacity of the distributed storage system.

[0168] Specifically, when the unusable storage capacity in a distributed storage system exceeds the first preset storage capacity, it indicates that the available storage capacity is insufficient, leading to a significant performance degradation or even a performance bottleneck that fails to meet user needs. Therefore, the distributed storage system needs to be expanded to meet user requirements (performance requirements, storage capacity requirements, etc.).

[0169] The expansion methods include disk expansion and storage node expansion. This means expanding the disk capacity of the distributed storage system or expanding the storage nodes within the distributed storage system. Disk expansion refers to adding additional disks to existing storage nodes to increase the storage capacity of a single storage node, thereby increasing the overall storage capacity of the distributed storage system. Storage node expansion refers to adding new storage nodes to the distributed storage system, thus increasing the overall storage capacity of the distributed storage system.

[0170] In one possible implementation, disk expansion of a distributed storage system specifically involves adding disks to each storage node in the distributed storage system to ensure a balance between capacity and performance. The number, type, and storage capacity of the added disks are the same in each storage node. For example, assuming the distributed storage system includes three storage nodes, a 17TB dual-arm disk is added to each of the three storage nodes.

[0171] In one possible implementation, expanding the storage nodes of a distributed storage system specifically involves adding at least one storage node to the distributed storage system in order to ensure a balance between capacity and performance. The newly added storage node is consistent with the original storage nodes in the distributed storage system, meaning that the number of disks, disk type, and disk capacity included in the newly added storage node are consistent with those of the original storage nodes.

[0172] In one possible implementation, the distributed storage system is expanded, with the expanded storage capacity exceeding a second preset storage capacity. The second preset storage capacity is obtained based on a second preset ratio and the total storage capacity of the distributed storage system.

[0173] The second preset storage capacity is greater than or equal to the first preset storage capacity, thereby ensuring that the expanded distributed storage system can meet the user's needs for the distributed storage system.

[0174] It should be noted that, in the possible implementation shown in Figure 7, taking "setting the disk replacement flag of the faulty disk to the second disk replacement flag and setting the health status corresponding to the faulty disk to sub-healthy" as an example, it is also possible to simply "set the disk replacement flag of the faulty disk to the second disk replacement flag", then "determine the unusable storage capacity in the distributed storage system based on the second disk replacement flag"; or it is possible to simply "set the health status corresponding to the faulty disk to the sub-healthy state", then "indicate that the faulty disk does not need to be replaced based on the sub-healthy state".

[0175] The above, in conjunction with Figure 7, details the fault handling method for a disk with a fault type of read / write head failure. In one possible implementation, after reconstructing at least one faulty logical unit in the faulty disk, a load balancing operation can be performed on the distributed storage system. The following, in conjunction with Figure 4B, describes a fault handling method provided by an embodiment of this application.

[0176] S405, Perform a capacity balancing operation on the distributed storage system.

[0177] Distributed storage systems rebalance their storage capacity to ensure balanced usage across all storage nodes and disks, preventing overload of any particular node or disk. This optimizes performance, improves resource utilization, and enhances the stability and reliability of the distributed storage system.

[0178] For example, assuming a distributed storage system like Ceph, Ceph uses the Controlled Replication Under Scalable Hashing (CRUSH) algorithm to determine how data is distributed across various OSDs (Object Storage Daesel), thereby dynamically adjusting data distribution based on the actual usage of storage nodes and disks, achieving load balancing. In Ceph, an OSD typically corresponds to a disk or logical unit.

[0179] This application provides a fault handling method, including: acquiring fault information; determining the fault type based on the fault information; reconstructing at least one faulty logical unit in the faulty disk based on the fault type; and expanding the storage capacity of the distributed storage system. In this application embodiment, when the fault type is a disk fault, the data stored in all logical units of the faulty disk is reconstructed; when the fault type is a read / write head fault, the faulty logical units in the faulty disk are reconstructed. Therefore, this application embodiment can effectively distinguish the fault type of the faulty disk, that is, it can distinguish whether a multi-reader disk is experiencing a complete disk failure or a partial read / write head failure, thereby determining the scope of data reconstruction, avoiding data loss or even EC degradation, and thus ensuring the reliability of the distributed storage system.

[0180] Furthermore, for a faulty disk with a fault type of "magnetic arm failure", it is determined whether the unusable storage capacity in the distributed storage system is greater than the first preset storage capacity. If the unusable storage capacity in the distributed storage system is greater than the first preset storage capacity, the distributed storage system is expanded to meet the user's needs for the distributed storage system (performance requirements, storage capacity requirements, etc.).

[0181] The above-described fault handling method provided by the present application embodiment is detailed in Embodiment 1. The fault handling method described in Embodiment 1 does not take into account specific application scenarios. The following describes another fault handling method provided by the present application embodiment in different application scenarios through Embodiment 2.

[0182] Example 2:

[0183] The following, with reference to Figures 9-13, details another fault handling method provided by the embodiments of this application under different application scenarios.

[0184] Before introducing another fault handling method provided by the embodiments of this application, let’s first introduce three application scenarios.

[0185] The three application scenarios are: performance-first scenario, cost-first scenario, and reliability-first scenario.

[0186] Performance-first scenarios prioritize ensuring the performance of the distributed storage system, targeting applications such as HPC, online big data analytics, and computing. In performance-first scenarios, maintaining high performance of the distributed storage system is crucial. This means that even if a logical unit within the distributed storage system fails (e.g., disk failure), the system's operational performance (e.g., CPU, memory, network performance) must be maximized to ensure that data reconstruction can proceed without impacting operational performance.

[0187] Cost-first scenarios prioritize minimizing the cost of distributed storage systems, such as HPC and big data archiving and backup. In cost-first scenarios, the goal is to maintain a low cost for the distributed storage system, meaning that even if a logical unit within the distributed storage system fails (e.g., a disk failure), the overall cost of the distributed storage system should be kept as low as possible.

[0188] In reliability-first scenarios, the priority is to ensure the reliability of the distributed storage system, specifically for core assets such as HPC core data and drawings. In other words, in reliability-first scenarios, the reliability of the distributed storage system must be guaranteed to the greatest extent possible, even when a logical unit within the distributed storage system fails (e.g., a disk fails).

[0189] For ease of understanding, the following describes the fault handling methods provided by the embodiments of this application in performance-priority, cost-priority, and reliability-priority scenarios, respectively, with reference to Figures 9, 10, and 11.

[0190] As shown in Figure 9, in a performance-priority scenario, the fault handling method provided in this application embodiment includes the following steps:

[0191] S901, Obtain fault information.

[0192] It should be noted that S901 is the same as S401 in Embodiment 1. For details on the specific implementation of S901, please refer to S401 in Embodiment 1, which will not be repeated here.

[0193] S902. Determine whether the business performance of the distributed storage system meets the preset conditions.

[0194] Among them, business performance includes: CPU utilization of the server corresponding to the storage node in the distributed storage system, memory utilization, disk utilization, network latency, etc.

[0195] In one possible implementation, if the CPU utilization, memory utilization, disk utilization, and network latency of the distributed storage system meet the corresponding preset requirements, it means that the business requirements of the distributed storage system meet the preset conditions; otherwise, it means that the business performance of the distributed storage system does not meet the preset conditions.

[0196] For example, if the CPU utilization rate of the distributed storage system is <70%, the memory utilization rate is <70%, the disk utilization rate is <70%, and the network latency is <1ms, then the business performance of the distributed storage system is determined to meet the preset conditions; if one or more of the CPU utilization rate, memory utilization rate, disk utilization rate, and network latency do not meet the preset requirements, for example: CPU utilization rate >70%, memory utilization rate <70%, disk utilization rate <70%, and network latency >1ms (i.e., CPU utilization rate and network latency do not meet the preset conditions), then the business performance of the distributed storage system is determined to not meet the preset conditions.

[0197] If it is determined that the business performance of the distributed storage system does not meet the preset conditions, then S902 is repeated, that is, the business performance of the distributed storage system is monitored until it is determined that the business performance of the distributed storage system meets the preset conditions.

[0198] If it is determined that the business performance of the distributed storage system does not meet the preset conditions, it means that the current business of the distributed storage system is relatively busy. If data reconstruction is performed at this time, it may affect the current business performance. Due to the performance priority strategy, the business performance of the distributed storage system needs to be guaranteed first, so data reconstruction is not performed.

[0199] If the business performance of the distributed storage system meets the preset conditions, then proceed with S903.

[0200] S903. Determine the fault type based on the fault information.

[0201] If the business performance of the distributed storage system meets the preset conditions, it means that the current business of the distributed storage system is relatively unbusy. If data reconstruction at this time will not affect the current business performance, then data reconstruction will continue.

[0202] When the fault type is disk fault, S904 is executed.

[0203] When the fault type is magnetic arm fault, proceed with S905.

[0204] S904. Reconstruct the data stored in all logical units of the faulty disk.

[0205] S905. Reconstruct the data stored in the faulty logical unit in the faulty disk.

[0206] S906, Perform a capacity balancing operation on the distributed storage system.

[0207] It should be noted that S903-S906 above are the same as S402-S405 in Embodiment 1. Therefore, for details of the specific implementation of S903-S906, please refer to S402-S405 in Embodiment 1, and will not be repeated here.

[0208] This application provides a fault handling method. In a performance-priority scenario, when a distributed storage system obtains fault information, it needs to determine whether the business performance of the distributed storage system meets preset conditions. If the preset conditions are met, at least one faulty logical unit in the faulty disk is reconstructed based on the fault type. In this application embodiment, before data reconstruction, the business performance of the distributed storage system is prioritized by judging the business performance of the distributed storage system against the preset conditions.

[0209] If the business performance of the distributed storage system meets the preset conditions, it means that the business performance of the distributed storage system will not be affected by data reconstruction, ensuring that the business performance of the distributed storage system is prioritized. If the business performance of the distributed storage system does not meet the preset conditions, the business performance of the distributed storage system will continue to be monitored until it is determined that the business performance of the distributed storage system meets the preset conditions. If it is determined that the business performance of the distributed storage system does not meet the preset conditions, it means that the current business performance of the distributed storage system may be affected by data reconstruction. Therefore, in order to prioritize the business performance of the distributed storage system, data reconstruction will not be performed.

[0210] The above description, in conjunction with Figure 9, illustrates a fault handling method provided by an embodiment of this application in a performance-priority scenario. The following description, in conjunction with Figure 10, illustrates a fault handling method provided by an embodiment of this application in a cost-priority scenario.

[0211] As shown in Figure 10, in a cost-priority scenario, the fault handling method provided in this application embodiment includes the following steps:

[0212] S1001, Obtain fault information.

[0213] S1002. Determine the fault type based on the fault information.

[0214] Specifically, the system determines whether the proportion of the faulty logical unit corresponding to the faulty disk among all corresponding logical units is not less than a preset proportion. If the proportion is not less than the preset proportion, the fault type is determined to be a disk fault; if the proportion is less than the preset proportion, the fault type is determined to be a read / write head fault. To maximize the cost of the distributed storage system, the preset proportion is set to 1 (100%), meaning that a disk fault occurs only when all logical units of the disk fail; otherwise, it is a read / write head fault.

[0215] When the fault type is disk fault, proceed to S1003.

[0216] When the fault type is magnetic arm fault, proceed to S1004.

[0217] S1003. Reconstruct the data stored in all logical units of the faulty disk.

[0218] S1004. Reconstruct the data stored in the faulty logical unit in the faulty disk.

[0219] S1005. Perform a capacity balancing operation on the distributed storage system.

[0220] It should be noted that S1001-S1005 above are the same as S401-S405 in Embodiment 1. Therefore, for details of the specific implementation of S1001-S1005, please refer to S401-S405 in Embodiment 1, and will not be repeated here.

[0221] This application provides a fault handling method. In a cost-priority scenario, when a distributed storage system obtains fault information and determines the fault type based on the fault information, a preset percentage is set to 1. That is, when all logical units of a faulty disk are faulty, all logical units of the faulty disk are subsequently reconstructed; when some logical units of a faulty disk are faulty due to a read / write head failure, only the faulty logical units of the faulty disk are reconstructed, and no additional non-faulty logical units are reconstructed, thereby ensuring the cost priority of the distributed storage system.

[0222] The above-described fault handling method, with reference to Figure 10, is an embodiment of this application providing a fault handling method in a cost-priority scenario. The following-described fault handling method, with reference to Figure 11, is an embodiment of this application providing a reliability-priority scenario.

[0223] As shown in Figure 11, in a reliability-first scenario, the fault handling method provided in this application embodiment includes the following steps:

[0224] S1101, Obtain fault information.

[0225] It should be noted that S1101 above is the same as S401 in Embodiment 1. For details on the specific implementation of S1101, please refer to S401 in Embodiment 1, which will not be repeated here.

[0226] S1102. Determine the fault type based on the fault information.

[0227] S1103 is executed when the fault type is either disk fault or read / write head fault.

[0228] S1103. Reconstruct the data stored in all logical units of the faulty disk.

[0229] In reliability-first scenarios, regardless of whether it is a disk failure or a read / write head failure, the distributed storage system reconstructs the data stored in all logical units of the disk with the faulty logical unit (i.e., the faulty disk), thereby ensuring the reliability of the distributed storage system.

[0230] Specifically, the data stored in all logical units of the faulty disk is reconstructed based on the remaining fragments (the remaining data fragments and parity fragments) in the non-faulty disk. This reconstruction applies to both data stored in faulty logical units and data stored in non-faulty logical units.

[0231] In one possible implementation, under a reliability-first scenario, regardless of whether the fault type is a disk fault or a read / write head fault, the data stored in all logical units of the faulty disk is reconstructed, and the disk swap flag corresponding to the faulty disk is set as the first disk swap flag; in response to the disk swap flag corresponding to the faulty disk being set as the first disk swap flag, a disk swap operation is performed on the faulty disk.

[0232] S1104. Perform a capacity balancing operation on the distributed storage system.

[0233] It should be noted that S1103 and S1104 above are the same as S403 and S405 in Embodiment 1. Therefore, for details of the specific implementation of S1103 and S1104, please refer to S403 and S405 in Embodiment 1, and will not be repeated here.

[0234] To facilitate understanding, the following examples, using Figure 12, illustrate the handling methods for disk failures and read / write head failures in a reliability-first scenario.

[0235] As shown in Figures 12(a) and (b), the distributed storage system includes three storage nodes: storage node 1, storage node 2, and storage node 3. Each storage node includes two dual-arm disks, and each dual-arm disk corresponds to four logical units. Specifically, four (N=4) data fragments are stored in logical unit 1211 of dual-arm disk 1210 in storage node 1, logical units 1241 and 1242 of dual-arm disk 1240 in storage node 2, and logical unit 1251 of dual-arm disk 1250 in storage node 3; two (M=2) parity fragments are stored in logical unit 1222 of dual-arm disk 1220 in storage node 1 and logical unit 1262 of dual-arm disk 1260 in storage node 3.

[0236] As shown in Figure 12(a), taking the dual-arm disk 1240 as a faulty disk, and the fault type of the dual-arm disk 1240 as a disk failure, that is, both logical units 1241 and 1242 corresponding to the dual-arm disk 1240 are faulty. Since the data (fragments) stored in logical units 1241 and 1242 are lost, the data fragments in logical units 1241 and 1242 are reconstructed based on the remaining data fragments and parity fragments (i.e., 2 data fragments and 2 parity fragments), and these fragments are saved to logical units 1212 of the dual-arm disk 1210 and 1232 of the dual-arm disk 1230, respectively. After data reconstruction, the disk swap flag of the dual-arm disk 1240 is set to the first disk swap flag (i.e., "Y"). In response to the disk swap flag of the dual-arm disk 1240 being set to the first disk swap flag, a disk swap operation is performed on the dual-arm disk 1240.

[0237] As shown in Figure 12(b), taking the dual-arm disk 1240 as the faulty disk, and the fault type of the dual-arm disk 1240 as an arm fault, i.e., the fault of the corresponding logical unit 1242 of the dual-arm disk 1240, as an example. Based on the remaining data fragments and parity fragments (i.e., 2 data fragments and 2 parity fragments), the data fragments in logical unit 1241 and logical unit 1242 are reconstructed, and they are saved to logical unit 1212 of dual-arm disk 1210 and logical unit 1232 of dual-arm disk 1230, respectively. After the data is reconstructed, the disk swap flag of dual-arm disk 1240 is set to the first disk swap flag (i.e., "Y"). In response to the disk swap flag of dual-arm disk 1240 being set to the first disk swap flag, a disk swap operation is performed on dual-arm disk 1240.

[0238] This application provides a fault handling method that, in a reliability-priority scenario, when a distributed storage system obtains fault information, regardless of whether the fault is a disk failure or a read / write head failure, reconstructs the data stored in all logical units of the faulty disk; in response to the completion of data reconstruction, a storage capacity balancing operation is performed on the distributed storage system. In this application embodiment, as long as a faulty disk exists, regardless of whether the fault is a disk failure or a read / write head failure, the data stored in all logical units of the faulty disk is reconstructed, thereby ensuring the reliability of the distributed storage system.

[0239] The above descriptions, in conjunction with Figures 9, 10, and 11, illustrate fault handling methods for performance-priority, cost-priority, and reliability-priority scenarios, respectively. The following section, with reference to Figure 13, details a fault handling method provided by an embodiment of this application.

[0240] S1301. Obtain the target scenario of user input / selection.

[0241] The target scenarios include: performance-priority scenarios, cost-priority scenarios, and reliability-priority scenarios.

[0242] In one possible implementation, the server corresponding to the storage node in the distributed storage system can display a scene selection interface through a display device, and the user inputs / selects the target scene based on the displayed scene selection interface.

[0243] S1302. Based on the target scenario, execute the corresponding fault handling process.

[0244] Specifically, when the target scenario is a performance-priority scenario, the fault handling process corresponding to the performance-priority scenario is executed, as shown in Figure 9. When the target scenario is a cost-priority scenario, the fault handling process corresponding to the cost-priority scenario is executed, as shown in Figure 10. When the target scenario is a reliability-priority scenario, the fault handling process corresponding to the reliability-priority scenario is executed, as shown in Figure 11.

[0245] In addition, this application embodiment also provides a computing device, the server including a processor and a memory, the processor being coupled to the memory, the memory storing computer execution instructions, and the processor executing the computer execution instructions to implement the fault handling method in the above embodiment.

[0246] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the fault handling method described in the above embodiments.

[0247] For explanations of the relevant content and descriptions of the beneficial effects in any of the computer-readable storage media provided above, please refer to the corresponding embodiments described above, which will not be repeated here.

[0248] This application also provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform any of the fault handling methods described in the above embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device including one or more servers, data centers, etc., that can be integrated with the medium. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD), etc.

[0249] It should be noted that the devices for storing computer instructions or computer programs provided in the embodiments of this application, such as but not limited to the memory, computer-readable storage medium and communication chip, are all non-transitory.

[0250] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks, SSDs).

[0251] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, the disclosure, and the appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0252] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A fault handling method, characterized in that, Applied to a distributed storage system, the distributed storage system including at least one storage node, the storage node including at least one disk, the disk including one or more magnetic arms, the method includes: Obtain fault information; wherein, the fault information includes: the identifier of the fault logic unit; Based on the fault information, the fault type is determined; wherein, the fault type includes: disk failure and read / write head failure; Based on the fault type, reconstruct at least one fault logic unit in the faulty disk.

2. The method according to claim 1, characterized in that, When the fault type is a disk fault, the step of reconstructing at least one fault logic unit in the faulty disk based on the fault type includes: Based on the fault type being a disk fault, the data stored in all logical units of the faulty disk is reconstructed.

3. The method according to claim 2, characterized in that, After reconstructing the data stored in all logical units of the faulty disk based on the fault type being a disk fault, the method further includes: The disk replacement flag corresponding to the faulty disk is set as the first disk replacement flag; wherein, the disk replacement flag corresponding to the faulty disk is used to indicate whether the faulty disk should be replaced, and the first disk replacement flag is used to indicate that the faulty disk should be replaced. In response to the disk swap flag corresponding to the faulty disk being designated as the first disk swap flag, a disk swap operation is performed on the faulty disk.

4. The method according to claim 1, characterized in that, When the fault type is a read / write head failure, the step of reconstructing at least one fault logic unit in the faulty disk based on the fault type includes: Based on the fault type being a magnetic arm fault, the data stored in the faulty logical unit of the faulty disk is reconstructed.

5. The method according to claim 4, characterized in that, After reconstructing the data stored in the faulty logical unit of the faulty disk based on the fault type being a read / write head fault, the method further includes: The disk replacement flag corresponding to the faulty disk is set as the second disk replacement flag; wherein, the disk replacement flag corresponding to the faulty disk is used to indicate whether to replace the faulty disk, and the second disk replacement flag is used to indicate whether to replace the faulty disk; And / or set the health status of the faulty disk to a sub-healthy state; wherein the health status is used to indicate the current state of the disk.

6. The method according to claim 4, characterized in that, After reconstructing at least one faulty logical unit in the faulty disk based on the fault type, the method further includes: When the unusable storage capacity in the distributed storage system exceeds the first preset storage capacity, the distributed storage system is expanded.

7. The method according to claim 1, characterized in that, Determining the fault type based on the fault information includes: Based on the fault information, determine whether the proportion of the faulty logical unit in the faulty disk to all logical units in the faulty disk is not less than a preset proportion; wherein, the preset proportion is a positive number less than or equal to 1; If so, the fault type is disk failure; If not, the fault type is magnetic arm fault.

8. The method according to claim 7, characterized in that, The step of determining whether the proportion of faulty logical units in the faulty disk to all logical units in the faulty disk is not less than a preset proportion based on the fault information includes: Traverse the identifiers of the fault logic units included in the fault information; Based on the identifier of the faulty logical unit and the stored annotation information, the corresponding faulty disk is determined; wherein, the annotation information is used to indicate all logical units belonging to the same disk; Based on the identifier of the faulty disk and the labeling information, all logical units belonging to the faulty disk are identified; Based on the fault information, determine whether the proportion of the faulty logical unit in the faulty disk to all logical units in the faulty disk is not less than a preset proportion.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: The storage capacity of the distributed storage system is balanced.

10. A computing device, characterized in that, include: Processor and memory; The processor runs storage system software, which is used to manage and maintain the distributed storage system. The memory is used to store computer program instructions; The processor is used to execute computer program instructions stored in the memory to implement the fault handling method according to any one of claims 1-9.

Images