Data storage method, apparatus and system
By storing replicas on cache nodes and sending copies of data blocks to disk nodes in the distributed storage system of EC storage, the problems of large data migration volume and slow processing speed caused by the lack of association between cache nodes and disk nodes are solved, achieving more efficient data storage and processing.
Patent Information
- Application Number
- PCT/CN2026/094514
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-08
- Filing Date
- 2026-04-30
- Publication Date
- 2026-11-12
AI Technical Summary
In EC storage's distributed storage system, the lack of connection between cache nodes and disk persistence nodes leads to a large amount of data migration, slow processing speed, and high consumption of computing resources.
In the distributed storage system of EC storage, replica storage is performed in the cache of storage nodes, and replicas of data blocks are sent to the disk-writing nodes to reduce the amount of subsequent data migration. Multiple storage nodes are used for caching to ensure that data blocks are replicated on different nodes.
This improves the processing speed of EC storage, reduces the computing resources consumed by communication, and achieves more efficient data storage and processing.
Smart Images

Figure CN2026094514_12112026_PF_FP_ABST
Abstract
Description
Data storage methods, devices and systems
[0001] This application claims priority to Chinese patent application filed on May 8, 2025, with application number 202510597325.0 and entitled "Data Storage Method, Apparatus and System", the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application relates to the field of storage technology, specifically to data storage methods, apparatus, and systems. Background Technology
[0003] In the field of data storage, data redundancy strategies are commonly used to ensure data reliability. These strategies can include various methods such as replica storage and EC (Extended Storage). In distributed storage systems using EC storage, input / output (I / O, primarily write operations) data can first be stored and aggregated in a cache, and then this aggregated data is written to disk via EC storage. In this way, multiple data sets can be aggregated together for EC storage, improving EC storage processing speed. Specifically, when storing data in the cache, replica storage can be used, meaning the data is stored in the caches of multiple storage nodes.
[0004] In current distributed storage systems that implement EC storage, since the data cache nodes and disk write nodes are not related and may be different computing nodes, a lot of data migration is required to migrate the data on the cache nodes to the disk write nodes, resulting in slow EC storage processing and high communication consumption of computing resources. Summary of the Invention
[0005] This application provides a data storage method, apparatus, and system that enables data to be replicated in the cache of storage nodes in a distributed storage system using EC storage, and reduces the amount of data migration during the subsequent EC storage process after replicating the cached data.
[0006] A first aspect provides a data storage method applied to a distributed storage system comprising multiple storage nodes, including a first storage node and a second storage node. The method includes a first storage node receiving a first data block and its logical block address (LBA), and storing the first data block in its cache. The first storage node calculates checksum data for the first data block and stores at least a portion of the checksum data. The method further includes the first storage node determining, based on the LBA, that the node where the first data block is written to disk is a second storage node, and the first storage node sending a copy of the first data block to the second storage node. Finally, the method also includes the second storage node storing the received copy of the first data block in its cache.
[0007] The data storage method provided in this application enables the replication of data in the cache of storage nodes within a distributed storage system employing EC storage. The first data block can be any data block requiring EC storage. Furthermore, in this embodiment, the second storage node serves as both the cache node for the first data block and the disk-writing node (also known as the data home node) used for EC storage of the first data block. Thus, during EC storage, the first data block in the cache of the second storage node can be directly written to the stripe unit on the second storage node (equivalent to flushing to the memory on the second storage node). This reduces the amount of data that needs to be migrated between different nodes during subsequent EC storage of cached data, improves the processing speed of EC storage, and reduces the computational resources consumed by communication.
[0008] Furthermore, in EC storage, both the data (including N copies of data) and the data verification data (including M copies of verification data) need to be stored. In this solution, the first storage node is a verification node used to store the verification data, making the first storage node and the second storage node (the node where the first data block is written to disk) different, thus ensuring that the first data block can be cached on two different storage nodes (the first storage node and the second storage node).
[0009] In one possible implementation, multiple storage nodes include multiple stripes, each stripe includes multiple stripe units, and the distributed storage system also includes compute nodes. The method further includes the compute node determining the target stripe of the first data block based on the LBA of the first data block. The compute node also determines the target stripe unit corresponding to the first data block based on the identifier ID of the target stripe and the number of stripe units in the target stripe. The compute node designates the storage node containing the target stripe unit as the first storage node. Finally, the compute node sends the first data block to the first storage node. Wherein, the target stripe of the first data block is the stripe storing the first data block; the target stripe unit corresponding to the first data block is the stripe unit in the target stripe that stores the first data block.
[0010] The LBA of a data block indicates the starting address where the data block is stored in the memory of the storage system. This starting address is located in one of the multiple stripes in the distributed storage system, that is, the stripe where the data block should be written in the storage system. This stripe can be called the target stripe of the data block. In other words, the target stripe of the data block is the stripe used to store the data block, or the target stripe of the data block is the stripe where the storage address indicated by the LBA of the data block is located. In this scheme, after accurately determining the target stripe of the first data block according to the LBA of the first data block, the target stripe unit is determined according to the ID of the target stripe and the number of stripe units in the stripe (for example, a stripe includes N+M stripe units). In one possible implementation, the target stripe unit corresponding to a data block is determined by the modulus between the target stripe ID and the number of stripe units in the target stripe. Different moduli correspond to different target stripe units (for example, if a stripe includes 6 stripe units, when the target stripe ID is 0, the modulus between the target stripe ID and the number of stripe units 6 is 0, and the ID of the target stripe unit corresponding to 0 is 0; when the target stripe ID is 1, the modulus between the target stripe ID and the number of stripe units 6 is 1, and the ID of the target stripe unit corresponding to 1 is 1). Furthermore, the storage node containing the target stripe unit is designated as the first storage node (also called the primary storage node), ensuring that data blocks with the same target stripe have the same primary storage node, while data blocks with different target stripes have different primary storage nodes. In this way, multiple data blocks written to different stripes can be sent to different primary storage nodes for replica caching, resulting in a more balanced distribution of storage and computational resources across multiple storage nodes.
[0011] In another possible implementation, the multiple storage nodes further include a third storage node for storing at least a portion of the verification data of the first data block. The method also includes the first storage node sending a copy of the first data block to the third storage node, and the third storage node storing the received copy of the first data block in its cache.
[0012] Depending on the requirements, storage nodes can perform k-replica caching of data blocks, where k can be greater than 2, such as 3, M+1 (where M is the number of parity data blocks in EC storage), etc. In this case, in addition to the node that writes the first data block to disk (i.e., the second storage node), the first storage node also determines other redundant nodes (including the third storage node) to cache replicas of the first data block. In a stripe, the storage node where the stripe unit used to store data is located can be called a data node, and the storage node where the stripe unit used to store parity data is located can be called a parity node. There can be multiple parity nodes. In this scheme, the third storage node is both the cache node for the first data block and the parity node (which can be any parity node other than the first storage node) used to store at least a portion of the parity data of the first data block when performing EC storage on the first data block. Thus, since the verification node used to store at least a portion of the parity data of the first data block and the node that writes the first data block to disk (the data node) are different nodes, the third storage node and the second storage node are different, ensuring that the first storage node can send the first data block to multiple different storage nodes (including the second and third storage nodes) to perform replica caching of the first data block.
[0013] Furthermore, in this scheme, for multiple data blocks with the same target stripe, these multiple data blocks are all sent to the same primary storage node (such as the first storage node), the check node used to store the check data of these multiple data blocks is the same, and the disk persistence nodes of these multiple data blocks are different. In this way, multiple data blocks with the same target stripe can be sent to the same primary storage node (such as the first storage node), the same check node (such as the third storage node) for caching, and these multiple data blocks are also sent to their respective disk persistence nodes (such as the second storage node) for caching. This allows for a more reasonable and balanced use of multiple storage nodes for caching multiple data blocks with the same target stripe. It also helps ensure that during the subsequent EC storage process, each data block is directly written from the cache of the disk persistence node to the stripe unit on the disk persistence node, reducing the amount of data migration between different nodes.
[0014] In another possible implementation, the check data of the first data block includes M check data blocks, and the number of the third storage nodes is M-1, where M is an integer greater than 1.
[0015] According to the N+M EC storage method (referring to storing N copies of data and M copies of check data for those N copies of data), when performing EC storage on N data blocks, M check data blocks can be generated. Correspondingly, among the multiple storage nodes, M check nodes can be used to store these M check data blocks. Based on this, in addition to the first storage node being a check node, this embodiment can also use the remaining M-1 check nodes to cache the first data block. This ensures that the first data block can be cached on multiple different storage nodes (i.e., the first storage node, the second storage node, and M-1 third storage nodes), allowing the number of cached replicas of the first data block to reach M, where M is an integer greater than 1, and can be 2, 3, 4, etc. Compared to existing two- or three-replica caching methods, this solution can increase the number of cached replicas, helping to improve the reliability of the first data block in the cache.
[0016] In another possible implementation, multiple storage nodes include multiple stripes, each stripe includes multiple stripe units, and the method in which the first storage node determines the disk destination node for the first data block based on the LBA of the first data block may include the first storage node determining the disk destination node for the first data block based on the LBA of the first data block, the data capacity of each stripe unit in the target stripe of the first data block, and the total data capacity of all stripe units in the target stripe of the first data block used for storing data.
[0017] The LBA of a data block indicates the starting address where the data block is stored in the memory of the storage system. This starting address is located in one of the multiple stripes in the distributed storage system, that is, the stripe where the data block should be written in the storage system, which can be called the target stripe of the data block. Based on this, the disk write node of the first data block can be accurately determined according to the LBA of the first data block, the data capacity of the stripe unit in the target stripe of the first data block, and the total data capacity of all stripe units used to store data in the target stripe of the first data block.
[0018] In another possible implementation, the multiple storage nodes include multiple stripes, and the multiple storage nodes further include a fourth storage node and a fifth storage node. The method further includes the fourth storage node receiving a second data block, the LBA of the second data block, and storing the second data block in its cache. The fourth storage node is used to calculate the checksum of the second data block and store at least a portion of the checksum. The method further includes the fourth storage node determining the disk-writing node for the second data block as the fifth storage node based on the LBA of the second data block. The method may also include the fourth storage node sending a copy of the second data block to the fifth storage node. And the method further includes the fifth storage node storing the received copy of the second data block in its cache.
[0019] The target stripes of the first data block and the second data block are different, and the first storage node and the fourth storage node are different.
[0020] The caching method for the second data block is similar to that of the first data block, and their technical effects are similar; please refer to the description of the technical effects of the caching method for the first data block. Furthermore, in this embodiment, the primary storage nodes for data blocks with different target stripes are different; the primary storage node for the first data block is the first storage node, and the primary storage node for the second data block is the fourth storage node. Thus, different data blocks written to different target stripes can be sent to different storage nodes for replica caching, resulting in a more balanced distribution of storage and computing resources across multiple storage nodes.
[0021] In another possible implementation, multiple storage nodes include multiple stripes, each stripe comprising multiple stripe units, and the distributed storage system also includes compute nodes. The method further includes the compute nodes determining the target stripe of the second data block based on the LBA of the second data block. The compute nodes also determine the target stripe unit corresponding to the second data block based on the identifier ID of the target stripe and the number of stripe units in the target stripe. The compute nodes designate the storage node containing the target stripe unit as a fourth storage node. Finally, the compute nodes send the second data block to the fourth storage node.
[0022] In another possible implementation, the multiple storage nodes also include a sixth storage node for storing at least a portion of the verification data of the second data block. The method further includes a fourth storage node sending a copy of the second data block to the sixth storage node, and the sixth storage node storing the received copy of the second data block in its cache.
[0023] The third and sixth storage nodes are different.
[0024] Depending on the requirements, storage nodes can perform k-replica caching of data blocks, where k can be greater than 2, such as 3, M+1 (where M is the number of parity data blocks in EC storage), etc. In this case, in addition to the primary storage node and the disk write node for the data block, other redundant nodes will be determined to cache replicas of the first data block. In this embodiment, for data blocks with different target stripes, the other redundant nodes can be different; the other storage node for the first data block is the third storage node, and the other storage node for the second data block is the sixth storage node. In this way, different data blocks written to different target stripes can be sent to different other redundant nodes for replica caching, making the storage resource usage and computing resource usage of multiple storage nodes more balanced.
[0025] In another possible implementation, the check data of the second data block includes M check data blocks, and the number of the sixth storage nodes is M-1, where M is an integer greater than 1.
[0026] In another possible implementation, multiple storage nodes include multiple stripes, each stripe including multiple stripe units. In the method, the fourth storage node determines the disk destination node for the second data block based on the LBA of the second data block. This may include the fourth storage node determining the disk destination node for the second data block based on the LBA of the second data block, the data capacity of each stripe unit in the target stripe of the second data block, and the total data capacity of all stripe units in the target stripe of the second data block used for storing data.
[0027] Secondly, a data storage system is provided, which is a distributed storage system comprising multiple storage nodes, including a first storage node and a second storage node. The first storage node is configured to receive a first data block and its Logical Block Address (LBA), store the first data block in its cache, calculate checksum data for the first data block, and store at least a portion of the checksum data. The first storage node is also configured to determine the disk-writing node for the first data block as the second storage node based on its LBA. The first storage node is further configured to send a copy of the first data block to the second storage node. The second storage node stores the received copy of the first data block in its cache.
[0028] In one possible implementation, multiple storage nodes include multiple stripes, each stripe includes multiple stripe units, and the data storage system also includes compute nodes. The compute nodes are used to determine the target stripe of the first data block based on the LBA of the first data block. The compute nodes are also used to determine the target stripe unit corresponding to the first data block based on the identifier ID of the target stripe and the number of stripe units in the target stripe. The compute nodes are further used to designate the storage node containing the target stripe unit as the first storage node. The compute nodes are also used to send the first data block to the first storage node.
[0029] In another possible implementation, the multiple storage nodes also include a third storage node, which stores at least a portion of the checksum data of the first data block. The first storage node is also used to send a copy of the first data block to the third storage node. The third storage node is further used to store the received copy of the first data block in its cache.
[0030] In another possible implementation, the check data of the first data block includes M check data blocks, and the number of the third storage nodes is M-1, where M is an integer greater than 1.
[0031] In another possible implementation, multiple storage nodes include multiple stripes, each stripe includes multiple stripe units, and the first storage node is further configured to determine the disk destination node for the first data block based on the LBA of the first data block, the data capacity of each stripe unit in the target stripe of the first data block, and the total data capacity of all stripe units in the target stripe of the first data block used for storing data.
[0032] In another possible implementation, the third storage node is also used in the event of a failure of the first storage node to: receive the first data block and the logical block address (LBA) of the first data block, store the first data block in the cache of the third storage node; calculate the checksum of the first data block; determine the disk-writing node of the first data block as the second storage node based on the LBA of the first data block; and send a copy of the first data block to the second storage node.
[0033] In another possible implementation, the multiple storage nodes include multiple stripes, and further include a fourth storage node and a fifth storage node. The fourth storage node is used to receive the second data block, the LBA of the second data block, store the second data block in its cache, and calculate and store at least a portion of the checksum of the second data block. The fourth storage node is also used to determine the disk write node for the second data block as the fifth storage node based on the LBA of the second data block.
[0034] The target stripes of the first data block and the second data block are different, and the first storage node and the fourth storage node are different.
[0035] In another possible implementation, multiple storage nodes comprise multiple stripes, each stripe comprising multiple stripe units, and the distributed storage system further comprises compute nodes. The compute nodes are used to determine the target stripe of the second data block based on its LBA. Furthermore, the compute nodes are also used to determine the target stripe unit corresponding to the second data block based on the identifier ID of the target stripe and the number of stripe units in the target stripe. Additionally, the compute nodes are used to designate the storage node containing the target stripe unit as a fourth storage node. Finally, the compute nodes are also used to send the second data block to the fourth storage node.
[0036] In another possible implementation, the multiple storage nodes also include a sixth storage node. The sixth storage node is used to store at least a portion of the checksum data of the second data block. The fourth storage node is also used to send a copy of the second data block to the sixth storage node, and the sixth storage node is also used to store the received copy of the second data block in its cache.
[0037] The third and sixth storage nodes are different.
[0038] In another possible implementation, the check data of the second data block includes M check data blocks, and the number of the sixth storage nodes is M-1, where M is an integer greater than 1.
[0039] In another possible implementation, multiple storage nodes include multiple stripes, each stripe includes multiple stripe units, and the fourth storage node is also used to determine the disk destination node for the second data block based on the LBA of the second data block, the data capacity of each stripe unit in the target stripe of the second data block, and the total data capacity of all stripe units in the target stripe of the second data block used for storing data.
[0040] Thirdly, a computer program product containing instructions is provided, which, when executed by a storage system comprising multiple storage nodes, causes the storage system to perform the data storage method as described in the first aspect.
[0041] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium including computer program instructions, which, when executed by a storage system including multiple storage nodes, cause the storage system to perform the data storage method as described in the first aspect.
[0042] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods.
[0043] The following description includes more specific details about the implementation methods provided for the above aspects. Attached Figure Description
[0044] Figure 1 is a schematic diagram of the data storage system provided in an embodiment of this application;
[0045] Figure 2 is a schematic flowchart of the first data storage method provided in an embodiment of this application;
[0046] Figure 3 is a schematic diagram of a second data storage method provided in an embodiment of this application;
[0047] Figure 4 is a schematic diagram of the third data storage method provided in the embodiments of this application;
[0048] Figure 5 is a schematic diagram of a first example of the data storage method provided in the embodiments of this application;
[0049] Figure 6 is a schematic diagram of a second example of the data storage method provided in the embodiments of this application;
[0050] Figure 7 is a schematic diagram of a third example of the data storage method provided in the embodiments of this application;
[0051] Figure 8 is a schematic diagram of a fourth example of the data storage method provided in the embodiments of this application;
[0052] Figure 9 is a schematic diagram of the structure of the storage device provided in an embodiment of this application;
[0053] Figure 10 is a schematic diagram of the structure of the computing device provided in the embodiment of this application. Detailed Implementation
[0054] In the field of data storage, data redundancy strategies can be employed to ensure data reliability. Data redundancy strategies involve adding redundant data to the existing data and storing both the original and redundant data. In the event of data corruption, the data can be recovered using the redundant data. Data redundancy strategies can include various methods such as replica storage and erasure coding (EC) storage.
[0055] Replica storage achieves data redundancy by storing data in multiple copies. For example, n-replica storage copies data A n times (n can be an integer greater than 1, such as 2, 3, etc.), storing them on n storage nodes in the storage system, such as storage node 0, storage node 1, storage node 2, etc. If n-1 storage nodes fail simultaneously, the remaining working storage node will still retain data A. Thus, n-replica storage allows for data preservation even if n-1 storage nodes fail simultaneously.
[0056] EC storage performs EC calculations on data A, which includes N copies of data (referred to as N original data), and adds M copies of check data (each of which can be called a check data block). When any M copies (including original data and check data) of the N original data + M check data fail, data A can be restored from the remaining data. Thus, storing N copies of data + M copies of check data achieves data redundancy. Here, N is an integer greater than 1, and M is an integer greater than or equal to 1. In a specific embodiment, EC storage uses an N+M scheme, dividing data A into N data blocks, generating M check data blocks from the N data blocks using the EC algorithm, and storing the N data blocks + M check data blocks on N+M storage nodes in the storage system. When any M storage nodes fail (or any M copies of data fail), the original data A can be restored from the remaining data.
[0057] EC storage has a higher storage space utilization rate than replica storage, reducing the storage space required for data redundancy. Taking the simultaneous failure of two storage nodes as an example: replica storage requires copying the data three times, with a storage space utilization rate of 1 / 3, or approximately 33%; EC storage requires adding 2 copies of checksum data to the data containing N copies (taking N=4 as an example), with a storage space utilization rate of 4 / (4+2), or approximately 67%.
[0058] In EC storage, data can be stored in stripes. For EC storage using an N+M scheme, multiple storage nodes in the storage system can include multiple stripes, each stripe comprising N+M stripe units, which are located on N+M storage nodes. N data blocks and M parity data blocks can be stored in the N+M stripe units of the stripe. For example, EC storage using a 4+2 scheme includes 6 hard drives (each hard drive is a storage node), and these 6 hard drives include six stripes: stripe 0, stripe 1, stripe 2, stripe 3, stripe 4, stripe 5, and stripe 6. Each stripe includes stripe units 0, 1, 2, 3, 4, and 5 located on hard drives 0, 1, 2, 3, 4, and 5, respectively. When data A is stored in EC, it is divided into 4 data blocks and 2 parity data blocks according to the data capacity of the stripe unit. The 4 data blocks are written to the 4 stripe units of stripe 0, and the 2 parity data blocks are written to the remaining 2 stripe units of stripe 0.
[0059] In the embodiments of this application, for any stripe, the storage node where the stripe unit used to store data blocks is located can be called a data node, and the storage node where the stripe unit used to store parity data blocks is located can be called a parity node.
[0060] The data capacity of a stripe unit can be configured according to requirements, such as 2KB, 4KB, 8KB, 16KB, 64KB, etc. The total data capacity of all stripe units used to store data blocks in a stripe is called the stripe size (e.g., 16KB, 32KB). For example, if a stripe includes 6 stripe units, with 4 units storing 4 data blocks and 2 units storing 2 parity blocks, and each stripe unit has a data capacity of 64KB, then the stripe size is 4 * 64KB = 256KB. Striping allows multiple processes to access different parts of data simultaneously without causing disk conflicts and balances the input / output (I / O, mainly read / write operations) load across multiple disks, thereby improving the I / O performance of the storage system.
[0061] EC (Extended EC) storage includes EC lowercase, which refers to writing a portion of the data blocks within a stripe. Performing an EC lowercase operation requires first requesting the storage nodes in the storage system to read the old data from the stripe, and then writing the new data into the stripe. For example, stripe 1 includes 10 stripe cells. The first EC lowercase operation divides data A into 4 data blocks and generates 2 parity blocks from these 4 data blocks using the EC algorithm. These 4 data blocks plus the 2 parity blocks are then written into 6 stripe cells of stripe 1. The second EC lowercase operation requires first reading the already stored data in stripe 1 (referred to as the old data, including the aforementioned 4 data blocks and 2 parity blocks of data A), then obtaining the original data A based on the old data. Next, the new data B to be stored is divided into 4 data blocks, and the EC algorithm generates 2 parity blocks from these 4 data blocks. The 4 data blocks of data A plus the 4 data blocks of new data B plus the 2 parity blocks are then written into the 10 stripe cells of stripe 1. It is evident that compared to writing all the stripe cells in each write operation (which can be called full stripe write, and full stripe write does not require reading the old data in the stripe), EC write is slower and consumes more computing power.
[0062] To improve the performance of storage systems, such as the speed of EC (Extended EC) storage, I / O data (primarily write operations) can be first stored and aggregated in the cache, and then this aggregated I / O data can be written to the hard drive via EC storage. In this way, multiple I / O data can be aggregated together for EC storage, improving the processing speed and performance of EC storage (including EC lowercase). When storing I / O data in the cache, I / O data can be replicated, meaning that I / O data is stored in the caches of multiple storage nodes, resulting in multiple copies of the I / O data cache (referred to as a replica cache), thereby improving the reliability of the I / O data cache.
[0063] EC storage can be implemented in centralized storage systems. In a centralized storage system, computing devices (referred to as hosts) can be connected to a Redundant Array of Independent Disks (RAID). A RAID can include multiple physical hard drives (such as Hard Disk Drives (HDDs), Solid State Drives (SSDs), etc.), and data is stored across these drives connected to the RAID. Each drive can be a storage node. In centralized storage, the RAID and computing devices are connected via a Peripheral Component Interconnect Express (PCIe) interface. All input / output (I / O, primarily read / write operations, hereinafter referred to as I / O) from the computing device must pass through the PCIe interface to reach the RAID. The maximum bandwidth of the PCIe interface can become a performance bottleneck for RAID in processing I / O requests. For example, the maximum bandwidth of a PCIe 4.0 x16 interface is 32 GB / s, meaning the maximum data throughput that the RAID can handle for I / O requests is limited to 32 GB / s.
[0064] EC storage can be implemented within traditional distributed storage. In traditional distributed storage, the storage system comprises multiple communicating computing devices, each connected to a hard drive. Data is stored across these connected hard drives. Each hard drive can be considered a storage node. Distributed storage enables simultaneous storage and retrieval of data across multiple computing devices, significantly improving the read / write speed of the storage system. Compared to centralized storage, distributed storage distributes all I / O across multiple computing devices, thus improving the I / O performance of the storage system. However, in a distributed storage system, during I / O processing, one or more computing devices may request read / write access to the hard drives connected to other computing devices. The processors of these devices consume computing power to perform this operation. Therefore, the maximum computing power of the processors can become a bottleneck in the I / O performance of the storage system.
[0065] In traditional distributed storage systems, after receiving I / O, computing devices store the I / O data in replicas across multiple device caches. For example, the I / O data is stored in the caches of computing node 1 and computing node 2 respectively, achieving two-copy storage. When writing the cached I / O data to disk using EC (Extended Checksum) storage, N data blocks of the I / O data and M parity data blocks obtained from EC calculation are written to N+M disks respectively to achieve EC storage. For example, they are written to disk 3 connected to computing device 3, disk 4 connected to computing device 4, and disk 5 connected to computing device 5. In traditional distributed storage systems, the computing node that caches I / O data can be called a cache node, and the computing node containing the disks storing the I / O data and its parity data can be called a disk write node. Thus, since the cache nodes (compute node 1 and compute node 2) and the disk-based nodes (compute nodes where hard disks 3, 4, and 5 are located) are not related and are different compute nodes, a lot of data migration is required to migrate the I / O data on the cache nodes to the disk-based nodes, resulting in a slow I / O processing process and communication consuming a lot of computing resources.
[0066] In some implementations of distributed storage systems, hard disks can integrate computing power to perform computational and communication functions, and can also include caches. In such distributed storage systems, compared to traditional distributed storage systems, some operations that were originally performed by computing devices can be moved to the hard disk, reducing the computing power consumption of computing devices and improving the I / O performance of the storage system.
[0067] As an example, Figure 1 is a schematic diagram of a data storage system provided in an embodiment of this application. As shown in Figure 1, the data storage system 100 includes multiple computing nodes (e.g., computing node 110-1, computing node 110-2, computing node 110-3...) and multiple storage nodes (e.g., storage node 0, storage node 1, storage node 2, storage node 3, storage node 4, storage node 5...) connected to the computing nodes.
[0068] Compute nodes are used to provide data storage services and receive and process I / O from user equipment and other devices. Compute nodes can be various devices or modules capable of performing their functions, such as applications, software modules, and hardware devices (e.g., servers, terminals).
[0069] Storage nodes are used to store data. Storage nodes may include storage devices such as hard drives for long-term data storage. Storage nodes integrate computing power to implement computing and communication functions, and include caches. As shown in Figure 1, a storage node includes a cache, a processing module, and memory. The cache may be a hardware unit for caching, such as a high-speed cache memory; the processing module may be a hardware unit such as a chip for implementing communication and computing functions; and the memory may be a hardware unit such as NAND flash memory for long-term data storage. Storage nodes can communicate with each other. As an example, a storage node includes a Solid State Unit (SSU), and each SSU includes a cache, a processing chip, and NAND flash memory. Storage nodes can also be other existing or future storage devices that integrate computing power and caches; this application does not limit this.
[0070] In this distributed storage system, compute nodes receive I / O and then distribute the I / O data to storage nodes. Storage nodes can perform operations such as storing data in a cache, performing computational operations required for EC storage of the data, communicating with other storage nodes to send and receive data and read / write data from other storage nodes, and writing data from the cache to storage. For details regarding the functions implemented by the compute nodes and each storage node, please refer to the relevant descriptions in the data storage methods provided in the embodiments of the following application.
[0071] In this embodiment, the data storage system 100 uses an N+M EC storage method for data storage. Furthermore, the multiple storage nodes of the data storage system 100 include multiple stripes, each stripe comprising N+M stripe units, which are located in the memories of the N+M storage nodes respectively. As an example, as shown in FIG1, each stripe includes 4+2 stripe units: stripe unit 0 is located in the memory of storage node 0, stripe unit 1 is located in the memory of storage node 1, stripe unit 2 is located in the memory of storage node 2, stripe unit 3 is located in the memory of storage node 3, stripe unit 4 is located in the memory of storage node 4, and stripe unit 5 is located in the memory of storage node 5.
[0072] It should be noted that Figure 1 is merely a schematic diagram of the architecture of a data storage system provided in an embodiment of this application, and the positional relationships between the devices, components, modules, etc. shown in the figure do not constitute any limitation.
[0073] In the distributed storage system, after a compute node receives I / O (mainly write requests) from other devices, it divides the data in the I / O into one or more data blocks based on the stripe size (the total data capacity of all stripe units used for data storage) and the stripe units in the distributed storage system. Then, the compute device sequentially sends each data block to the storage node, which caches the received data blocks. Subsequently, the storage node writes the cached data blocks into the stripe using EC storage (equivalent to writing to memory).
[0074] In the distributed storage system, the storage node used to cache I / O data can be called a cache node, and the storage node where the memory used to store I / O data and I / O data verification data is located can be called a disk persistence node.
[0075] As an example, using an N+M EC storage configuration, each stripe consists of 6 stripe units, with a data capacity of 64KB per unit, resulting in a stripe size of 4 * 64KB = 256KB. The compute node receives I / O from other devices, which includes the data to be stored and the logical block address (LBA, indicating the starting address of the data in the storage system's memory). The compute node can partition the I / O data as follows:
[0076] When the size of the I / O data is less than the stripe size, for example, the data size is 200KB, the compute node will divide the 200KB data into multiple data blocks (64KB data block 1, 64KB data block 2, 64KB data block 3, and 8KB data block 4) according to the data capacity of the stripe unit (64KB), and obtain the LBA of each data block.
[0077] When the size of the I / O data is greater than the stripe size, for example, the data size is 300KB, the compute node will first divide the data into multiple segments according to the stripe size (data segment 1 with a size of 256KB and data segment 2 with a size of 44KB). Then, according to the data capacity of the stripe unit of 64KB, each segment of data will be divided into one or more data blocks (data segment 1 with a size of 256KB is divided into data block 1, data block 2, data block 3, and data block 4 with a size of 64KB, and data segment 2 with a size of 44KB is divided into data block 5 with a size of 44KB).
[0078] The compute node sequentially sends the aforementioned data blocks and their LBAs to the storage node, which then caches the received data blocks. Subsequently, the storage node writes the cached data blocks into the stripe using EC (Extra Check) storage. For example, if the storage node's cache contains N data blocks that should be written to a stripe, the storage node will perform EC calculations on these N data blocks to obtain M parity data blocks, and then write these N data blocks and M parity data blocks into N+M stripe cells of a stripe.
[0079] In this embodiment, to distinguish different data blocks, a data block may be referred to as a first data block, a second data block, or a third data block, etc. Similarly, a storage node may be referred to as a first storage node, a second storage node, a third storage node, a fourth storage node, a fifth storage node, a sixth storage node, etc. The terms first, second, third, fourth, etc., are only used to distinguish different data blocks / storage nodes and do not constitute any limitation on the data blocks / storage nodes.
[0080] This application provides a data storage method applicable to distributed storage systems of the aforementioned type, including multiple storage nodes (where storage nodes integrate computing power to perform computation and communication functions, and storage nodes include caches). This method enables the replication and caching of I / O data across multiple storage nodes. Furthermore, by implementing replication caching through the data storage method provided in this application, the amount of data migration during the subsequent writing of cached I / O data to memory via EC storage is reduced, thereby accelerating I / O processing speed and reducing computational resource consumption during communication. The data stored in the data storage method provided in this application can be of various types, such as images, text, and audio data.
[0081] The data storage method provided in the embodiments of this application is described in detail below. The data storage method provided in the embodiments of this application can be jointly executed by multiple nodes such as storage nodes and computing nodes in a distributed storage system, wherein the method steps executed by each storage node / computing node can constitute the data storage method executed by that storage node / computing node.
[0082] The following figures 2-4 illustrate various method flows for data storage through multiple embodiments.
[0083] Figure 2 is a schematic flowchart of a first method for storing data according to an embodiment of this application. In this first method flowchart, the storage method for any data block is described in detail. The data storage method in this first method flowchart may include the following steps.
[0084] Step 210: The first storage node receives the first data block and the LBA of the first data block, and saves the first data block in the cache of the first storage node.
[0085] For each data block, a storage node can be determined from among multiple storage nodes, which can be called the primary storage node. Compute nodes can send data blocks to the corresponding primary storage node. The primary storage node can perform replica caching of the data blocks, including the primary storage node keeping a copy of the received data block in its local cache, and also sending a copy of the received data block to other storage nodes so that other storage nodes also keep a copy of the data block in their local cache.
[0086] In this embodiment, any data block is referred to as the first data block, and the primary storage node of the first data block is referred to as the first storage node. The primary storage node of the data block can be determined by the computing node receiving I / O, or it can be determined by other computing nodes, which can then send the indication information of the primary storage node to the computing node receiving I / O, so that the computing node receiving I / O can send the data block to the corresponding primary storage node.
[0087] In some embodiments, the primary storage node for a data block can be determined based on the target stripe of the data block. The specific implementation of this determination method is described in detail below.
[0088] As mentioned earlier, each data block has a corresponding LBA, which indicates the starting address where the data block is stored in the memory of the storage system. This starting address is located in one of the multiple stripes in the distributed storage system, that is, the stripe where the data block should be written in the storage system. This stripe can be called the target stripe of the data block. In other words, the target stripe of the data block is the stripe used to store the data block, or the target stripe of the data block is the stripe where the storage address indicated by the data block's LBA is located. In a specific embodiment, the target stripe of the data block can be determined based on the data block's LBA. Then, the target stripe unit corresponding to the data block can be determined based on the identity document (ID) of the target stripe and the number of stripe units in the target stripe. Finally, the storage node where the target stripe unit is located is determined as the primary storage node of the data block. The target stripe unit corresponding to a data block can be determined based on the modulus between the target stripe ID and the number of stripe units in the target stripe (the modulus between the target stripe ID and the number of stripe units in the target stripe is equivalent to the remainder when the target stripe ID is divided by the number of stripe units in the target stripe). Different moduli correspond to different target stripe units. For example, if a stripe contains 6 stripe units, when the target stripe ID is 0, the modulus between the target stripe ID and the number of stripe units 6 is 0, and the ID of the target stripe unit corresponding to 0 is 0. When the target stripe ID is 1, the modulus between the target stripe ID and the number of stripe units 6 is 1, and the ID of the target stripe unit corresponding to 1 is 1, and so on. For example, a stripe consists of 6 stripe units. When the target stripe ID is 0, the modulus between the target stripe ID and the number of stripe units 6 is 0, and the ID of the target stripe unit corresponding to 0 is 5. When the target stripe ID is 1, the modulus between the target stripe ID and the number of stripe units 6 is 1, and the ID of the target stripe unit corresponding to 1 is 4, and so on.
[0089] The target stripe ID of a data block can be determined based on the data block's LBA, the data capacity of the stripe unit, and the stripe size (the total data capacity of all stripe units used to store data in the stripe). For example, the target stripe ID can be calculated as follows: ((LBA – LBA%[stripe unit]) / [stripe size]). Furthermore, the target stripe unit ID corresponding to the data block can be determined based on the remainder when the target stripe ID is divided by the number of stripe units in the target stripe (equal to N+M).
[0090] As an example, the ID of the target stripe unit corresponding to the data block can be determined by the following calculation: ((LBA – LBA%[stripe unit]) / [stripe size])%(N+M). Taking the first data block with LBA = 500KB, stripe unit = 64KB, N = 4, M = 2, and stripe size = 4 * 64KB = 256KB as an example, according to the aforementioned calculation method, 500KB%64 = 52KB, 500KB - 52KB = 448KB, 448KB / 256KB = 1 (rounded to the nearest integer), 1%6 = 1, that is, the ID of the target stripe of the first data block is 1, and the ID of the target stripe unit corresponding to the first data block is 1.
[0091] As mentioned earlier, each stripe comprises N+M stripe units, which are located on N+M storage nodes. Taking N+M = 4+2 as an example, each stripe includes six stripe units: stripe unit 0, stripe unit 1, stripe unit 2, stripe unit 3, stripe unit 4, and stripe unit 5. These six stripe units are located on storage nodes 0, 1, 2, 3, 4, and 5, respectively. Thus, the ID of the target stripe unit corresponding to the first data block is 1, and the storage node where stripe unit 1 is located is storage node 1. Therefore, storage node 1 is the primary storage node (or primary storage node) for the first data block.
[0092] As can be seen from the aforementioned method, in some embodiments, for any number of different data blocks, if the target stripes of the data blocks are the same, then the primary storage nodes of the two data blocks are the same; if the target stripes of the data blocks are different, then the primary storage nodes of the data blocks can be different.
[0093] As described above, for multiple data blocks with the same target stripe, the compute node will send these data blocks to the primary storage node (e.g., the first storage node) corresponding to that target stripe for replica caching. That is, for a stripe, all data blocks that should be written to that stripe can correspond to the same primary storage node. In other words, one stripe can correspond to one primary storage node, and the primary storage node for all data blocks that should be written to that stripe is also the primary storage node for that stripe. Furthermore, all data blocks that should be written to a stripe can be sent to the primary storage node corresponding to that stripe for replica caching, and multiple data blocks written to different stripes can be sent to different primary storage nodes for replica caching. Based on this, the storage resource usage and compute resource usage of multiple storage nodes are relatively balanced.
[0094] In this embodiment, the primary storage node corresponding to the stripe can also be used to perform EC storage on the data blocks in the local cache when the data blocks in the local cache meet the flush conditions. The flush conditions may include the number of data blocks stored in the local cache of the primary storage node corresponding to the stripe being equal to N (EC storage adopts an N+M scheme), the usage time of the local cache of the primary storage node corresponding to the stripe being equal to or greater than a preset time, etc. EC storage includes performing EC calculation on the data blocks in the local cache to obtain the check data of the data blocks, and communicating with other storage nodes to write the data blocks and check data into the stripe for storage. The primary storage node can also be a check node used to store the check data blocks of the data blocks in the local cache, that is, the primary storage node is used to store at least part of the check data of the first data block in the local cache. For more detailed descriptions and examples of this step, please refer to the relevant description of the example shown in Figure 5.
[0095] In this embodiment, the number of data copies k (where k is an integer greater than or equal to 2) that the first storage node caches for a data block can be configured according to requirements. Cached k data blocks across k storage nodes can be called k-replica caching. For example, the data block can be cached with two replicas, three replicas, or M+1 replicas (where M is the number of checksum data blocks stored in EC), etc. Correspondingly, for k-replica caching of a data block, the first storage node also needs to determine k-1 storage nodes among multiple storage nodes to cache the replicas of the data block.
[0096] In some embodiments, if k=2, that is, the first storage node caches two copies of the data block, the first storage node may determine a second storage node among multiple storage nodes to cache a copy of the first data block. In this case, the method may further include the following steps 220-240.
[0097] In other embodiments, if k > 2, such as when the first storage node performs three-replica caching or M+1-replica caching (M ≥ 2) on the data block, the first storage node can determine a second storage node and k-2 third storage nodes from among multiple storage nodes to cache replicas of the first data block. In this case, the method may further include steps 220-270. There is no execution order restriction between steps 220-240 and steps 250-270.
[0098] Step 220: The first storage node determines the disk-writing node of the first data block as the second storage node based on the LBA of the first data block.
[0099] Step 230: The first storage node sends a copy of the first data block to the second storage node.
[0100] Step 240: The second storage node saves a copy of the received first data block in the cache of the second storage node.
[0101] The disk write node of the first data block can also be called the data ownership node of the first data block. In EC storage, it refers to the storage node where the corresponding stripe unit is located when the first data block is written from the cache to the target stripe.
[0102] The disk write node for a data block can be determined based on the LBA of the data block. In a specific embodiment, the disk write node for the first data block can be determined based on the LBA of the first data block, the data capacity of each stripe unit in the target stripe of the first data block, and the stripe size of the target stripe of the first data block (the total data capacity of all stripe units in the target stripe used to store data).
[0103] As an example, the disk write node for the first data block can be determined by the following calculation: ((LBA%[stripe size]) / [stripe unit]). Taking the first data block with LBA = 500KB, stripe unit = 64KB, and stripe size = 4 * 64KB = 256KB as an example, according to the above calculation method, 500KB%256KB = 244KB, 244KB / 64KB = 3 (rounded to the nearest integer), which means the disk write node for the first data block is the 3rd storage node in reverse order of the IDs of the multiple storage nodes. For example, if the IDs of the 6 storage nodes are storage node 0, storage node 1, storage node 2, storage node 3, storage node 4, and storage node 5, the disk write node for the first data block is the 3rd storage node in reverse order of the IDs, which is storage node 3.
[0104] As can be seen from the aforementioned method, different data blocks have different LBAs, and the corresponding disk write nodes for different data blocks can be different.
[0105] In this embodiment, the second storage node serves as both a cache node for the first data block and a disk-writing node used to store the first data block during EC storage. During EC storage, the first data block from the cache of the second storage node can be directly written to the stripe unit on the second storage node (equivalent to writing to the memory on the second storage node). For more detailed explanations and examples of this step, please refer to the relevant descriptions of the example shown in Figure 5.
[0106] In this way, the cache node and disk write node of the first data block are associated (for example, both the cache node and the disk write node are second storage nodes). The data blocks in the cache of the cache node are directly written to the stripe unit of the cache node, which can reduce the amount of data that needs to be migrated between different nodes when performing EC storage and improve the processing speed of I / O data for EC storage.
[0107] Step 250: The first storage node determines the third storage node.
[0108] The number of third storage nodes can be k-2. For example, if the first storage node caches the data block with three copies, then k=3, and the number of third storage nodes is 1. If the first storage node caches the data block with M+1 copies (M≥2), then k=M+1, and the number of third storage nodes is M-2.
[0109] In the embodiments of this application, in addition to the primary storage node of the data block (such as the first storage node of the first data block) and the disk-writing node (such as the second storage node of the first data block), other storage nodes used to cache copies of the data block can be referred to as other redundant nodes (such as the third storage node of the first data block).
[0110] The first storage node can determine the third storage node in various feasible ways, such as arbitrarily selecting a storage node other than the first and second storage nodes from among multiple storage nodes as the third storage node.
[0111] In some embodiments, third storage nodes can be selected according to certain rules, such that the k-2 third storage nodes corresponding to any two data blocks written to different stripes may not be completely identical. This non-identical nature can include the k-2 third storage nodes corresponding to the two data blocks not overlapping, or it can include some overlapping storage nodes among the k-2 third storage nodes corresponding to the two data blocks.
[0112] As an example, the third storage node can be determined according to the following rules: If k-2 = 1, then among the remaining storage nodes excluding the first and second storage nodes, the storage node offset one position from the ID of the first storage node will be designated as the third storage node; if k-2 ≥ 2, then among the remaining storage nodes excluding the first, second, and already determined third storage nodes, the storage node offset one position from the ID of the i-th third storage node will be designated as the (i+1)-th third storage node. The offset can be in the order of the IDs or in the reverse order of the IDs.
[0113] In this embodiment, when the primary storage node corresponding to the first data block performs EC storage on the data block in its local cache, the third storage node can also be a check node used to store check data blocks. That is, the third storage node is used to store at least a portion of the check data of the first data block. For example, when the primary storage node performs EC storage on the data block in its local cache, it calculates M check data blocks. When the number of third storage nodes is M-2, the M-2 third storage nodes are used to store the M-2 check data blocks. For more detailed explanations and examples of this step, please refer to the relevant description of the example shown in Figure 5.
[0114] Step 260: The first storage node sends a copy of the first data block to the third storage node.
[0115] When k-2 = 1, there is one third storage node. In this case, the first storage node only needs to send a copy of the first data block to one third storage node. When k-2 ≥ 2, there are multiple third storage nodes. In this case, the first storage node sends a copy of the first data block to each of the third storage nodes.
[0116] Step 270: The third storage node saves a copy of the received first data block in the cache of the third storage node.
[0117] In the data storage method provided in this application embodiment, for different data blocks, if the target stripes of the data blocks are different, the computing device can send different data blocks to different primary storage nodes, and the storage nodes used to cache copies of the data blocks can also be different or not completely the same. The data storage method for different data blocks is described in detail below through the second method flow in Figure 3.
[0118] Figure 3 is a schematic diagram of a second flow of the data storage method provided in an embodiment of this application. In this second flow, the storage method for different multiple data blocks is described in detail. The multiple data blocks include a first data block and a second data block, which can be any two data blocks with different target stripes. Taking the example of caching more than 2 copies of data blocks, the data storage method may include the following steps.
[0119] Step 310: The first storage node receives the first data block and the LBA of the first data block, and saves the first data block in the cache of the first storage node.
[0120] Step 312: The first storage node determines the disk write node of the first data block as the second storage node based on the LBA of the first data block.
[0121] Step 314: The first storage node sends a copy of the first data block to the second storage node.
[0122] Step 316: The second storage node saves a copy of the received first data block in the cache of the second storage node.
[0123] Step 318: The first storage node determines the third storage node. The number of third storage nodes is k-2.
[0124] Step 320: The first storage node sends a copy of the first data block to the third storage node.
[0125] Step 322: The third storage node saves a copy of the received first data block in the cache of the third storage node.
[0126] Step 324: The fourth storage node receives the second data block and its LBA, and stores the second data block in the cache of the fourth storage node.
[0127] The method for determining the fourth storage node is similar to that for determining the first storage node. However, because the target stripes of the first and second data blocks are different, the first and fourth storage nodes are different.
[0128] Step 326: The fourth storage node determines the disk write node of the second data block as the fifth storage node based on the LBA of the second data block.
[0129] The method for determining the fifth storage node is similar to that for determining the second storage node. However, if the disk write node of the first data block is different from that of the second data block, then the second and fifth storage nodes are different. If the disk write node of the first data block is the same as that of the second data block, then the second and fifth storage nodes are the same.
[0130] Step 328: The fourth storage node sends a copy of the second data block to the fifth storage node.
[0131] Step 330: The fifth storage node saves a copy of the received second data block in the cache of the fifth storage node.
[0132] Step 332: The fourth storage node determines the sixth storage node. The number of sixth storage nodes is k-2.
[0133] The method for determining the sixth storage node is similar to that for determining the third storage node. However, because the target stripes of the first and second data blocks are different, the first and fourth storage nodes are different, and the k-2 sixth storage nodes and k-2 third storage nodes may not be completely identical.
[0134] Step 334: The fourth storage node sends a copy of the first data block to the sixth storage node.
[0135] Step 336: The sixth storage node saves a copy of the received first data block in the cache of the sixth storage node.
[0136] According to the aforementioned method, in the embodiments of this application, for a storage node among multiple storage nodes in a distributed storage system, it can be the master storage node of a data block, or the data ownership node of other data blocks, or a verification node used to store verification data blocks of other data blocks.
[0137] In this embodiment, for any two data blocks written to different stripes, the k storage nodes caching the data block replicas do not need to be completely identical. For example, all k storage nodes may be different, or some of the k storage nodes may overlap. In this way, replicas of any number of data blocks written to different stripes can be stored relatively evenly across multiple storage nodes in the distributed storage system, resulting in a more balanced utilization of storage resources across multiple storage nodes.
[0138] This application also provides a data storage method for replicating and caching I / O data across multiple storage nodes when some storage nodes in a distributed storage system fail. Furthermore, this data storage method can accelerate the fault recovery speed of a distributed storage system.
[0139] Figure 4 is a schematic diagram of a third method for data storage provided in an embodiment of this application. In this third method flow, a detailed explanation is provided of the data block storage method in the event of partial storage node failure in a distributed storage system.
[0140] In the first scenario, for any given stripe, the primary storage node corresponding to that stripe fails. In this case, the functions performed by the primary storage node for that stripe are transferred to the backup storage node for the same stripe.
[0141] That is, for any first data block, in the event of a failure of the primary storage node (which can be referred to as the first storage node) corresponding to the target stripe of the first data block, the backup storage node corresponding to the target stripe is used for:
[0142] The system receives the first data block and its LBA, stores the first data block in the cache of the backup storage node, and calculates the checksum of the first data block. Based on the LBA of the first data block, it determines the disk-writing node of the first data block as the second storage node. It sends a copy of the first data block to the second storage node and stores at least a portion of the checksum of the first data block. When it is necessary to cache k copies of the data block, where k > 2, for example, k = 4, the backup storage node is also used to determine other redundant nodes and send copies of the first data block to other redundant nodes so that they can be stored in the cache of other redundant nodes.
[0143] Thus, in the event of a failure of the primary storage node corresponding to a stripe, the storage system can quickly restore its functionality and continue the normal data storage process by transferring the functions performed by the primary storage node to the backup storage node. Data that was originally intended to be stored on the primary storage node can be transferred to other storage nodes.
[0144] For any first data block, the third storage node (also a check node) shown in Figure 4 can serve as the backup storage node corresponding to the target stripe where the first data block is located.
[0145] Multiple storage nodes can comprise multiple stripes, which can include multiple stripe periods. One stripe period consists of N+M stripes. For example, if multiple storage nodes comprise 36 stripes with stripe IDs 0-35, and N+M = 4+2 = 6, then the 36 stripes comprise 36 / 6 = 6 stripe periods. The first stripe period includes stripes 0-5, the second includes stripes 6-11, the third includes stripes 12-17, the fourth includes stripes 18-23, the fifth includes stripes 24-29, and the sixth includes stripes 30-35.
[0146] In some embodiments, the backup storage nodes corresponding to a stripe can also be determined based on the stripe period. For example: for the i-th stripe period, the backup storage nodes corresponding to the stripe [(i-1)*(N+M)]–[i*(N+M)-1] are storage nodes [j]-[j+5] respectively; for the (i+1)-th stripe period, based on the backup storage nodes [j]-[j+N+M-1] corresponding to the N+M stripes of the previous stripe period, each backup storage node in the backup storage nodes [j]-[j+N+M-1] is offset by one bit according to the storage node ID, resulting in N+M storage nodes as the backup storage nodes corresponding to the stripes [i*(N+M)]–[(i+1)*(N+M)-1] in the (i+1)-th stripe period. The offset can be in the direction of ID order or in the direction of ID reversal.
[0147] This embodiment enables the functions of the primary storage nodes of multiple stripes to be evenly transferred to the remaining storage nodes for execution in multiple stripe cycles, rather than all being transferred to a single storage node, thereby making the computing and storage resource usage of multiple storage nodes more balanced.
[0148] In the second scenario, for any data block, if any storage node other than the primary storage node (the first storage node) fails, such as the second or third storage node, the primary storage node continues to function normally. Data that was originally stored on the failed storage node can be transferred to other storage nodes. Thus, the storage system can still execute the data storage process normally, without affecting normal business operations.
[0149] For ease of understanding, the following examples of the data storage method provided in this application are illustrated with Figures 5-8. In these examples, the distributed storage system includes two computing nodes, computing node 1 and computing node 2, which are connected to six storage nodes, n0-n5. The storage system employs a 4+2 EC storage configuration. The six storage nodes comprise six stripes, s0-s5, and each stripe includes six stripe units u0-u5 located on the six storage nodes.
[0150] Figure 5 is a schematic diagram of a first example of the data storage method provided in this application. In the example shown in Figure 5, the storage node performs three-copy caching of the data blocks. Compute node 1 and compute node 2 respectively receive I / O, and the I / O data includes four data blocks b0-b3 with the target stripe being stripe 0. The primary storage node corresponding to stripe 0 (that is, the primary storage node corresponding to the four data blocks) is storage node n0.
[0151] Thus, compute nodes 1 and 2 send data blocks b0-b3 and their LBAs to storage node n0. Then, storage node n0 determines the data ownership nodes for data blocks b0-b3 as storage nodes n2-n5, and the other redundant nodes for data blocks b0-b3 as storage node n1. Data blocks b0-b3 are then cached using a three-replica cache as follows:
[0152] Storage node n0 stores data block b0 in its local cache. Storage node n0 sends a copy of data block b0 to storage node n1 (other redundant nodes) and stores it in storage node n1's local cache. Storage node n0 sends a copy of data block b0 to storage node n2 (the data owner node) and stores it in storage node n2's local cache.
[0153] Storage node n0 stores data block b1 in its local cache. Storage node n0 sends a copy of data block b1 to storage node n1 (other redundant nodes) and stores it in storage node n1's local cache. Storage node n0 sends a copy of data block b1 to storage node n3 (the data owner node) and stores it in storage node n3's local cache.
[0154] Storage node n0 stores data block b2 in its local cache. Storage node n0 sends a copy of data block b2 to storage node n1 (other redundant nodes) and stores it in storage node n1's local cache. Storage node n0 sends a copy of data block b2 to storage node n4 (the data owner node) and stores it in storage node n4's local cache.
[0155] Storage node n0 stores data block b3 in its local cache. Storage node n0 sends a copy of data block b3 to storage node n1 (other redundant nodes) and stores it in storage node n1's local cache. Storage node n0 sends a copy of data block b3 to storage node n5 (the data owner node) and stores it in storage node n5's local cache.
[0156] For a 4+2 EC storage configuration, taking the local cache flush condition of storage node n0 as an example where the number of data blocks stored in the cache is n=4, after storage node n0's local cache stores 4 data blocks b0-b3, storage node n0 can also perform EC calculation on the data blocks b0-b3 in the local cache to obtain 2 parity data blocks, and write the data blocks b0-b3 and the 2 parity data blocks to stripe 0 storage in the following way:
[0157] Storage node n0 writes a parity data block to stripe cell u0 of stripe 0 on storage node n0;
[0158] Storage node n0 sends a parity data block to storage node n1 and writes it into stripe cell u1 of stripe 0 on storage node n1.
[0159] Storage node n0 notifies storage node n2 to write a copy of data block b0 to stripe cell u2 of stripe s0 on storage node n2;
[0160] Storage node n0 notifies storage node n3 to write a copy of data block b1 into stripe cell u3 of stripe s0 on storage node n3;
[0161] Storage node n0 notifies storage node n4 to write a copy of data block b2 into stripe cell u4 of stripe s0 on storage node n4;
[0162] Storage node n0 notifies storage node n5 to write a copy of data block b3 to stripe cell u5 of stripe s0 on storage node n5.
[0163] Figure 6 is a schematic diagram of a second example of the data storage method provided in this application. In the example shown in Figure 6, the storage node performs two-copy caching of the data blocks. Compute node 1 and compute node 2 respectively receive I / O, and the I / O data includes four data blocks b0-b3 with target stripes s0-s3. The primary storage node for data block b0 is n0, the primary storage node for data block b1 is n1, the primary storage node for data block b2 is n4, and the primary storage node for data block b3 is n5.
[0164] The primary storage nodes of stripes s0-s5 are the storage nodes marked with the word "primary" in Figure 6.
[0165] Thus, the compute node sends data block b0 and its LBA to storage node n0, the compute node sends data block b1 and its LBA to storage node n1, the compute node sends data block b2 and its LBA to storage node n4, and the compute node sends data block b3 and its LBA to storage node n5.
[0166] Storage nodes n0-n3 determine the data ownership nodes for data blocks b0-b3 as storage nodes n2, n3, n0, and n1, respectively. Then, data blocks b0-b3 are cached using two replicas each, as follows:
[0167] Storage node n0 stores data block b0 in its local cache. Storage node n0 sends a copy of data block b0 to storage node n2 (the data owner node) and stores it in storage node n2's local cache.
[0168] Storage node n1 stores data block b1 in its local cache. Storage node n1 sends a copy of data block b1 to storage node n3 (the data owner node) and stores it in storage node n3's local cache.
[0169] Storage node n4 stores data block b2 in its local cache. Storage node n4 sends a copy of data block b2 to storage node n0 (the data owner node) and stores it in storage node n0's local cache.
[0170] Storage node n5 stores data block b3 in its local cache. Storage node n5 sends a copy of data block b3 to storage node n1 (the data owner node) and stores it in the local cache of storage node n2.
[0171] Figure 7 is a schematic diagram of a third example of the data storage method provided in this application. In the example shown in Figure 7, the storage node performs three-copy caching of the data blocks. Compute node 1 and compute node 2 respectively receive I / O, and the I / O data includes four data blocks b0-b3 with target stripes s0-s3. The primary storage node for data block b0 is n0, the primary storage node for data block b1 is n1, the primary storage node for data block b2 is n4, and the primary storage node for data block b3 is n5.
[0172] The primary storage nodes of stripes s0-s5 are the storage nodes marked with the word "primary" in Figure 6.
[0173] In this example, based on the instance shown in Figure 6, storage nodes n0, n1, n4, and n5 further determine the other redundant nodes for data blocks b0-b3 as storage nodes n1, n2, n5, and n0, respectively. Then, data blocks b0-b3 are cached using a three-copy method as follows:
[0174] Storage node n0 stores data block b0 in its local cache. Storage node n0 sends a copy of data block b0 to storage node n1 (other redundant nodes) and stores it in storage node n1's local cache. Storage node n0 sends a copy of data block b0 to storage node n2 (the data owner node) and stores it in storage node n2's local cache.
[0175] Storage node n1 stores data block b1 in its local cache. Storage node n1 sends a copy of data block b1 to storage node n2 (other redundant nodes) and stores it in storage node n2's local cache. Storage node n1 sends a copy of data block b1 to storage node n3 (the data owner node) and stores it in storage node n3's local cache.
[0176] Storage node n4 stores data block b2 in its local cache. Storage node n4 sends a copy of data block b2 to storage node n5 (other redundant nodes) and stores it in storage node n5's local cache. Storage node n4 sends a copy of data block b2 to storage node n0 (the data owner node) and stores it in storage node n0's local cache.
[0177] Storage node n5 stores data block b3 in its local cache. Storage node n5 sends a copy of data block b3 to storage node n0 (other redundant nodes) and stores it in storage node n0's local cache. Storage node n50 sends a copy of data block b3 to storage node n1 (the data owner node) and stores it in storage node n1's local cache.
[0178] Figure 8 is a schematic diagram of a fourth example of the data storage method provided in the embodiments of this application. In the example shown in Figure 8, multiple storage nodes include 36 stripes with stripe IDs of 0-35. Since N+M=4+2=6, the 36 stripes include 36 / 6=6 stripe periods. The first stripe period includes stripes 0-5, the second stripe period includes stripes 6-11, the third stripe period includes stripes 12-17, the fourth stripe period includes stripes 18-23, the fifth stripe period includes stripes 24-29, and the sixth stripe period includes stripes 30-35.
[0179] The primary storage nodes for target stripes s0-s35 are the storage nodes marked with the word "primary" in Figure 8.
[0180] In the first implementation, the backup storage nodes for the 6 stripes in the (i+1)th stripe period are obtained by migrating the backup storage nodes for the 6 stripes in the ith stripe period by one position according to their storage node IDs. In this method, as shown in 810 of Figure 8, the backup storage nodes for the target stripes s0-s35 are the storage nodes marked with "backup" in 810.
[0181] In the second implementation, the backup storage nodes for the 6 stripes in the (i+1)th stripe period are obtained by migrating the backup storage nodes for the 6 stripes in the ith stripe period by one position in reverse order of their storage node IDs. In this method, as shown in 820 of Figure 8, the backup storage nodes for the target stripes s0-s35 are the storage nodes marked with "backup" in 820.
[0182] It is understood that, in order to achieve the functions in the above embodiments, the computing device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.
[0183] The data processing method provided according to the embodiments of this application has been described in detail above with reference to Figures 2 to 8. The apparatus provided according to this application will now be described with reference to Figures 9 and 10. These apparatuses can be used to implement the functions of storage nodes and computing nodes in the above method embodiments, and thus can also achieve the beneficial effects of the above method embodiments.
[0184] Figure 9 is a schematic diagram of a storage device provided in an embodiment of this application. The storage device 900 can be implemented as any of the aforementioned storage nodes, such as a first storage node (i.e., the primary storage node of the first data block), a second storage node (i.e., the data ownership node of the first data block), a third storage node (i.e., a verification node), etc. As shown in Figure 9, the storage device 900 includes a transceiver module 910 and a processing module 920.
[0185] In the first implementation, taking any data block as the first data block as an example, the storage device 900 can be implemented as the first storage node.
[0186] The transceiver module 910 is used to receive a first data block and its logical block address (LBA). The processing module 920 is used to store the first data block in the cache of a first storage node. The first storage node is used to calculate the checksum of the first data block and store at least a portion of the checksum. The processing module 920 is also used to determine the disk-writing node of the first data block as a second storage node based on its LBA. The transceiver module 910 is also used to send a copy of the first data block to the second storage node.
[0187] In some embodiments, the multiple storage nodes include multiple stripes, each stripe includes multiple stripe units, and the processing module 920 is further configured to determine the disk-writing node of the first data block based on the LBA of the first data block, the data capacity of each stripe unit in the target stripe of the first data block, and the total data capacity of all stripe units in the target stripe of the first data block used for storing data.
[0188] In the second implementation, taking any data block as the first data block as an example, the storage device 900 can be implemented as the second storage node.
[0189] The transceiver module 910 is used to receive a copy of the first data block sent by the first storage node. The processing module 920 is used to store the received copy of the first data block in the cache of the second storage node.
[0190] In the third implementation, taking any data block as the third data block as an example, the storage device 900 can be implemented as the third storage node.
[0191] The transceiver module 910 is used to receive a copy of the first data block sent by the first storage node. The processing module 920 is used to store the received copy of the first data block in the cache of the third storage node. The transceiver module 910 can also be used to receive at least a portion of the checksum data of the first data block sent by the first storage node, and the processing module 920 can also be used to store at least a portion of the checksum data of the first data block in the memory of the third storage node.
[0192] In some embodiments, in the event of a failure of the first storage node: the transceiver module 910 can also be used to receive a first data block and the logical block address (LBA) of the first data block; the processing module 920 can also be used to store the first data block in the cache of the third storage node; the processing module 920 can also be used to calculate the check data of the first data block and store at least a portion of the check data of the first data block; the processing module 920 can also be used to determine the disk-writing node of the first data block as the second storage node based on the LBA of the first data block; the transceiver module 910 can also be used to send a copy of the first data block to the second storage node.
[0193] For more details regarding the functions implemented by the transceiver module 910 and the processing module 920 included in the storage device 900, which is implemented as a storage node, please refer to Figures 2-8 and their related descriptions.
[0194] Figure 10 is a schematic diagram of a computing device provided in an embodiment of this application. The computing device 1000 can be implemented as any of the aforementioned computing nodes. As shown in Figure 10, the computing device 1000 includes a transceiver module 1010 and a processing module 1020.
[0195] Taking any data block as the first data block as an example: the transceiver module 1010 can be used to receive I / O data sent by other devices to obtain the first data block and its LBA; the processing module 1020 can be used to determine the target stripe of the first data block according to its LBA, and to determine the target stripe unit corresponding to the first data block according to its identifier ID and the number of stripe units in the target stripe, and to designate the storage node where the target stripe unit is located as the first storage node; the transceiver module 1010 can also be used to send the first data block to the first storage node.
[0196] For more details regarding the functions implemented by the transceiver module 1010 and the processing module 1020 included in the computing device 1000, which is implemented as a computing node, please refer to Figures 2-8 and their related descriptions.
[0197] Optionally, the transceiver module 910, the processing module 920, the transceiver module 1010, and the processing module 1020 may each include multiple sub-modules. The multiple sub-modules may be deployed separately to implement some of the functions of the corresponding module, such as implementing one or more steps in the aforementioned data storage method.
[0198] All modules can be implemented in software or hardware. For example, the implementation methods of transceiver module 910 and processing module 920 will be described below. The implementation methods of transceiver module 1010 and processing module 1020 can be similar to those of transceiver module 910 and processing module 920.
[0199] As an example of a software functional unit, the transceiver module 910 or the processing module 920 may include code running on a device instance. The device instance may be at least one of a physical device (e.g., a switch, router, server, etc.) or a virtualized device (e.g., a virtual machine, etc.). Furthermore, the aforementioned device instance may be one or more. For example, the transceiver module 812 or the processing module 814 may include code running on multiple physical / virtualized devices.
[0200] As an example of a hardware functional unit, the transceiver module 910 or the processing module 920 may include at least one hardware device. For example, the transceiver module 910 may include a network interface card, a transceiver, etc., and the processing module 920 may include any one or more processors such as a central processing unit (CPU), a digital signal processor (DSP), a network processor (NP), a neural network processing unit (NPU), a data processing unit (DPU), or a microprocessor (MP). Alternatively, the processing module 920 may be a device or component implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD).
[0201] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the steps or operations of the data storage method provided in Figures 2-8 of this application, such as instructions for performing the steps or operations of the data storage method provided in Figures 2-8 of this application.
[0202] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform the steps or operations of the data storage method provided in Figures 2-8 of this application, for example, instructions for performing the steps or operations of the data storage method provided in Figures 2-8 of this application.
[0203] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.
[0204] The terms “first,” “second,” “third,” and “fourth,” etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to define a specific order.
[0205] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
Claims
1. A data storage method, characterized in that, The method is applied to a distributed storage system comprising multiple storage nodes, wherein the multiple storage nodes include a first storage node and a second storage node, and the method includes: The first storage node receives a first data block and the logical block address (LBA) of the first data block, and stores the first data block in the cache of the first storage node. The first storage node is used to calculate the check data of the first data block and store at least a portion of the check data of the first data block. The first storage node determines the disk write node of the first data block as the second storage node based on the LBA of the first data block; The first storage node sends a copy of the first data block to the second storage node; The second storage node stores a copy of the first data block it receives in its cache.
2. The method according to claim 1, characterized in that, The plurality of storage nodes include a plurality of stripes, each stripe including a plurality of stripe units, the distributed storage system further includes computing nodes, and the method further includes: The computing node determines the target stripe of the first data block based on the LBA of the first data block; The computing node determines the target stripe unit corresponding to the first data block based on the target stripe's identifier ID and the number of stripe units in the target stripe; The computing node uses the storage node where the target stripe unit is located as the first storage node; The computing node sends the first data block to the first storage node.
3. The method according to claim 1 or 2, characterized in that, The plurality of storage nodes further includes a third storage node, the third storage node being used to store at least a portion of the verification data of the first data block, and the method further includes: The first storage node sends a copy of the first data block to the third storage node; The third storage node stores a copy of the received first data block in its cache.
4. The method according to claim 3, characterized in that, The first data block's verification data includes M verification data blocks, and the number of the third storage nodes is M-1, where M is an integer greater than 1.
5. The method according to any one of claims 1-4, characterized in that, The plurality of storage nodes includes a plurality of stripes, and the plurality of storage nodes further includes a fourth storage node and a fifth storage node; The fourth storage node receives the second data block and the LBA of the second data block, and stores the second data block in the cache of the fourth storage node. The fourth storage node is used to calculate the check data of the second data block and store at least a portion of the check data of the second data block. The fourth storage node determines the disk write node of the second data block as the fifth storage node based on the LBA of the second data block; The target stripe of the first data block is different from that of the second data block, and the first storage node is different from that of the fourth storage node.
6. The method according to any one of claims 1-5, characterized in that, The plurality of storage nodes include a plurality of stripes, each stripe including a plurality of stripe units, and the step of determining the disk persistence node of the first data block based on the LBA of the first data block includes: The disk dump node for the first data block is determined based on the LBA of the first data block, the data capacity of each stripe unit in the target stripe of the first data block, and the total data capacity of all stripe units in the target stripe of the first data block used for storing data.
7. A data storage system, characterized in that, The system is a distributed storage system comprising multiple storage nodes, including a first storage node and a second storage node; The first storage node is configured to receive a first data block and the logical block address (LBA) of the first data block, store the first data block in the cache of the first storage node, and calculate the check data of the first data block and store at least a portion of the check data of the first data block. The first storage node is further configured to determine the disk write node of the first data block as the second storage node based on the LBA of the first data block; The first storage node is also used to send a copy of the first data block to the second storage node; The second storage node is used to store a copy of the received first data block in the cache of the second storage node.
8. The system according to claim 7, characterized in that, The plurality of storage nodes include a plurality of stripes, each stripe including a plurality of stripe units, and the distributed storage system further includes computing nodes; The computing node is used to determine the target stripe of the first data block based on the LBA of the first data block. The computing node is also used to determine the target stripe unit corresponding to the first data block based on the identifier ID of the target stripe and the number of stripe units in the target stripe; The computing node is also used to use the storage node where the target stripe unit is located as the first storage node; The computing node is also used to send the first data block to the first storage node.
9. The system according to claim 7 or 8, characterized in that, The plurality of storage nodes further includes a third storage node, which is used to store at least a portion of the verification data of the first data block; The first storage node is also used to send a copy of the first data block to the third storage node; The third storage node is also used to store a copy of the received first data block in the cache of the third storage node.
10. The system according to any one of claims 7-9, characterized in that, The plurality of storage nodes includes a plurality of stripes, and the plurality of storage nodes further includes a fourth storage node and a fifth storage node; The fourth storage node is used to receive the second data block, the LBA of the second data block, store the second data block in the cache of the fourth storage node, and calculate the check data of the second data block and store at least a portion of the check data of the second data block. The fourth storage node is also used to determine the disk write node of the second data block as the fifth storage node based on the LBA of the second data block; The target stripe of the first data block is different from that of the second data block, and the first storage node is different from that of the fourth storage node.
11. The system according to any one of claims 7-10, characterized in that, The plurality of storage nodes include a plurality of stripes, each stripe including a plurality of stripe units, and the first storage node is further configured to: The disk dump node for the first data block is determined based on the LBA of the first data block, the data capacity of each stripe unit in the target stripe of the first data block, and the total data capacity of all stripe units in the target stripe of the first data block used for storing data.
12. The system according to claim 9, characterized in that, The third storage node is also used for: In the event of a failure of the first storage node, the system receives the first data block, the logical block address (LBA) of the first data block, stores the first data block in the cache of the third storage node, and calculates the checksum of the first data block. The disk-writing node of the first data block is determined to be the second storage node based on the LBA of the first data block; Send a copy of the first data block to the second storage node.
13. A computer program product containing instructions, characterized in that, When the instruction is executed by a storage system comprising multiple storage nodes, the storage system performs the method as described in any one of claims 1-6.
14. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a storage system comprising multiple storage nodes, enable the storage system to perform the method as described in any one of claims 1-6.
