Data transmission method, system and apparatus, and non-volatile readable storage medium and program product
By splitting the data set into multiple data storage devices in parallel within the storage system, and utilizing a multi-channel transmission channel, the problem of low data transmission efficiency between controllers is solved, thus achieving efficient data transmission.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- INSPUR SUZHOU INTELLIGENT TECH CO LTD
- Filing Date
- 2025-11-28
- Publication Date
- 2026-06-04
Smart Images
Figure CN2025138676_04062026_PF_FP_ABST
Abstract
Description
Data transmission methods and systems, apparatus, non-volatile readable storage media and program products
[0001] Cross-reference to related applications
[0002] This application claims priority to Chinese Patent Application No. 202411728833.X, filed on November 28, 2024, entitled “Data Transmission Method and System, Apparatus, Storage Medium and Program Product”, the entire contents of which are incorporated herein by reference. Technical Field
[0003] This application relates to the field of storage technology, and in particular to a data transmission method and system, apparatus, non-volatile readable storage medium, and program product. Background Technology
[0004] In centralized storage systems, storage architectures can be implemented using hardware such as servers, controllers, and hard disk enclosures. Multiple controllers can be supported to improve availability and performance. The head unit can connect to servers and disk array devices, providing high-speed data transfer. The head unit can manage disks, abstract disks into storage resource pools, and allocate logical unit numbers for client use.
[0005] In related technologies, after the head unit receives an I / O (Input / Output) request from the server, it processes the I / O request, converting the logical address in the I / O request into a physical address and transmitting the data to the backend disk array. Each data transmission requires adherence to a specific communication protocol, including connection establishment, transmission control, and error detection. However, this process needs to be performed for I / O requests of varying sizes. During the transmission of small I / O requests, the overhead of these protocols is relatively large compared to the amount of data transmitted, resulting in lower data transmission efficiency for the same bandwidth.
[0006] There is currently no effective solution to the problem of low data transfer efficiency between controllers in existing storage systems. Summary of the Invention
[0007] This application provides a data transmission method and system, apparatus, non-volatile readable storage medium, and program product to at least solve the technical problem of low data transmission efficiency between controllers in current storage systems.
[0008] According to one embodiment of this application, a data transmission method is provided, comprising: in a first controller, sending a first data set to a data processing module, wherein the first data set includes a plurality of first data blocks, and the data processing module is configured to process the data blocks; under the condition of satisfying a preset condition, the data processing module splits the first data set into a plurality of second data sets, and sends the plurality of second data sets to corresponding data storages respectively, wherein the number of second data sets is the same as the number of data storages; and sending the second data sets to a second controller through a transmission channel corresponding to the data storage.
[0009] In an exemplary embodiment, when the preset conditions are met, the data processing module splits the first data set into multiple second data sets and sends the multiple second data sets to the corresponding data storage, including: in the data storage, aggregating the multiple first data blocks included in the stored second data sets to obtain aggregated second data blocks, and using the second data blocks to update the second data sets to obtain updated second data sets.
[0010] In one exemplary embodiment, the above-described aggregation of the plurality of first data blocks included in the stored second data set includes one of the following: aggregating the plurality of first data blocks included in the second data set stored in the data storage into a second data block; aggregating the plurality of first data blocks included in the second data set stored in the data storage into a plurality of second data blocks, wherein the data size of the second data block is greater than a preset data size.
[0011] In an exemplary embodiment, the data processing module splits the first data set into multiple second data sets and sends the multiple second data sets to corresponding data storage devices, including: sending the multiple split second data sets to data storage devices connected to the data processing module via the chip of the data processing module, wherein the number of chips of the data processing module is the same as the number of data storage devices.
[0012] In an exemplary embodiment, when the preset conditions are met, the data processing module splits the first data set into multiple second data sets, including one of the following: when the total size of the first data blocks included in the first data set is greater than a preset data threshold, the first data set is split into multiple second data sets; or the first data set composed of the first data blocks received in the first time period is split into multiple second data sets.
[0013] In an exemplary embodiment, the data processing module splits the first data set into multiple second data sets, including: determining the total size of the first data blocks included in the first data set, and determining the average data size based on the total size of the first data blocks and the number of data storage devices; adding the first data blocks included in the first data set sequentially to form the i-th second data set, until the total size of the first data blocks included in the i-th second data set is greater than the average data size, and determining the i-th second data set; after determining the p-1-th second data set, determining the remaining first data blocks as the p-th second data set, where i is a positive integer greater than 0 and less than p, and p is the number of data storage devices.
[0014] In one exemplary embodiment, the above-described method of sending the second data set to the second controller via the transmission channel corresponding to the data storage includes: sending the second data set to the bridging port; and sending the second data set to the second controller via the bridging port and the transmission channel, wherein the transmission channel is connected to the data processing module in the second controller.
[0015] In one exemplary embodiment, the above-described sending of the second data set to the bridge port includes: when the second data set meets the sending conditions, the memory access module sends the second data set in the data memory to the bridge port.
[0016] In an exemplary embodiment, sending the second data set to the bridge port includes mapping the sending address of the second data set to obtain the receiving address corresponding to the second controller.
[0017] In one exemplary embodiment, before sending the first data set to the data processing module, the process includes: acquiring the first data set at the input port.
[0018] According to another embodiment of this application, a data transmission system is provided, comprising: a first controller, wherein the first controller includes a data processing module, a data storage device, a first PCIe switch, and an input port, the first PCIe switch being configured to perform multi-channel data transmission with a second PCIe switch of a second controller; and a second controller, wherein the second controller includes a second PCIe switch, the second PCIe switch being configured to perform multi-channel data transmission with the first PCIe switch.
[0019] In an exemplary embodiment, the first PCIe switch includes multiple partitions. The uplink port of each partition of the first PCIe switch is connected to a data processing module, wherein the uplink port is configured to transmit data with the data processing module. The downlink port of each partition of the first PCIe switch is connected to the downlink port of the corresponding partition of the second PCIe switch, wherein the downlink port is configured to enable data transmission between the partitions of the first PCIe switch and the corresponding partitions of the second PCIe switch.
[0020] In an exemplary embodiment, the uplink port of each partition of the first PCIe switch is connected to the corresponding chip in the data processing module, wherein the number of partitions of the first PCIe switch is the same as the number of chips, and the partitions of the first PCIe switch correspond one-to-one with the chips.
[0021] In one exemplary embodiment, each partition of the second PCIe switch is connected to a corresponding chip in the data processing module of the second controller, wherein the number of partition data of the second PCIe switch is the same as the number of chips of the second controller, and the partitions of the second PCIe switch correspond one-to-one with the chips of the second controller.
[0022] In one exemplary embodiment, the chips of the data processing module are interconnected via an off-chip bus, and each chip is connected to a corresponding data memory, wherein the data memory is configured to store data blocks.
[0023] In one exemplary embodiment, the first PCIe switch includes multiple DMA engines, the number of which is the same as the number of chips, and the DMA engines are configured to retrieve data from their corresponding data memories.
[0024] In one exemplary embodiment, the first PCIe switch and the second PCIe switch are connected via a non-transparent bridge port, which is configured to translate data addresses in the first PCIe switch to addresses in the second PCIe switch or to translate addresses in the second PCIe switch to addresses in the first PCIe switch.
[0025] In one exemplary embodiment, the aforementioned input port is connected to a server and is configured to receive data sent by the server.
[0026] According to another embodiment of this application, a data transmission apparatus is provided, comprising: a first transmitting module, configured in a first controller to transmit a first data set to a data processing module, wherein the first data set includes a plurality of first data blocks, and the data processing module is configured to process the data blocks; a splitting module, configured to, under preset conditions, split the first data set into a plurality of second data sets, and transmit the plurality of second data sets to corresponding data storages respectively, wherein the number of second data sets is the same as the number of data storages; and a transmission module, configured to transmit the second data sets to a second controller through a transmission channel corresponding to the data storages.
[0027] According to yet another embodiment of this application, a non-volatile readable storage medium is also provided, wherein a computer program is stored in the non-volatile readable storage medium, and the computer program is configured to execute the steps in any of the above method embodiments when running.
[0028] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein a computer program is stored in the memory and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.
[0029] According to yet another embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the methods in various embodiments of this application.
[0030] According to this application, in a first controller, a first data set is sent to a data processing module. The first data set includes multiple first data blocks, and the data processing module is configured to process these data blocks. Under preset conditions, the data processing module splits the first data set into multiple second data sets and sends each of the second data sets to a corresponding data storage device. The number of second data sets is the same as the number of data storage devices. The second data sets are then sent to the second controller via the corresponding transmission channel of the data storage device. In the first controller, after acquiring the first data set, it can send it to the data processing module for splitting. The data processing module then sends the first data set to a corresponding data storage device. The second data sets in the data storage device can be sent to the second controller via the corresponding transmission channel. This enables parallel transmission of the first data set, improving data transmission efficiency and solving the technical problem of low data transmission efficiency between controllers in current storage systems. Attached Figure Description
[0031] Figure 1 is a hardware structure block diagram of an optional data transmission method according to an embodiment of this application;
[0032] Figure 2 is a flowchart of an optional data transmission method according to an embodiment of this application;
[0033] Figure 3 is a schematic diagram of an optional data transmission method according to an embodiment of this application;
[0034] Figure 4 is a schematic diagram of another optional data transmission method according to an embodiment of this application;
[0035] Figure 5 is a schematic diagram of another optional data transmission method according to an embodiment of this application;
[0036] Figure 6 is a schematic diagram of another optional data transmission method according to an embodiment of this application;
[0037] Figure 7 is an architecture diagram of an optional data transmission system according to an embodiment of this application;
[0038] Figure 8 is a structural block diagram of an optional data transmission device according to an embodiment of this application. Detailed Implementation
[0039] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.
[0040] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0041] Figure 1 is a hardware structure block diagram of an optional data transmission method according to an embodiment of this application. As shown in Figure 1, a centralized area network (SAN) is a dedicated storage network that can connect servers and storage devices via high-speed fiber optic channels. As shown in Figure 1, the controller is the core component of the storage system, including two controllers (controller one 114 and controller two 122). The controllers can be connected to servers one 102, server two 104, etc., via a fiber optic switch 106. The number of servers in Figure 1 is only an example, and the number of servers is not limited. Different controllers can be connected through a communication bus 124 to achieve data communication between controllers. The controllers can be connected to a disk array 126 to implement read / write commands from the servers.
[0042] The fiber optic switch 106 can be a high-speed Fibre Channel (such as FC or RoCE) to connect the controller and the server. The disk array 126 can be a physical device for storing data, and multiple disks can be connected in series to achieve data redundancy and fault tolerance. The controller can implement various functions in the centralized SAN storage system, such as disk management, disk abstraction into storage resource pools, and allocation of logical unit numbers for client use.
[0043] Optionally, controller 114 may include front-end host interface 108, cache 110, and back-end bus 112, while controller 2 122 may include front-end host interface 2 116, cache 2 118, and back-end bus 2 120, etc. Data transmission between different controllers is achieved through communication bus 124. The front-end host interface is the part of the controller that communicates with the server. It can handle I / O requests from the server and translate these requests into operations on the disk array. The front-end interface can be of different interface types (such as PCIe (Peripheral Component Interconnect Express)), depending on the needs of the connected server and disk array, allowing data to flow from the server to the disk array or from the disk array to the server.
[0044] A cache is a fast storage resource used to temporarily store data that is about to be written to or read from a disk array (write cache). Caching can significantly improve data access speed because it is much faster than accessing a disk array. Using a cache reduces the number of accesses to the storage medium, thereby improving overall performance.
[0045] The back-end bus, or back-end interface, is a channel within the controller configured to connect the cache and the disk array, enabling data transfer between the cache and the disk array.
[0046] Optionally, in a centralized storage system, the two controllers can be designed as backups for each other to improve system availability and fault tolerance. The two controllers can operate in the following modes: Active-Passive mode: In this mode, typically only one controller (the active controller) handles all I / O operations and storage management tasks, while the other controller (the standby controller) is in standby mode, waiting to take over if the active controller fails. This mode ensures that the standby controller can quickly take over in the event of an active controller failure, achieving seamless switching and minimizing service interruption. Active-Active mode: In this mode, both controllers participate in storage operations and can handle I / O requests simultaneously, thereby improving the overall performance of the storage system. Active-Active configurations can provide load balancing and higher throughput because the two controllers can work in parallel, sharing the I / O load. Load Sharing mode: The two controllers can share different storage tasks or manage different storage resource pools, thus optimizing resource utilization and improving efficiency.
[0047] Optionally, regardless of the operating mode of the two controllers, when one controller fails, the other controller should be able to automatically take over the tasks of the failed controller to ensure the continuous operation of the storage system, i.e., to achieve failover. Therefore, it is necessary to maintain the consistency of the caches of the two controllers to ensure the accuracy and consistency of the data; that is, the two controllers can synchronize data, and the active controller can synchronize its state to the standby controller.
[0048] In an optional implementation, when data is transmitted between the two controllers, the master controller can store the data received from the server in a cache and then synchronously send the cached data to the other controller to achieve data synchronization between the controllers. During the transmission of data from the master controller to the other controller, the data can be split and sent to the other controller through multiple channels to achieve efficient data transmission.
[0049] According to this application, in a first controller, a first data set is sent to a data processing module. The first data set includes multiple first data blocks, and the data processing module is configured to process these data blocks. Under preset conditions, the data processing module splits the first data set into multiple second data sets and sends each of the second data sets to a corresponding data storage device. The number of second data sets is the same as the number of data storage devices. The second data sets are then sent to the second controller via the corresponding transmission channel of the data storage device. In the first controller, after acquiring the first data set, it can send it to the data processing module for splitting. The data processing module then sends the first data set to a corresponding data storage device. The second data sets in the data storage device can be sent to the second controller via the corresponding transmission channel. This enables parallel transmission of the first data set, improving data transmission efficiency and solving the technical problem of low data transmission efficiency between controllers in current storage systems.
[0050] This embodiment provides a data transmission method. Figure 2 is a flowchart of an optional data transmission method according to an embodiment of this application. As shown in Figure 2, the data transmission method includes:
[0051] Step S202: In the first controller, the first data set is sent to the data processing module, wherein the first data set includes multiple first data blocks, and the data processing module is configured to process the data blocks;
[0052] It should be noted that the first controller can be a controller in a different working state, such as the main controller or either of the two controllers that are performing storage operations.
[0053] It should be noted that the first data set can be the collection of data sent by the server to the controller. This first data can be read or write requests sent by the server, which may include the location of the data to be accessed (such as logical unit number, file path, or block address) and the data content to be transferred. The first data can also be control commands sent by the server to the controller to instruct the controller on specific operations, such as creating or deleting a file system, performing a snapshot, or resizing a volume. The first data can also be metadata related to file system operations, application data, database data, management commands, etc. There are no restrictions on the format of the first data.
[0054] In an optional implementation, the server can send multiple first data sets to the controller. The first data set may include multiple first data blocks. The first data block may be a unit set for storing data in the file system. The size of the first data block is not limited. It may be a small I / O request such as 512 bytes or 4KB, or a large I / O request such as 8KB, 16KB, or 1MB.
[0055] In an optional implementation, the controller may include a data processing module that can process the first data set, such as managing the transfer of the first data set between the cache and the disk array.
[0056] Step S204: Under the condition of satisfying the preset conditions, the data processing module splits the first data set into multiple second data sets and sends the multiple second data sets to the corresponding data storages respectively, wherein the number of second data sets is the same as the number of data storages;
[0057] It should be noted that the data processing module can split the first data set to obtain multiple second data sets. Splitting the first data set involves regrouping the first data blocks within the first data set to obtain multiple new sets, without altering the first data blocks themselves.
[0058] It should be noted that the data processing module can be the controller's CPU, which can process instructions sent by the server and process the sent data. It can connect to the controller's cache and send the received first data set to the cache. The data processing module can split the first data set according to pre-defined rules, such as the number of second data sets obtained from the split.
[0059] In an optional implementation, after splitting the first data set into multiple second data sets, the multiple second data sets can be sent to a data storage device, which serves as a cache for the controller. The data storage device can be configured to temporarily store frequently accessed data to reduce the number of accesses to main memory, thereby improving data access speed and overall system performance. For a first data block of write type, the first data block can be first written to the data storage device and then asynchronously written to main memory (such as a disk array).
[0060] In an optional implementation, the controller's data processing module can be connected to multiple data storage devices, and the number of second data sets can be the same as the number of data storage devices, that is, one second data set can be stored in each data storage device.
[0061] Step S206: The second data set is sent to the second controller through the transmission channel corresponding to the data storage.
[0062] It should be noted that the data storage device can physically have a corresponding transmission channel. The transmission channel can be a data transmission channel constructed using the PCIe bus standard. Two controllers can be treated as two PCIe devices. Each controller can include a PCIe switch. The switch can forward data from one port of a controller to another port, realizing data transmission between the two controllers.
[0063] It's important to note that a transmission channel can be a data transfer channel created by interconnecting two controllers using high-speed data transfer technologies such as PCIe bus, Infiniband, or RoCE network. Different transmission technologies can be used to construct transmission channels for different application scenarios. PCIe bus can be used for connections between hardware components, supporting point-to-point connections; Infiniband can connect servers and storage clusters, ensuring reliable data transfer between hardware components, but its cost is higher; RoCE network is a remote direct memory access technology based on Ethernet, and the RoCE protocol has two versions. Compared to PCIe technology, Infiniband has the advantage of lower cost, and compared to RoCE network, it has the advantages of no protocol conversion and low latency. Therefore, PCIe bus can be considered the preferred solution for constructing a data synchronization transmission channel between controllers.
[0064] In an optional implementation, PCIe interfaces can be configured on both controllers. These interfaces can be upstream ports (USP) and downstream ports (DSPs). Each controller can have a PCIe switch, which connects the first and second controllers. The connection method involves physically connecting the PCIe ports of the two controllers and configuring the PCIe devices using software for management and identification. How to utilize the transmission channel for data transmission will be described in detail below and will not be repeated here.
[0065] It should be noted that data storage can be different types of memory, such as DDR (Double Data Rate), LPDDR (Low-Power Double Data Rate), SRAM (Static Random Access Memory), NAND Flash, ROM, etc.
[0066] In an optional implementation, the second data set can be sent to the second controller through a transmission channel corresponding to the data storage device. Each data storage device can be configured with a separate transmission channel to send the second data set to the second controller. The transmission channel corresponds to the data storage device, and each data storage device can have an independent transmission channel. How to establish independent transmission channels will be explained in detail later and will not be repeated here.
[0067] According to this application, in a first controller, a first data set is sent to a data processing module. The first data set includes multiple first data blocks, and the data processing module is configured to process these data blocks. Under preset conditions, the data processing module splits the first data set into multiple second data sets and sends each of the second data sets to a corresponding data storage device. The number of second data sets is the same as the number of data storage devices. The second data sets are then sent to the second controller via the corresponding transmission channel of the data storage device. In the first controller, after acquiring the first data set, it can send it to the data processing module for splitting. The data processing module then sends the first data set to a corresponding data storage device. The second data sets in the data storage device can be sent to the second controller via the corresponding transmission channel. This enables parallel transmission of the first data set, improving data transmission efficiency and solving the technical problem of low data transmission efficiency between controllers in current storage systems.
[0068] In an optional implementation, under the condition that the preset conditions are met, the data processing module splits the first data set into multiple second data sets and sends the multiple second data sets to the corresponding data storage, including: in the data storage, aggregating the multiple first data blocks included in the stored second data sets to obtain aggregated second data blocks, and using the second data blocks to update the second data sets to obtain updated second data sets.
[0069] In an optional implementation, the first data block can be an I / O request sent by the server, which can be the amount of data the server reads or writes. The size of the I / O request is arbitrary, depending on the application's requirements and the operating system's design. When writing data, the location of the data block on the storage device can be determined, and the logical block address can be translated into a physical block address, which can be done by the file system's block manager or the storage device's firmware.
[0070] It's important to note that I / O requests are instructions issued by the operating system to read or write data to storage devices. In block storage systems, I / O requests are typically based on data blocks.
[0071] Each data transmission must adhere to a specific communication protocol, including connection establishment, transmission control, error detection, and correction. For small I / O requests, these protocol overheads account for a larger proportion of the data volume being transmitted, as the protocol overhead is essentially fixed regardless of the packet size. Furthermore, each packet needs to encapsulate header information, including source address, destination address, and checksum. For small I / O requests, this header information constitutes a significant portion of the data itself, resulting in a lower payload ratio for each packet. In network transmission, each packet requires a certain amount of time to process and transmit, including queuing and processing time within the network device. For small I / O requests, due to the small data volume, less data can be transmitted per unit time, thus requiring more packets to complete the same amount of data transmission. Therefore, small I / O requests require more interactions than large I / O requests, impacting overall transmission efficiency.
[0072] Through the above-described embodiments of this application, a second data set in a data storage device can be aggregated to obtain an aggregated second data block. Aggregation can transform random small I / O into sequential large I / O, thereby improving transmission performance in scenarios with high throughput and low latency requirements.
[0073] In an optional implementation, the above-mentioned aggregation of the multiple first data blocks included in the stored second data set includes one of the following: aggregating the multiple first data blocks included in the second data set stored in the data storage into a second data block; aggregating the multiple first data blocks included in the second data set stored in the data storage into multiple second data blocks, wherein the data size of the second data block is greater than a preset data size.
[0074] In an optional implementation, the data processing module can be connected to a data storage device, the type of which is as described above; in one example, it could be DDR. After storing the second data set in the corresponding data storage device, the second data set can be aggregated in each data storage device to obtain large I / O requests. The size of the first data block being merged is irrelevant; regardless of the size of the first data block, it is necessary to aggregate multiple first data blocks included in the second data set in one data storage device. That is, the size of the first data block does not need to be identified during the aggregation process.
[0075] In an optional implementation, the first data blocks in the data storage can be aggregated. All the first data blocks in the second data set of a data storage can be directly aggregated into a second data block without identifying the data blocks. This allows for the rapid acquisition of large I / O requests. Since the data contained in each second data set is of similar size, the size of the second data blocks obtained in different data storages is similar. Parallel transmission of the second data blocks can accelerate data transmission efficiency.
[0076] In an optional implementation, the second data set in the data storage can be aggregated according to a preset data size, and different second data sets can be aggregated into multiple data blocks in the data storage, with the data blocks having the same size, so as to improve the transmission efficiency.
[0077] Through the above-described embodiments of this application, the first data block can be directly aggregated in the data storage, and the second data set stored in each data storage can be directly aggregated. The resulting second data block sizes are similar, recalculation is avoided during the aggregation process, and transmission efficiency is improved through large I / O requests during transmission.
[0078] In an optional implementation, the data processing module splits the first data set into multiple second data sets and sends the multiple second data sets to corresponding data storage devices, including: sending the multiple split second data sets to data storage devices connected to the data processing module via the chip of the data processing module, wherein the number of chips of the data processing module is the same as the number of data storage devices.
[0079] It should be noted that the data processing module can be a CPU, and the data processing module can be composed of a die. A die is a small square cut from a wafer, which is the core part of the CPU and contains the circuit logic set up to calculate, receive / store commands, and process data.
[0080] It should be noted that a chip can be a large-area chip, or it can be multiple chips with specific functions obtained by splitting them up. Multiple chips can be packaged into a data processing module. Chips can be interconnected via an off-chip bus, and each chip can be connected to a data memory. Data stored in different data memories can be shared or stored separately.
[0081] Figure 3 is a schematic diagram of an optional data transmission method according to an embodiment of this application. As shown in Figure 3, the controller may include a data processing module and an I / O card 302 and multiple data storage devices connected thereto. The data processing module may include multiple chips (such as chip one 304, chip two 306, chip three 308, and chip four 310). After receiving a first data set from the server via the I / O card 302, the first data set is split into second data sets, which are then sent to the multiple chips. These second data sets are then sent to the multiple connected data storage devices (storage one 312, storage two 314, storage three 316, and storage four 318). The stored second data sets in the multiple data storage devices can be aggregated to obtain a second data block, which is then sent to the second controller through the corresponding transmission channel.
[0082] Through the above-described embodiments of this application, the data processing module can have multiple chips, and each chip has a directly connected data memory. Multiple second data sets obtained from the splitting process can be sent to the multiple data memories respectively, where data aggregation is performed to obtain a second data block. The multiple chips and multiple data memories can achieve parallel transmission during subsequent transmission, thereby improving transmission efficiency.
[0083] In an optional implementation, under the condition that the preset conditions are met, the data processing module splits the first data set into multiple second data sets, including one of the following:
[0084] (1) If the total size of the first data blocks included in the first data set is greater than the preset data threshold, the first data set is split into multiple second data sets;
[0085] (2) The first data set consisting of the first data blocks received in the first time period is split into multiple second data sets.
[0086] It should be noted that after receiving data from the server, it is necessary to determine whether the data needs to be split. As mentioned earlier, splitting the data does not mean splitting a single data block, but rather splitting a data set into multiple data sets. Therefore, the first data set to be split needs to be determined before splitting.
[0087] In optional implementations, the first data set can be determined by data volume or time. For example, if the first controller receives a first data block from the server, it can calculate the size of the received multiple first data blocks. If the total size of the multiple first data blocks exceeds a preset data threshold, the multiple first data blocks can be defined as the first data set. The preset data threshold can be a threshold preset according to actual needs. For example, to ensure timely transmission, the preset data threshold can be set relatively small to facilitate the determination of the first data set. To improve the utilization of hardware resources, the preset data threshold can be set relatively large to increase the size of the subsequently received second data blocks, thereby increasing the amount of data transmitted each time. In one example, the first data set can also be determined by time, defining the first data blocks received within a certain time period as the first data set.
[0088] Through the above-described embodiments of this application, the method for determining the first data set can be selected according to the requirements. The first data set can be determined according to the time or the total size of the data. The preset data threshold and the value of the first time can be freely set according to actual needs to obtain a first data set that meets the project requirements, so as to facilitate subsequent splitting and transmission and improve the reliability of data transmission.
[0089] In an optional implementation, the data processing module splits the first data set into multiple second data sets, including: determining the total size of the first data blocks included in the first data set, and determining the average data size based on the total size of the first data blocks and the number of data storage devices; adding the first data blocks included in the first data set in sequence to form the i-th second data set, until the total size of the first data blocks included in the i-th second data set is greater than the average data size, and determining the i-th second data set; after determining the p-1-th second data set, determining the remaining first data blocks as the p-th second data set, where i is a positive integer greater than 0 and less than p, and p is the number of data storage devices.
[0090] It should be noted that, when the first data set is determined by a preset data threshold, the mean of the preset data threshold can be calculated in advance and used as the basis for splitting to obtain the second data set.
[0091] It should be noted that although the first data set is split, the size of the second data set may be different. When the first data set is split according to the average data size, the size of the first data block does not change. Therefore, for the first p-1 second data sets, the size is greater than or equal to the average data size, while for the p-th second data set, the size is less than the average data size.
[0092] Figure 4 is a schematic diagram of another optional data transmission method according to an embodiment of this application. As shown in Figure 4, in the first controller, after receiving the first data set from the server, it can be divided equally and sent to the connected data memory through the chip in the data processing module to realize the storage of the second data set. The splitting process of the first data set can be performed in the data processing module, which may also have a dedicated logic circuit for splitting, or the data can be split using any one of the multiple chips connected to the memory.
[0093] Through the above-described embodiments of this application, different first data sets can be split according to their mean to obtain approximately the same second data set, so that the data sent by each subsequent transmission channel is similar in size, thereby improving transmission efficiency.
[0094] In an optional implementation, the above-mentioned sending the second data set to the second controller through the transmission channel corresponding to the data storage includes: sending the second data set to the bridging port; and sending the second data set to the second controller through the bridging port and the transmission channel, wherein the transmission channel is connected to the data processing module in the second controller.
[0095] In an optional implementation, a physical PCIe switch can be divided into multiple virtual partitions. Each partition's uplink port can connect to a single chip, and its downlink port can connect to the data processing module of a second controller, forming a transmission channel corresponding to each chip and data memory. Each partition's uplink port can connect to the chip, meaning each partition can communicate directly with the data processing module without the need for intermediate devices. This allows each partition to have an independent data transmission path, reducing data transmission latency and congestion. The independence between partitions means they can operate independently without interference. This design improves system flexibility and scalability while also enhancing fault tolerance. If one partition fails, the others can still function normally.
[0096] Figure 5 is a schematic diagram of another optional data transmission method according to an embodiment of this application; as shown in Figure 5, each chip can be connected to a transmission channel (such as transmission channel three 502, transmission channel one 504, transmission channel two 506, and transmission channel four 508). The second data block in the data memory can be sent to the second controller through the corresponding transmission channel, thereby realizing parallel communication.
[0097] Through the above-described embodiments of this application, the aggregated second data block can be sent to the second controller in parallel through the corresponding transmission channel, which improves the efficiency of data transmission and the bandwidth of transmission, thereby improving the efficiency of the storage system.
[0098] In an optional implementation, sending the second data set to the bridge port includes: when the second data set meets the sending conditions, the memory access module sends the second data set in the data memory to the bridge port.
[0099] It's important to note that a DMA (Direct Memory Access) engine can exist within a PCIe switch. This engine allows direct data transfer with the data storage without intervention from the data processing module. The DMA engine can be configured with specific channels to handle interrupts related to transfer completion, partial transfer, and transfer errors. After the transfer channel between controllers is set up, the data processing module can configure the DMA controller's source address (data source), destination address (data destination), and the size of the data block to be transferred. During storage system operation, after splitting the first data set into a second data set, the DMA controller can take over control and initiate the data transfer process. During this process, the data processing module does not need to perform data movement operations. The DMA controller continuously reads data from the corresponding data storage and writes it to the corresponding address on the second controller until the second data set is completely transferred. Upon completion of the transfer, the DMA controller will notify the data processing module via an interrupt signal, allowing the data processing module to handle other tasks or perform further processing on the transferred data.
[0100] Figure 6 is a schematic diagram of another optional data transmission method according to an embodiment of this application. As shown in Figure 6, the existence of data in the data memory can be detected by a DMA engine. For example, a data pointer can be set to identify the data. When the pointer is not null, the second data set can be actively moved. As shown by the dashed lines in Figure 6, each data memory can send the second data set to the second controller through a corresponding transmission channel.
[0101] Through the above-described embodiments of this application, the second data set in the corresponding data storage can be directly sent to the second controller through the transmission channel via the DMA engine without the intervention of the data processing module. This allows for rapid data movement, saves the resources of the data processing module for other operations, and improves the overall performance of the storage system.
[0102] In an optional implementation, sending the second data set to the bridge port includes mapping the sending address of the second data set to obtain the receiving address corresponding to the second controller.
[0103] It's important to note that the two controllers are two systems with independent address domains; therefore, one side cannot directly observe the other's complete memory or I / O space. Thus, a port capable of translating between the two independent address domains is needed—a non-transparent bridge (NTB). Data communication between the two controllers can be achieved through the NTB's special registers and address translation functions.
[0104] In an optional implementation, after the storage system is determined, two sets of base address registers can be placed for the two controllers, one for the first controller and the other for the second controller. The base address registers can be configured to define address translation windows to map the translated addresses to data memory or I / O space, thereby enabling data transfer between the first and second controllers.
[0105] Through the above-described embodiments of this application, address translation can be achieved via a non-transparent bridge during data transmission between two controllers, avoiding differences in address domains and enabling normal communication between the two controllers. This improves the accuracy of data transmission.
[0106] In an optional implementation, before sending the first data set to the data processing module, the process includes: acquiring the first data set at the input port.
[0107] This embodiment provides a data transmission system. Figure 7 is an architecture diagram of an optional data transmission system according to an embodiment of this application. As shown in Figure 7, the data transmission system includes: a first controller 708, wherein the first controller 708 includes a data processing module 704, a memory 706, a first PCIe switch 710, and an input port 702, and the first PCIe switch 710 is configured to perform multi-channel data transmission with a second PCIe switch 714 of the second controller; and a second controller 712, wherein the second controller 712 includes a second PCIe switch 714, and the second PCIe switch 714 is configured to perform multi-channel data transmission with the first PCIe switch 710.
[0108] In an optional implementation, the first PCIe switch 710 includes multiple partitions. The uplink port of each partition of the first PCIe switch 710 is connected to the data processing module 704, wherein the uplink port is configured to perform data transmission with the data processing module 704. The downlink port of each partition of the first PCIe switch 710 is connected to the downlink port of the corresponding partition of the second PCIe switch 714, wherein the downlink port is configured to realize data transmission between the partitions of the first PCIe switch 710 and the corresponding partitions of the second PCIe switch 714.
[0109] In an optional implementation, the uplink port of each partition of the first PCIe switch 710 is connected to the corresponding chip in the data processing module 704, wherein the number of partitions of the first PCIe switch 710 is the same as the number of chips, and the partitions of the first PCIe switch 710 correspond one-to-one with the chips.
[0110] In an optional implementation, each partition of the second PCIe switch 714 is connected to the corresponding chip in the data processing module of the second controller 712, wherein the number of partition data of the second PCIe switch 714 is the same as the number of chips of the second controller 712, and the partitions of the second PCIe switch 714 correspond one-to-one with the chips of the second controller 712.
[0111] In an optional implementation, the chips of the data processing module 704 are interconnected via an off-chip bus, and the chips are respectively connected to the corresponding memory 706, wherein the memory 706 is configured to store data blocks.
[0112] In an optional implementation, the first PCIe switch 710 includes multiple DMA engines, the number of which is the same as the number of chips, and the DMA engines are configured to retrieve data from the corresponding memory 706.
[0113] In an optional implementation, the first PCIe switch 710 and the second PCIe switch 714 are connected via a non-transparent bridge port, which is configured to translate the data address in the first PCIe switch 710 into the address of the second PCIe switch 714 or translate the address of the second PCIe switch 714 into the address of the first PCIe switch 710.
[0114] In an optional implementation, the aforementioned input port 702 is connected to a server and configured to receive data sent by the server.
[0115] Optionally, the first controller 708 and the second controller 712 described above can implement most of the advanced functions in the centralized storage system, such as disk management, abstracting disks into storage resource pools, and allocating Logical Unit Numbers (LUNs) for client use. The first controller 708 and the second controller 712 serve as primary and backup to each other, preventing hardware failures from causing the entire storage system to become unavailable. The controllers include front-end ports and back-end ports. The front-end ports connect to the outside world to provide storage services, while the back-end ports connect to more storage devices, forming a larger storage resource pool and expanding the capacity of the storage system.
[0116] Optionally, the data processing module 704 can be a CPU (Central Processing Unit), which is the core of a computer's processing and control. A CPU consists of an arithmetic logic unit (ALU), a control unit, registers, and data, control, and status buses that connect them. Its main functions are interpreting computer instructions and processing data in computer software. A CPU can be composed of multiple CPU dies, each containing multiple CPU cores. CPU dies are interconnected via an off-chip bus.
[0117] Optionally, input port 702 can be an I / O card, also known as an input / output card or expansion card, which can be configured as a hardware device to expand the server's input / output interfaces. It primarily acts as a bridge connecting the server motherboard and external devices, responsible for transmitting data and control signals. I / O cards are divided into front-end I / O cards and back-end I / O cards; front-end I / O cards, such as FC cards and RoCE cards, are configured to connect to the server; back-end I / O cards, such as RoCE cards and SAS (Serial Attached SCSI, SCSI (Small Computer System Interface)) cards, are configured to connect to hard drive expansion enclosures.
[0118] Optionally, the memory 706 can be DDR SDRAM (Double Data Rate Synchronous Dynamic Random Access Memory). The data memory 706 can be triggered once on the rising edge and once on the falling edge of each clock cycle, thus enabling it to work twice in one clock cycle and complete twice the workload of ordinary memory in the same amount of time.
[0119] Optionally, the first PCIe switch 710 and the second PCIe switch 714 can internally consist of multiple PCIe ports and a switch core. When a PCIe device is connected to the PCIe switch, the switch connects the device's PCIe port to the switch core. When multiple devices are connected to the switch, the switch forwards data from one port to another as needed, enabling data transmission between multiple devices. In this way, the PCIe switch can connect multiple PCIe lanes to a single high-speed switch chip, thereby enabling more PCIe lanes.
[0120] Optionally, the first PCIe switch 710 and the second PCIe switch 714 may have DMA engines, which allow for rapid data movement without CPU intervention, saving CPU resources for other operations.
[0121] Alternatively, a non-transparent bridge (NTB) can be used on the transmission channel. Since the two controllers belong to different address spaces, communication between them can be achieved by translating the address mapping of the corresponding data packets inside the PCIe Switch, which is located in two different partitions.
[0122] Through the above-described embodiments of this application, the first data set transmitted from the server can be evenly distributed to the corresponding data memory of the chip in the data processing module, and small I / O requests can be aggregated in the data memory to obtain large I / O requests, thereby improving the transmission rate. During the transmission of the second data block between the first controller and the second controller, each chip has an independent PCIe channel, ensuring parallel transmission and improving concurrent communication capabilities. During the transmission of the second data block to the second controller, the PCIe switch within the first controller has an independent DMA engine for data transfer, eliminating the need for the data processing module and saving its resources. This improves data transmission efficiency, thus solving the technical problem of low data transmission efficiency between controllers in current storage systems.
[0123] In this application, front-end I / O from the server is evenly distributed across each DIE of the CPU, and small I / O and system messages are aggregated into large I / O to improve transmission efficiency. Without increasing hardware costs, the wide port (x16) of the PCIe switch is split into narrow ports (x8), using the PCIe protocol to reduce cross-DIE transmission latency; mirrored PCIe channels are allocated to each DIE to increase concurrent transmission bandwidth. This increased concurrent transmission bandwidth solves the single-channel performance bottleneck problem and enables higher bandwidth.
[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0125] This embodiment also provides a data transmission device configured to implement the above embodiments and optional implementations; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0126] Figure 8 is a structural block diagram of an optional data transmission device according to an embodiment of this application. As shown in Figure 8, the device includes:
[0127] The first sending module 82 is configured in the first controller to send a first data set to the data processing module, wherein the first data set includes multiple first data blocks, and the data processing module is configured to process the data blocks;
[0128] The splitting module 84 is configured to, under preset conditions, split the first data set into multiple second data sets by the data processing module, and send the multiple second data sets to their respective data storage devices, wherein the number of second data sets is the same as the number of data storage devices.
[0129] The transmission module 86 is configured to send the second data set to the second controller through the transmission channel corresponding to the data storage.
[0130] According to this application, in a first controller, a first data set is sent to a data processing module. The first data set includes multiple first data blocks, and the data processing module is configured to process these data blocks. Under preset conditions, the data processing module splits the first data set into multiple second data sets and sends each of the second data sets to a corresponding data storage device. The number of second data sets is the same as the number of data storage devices. The second data sets are then sent to the second controller via the corresponding transmission channel of the data storage device. In the first controller, after acquiring the first data set, it can send it to the data processing module for splitting. The data processing module then sends the first data set to a corresponding data storage device. The second data sets in the data storage device can be sent to the second controller via the corresponding transmission channel. This enables parallel transmission of the first data set, improving data transmission efficiency and solving the technical problem of low data transmission efficiency between controllers in current storage systems.
[0131] Optionally, the splitting module 84 is further configured to: aggregate multiple first data blocks included in the stored second data set in the data storage to obtain an aggregated second data block, and use the second data block to update the second data set to obtain an updated second data set.
[0132] Optionally, the above-mentioned aggregation of the multiple first data blocks included in the stored second data set includes one of the following: aggregating the multiple first data blocks included in the second data set stored in the data storage into a second data block; aggregating the multiple first data blocks included in the second data set stored in the data storage into multiple second data blocks, wherein the data size of the second data block is greater than a preset data size.
[0133] Optionally, the splitting module 84 is further configured to: send the multiple second data sets obtained from the splitting to a data memory connected to the chip via the chip of the data processing module, wherein the number of chips in the data processing module is the same as the number of data memories.
[0134] Optionally, the above-mentioned splitting module 84 is further configured to: split the first data set into multiple second data sets when the total size of the first data blocks included in the first data set is greater than a preset data threshold; and split the first data set composed of the first data blocks received in the first time period into multiple second data sets.
[0135] Optionally, the splitting module 84 is further configured to: determine the total size of the first data blocks included in the first data set, and determine the average data size based on the total size of the first data blocks and the number of data storage devices; add the first data blocks included in the first data set in sequence to form the i-th second data set, until the total size of the first data blocks included in the i-th second data set is greater than the average data size, and determine the i-th second data set; after determining the p-1-th second data set, determine the remaining first data blocks as the p-th second data set, where i is a positive integer greater than 0 and less than p, and p is the number of data storage devices.
[0136] Optionally, sending the second data set to the bridge port includes: when the second data set meets the sending conditions, the memory access module sends the second data set in the data memory to the bridge port.
[0137] Optionally, sending the second data set to the bridge port includes mapping the sending address of the second data set to obtain the receiving address corresponding to the second controller.
[0138] Optionally, the first sending module 82 described above is further configured to: acquire a first data set at the input port.
[0139] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.
[0140] Embodiments of this application also provide a computer-readable storage medium (i.e., a non-volatile readable storage medium) storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.
[0141] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.
[0142] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.
[0143] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.
[0144] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium storing the computer program product, wherein the computer program, when executed by a processor, implements the steps of the methods described in various embodiments of this application.
[0145] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.
[0146] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.
[0147] The above description is merely an optional embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.
Claims
1. A data transmission method, characterized in that, The method includes: In the first controller, a first data set is sent to the data processing module, wherein the first data set includes a plurality of first data blocks, and the data processing module is configured to process the data blocks; Under preset conditions, the data processing module splits the first data set into multiple second data sets and sends the multiple second data sets to their respective data storage devices, wherein the number of second data sets is the same as the number of data storage devices. The second data set is sent to the second controller through the transmission channel corresponding to the data storage.
2. The method according to claim 1, characterized in that, Under the condition that preset conditions are met, the data processing module splits the first data set into multiple second data sets, and sends the multiple second data sets to their respective data storage locations, including: In the data storage, the multiple first data blocks included in the stored second data set are aggregated to obtain an aggregated second data block, and the second data set is updated using the second data block to obtain an updated second data set.
3. The method according to claim 2, characterized in that, The aggregation of the multiple first data blocks included in the stored second data set includes one of the following: The second data set stored in the data storage device is aggregated into a second data block, comprising multiple first data blocks. The second data set stored in the data storage device includes multiple first data blocks, which are aggregated into multiple second data blocks, wherein the data size of the second data block is greater than a preset data size.
4. The method according to claim 2, characterized in that, The data processing module splits the first data set into multiple second data sets, and sends the multiple second data sets to their respective data storage locations, including: The multiple second data sets obtained from the splitting are sent to a data memory connected to the chip via the chip of the data processing module, wherein the number of chips in the data processing module is the same as the number of data memories.
5. The method according to claim 4, characterized in that, Under the condition that preset conditions are met, the data processing module splits the first data set into multiple second data sets, including one of the following: If the total size of the first data blocks included in the first data set is greater than a preset data threshold, the first data set is split to obtain the plurality of second data sets; The first data set, composed of the first data blocks received in the first time period, is split into the plurality of second data sets.
6. The method according to claim 4, characterized in that, The data processing module splits the first data set into multiple second data sets, including: Determine the total size of the first data blocks included in the first data set, and determine the average data size based on the total size of the first data blocks and the number of data storage devices; The first data blocks included in the first data set are added sequentially to form the i-th second data set until the sum of the sizes of the first data blocks included in the i-th second data set is greater than the average size of the data, and the i-th second data set is determined. After determining the (p-1)th second data set, the remaining first data block is determined as the p-th second data set, where i is a positive integer greater than 0 and less than p, and p is the number of data storage devices.
7. The method according to any one of claims 1 to 6, characterized in that, The step of sending the second data set to the second controller through the transmission channel corresponding to the data storage includes: Send the second data set to the bridge port; The second data set is sent to the second controller through the bridging port and the transmission channel, wherein the transmission channel is connected to the data processing module in the second controller.
8. The method according to claim 7, characterized in that, Sending the second data set to the bridge port includes: If the second data set meets the transmission conditions, the memory access module sends the second data set in the data memory to the bridge port.
9. The method according to claim 7, characterized in that, Sending the second data set to the bridge port includes: The sending address of the second data set is mapped to obtain the receiving address corresponding to the second controller.
10. The method according to claim 1, characterized in that, Before sending the first data set to the data processing module, the process includes: Obtain the first data set at the input port.
11. The method according to claim 1, characterized in that, The first controller is configured to receive the first data set sent by the server, and the second controller synchronizes its data with the first controller.
12. A data transmission system, characterized in that, The system includes: The first controller includes a data processing module, a data storage device, a first PCIe switch, and an input port. The first PCIe switch is configured to perform multi-channel data transmission with the second PCIe switch of the second controller. The second controller includes a second PCIe switch configured to perform multi-channel data transmission with the first PCIe switch.
13. The system according to claim 12, characterized in that, The first PCIe switch includes multiple partitions, and the uplink port of each partition of the first PCIe switch is connected to the data processing module, wherein the uplink port is configured to transmit data with the data processing module; Each partition of the first PCIe switch has a downlink port connected to the corresponding partition of the second PCIe switch. The downlink port is configured to enable data transmission between the partitions of the first PCIe switch and the corresponding partitions of the second PCIe switch.
14. The system according to claim 13, characterized in that, Each uplink port of the first PCIe switch is connected to the corresponding chip in the data processing module. The number of partitions of the first PCIe switch is the same as the number of chips, and the partitions of the first PCIe switch correspond one-to-one with the chips.
15. The system according to claim 14, characterized in that, Each partition of the second PCIe switch is connected to a corresponding chip in the data processing module of the second controller. The number of partitions in the second PCIe switch is the same as the number of chips in the second controller, and there is a one-to-one correspondence between the partitions of the second PCIe switch and the chips in the second controller.
16. The system according to claim 14, characterized in that, The chips of the data processing module are interconnected via an off-chip bus, and each chip is connected to a corresponding data memory, wherein the data memory is configured to store data blocks.
17. The system according to claim 14, characterized in that, The first PCIe switch includes multiple DMA engines, the number of which is the same as the number of chips, and the DMA engines are configured to retrieve data from the corresponding data memory.
18. The system according to claim 17, characterized in that, The first PCIe switch and the second PCIe switch are connected via a non-transparent bridge port, which is configured to translate the data address in the first PCIe switch to the address of the second PCIe switch or to translate the address of the second PCIe switch to the address of the first PCIe switch.
19. The system according to any one of claims 12 to 18, characterized in that, The input port is connected to the server and is configured to receive data sent by the server.
20. A data transmission device, characterized in that, The device includes: A first sending module is configured in a first controller to send a first data set to a data processing module, wherein the first data set includes a plurality of first data blocks, and the data processing module is configured to process the data blocks; The splitting module is configured such that, under preset conditions, the data processing module splits the first data set into multiple second data sets and sends the multiple second data sets to corresponding data storage devices, wherein the number of second data sets is the same as the number of data storage devices; The transmission module is configured to send the second data set to the second controller through the transmission channel corresponding to the data storage.
21. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.
22. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.
23. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.