Data caching method and system, and product, device and storage medium

Abstract

The present application relates to the technical field of data processing. Provided are a data caching method and system, and a product, a device and a storage medium. The method is applied to a graphics processing unit, and comprises: acquiring, from a current pending task sent by a host, a target computing task that is required to be executed by the graphics processing unit, wherein the graphics processing unit pre-establishes a connection with a memory pool, the memory pool comprising a shared first memory partition and a second memory partition exclusive to the graphics processing unit; determining, when the graphics processing unit itself executes the target computing task, the data type of each piece of first data to be cached; on the basis of the data type, determining a corresponding target memory partition; and caching the first data to be cached into the corresponding target memory partitions, in order to use the cached data to complete a subsequent computing process. In this way, the graphics processing unit of the present application can implement data caching using a mounted memory pool, and can flexibly cache data into different memory partitions on the basis of data types and a memory partition set, thereby improving the data exchange performance and reducing the data latency.

Description

A data caching method, system, product, device, and storage medium

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411534785.0, filed on October 31, 2024, entitled “A Data Caching Method, System, Product, Device and Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of data processing technology, and in particular to a data caching method, system, product, device and storage medium. Background Technology

[0004] Current video memory capacity is entirely limited by the GPU (Graphics Processing Unit) memory controller design. Inference calculations require massive amounts of cached data, but due to GPU specifications and hardware limitations, customers cannot flexibly expand video memory capacity. Increasing the number of GPUs leads to increased costs and reduced economic efficiency. Furthermore, during GPU computation, when retrieving cached data from another GPU, GPUDirect P2P (Peer-to-Peer) technology (a high-speed communication technology between single-machine GPUs) utilizes the characteristics of the PCIe (Peripheral Component Interconnect express) bus to achieve direct data transfer between GPUs. In systems supporting GPUDirect P2P, GPUs can communicate directly via the PCIe bus without going through host memory. For example, video memory data is uploaded to PCIE_Switch via GPU0, and then forwarded to the video memory of GPU1 via PCIE_Switch. This can significantly reduce data transmission latency and increase transmission bandwidth. However, in actual cross-domain transmission, it will be affected by PCIE link loss, and the latency will reach several milliseconds or even tens of milliseconds.

[0005] It is evident that the relevant technologies suffer from the inability to effectively expand video memory and low data exchange performance. Summary of the Invention

[0006] The purpose of this application is to provide a data caching method, system, product, device, and storage medium that can solve the problems of ineffective expansion of video memory and low data exchange performance.

[0007] To address the aforementioned technical problems, in a first aspect, this application provides a data caching method applied to a graphics processor, comprising:

[0008] The target computation task that the graphics processor needs to execute is obtained from the current pending tasks sent by the host; the graphics processor establishes a connection with the memory pool in advance, which contains a first memory partition shared by the host and the graphics processor and a second memory partition exclusively for the graphics processor;

[0009] Determine the data type of the first data to be cached when performing the target computation task;

[0010] The target memory partitions corresponding to each first data to be cached are determined from the preset memory partition set based on the data type; wherein, the preset memory partition set includes the first memory partition, the second memory partition and the third memory partition located in the local memory of the graphics processor, and different data types correspond to different memory partitions;

[0011] The first data to be cached is cached to the corresponding target memory partition so that the cached data in the target memory partition can be used to complete the subsequent calculation process of the current task to be executed.

[0012] Optionally, determine the data type of each first piece of data to be cached when performing the target computation task, including:

[0013] During the execution of the target computation task, at least one target parameter is determined for each first data to be cached, and a weighted calculation is performed based on each target parameter and its corresponding weight for each first data to be cached to obtain the weighted result for each first data to be cached.

[0014] Based on the weighted results, each first piece of data to be cached is classified to determine the data type of each piece of data to be cached.

[0015] Optionally, determine at least one target parameter corresponding to each first piece of data to be cached during the execution of the target computation task, including:

[0016] During the execution of the target computation task, the task level, parameter model size, and matrix operation method corresponding to each first data to be cached are determined as the target parameters.

[0017] Optionally, the data type of the first data to be cached includes:

[0018] The first type of data to be cached is the data that needs to be transferred between the host and the graphics processor during the execution of the target computing task; the second type of data to be cached is the data that has a data dependency level with the target computing task that meets the first preset low dependency condition during the execution of the target computing task; and the third type of data to be cached is the data that has a data dependency level with the target computing task that meets the first preset high dependency condition during the execution of the target computing task.

[0019] Among them, the target memory partitions corresponding to the first type of data to be cached, the second type of data to be cached, and the third type of data to be cached are the first memory partition, the second memory partition, and the third memory partition, respectively.

[0020] Optionally, the number of graphics processors can be one or more, the host establishes a connection with each graphics processor through a first switch, and each graphics processor establishes a connection with the memory pool through a second switch;

[0021] Correspondingly, data caching methods also include:

[0022] Based on the number of graphics processors, a corresponding number of memory partitions are created in the memory pool to obtain a second memory partition corresponding to each graphics processor, so that the graphics processor can send the second type of data to be cached when it executes the target computing task to the corresponding second memory partition for caching.

[0023] Optionally, the first switch is a switch based on the high-speed serial computer expansion bus standard, and the second switch is a switch based on the computing fast link.

[0024] Optionally, the method also includes:

[0025] The memory pool is attached to the second switch via a pre-defined extender;

[0026] The memory pool is allocated a corresponding memory address space through the second switch, so as to divide the memory pool into a first memory partition shared by the host and the graphics processor, a second memory partition exclusively used by the graphics processor, and a fourth memory partition exclusively used by the host, and the first memory partition and the fourth memory partition are mounted to the host.

[0027] The preset memory partition set also includes a fourth memory partition.

[0028] Optionally, the method also includes:

[0029] While the host sends the current task to be executed to the graphics processor, the host determines the data type of the second data to be cached during the processing of the current task, and determines the target memory partition corresponding to each second data to be cached from the preset memory partition set according to the data type of the second data to be cached, and then caches the second data to be cached to the corresponding target memory partition.

[0030] Different data types of second data to be cached correspond to different memory partitions.

[0031] Optionally, the data type of the second data to be cached includes:

[0032] The fourth type of cached data requires data transfer between the host and the graphics processor during the processing of the current task to be executed; the fifth type of cached data whose data dependency level with the current task to be executed meets the second preset low dependency condition during the processing of the current task to be executed; and the sixth type of cached data whose data dependency level with the current task to be executed meets the second preset high dependency condition during the processing of the current task to be executed.

[0033] Among them, the target memory partitions corresponding to the fourth type of data to be cached, the fifth type of data to be cached, and the sixth type of data to be cached are the first memory partition, the fourth memory partition, and the fifth memory partition, respectively. The fifth memory partition is the host's local memory.

[0034] Optionally, the number of hosts can be one or more;

[0035] The data caching methods also include:

[0036] Based on the number of hosts, a corresponding number of memory partitions are created in the memory pool to obtain a fourth memory partition corresponding to each host, so that the host can send the fifth type of data to be cached in the processing of the current task to the corresponding fourth memory partition for caching.

[0037] Accordingly, the first and fourth memory partitions are mounted to the host, including:

[0038] The first memory partition is mounted to each host via the second switch, and each fourth memory partition is mounted to the corresponding host.

[0039] Optionally, the method also includes:

[0040] The first memory partition is used to respond to the data access request initiated by the target data access party, so as to transfer the data corresponding to the data access request in the first memory partition to the target data access party;

[0041] The target data access party is any host or any graphics processor, and the data corresponding to the data access request includes data cached in the first memory partition by any host and / or any graphics processor.

[0042] Secondly, this application provides a data caching system, including a host, a memory pool, and a graphics processor pre-connected to the host and the memory pool;

[0043] The graphics processor implements the following data caching methods by executing computer programs:

[0044] The system retrieves the target computation task that the graphics processor needs to execute from the current pending task sent by the host; the memory pool contains a first memory partition shared by the host and the graphics processor and a second memory partition exclusively used by the graphics processor; it determines the data type of each first data to be cached when executing the target computation task; based on the data type, it determines the target memory partition corresponding to each first data to be cached from a preset memory partition set; wherein, the preset memory partition set includes the first memory partition, the second memory partition, and a third memory partition located in the local memory of the graphics processor, and different data types correspond to different memory partitions; it caches the first data to be cached to the corresponding target memory partition so that the data cached in the target memory partition can be used to complete the subsequent computation process of the current pending task.

[0045] Optionally, the number of graphics processors can be one or more, the number of hosts can be one or more, the hosts establish connections with each graphics processor through a first switch, and each graphics processor and each host establish connections with the memory pool through a second switch;

[0046] The first switch is a switch based on the high-speed serial computer expansion bus standard, and the second switch is a switch based on the computing fast link.

[0047] Thirdly, this application provides an electronic device, comprising:

[0048] Memory, configured to store computer programs;

[0049] The processor is configured to execute computer programs to implement the steps of the data caching method described above.

[0050] Fourthly, this application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the data caching method described above.

[0051] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the data caching method described above.

[0052] Therefore, this application is applied to a graphics processing unit (GPU). First, it obtains the target computation task that the GPU needs to execute from the current task to be executed, sent by the host. The GPU establishes a connection with a memory pool beforehand, which contains a first memory partition shared by the host and the GPU, and a second memory partition exclusively for the GPU. Then, it determines the data type of each first data to be cached when executing the target computation task. Next, based on the data type, it determines the target memory partition corresponding to each first data to be cached from a preset memory partition set. The preset memory partition set includes a first memory partition, a second memory partition, and a third memory partition located in the GPU's local memory, with different data types corresponding to different memory partitions. Finally, the first data to be cached is cached to the corresponding target memory partition so that the cached data in the target memory partition can be used to complete the subsequent computation process of the current task to be executed. In this way, in this application, the GPU can achieve data caching through a mounted memory pool. By combining data type and memory partition set, data can be flexibly cached to different memory partitions, which can improve data exchange performance and reduce data latency. Attached Figure Description

[0053] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 is a flowchart of a data caching method disclosed in this application;

[0055] Figure 2 is a schematic diagram of a graphics processor structure disclosed in this application;

[0056] Figure 3 is a schematic diagram of an optional graphics processor structure disclosed in this application;

[0057] Figure 4 is a schematic diagram of a data caching system structure disclosed in this application;

[0058] Figure 5 is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0059] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0060] The terms "comprising" and "having," and any variations thereof, in the specification and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may include steps or units not listed.

[0061] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] As shown in Figure 1, this application discloses a data caching method applied to a graphics processor, comprising:

[0063] Step S11: Obtain the target computing task that the graphics processor needs to execute from the currently pending tasks sent by the host; the graphics processor establishes a connection with the memory pool in advance, which contains a first memory partition shared by the host and the graphics processor and a second memory partition exclusively for the graphics processor.

[0064] In this embodiment, the graphics processor (GPU) establishes a connection with the memory pool in advance. The memory pool is pre-divided into a shared first memory partition for the host and the GPU, and a second memory partition exclusively for the GPU. It is understood that the exclusive second memory partition can only be read and written by the GPU, while the shared first memory partition can be read and written by both the host and the GPU. Furthermore, the task to be executed is first obtained by the host, which then sends the target computation task requiring GPU execution to the GPU so that the GPU can execute the corresponding target computation task.

[0065] Step S12: Determine the data type of the first data to be cached when performing the target computing task.

[0066] In this embodiment of the application, during the execution of the target computing task by the graphics processor, several computing results are generated, namely the first data to be cached. These data need to be temporarily cached, and some of them may be used for subsequent computing processes. In order to cache the first data to be cached generated by these computing processes, the graphics processor here first needs to determine the data type of each first data to be cached.

[0067] In one optional embodiment, determining the data type of each first data to be cached during the execution of the target computation task may include: determining at least one target parameter corresponding to each first data to be cached during the execution of the target computation task, and performing a weighted calculation based on each target parameter and its corresponding weight to obtain a weighted result for each first data to be cached; classifying each first data to be cached based on the weighted result to determine the data type of each first data to be cached. Further, in another optional embodiment, determining at least one target parameter corresponding to each first data to be cached during the execution of the target computation task may include: determining the task level, parameter model size, and matrix operation method corresponding to each first data to be cached as target parameters during the execution of the target computation task. Optionally, in determining the data type of each first data to be cached generated during the execution of the target computation task, the graphics processor first determines at least one target parameter corresponding to each first data to be cached. It is understood that the target parameter mentioned here can be one or any combination of the task level, parameter model size, and matrix operation method corresponding to each first data to be cached. Further, after determining the target parameter corresponding to each first data to be cached, a weighted calculation is performed based on each target parameter and its corresponding weight, thus obtaining the weighted result corresponding to each first data to be cached. It is understood that, based on the different weighted results, each first data to be cached can be divided into different categories, thus ultimately obtaining the data type of each first data to be cached.

[0068] Step S13: Determine the target memory partition corresponding to each first data to be cached from the preset memory partition set based on the data type; wherein, the preset memory partition set includes the first memory partition, the second memory partition and the third memory partition located in the local memory of the graphics processor, and different data types correspond to different memory partitions.

[0069] In this embodiment of the application, after obtaining the data types corresponding to the first data to be cached through the above steps, the target memory partitions corresponding to each first data to be cached can be determined from the preset memory partition set based on the data type. It should be noted that the preset memory partition set may include the first memory partition shared by the host and the graphics processor in the memory pool, the second memory partition exclusively occupied by the graphics processor in the memory pool, and the third memory partition in the local memory of the graphics processor. Each data type corresponds to a different memory partition.

[0070] In an optional embodiment, the data type of the first data to be cached may include: a first type of data to be cached that requires data transfer between the host and the graphics processor during the execution of the target computing task; a second type of data to be cached that meets a first preset low dependency condition in terms of data dependency level with the target computing task during the execution of the target computing task; and a third type of data to be cached that meets a first preset high dependency condition in terms of data dependency level with the target computing task during the execution of the target computing task; wherein the target memory partitions corresponding to the first type of data to be cached, the second type of data to be cached, and the third type of data to be cached are the first memory partition, the second memory partition, and the third memory partition, respectively. Optionally, the above steps can be used to classify the first type of data to be cached into three types. Data that needs to be transferred during the execution of a computational task is identified as the first type of data to be cached. This type of data may require the host or graphics processor to read and access it during subsequent computations. The second type of data to be cached is data whose data dependency level with the current computational task meets the first preset low dependency condition. This type of data is unlikely to be read and accessed during subsequent computational task executions and has a low correlation level with the computational task. Furthermore, the third type of data to be cached is data whose data dependency level with the computational task meets the first preset high dependency condition. This type of data is necessary and frequently used during the execution of the computational task. It should be noted that the first memory partition in the memory pool, shared by the host and the graphics processor, is the target memory partition corresponding to the first type of data to be cached. This first memory partition enables efficient transfer of the first type of data to be cached during the execution of computing tasks. Correspondingly, the second memory partition in the memory pool, exclusively used by the graphics processor, is the target memory partition corresponding to the second type of data to be cached, and the third memory partition in the graphics processor's local memory is the target partition corresponding to the third type of data to be cached. It can be understood that data with a high dependency level on the computing task is directly cached in local memory for easy access by subsequent computing tasks, while data with a low dependency level on the computing task can be cached in the second memory partition in the memory pool exclusively used by the graphics processor to reduce the impact on subsequent computing processes.

[0071] In one optional embodiment, the number of graphics processors (GPUs) is one or more. The host establishes a connection with each GPU through a first switch, and each GPU establishes a connection with a memory pool through a second switch. Correspondingly, the data caching method may further include: creating a corresponding number of memory partitions in the memory pool based on the number of GPUs, to obtain second memory partitions corresponding to each GPU, so that the GPUs can send the second type of data to be cached generated during the execution of the target computing task to the corresponding second memory partitions for caching. Optionally, the host can connect to several GPUs through the first switch, and each GPU establishes a connection with the mounted memory pool through a second switch. In this case, the mounted memory pool needs to allocate memory for each GPU. This can be achieved by creating a corresponding number of memory partitions in the memory pool based on the number of GPUs, thus obtaining several second memory partitions corresponding to each GPU and exclusively occupied by each GPU. In this way, subsequent GPUs can send the second type of data to be cached generated during the execution of the target computing task to the corresponding second memory partitions for caching.

[0072] Step S14: Cache the first data to be cached to the corresponding target memory partition so that the cached data in the target memory partition can be used to complete the subsequent calculation process of the current task to be executed.

[0073] In this embodiment of the application, through the above steps, the graphics processor can determine the target memory partition corresponding to the first data to be cached during the execution of the target computing task, and then cache the first data to be cached to the corresponding target memory partitions respectively. Subsequently, the cached data in the target memory partitions can be used to continue to execute the relevant computing process.

[0074] In an optional embodiment, the method may further include: mounting the memory pool to a second switch via a preset expander; allocating corresponding memory address space to the memory pool via the second switch to divide the memory pool into a first memory partition shared by the host and the graphics processor, a second memory partition exclusively for the graphics processor, and a fourth memory partition exclusively for the host; and mounting the first memory partition and the fourth memory partition to the host; wherein the preset memory partition set also includes the fourth memory partition. Furthermore, the first switch is a switch based on the High-Speed ​​Serial Computer Expansion Bus (PCIE) standard, and the second switch is a switch based on Compute Express Link (CXL). It is understood that the first switch establishes connections with the host and the graphics processor respectively via the PCIE standard, while the second switch establishes connections with the host and the graphics processor respectively via Compute Express Link (CXL); during the mounting of the memory pool, the memory pool is mounted to the second switch via an expander; it is understood that the memory pool is also connected to the second switch via CXL, and the host and the graphics processor can directly access the memory pool via CXL. Furthermore, the memory pool can be divided into a first memory partition shared by the host and the graphics processor (GPU), a second memory partition exclusively for the GPU, and a fourth memory partition exclusively for the host via a second switch. The first and fourth memory partitions need to be mounted to the host so that the host can cache data generated during the execution of computing tasks to the corresponding memory partitions. It should be noted that the preset memory partition set may also include a fourth memory partition; that is, data generated by the host during the execution of computing tasks that needs to be cached can be cached in the memory pool in the first memory partition shared by the host and the GPU, and the fourth memory partition exclusively for the host. In an optional embodiment, a CXL Type 3 device (a memory expansion device) can be mounted to achieve memory pooling, dividing the memory into several memory partitions, each corresponding to the host and the GPU.

[0075] In an optional embodiment, the method may further include: simultaneously sending the currently pending task to the graphics processor, determining the data type of the second data to be cached during the processing of the currently pending task, determining the target memory partition corresponding to each second data to be cached from a preset memory partition set based on the data type of the second data to be cached, and then caching the second data to be cached to the corresponding target memory partition; wherein, different data types of second data to be cached correspond to different memory partitions. Optionally, the host may send the acquired currently pending task to the graphics processor, and simultaneously determine the second data to be cached during the processing of the currently pending task, and determine the data type of the second data to be cached; accordingly, the host may determine the target memory partition corresponding to each second data to be cached from a preset memory partition set based on the data type of these data to be cached; it is understood that the target memory partition here may be a first memory partition shared by the host and the graphics processor in the memory pool, a fourth memory partition exclusively occupied by the host, and a fifth memory partition in the host's local memory; and different data types of second data to be cached correspond to different target memory partitions. Furthermore, the data types of the second cached data may include: a fourth type of cached data that requires data transfer between the host and the graphics processor during the processing of the current task to be executed; a fifth type of cached data whose data dependency level with the current task to be executed meets the second preset low dependency condition during the processing of the current task to be executed; and a sixth type of cached data whose data dependency level with the current task to be executed meets the second preset high dependency condition during the processing of the current task to be executed. The target memory partitions corresponding to the fourth, fifth, and sixth types of cached data are the first memory partition, the fourth memory partition, and the fifth memory partition, respectively, with the fifth memory partition being the host's local memory. Optionally, the data type of the second cached data generated by the host during the execution of the computing task can be the fourth type of cached data that needs to be transferred between hosts or between the host and the graphics processor during the processing of the computing task, i.e., computing parameters, computing results, and other data that need to be exchanged; it can also be the fifth type of cached data whose data dependency level with the current computing task meets the second preset low dependency condition, i.e., data with a low association level with the computing task; or it can be the sixth type of cached data whose data dependency level with the current computing task meets the second preset high dependency condition, i.e., some of the most necessary and most frequently used data to achieve the computing task.Accordingly, the sixth type of data to be cached needs to be cached in the fifth memory partition of the host's local memory to facilitate efficient execution of computing tasks; the fifth type of data to be cached needs to be cached in the fourth memory partition of the memory pool to reduce the impact on the execution process of computing tasks; the fourth type of data to be cached needs to be cached in the first memory partition of the memory pool, which can be directly read and called by the host or graphics processor, thereby improving the execution efficiency of computing tasks.

[0076] In one optional embodiment, the number of hosts is one or more; wherein, the data caching method may further include: creating a corresponding number of memory partitions in a memory pool according to the number of hosts, to obtain a fourth memory partition corresponding to each host, so that the host sends the fifth type of data to be cached during the processing of the currently executed task to the corresponding fourth memory partition for caching; correspondingly, mounting the first memory partition and the fourth memory partition to the host includes: mounting the first memory partition to each host through a second switch, and mounting each fourth memory partition to the corresponding host respectively. Optionally, it can be understood that there may also be several hosts, each connected to a memory pool mounted to the second switch, wherein the memory pool contains memory partitions with a corresponding amount of data created according to the number of hosts, i.e., fourth memory partitions corresponding to each host, and several fourth memory partitions in the memory pool, and the first memory partition shared by the host and the graphics processor, are all mounted to each host through the second switch, so that the host can cache the fifth type of data to be cached during the processing of the computing task to the fourth memory partition of the memory pool, and cache the fourth type of data to be cached to the first memory partition of the memory pool.

[0077] In another optional embodiment, the method may further include: responding to a data access request initiated by a target data access party using a first memory partition, thereby transferring the data corresponding to the data access request in the first memory partition to the target data access party; wherein, the target data access party is any host or any graphics processor, and the data corresponding to the data access request includes data cached in the first memory partition by any host and / or any graphics processor. Optionally, during the process of executing the computation task through the steps of the above embodiments, the memory pool also caches data related to the computation task. Subsequently, the data in the first memory partition can be used to respond to a data access request initiated by the target data access party, transferring the cached data corresponding to the data access request to the target data access party; it is understood that the target data access party here can be either a host or a graphics processor, both of which can obtain data from the first memory partition of the memory pool to perform the computation process according to the computation requirements, and data exchange between the host and the graphics processor can be achieved through the memory pool.

[0078] In another alternative embodiment, if the current computing task to be executed by the current host does not require the aid of a graphics processor, the current host can directly execute the computing task and simultaneously cache the data generated during the computing process in a hierarchical manner to the fifth memory partition of the local memory and the first and fourth memory partitions in the memory pool. In this way, other hosts can directly access the data cached by the current host from the first memory partition of the memory pool according to the computing requirements of the relevant computing tasks, which facilitates the execution of the relevant computing tasks. This achieves memory expansion of the host and can improve the data transmission efficiency between hosts.

[0079] It should be noted that the graphics processor (graphics processing accelerator card) in this application is responsible for data processing and high-performance computing. As shown in Figures 2 and 3, the accelerator card mainly includes HBM (High Bandwidth Memory), RJ45 (an information socket connector in a cabling system), DPU (Data Processing Unit) chip, GPU chip, CPLD (Complex Programming logic device) logic controller, QSFP (Quad Small Form-factor Pluggable) optical port connector, gold finger connector, CDFP (a type of connector), and M.2 (a type of hard disk protocol) hard drive, etc. The DPU chip can expand its serial data interface to connect to an external system hard drive, and interconnects with the GPU chip (graphics processing unit) via PCIe (high-speed serial bus). Internally, it can be managed and controlled via an ARM (a type of processor) core to achieve task allocation. It can also connect to an RJ45 (e.g., fiber optic interface) management network port, routing information with the host and various switch boxes via a switch. In other words, the graphics processor in this application can have an ARM core and operating system, possessing system management, memory management, and other functions, essentially acting as a host handling its own computing tasks. It interconnects with the host via the PCIe protocol, functioning as a downstream device; and interconnects with a CXL switch via the CXL protocol, functioning as an upstream device. The DPU chip can also provide a QSFP interface, connecting to a data plane switch. Through GPU Direct RDMA (Remote Direct Memory Access, a technology that allows remote devices or nodes to directly access the memory of another device), it can forward GPU computing data, improving GPU performance. The CPLD (Programmable Logic Device) is responsible for the management of the accelerator card, including thermal control, power-on / off timing, and status indication. The GPU chip is responsible for data processing and high-performance computing tasks. Its PCIe link is connected to the PCIe Switch (first switch) through the gold finger connector, and the CXL link is interconnected with the CXL Switch through the CDFP (connector). The GPU accelerator card comes with HBM (high bandwidth memory), which can improve data transmission bandwidth and reduce latency.

[0080] In an optional embodiment, the GPU in Figure 3 can obtain the target computing task to be executed from the host via a Gold Finger, or it can obtain the computing task to be executed via its own operating system. Then, during the execution of the target computing task, the GPU determines the data type of the data to be cached for each computation to determine the importance of this data to the current target computing task, or whether data transfer with other hosts or other GPUs is required. The process of determining the data type can divide the data to be cached generated by the computation into three types: data L1, which is the most frequently used data in the computing task; data L2, which needs to be exchanged; and data L3, which has a low association level with the computing task. Furthermore, the GPU in Figure 3 can use CXL (Compute Fast Link) to store the data to be cached with data type L2 that needs to be exchanged to the first memory partition in the mounted memory pool, store the data to be cached with data type L3 that has a low association level with the computing task to the second memory partition in the memory pool, and store the data to be cached with data type L1, which is the most frequently used data in the computing task, to the third memory partition (i.e., the high-bandwidth memory in the GPU). In this way, the GPU can use the memory pool pre-connected via the CXL link to cache data related to the computing task, thereby realizing the memory expansion of the GPU.

[0081] Furthermore, the GPU (Graphics Processing Unit) in Figure 3 can expand the system hard disk through the DPU (Data Processing Unit), and manage the acquisition and execution of computing tasks through the operating system. In other words, the GPU in this application can act as a downlink device for the host, executing computing tasks issued by the host with the help of a pre-mounted memory pool; and the GPU can also act as an uplink device, acquiring computing tasks through its own operating system and executing the relevant computing tasks with the help of the mounted memory pool.

[0082] Therefore, the graphics processor in this application can achieve data caching through the mounted memory pool. By combining data types and memory partition sets, data can be flexibly cached to different memory partitions, realizing the functions of expanding, allocating, and controlling the memory capacity of the host and the graphics processor. Furthermore, the graphics processor can come with its own ARM core and operating system, and can implement data partition storage just like the host. In this way, data can be stored between different memory partitions, improving data exchange efficiency.

[0083] This application discloses a data caching system, including a host, a memory pool, and a graphics processor that has been pre-connected to the host and the memory pool;

[0084] The graphics processor implements the following data caching methods by executing computer programs:

[0085] The system retrieves the target computation task that the graphics processor needs to execute from the current pending task sent by the host; the memory pool contains a first memory partition shared by the host and the graphics processor and a second memory partition exclusively used by the graphics processor; it determines the data type of each first data to be cached when executing the target computation task; based on the data type, it determines the target memory partition corresponding to each first data to be cached from a preset memory partition set; wherein, the preset memory partition set includes the first memory partition, the second memory partition, and a third memory partition located in the local memory of the graphics processor, and different data types correspond to different memory partitions; it caches the first data to be cached to the corresponding target memory partition so that the data cached in the target memory partition can be used to complete the subsequent computation process of the current pending task.

[0086] In an optional embodiment, the number of graphics processors is one or more, the number of hosts is one or more, the hosts establish connections with each graphics processor through a first switch, and each graphics processor and each host establish connections with the memory pool through a second switch.

[0087] The first switch is a switch based on the high-speed serial computer expansion bus standard, and the second switch is a switch based on the computing fast link.

[0088] Furthermore, Figure 4 shows a schematic diagram of a data caching system architecture, including:

[0089] In this application, several hosts are connected to a first switch via a PCIe bus, and the first switch is connected to several graphics processing units (GPUs) via a PCIe bus. The local memory regions corresponding to HOST0 and HOST1 are denoted as DIMMs (Dual-Inline-Memory-Modules), and the local memory regions corresponding to GPU0 and GPU1 are denoted as HBMs. Furthermore, the hosts and GPUs are connected to a second switch via CXL links. The second switch is pre-loaded with a memory pool via the CXL protocol. This memory pool includes a first memory pool (Sharing memory area, i.e., C area memory pool) shared by the hosts and GPUs, a fourth memory pool exclusively used by the hosts (Host0 memory area and Host1 memory area, i.e., B area and D area memory pools), and a second memory partition exclusively used by the GPUs (GPU0 memory area and GPU1 memory area, i.e., A area and E area memory pools). This allows for memory capacity expansion. Furthermore, both the host computer and the graphics processor have their own local memory, designated as the fifth and third memory partitions, respectively. When users execute computational tasks under the host computer's operating system, they can weight various parameter data corresponding to the task based on task priority, parameter model size, matrix operation method, etc., and classify the corresponding data into three levels, such as L1, L2, and L3. Among them, L1 is the parameter data that is most necessary for realizing the computational task and is used most frequently; L2 is the computational parameters and calculation results that need to be exchanged in subsequent calculation processes; and L3 is the data with a lower correlation level to the computational task.

[0090] Optionally, when a computational task to be executed arrives at the host, the host first distributes the task to the GPU accelerator card through the first switch (PCIe Switch), enabling it to start parallel computation. At the same time, the host's operating system automatically classifies the task data (data to be cached) of the computational task to be executed. At this time, L1 level data can be stored in the fifth memory partition (DIMM) of the host's local memory, L2 level data can be stored in the Sharing memory area of ​​the memory pool, and L3 level data can be stored in the fourth memory partition exclusively used by the host.

[0091] Correspondingly, after receiving the computing task sent by the host, the graphics processor can execute the relevant computing business. At the same time, it can verify and distinguish the data generated during the computing process, that is, partition the data generated during the computing process (data to be cached) to obtain L1, L2 and L3 level data. The L1 level data can be stored in the third memory partition of the graphics processor's local memory, the L2 level data can be stored in the first memory partition of the memory pool, and the L3 level data can be stored in the second memory partition of the memory pool exclusively used by the graphics processor.

[0092] After the above steps, during the execution of the computing task, the data that needs to be transferred is cached by the host or graphics processor in the first memory partition of the memory pool. If a computing task executed by a host requires computing data from another host as support, it can be obtained in real time through the first memory partition of the memory pool; similarly, if a computing task executed by a graphics processor requires computing data from another graphics processor as support, it can also be obtained in real time through the first memory partition of the memory pool, which can improve the efficiency of data exchange.

[0093] Therefore, in this application, when a computing task begins execution, both the host and the graphics processor (GPU) can cache the latest computing results in the memory pool, specifically in the first memory partition shared by both the host and the GPU, and the fourth and second memory partitions, which are exclusively used by the host and the GPU respectively. The GPU can then retrieve the latest computing results from different cache spaces according to computing needs and continue the computing process. From a system-level perspective, data exchange between hosts, between GPUs, and between the host and the GPU can be achieved solely through the memory pool mounted on the fast computing link under the second switch, reducing data transmission latency and thus improving data exchange performance.

[0094] Furthermore, this application also discloses an electronic device. FIG5 is a structural diagram of the electronic device according to an exemplary embodiment. The content in the figure should not be considered as any limitation on the scope of use of this application. The electronic device may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is configured to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data caching method disclosed in any of the foregoing embodiments. Additionally, the electronic device in this embodiment may be an electronic computer.

[0095] In this embodiment, the power supply 23 is configured to provide operating voltage for each hardware device on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not limited here; the input / output interface 25 is configured to acquire external input data or output data to the outside world, and its optional interface type can be selected according to specific application needs, and is not limited here.

[0096] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0097] The operating system 221 is configured to manage and control the various hardware devices on the electronic device and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the data caching method executed by the electronic device as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0098] Furthermore, this application also discloses a computer-readable storage medium configured to store a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed data caching method. Optional steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here. The computer-readable storage medium may be, but is not limited to, a non-volatile readable storage medium.

[0099] This application discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the data caching method disclosed in any of the foregoing embodiments.

[0100] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0101] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0102] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0103] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0104] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data caching method, characterized in that, Applied to graphics processors, including: The target computation task that the graphics processor needs to execute is obtained from the current pending tasks sent by the host; the graphics processor establishes a connection with the memory pool in advance, the memory pool contains a first memory partition shared by the host and the graphics processor and a second memory partition exclusively used by the graphics processor; Determine the data type of each first data to be cached when performing the target computing task; Based on the data type, the target memory partitions corresponding to each of the first data to be cached are determined from the preset memory partition set; wherein, the preset memory partition set includes the first memory partition, the second memory partition, and a third memory partition located in the local memory of the graphics processor, and different data types correspond to different memory partitions; The first data to be cached is cached in the corresponding target memory partition so that the data cached in the target memory partition can be used to complete the subsequent calculation process of the current task to be executed.

2. The data caching method according to claim 1, characterized in that, Determining the data type of each first data to be cached when performing the target computing task includes: During the execution of the target computation task, at least one target parameter corresponding to each first data to be cached is determined, and a weighted calculation is performed based on each target parameter corresponding to each first data to be cached and the corresponding weight to obtain a weighted result corresponding to each first data to be cached. Based on the weighted results, each of the first data to be cached is classified to determine the data type of each of the first data to be cached.

3. The data caching method according to claim 2, characterized in that, Determining at least one target parameter corresponding to each first data to be cached during the execution of the target computation task includes: During the execution of the target computation task, the task level, parameter model size, and matrix operation method corresponding to each first data to be cached are determined as the target parameters.

4. The data caching method according to claim 1, characterized in that, The data types of the first data to be cached include: The data to be cached includes a first type of data that needs to be transferred between the host and the graphics processor during the execution of the target computing task; a second type of data that needs to be cached with the target computing task during the execution of the target computing task and whose data dependency level meets a first preset low dependency condition; and a third type of data that needs to be cached with the target computing task during the execution of the target computing task and whose data dependency level meets a first preset high dependency condition. The target memory partitions corresponding to the first type of data to be cached, the second type of data to be cached, and the third type of data to be cached are the first memory partition, the second memory partition, and the third memory partition, respectively.

5. The data caching method according to claim 4, characterized in that, The number of graphics processors is one or more, the host establishes a connection with each of the graphics processors through a first switch, and each of the graphics processors establishes a connection with the memory pool through a second switch; Accordingly, the data caching method further includes: Based on the number of graphics processors, a corresponding number of memory partitions are created in the memory pool to obtain each second memory partition corresponding to each graphics processor, so that the graphics processor can send the second type of cached data generated when it executes the target computing task to the corresponding second memory partition for caching.

6. The data caching method according to claim 5, characterized in that, The first switch is a switch based on the high-speed serial computer expansion bus standard, and the second switch is a switch based on the computing fast link.

7. The data caching method according to claim 5, characterized in that, Also includes: The memory pool is attached to the second switch via a preset extender; The second switch allocates a corresponding memory address space to the memory pool, dividing the memory pool into a first memory partition shared by the host and the graphics processor, a second memory partition exclusively used by the graphics processor, and a fourth memory partition exclusively used by the host, and then mounts the first memory partition and the fourth memory partition to the host. The preset memory partition set also includes the fourth memory partition.

8. The data caching method according to claim 7, characterized in that, Also includes: While the host sends the current task to be executed to the graphics processor, the host determines the data type of the second data to be cached during the processing of the current task, and determines the target memory partition corresponding to each second data to be cached from the preset memory partition set according to the data type of the second data to be cached, and then caches the second data to be cached to the corresponding target memory partition. The second data to be cached, which has different data types, corresponds to different memory partitions.

9. The data caching method according to claim 8, characterized in that, The data types of the second data to be cached include: The fourth type of cached data that requires data transfer between the host and the graphics processor during the processing of the current task to be executed; the fifth type of cached data whose data dependency level with the current task to be executed meets the second preset low dependency condition during the processing of the current task to be executed; and the sixth type of cached data whose data dependency level with the current task to be executed meets the second preset high dependency condition during the processing of the current task to be executed. The target memory partitions corresponding to the fourth type of data to be cached, the fifth type of data to be cached, and the sixth type of data to be cached are the first memory partition, the fourth memory partition, and the fifth memory partition, respectively, and the fifth memory partition is the local memory of the host.

10. The data caching method according to claim 9, characterized in that, The number of hosts is one or more; The data caching method further includes: Based on the number of hosts, a corresponding number of memory partitions are created in the memory pool to obtain each of the fourth memory partitions corresponding to each host, so that the host can send the fifth type of data to be cached during the processing of the current task to the corresponding fourth memory partition for caching; Accordingly, mounting the first memory partition and the fourth memory partition to the host includes: The first memory partition is mounted to each of the hosts via the second switch, and each of the fourth memory partitions is mounted to the corresponding hosts.

11. The data caching method according to any one of claims 1 to 10, characterized in that, Also includes: The first memory partition is used to respond to the data access request initiated by the target data access party, so as to transfer the data in the first memory partition corresponding to the data access request to the target data access party. The target data access party is any of the hosts or any of the graphics processors, and the data corresponding to the data access request includes data cached in the first memory partition by any of the hosts and / or any of the graphics processors.

12. The data caching method according to claim 10, characterized in that, The method further includes: When the host obtains that the current computing task does not require a graphics processor, the host executes the current computing task and caches the data generated during the computing process in the fifth memory level of local memory and the first and fourth memory partitions in the memory pool.

13. A data caching system, characterized in that, Includes a host, a memory pool, and a graphics processor pre-connected to the host and the memory pool; The graphics processor implements the following data caching method by executing a computer program: The system retrieves the target computation task that the graphics processor needs to execute from the current pending task sent by the host; the memory pool contains a first memory partition shared by the host and the graphics processor and a second memory partition exclusively used by the graphics processor; it determines the data type of each first data to be cached when executing the target computation task; based on the data type, it determines the target memory partition corresponding to each first data to be cached from a preset memory partition set; wherein, the preset memory partition set includes the first memory partition, the second memory partition, and a third memory partition located in the local memory of the graphics processor, and different data types correspond to different memory partitions; it caches the first data to be cached to the corresponding target memory partition so as to use the cached data in the target memory partition to complete the subsequent computation process of the current pending task.

14. The data caching system according to claim 13, characterized in that, The number of graphics processors is one or more, the number of hosts is one or more, the hosts establish a connection with each graphics processor through a first switch, and each graphics processor and each host establish a connection with the memory pool through a second switch; The first switch is a switch based on the high-speed serial computer expansion bus standard, and the second switch is a switch based on the computing fast link.

15. The data caching system according to claim 14, characterized in that, The graphics processor is configured to create a corresponding number of memory partitions in the memory pool based on the number of graphics processors, so as to obtain each second memory partition corresponding to each graphics processor, so that the graphics processor sends the second type of cached data generated when it executes the target computing task to the corresponding second memory partition for caching.

16. The data caching system according to claim 14, characterized in that, The memory pool is mounted on the second switch via a preset extender; The image processor is configured to allocate a corresponding memory address space to the memory pool through the second switch, so as to divide the memory pool into a first memory partition shared by the host and the graphics processor, a second memory partition exclusively used by the graphics processor, and a fourth memory partition exclusively used by the host, and to mount the first memory partition and the fourth memory partition to the host. The preset memory partition set also includes the fourth memory partition.

17. The data caching system according to claim 14, characterized in that, The data caching system further includes: while the host sends the currently to-be-executed task to the graphics processor, the host determines the data type of the second data to be cached during the processing of the currently to-be-executed task, and determines the target memory partition corresponding to each second data to be cached from the preset memory partition set according to the data type of the second data to be cached, and then caches the second data to be cached to the corresponding target memory partition; wherein, the second data to be cached of different data types corresponds to different memory partitions.

18. An electronic device, characterized in that, include: Memory, configured to store computer programs; A processor is configured to execute the computer program to implement the steps of the data caching method as described in any one of claims 1 to 12.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data caching method as described in any one of claims 1 to 12.

20. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the data caching method as described in any one of claims 1 to 12.

Images