Access control method and apparatus

Abstract

Provided in the present application are an access control method and an apparatus. In the method, a computing node maps a target physical memory to a pre-created CCL Window, so that another computing node can access the target physical memory on the basis of IO virtual address information of the CCL Window, thereby improving memory access efficiency, avoiding additional communication overheads, and improving the resource utilization rate.

Description

Access control methods and devices

[0001] This application claims priority to Chinese Patent Application No. 202411988234.1, filed with the China National Intellectual Property Administration on December 30, 2024, entitled “Access Control Method and Apparatus”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computers, and more particularly to an access control method and apparatus. Background Technology

[0003] In a collective communication (COLL) scenario, compute nodes can directly access memory through communication I / O. However, the virtual address corresponding to the dynamically allocated memory of a compute node is unpredictable, while COLL requires obtaining the virtual address corresponding to the memory of the peer in advance during direct memory access.

[0004] To address the aforementioned issues, existing technologies employ either setting up an additional Cluster Communication Library Buffer (CCL Buffer) or informing the peer of the virtual memory address via communication messages to achieve memory access. The former incurs additional data copying overhead, while the latter increases communication overhead. Summary of the Invention

[0005] This application provides an access control method and apparatus. In this method, physical memory is mapped onto a CCL Window to reduce system overhead and improve the efficiency of collection communication.

[0006] In a first aspect, this application provides an access control method, comprising: a first computing node creating a CCL Window, the CCL Window being allocated a subset of an I / O virtual address space, and the CCL Window not being allocated physical memory; the subset of the I / O virtual address space being allocated from an input / output I / O virtual address space, the subset of the I / O virtual address space including at least one I / O virtual address; the first computing node sending I / O virtual address information of the CCL Window to a second computing node, the I / O virtual address information being used to indicate the subset of the I / O virtual address space of the CCL Window; the first computing node establishing a mapping relationship between the physical address of the target physical memory and the subset of the I / O virtual address space of the CCL Window, so that the second computing node can access the target physical memory based on the I / O virtual address information of the CCL Window. In this way, by mapping physical memory to a CCL Window that only has I / O virtual addresses but has not allocated physical memory, other compute nodes that have already obtained the I / O virtual addresses of the CCL Window can directly access the physical memory mapped to the CCL Window based on the CCL Window's virtual addresses, without performing memory copying. This allows other compute nodes to directly access the physical memory of this compute node, effectively saving overhead and improving the efficiency of aggregated communication. Furthermore, in aggregated communication, compute nodes can obtain the I / O virtual address information of the CCL Window in advance to meet the requirement of pre-obtaining the I / O virtual address in aggregated communication.

[0007] For example, the IOVA virtual address space is an ordered set of IO virtual addresses.

[0008] For example, IO virtual address information includes, but is not limited to, starting IO virtual address and length information.

[0009] For example, the target physical memory is in physical memory.

[0010] For example, the IO virtual address may optionally be IOVA.

[0011] For example, the subset of the IO virtual address space includes IO virtual addresses that are contiguous.

[0012] In one possible implementation, the target physical memory corresponds to a virtual address, which comes from a virtual address space. This virtual address space is independent of the IO virtual address space. Thus, a compute node can allocate a corresponding IOVA for the CCL Window from the IO virtual address space and a corresponding VA for the target physical memory from the virtual address space. The IOVA and VA can be different. That is, in this embodiment, the IOVA corresponding to the target physical memory provided by the compute node to other compute nodes is the IOVA of the CCL Window, and it does not need to be forced to be equal to the VA of the target physical memory. Correspondingly, in a set communication scenario, it is required to inform other compute nodes in advance of the IOVA corresponding to the target physical memory. Since the IOVA and VA are not forced to be equal, it is not necessary to wait until the target physical memory is dynamically allocated and the corresponding VA is obtained before obtaining the IOVA of the target physical memory. Therefore, by configuring the CCL Window and informing other compute nodes of the CCL Window's IOVA, other compute nodes can directly access the target physical memory based on the CCL Window's IOVA.

[0013] In one possible implementation, enabling other compute nodes to access target physical memory based on the I / O virtual address information of the CCL Window includes: a first compute node receiving an access request from a second compute node, the access request requesting access to physical memory mapped to a target I / O virtual address within the target physical memory, where the target I / OVA belongs to a subset of the I / O virtual address space. The first compute node then performs the corresponding access operation on the physical memory mapped to the target I / O virtual address based on the access request and the mapping relationship. In this way, this application can directly access the physical memory mapped to the CCL Window based on the virtual address of the CCL Window, without performing memory copying, allowing other compute nodes to directly access the physical memory of this compute node, avoiding additional communication overhead and improving memory access efficiency.

[0014] In one possible implementation, the access operation includes at least one of a read operation, a write operation, and an atomic operation.

[0015] In one possible implementation, the first compute node creates a CCL Window, including: allocating a subset of the I / O virtual address space for the CCL Window from the I / O virtual address space based on the size of the CCL Window; wherein the size of the CCL Window is greater than or equal to the size of the accessed data corresponding to the access operation; and generating an Input / Output Memory Management Unit (IOMMU) page table corresponding to the CCL Window, wherein the IOMMU page table includes address translation table entries indicating mapping relationships, and the physical addresses in the address translation table entries of the IOMMU page table are invalid. Thus, by pre-creating the CCL Window and allocating the corresponding I / O virtual address, this application can inform other compute nodes of the CCL Window's IOVA before dynamically allocating the target physical memory, enabling other compute nodes to access the target physical memory based on the CCL Window's IOVA. This effectively improves the efficiency of aggregated communication and avoids additional communication overhead.

[0016] In one possible implementation, the first compute node establishes a mapping between the physical addresses of the target physical memory and a subset of the CCL Window's I / O virtual address space. This includes copying the physical addresses from the address translation table entries of the MMU page table to the address translation table entries of the IOMMU page table. The address translation table entries of the MMU page table indicate the mapping between the physical addresses and virtual addresses of the target physical memory. Thus, by copying the MMU page table to the IOMMU page table, the physical addresses in the IOMMU page table and the physical addresses in the MMU page table are identical, meaning they point to the same physical page, thereby achieving the mapping between the CCL Window and the target physical memory.

[0017] In one possible implementation, the first compute node creates a CCL Window, including: allocating a subset of the I / O virtual address space for the CCL Window from the I / O virtual address space based on the size of the CCL Window; wherein the size of the CCL Window is greater than or equal to the size of the accessed data corresponding to the access operation; and generating an IOMMU page table corresponding to the CCL Window, wherein the IOMMU page table includes offset information, and the current content of the offset information is invalid. Thus, by pre-creating the CCL Window and allocating the corresponding I / O virtual address, this application can inform other compute nodes of the IOVA of the CCL Window before dynamically allocating the target physical memory, enabling other compute nodes to access the target physical memory based on the IOVA of the CCL Window. This effectively improves the efficiency of aggregated communication and avoids additional communication overhead.

[0018] In one possible implementation, the first compute node establishes a mapping relationship between the physical address of the target physical memory and a subset of the CCL Window's I / O virtual address space. This includes updating offset information, where the updated offset information indicates the offset between the virtual address of the target physical memory and the I / O virtual address of the CCL Window, or the offset between the I / O virtual address of the target physical memory and the I / O virtual address of the CCL Window. By updating the offset information, the IOMMU page table and the MMU page table can share a common address translation table. During the mapping process, there is no need to switch to kernel execution, thus reducing processing time, shortening the set communication completion time, and further improving overall processing efficiency.

[0019] Secondly, this application provides an access control device applied to a first computing node. The device includes: a creation module for creating a Cluster Communication Library Window (CCL Window), wherein the CCL Window is allocated a subset of an I / O virtual address space and is not allocated physical memory; the subset of the I / O virtual address space is allocated from an input / output I / O virtual address space, and the subset of the I / O virtual address space includes at least one I / O virtual address; a communication module for sending I / O virtual address information of the CCL Window to a second computing node, the I / O virtual address information indicating the subset of the I / O virtual address space of the CCL Window; and a mapping module for establishing a mapping relationship between the physical address of a target physical memory and the subset of the I / O virtual address space of the CCL Window, so that the second computing node accesses the target physical memory based on the I / O virtual address information of the CCL Window.

[0020] In one possible implementation, the target physical memory is allocated virtual addresses, which are allocated from a virtual address space, and the virtual address space is independent of the I / O virtual address space.

[0021] In one possible implementation, the device further includes an access module and a communication module, which are also used to receive an access request sent by the second computing node. The access request is used to request access to the physical memory in the target physical memory that is mapped to the target IO virtual address, and the target IOVA belongs to a subset of the IO virtual address space. The access module is used to perform corresponding access operations on the physical memory mapped to the target IO virtual address based on the access request and the mapping relationship.

[0022] In one possible implementation, the access operation includes at least one of a read operation, a write operation, and an atomic operation.

[0023] In one possible implementation, a module is created specifically for: allocating a subset of the I / O virtual address space for the CCL Window from the I / O virtual address space based on the size of the CCL Window; wherein the size of the CCL Window is greater than or equal to the size of the access data corresponding to the access operation; and generating an Input / Output Memory Management Unit (IOMMU) page table corresponding to the CCL Window, wherein the IOMMU page table includes address translation table entries for indicating mapping relationships, wherein the physical addresses in the address translation table entries of the IOMMU page table are invalid content.

[0024] In one possible implementation, the mapping module is specifically used to: copy the physical address from the address translation table entry of the MMU page table to the address translation table entry of the IOMMU page table; wherein the address translation table entry of the MMU page table is used to indicate the mapping relationship between the physical address of the target physical memory and the virtual address of the target physical memory.

[0025] In one possible implementation, a module is created specifically for: allocating a subset of the I / O virtual address space for the CCL Window from the I / O virtual address space based on the size of the CCL Window; wherein the size of the CCL Window is greater than or equal to the size of the access data corresponding to the access operation; and generating an IOMMU page table corresponding to the CCL Window, wherein the IOMMU page table includes offset information, and the current content of the offset information is invalid.

[0026] In one possible implementation, the mapping module is specifically used to: update offset information, wherein the updated offset information is used to indicate the offset between the virtual address of the target physical memory and the IO virtual address of the CCL Window, or the offset between the IO virtual address of the target physical memory and the IO virtual address of the CCL Window.

[0027] Thirdly, this application provides a computer-readable medium for storing a computer program including instructions for performing the methods in the first aspect or any possible implementation thereof.

[0028] Fourthly, this application provides a computer program including instructions for performing the method in the first aspect or any possible implementation thereof.

[0029] Fifthly, this application provides a chip including a processing circuit and transceiver pins. The transceiver pins and the processing circuit communicate with each other via an internal connection path. The processing circuit executes the method in the first aspect or any possible implementation of the first aspect to control the receiving pin to receive signals and to control the transmitting pin to transmit signals.

[0030] In a sixth aspect, this application provides a computer cluster including at least one computer device for performing the methods in the first aspect or any possible implementation thereof. Attached Figure Description

[0031] Figure 1 is a schematic diagram illustrating virtual addressing as an example;

[0032] Figure 2a is an exemplary diagram of IOVA addressing;

[0033] Figure 2b is an example illustrating the intended meaning of MTT;

[0034] Figure 3a is an exemplary diagram of IOVA addressing;

[0035] Figure 3b is an example illustrating the intended meaning of MATT;

[0036] Figure 4 is a schematic diagram of an exemplary communication system;

[0037] Figure 5 is a schematic diagram of an exemplary collection communication process;

[0038] Figure 6 is a schematic diagram of an exemplary collection communication process;

[0039] Figure 7 is a flowchart illustrating an exemplary access control method;

[0040] Figure 8 is an exemplary mapping diagram;

[0041] Figure 9 is an example of a comparison of the effects;

[0042] Figure 10 is an exemplary schematic diagram of the access control flow;

[0043] Figure 11 is an exemplary mapping diagram;

[0044] Figure 12 is an exemplary mapping diagram;

[0045] Figure 13 is an exemplary mapping diagram;

[0046] Figure 14a is an exemplary schematic diagram of data writing;

[0047] Figure 14b is an exemplary schematic diagram of data writing;

[0048] Figure 15 is an exemplary diagram illustrating data writing;

[0049] Figure 16 is a schematic diagram of an exemplary communication system;

[0050] Figure 17 is an exemplary mapping diagram;

[0051] Figure 18 is a schematic diagram of an exemplary communication system;

[0052] Figure 19 is a flowchart illustrating an exemplary access control method;

[0053] Figure 20 is a schematic diagram of the device structure as an example;

[0054] Figure 21 is a schematic diagram of the device structure as an example. Detailed Implementation

[0055] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0056] The following is a brief explanation of some background technologies and terms involved in the embodiments of this application.

[0057] (1) Physical memory: refers to the actual memory installed in a computer, which is used to store data for the central processing unit (CPU). It is generally Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM) or High Bandwidth Memory (HBM, also known as high bandwidth video memory). The memory mentioned in the embodiments of this application refers to physical memory, which will not be repeated below.

[0058] (2) Physical Address (PA): Also known as real address, it is the memory address used to access a specific storage unit of physical memory.

[0059] (3) Logical Address: The address used by the application. A logical address, also called a virtual address, is the address used in the program code. In memory access scenarios, a logical address is translated into a physical address (PA). In this embodiment, the logical address may include, but is not limited to, virtual addresses (VA) and input / output (IO) virtual addresses. In this embodiment, the IO virtual address may be IOVA (Input / Output Virtual Address). Of course, in some other instances, the IO virtual address may have other names, which are not limited in this application. The Memory Management Unit (MMU) can find the corresponding PA based on the VA. The Input / Output Memory Management Unit (IOMMU) can find the corresponding PA based on the IOVA.

[0060] (4) Address space: An ordered set of non-negative integer addresses. A system can have multiple address spaces, each independent. In this embodiment, the address space may include, but is not limited to: physical address space, virtual address space, and input / output virtual address space (i.e., I / O virtual address space). Optionally, the physical address, virtual address, and I / O virtual address involved in this embodiment belong to the physical address space, virtual address space, and I / O virtual address space, respectively.

[0061] (5) Page: Also known as a page, it is a fixed-length contiguous block of virtual memory, described by a single entry in the page table. It is the smallest unit of data used for memory management in operating systems that use virtual memory.

[0062] (6) Page table: A page table is a data structure used in the virtual memory system of a computer operating system. It stores the mapping between virtual addresses and physical addresses.

[0063] (7) Direct Memory Access (DMA): is a memory access technology that allows the computer’s internal hardware subsystems (peripherals) to read and write system memory independently without the intervention of the central processing unit (CPU).

[0064] (8) Remote Direct Memory Access (RDMA): is a high-speed network interconnection technology that enables computers to directly access the memory of remote computers over a network without frequent CPU intervention, thereby significantly enhancing network communication performance.

[0065] (9) Memory region (MR): A virtual contiguous region of arbitrary size in the registered consumer address space, supporting local and remote access by the host channel adapter (HCA).

[0066] (10) Memory Registration: Memory registration is to enable the device to access host memory using DMA. The memory registration process mainly includes: creating an Input / Output Memory Management Unit (IOMMU) page table for the device to use, and locking the memory region by the kernel to prevent the corresponding memory pages from being swapped out.

[0067] (11) Physical addressing and virtual addressing (also known as logical addressing): The way to access memory using physical addresses is called physical addressing.

[0068] Figure 1 is a schematic diagram illustrating virtual addressing. Referring to Figure 1, the CPU uses virtual addresses (VA) to access memory. The MMU is responsible for handling the CPU's memory access requests, implementing functions such as virtual address to physical address translation and access control. The IOMMU is responsible for handling the device's memory access requests, implementing functions such as I / O virtual address to physical address translation and access control.

[0069] For example, before a compute node exposes the memory corresponding to a virtual contiguous region of arbitrary size in its virtual address space to other compute nodes for direct memory access via communication I / O, it needs to perform memory registration to generate corresponding IOVA and Token, and generate corresponding IOMMU page tables. In existing implementations, IOVA and VA are equal. The IOMMU page tables include, but are not limited to, a permission control table and an address translation table. The permission control table is used to control access behavior, and the address translation table indicates the mapping relationship between IOVA and PA.

[0070] In one possible implementation, the RDMA network card uses a Memory Protection Table (MPT) + Memory Translation Table (MTT) mechanism to implement the IOMMU function. The MPT table serves as the IOMMU's access control table, and the MTT table is the address translation table. A memory region (MR) registration and deregistration mechanism (using a standard Application Programming Interface (API)) is provided. When registering an MR, corresponding MPT and MTT entries are created; when deregistering an MR, the corresponding MPT and MTT entries are destroyed.

[0071] For example, when registering a memory region specified by {VA, Length} as an MR, the RDMA software stack requests and initializes the MTT table and generates an MKEY. Based on this MKEY, an MPT table entry is created. The MPT table entry includes, but is not limited to: the starting address (Base) of the MR, the size of the MR, the open read / write permission identifier (Permission), and a pointer to the MTT table (MTT Pointer).

[0072] Figure 2a is an exemplary IOVA addressing diagram. Please refer to Figure 2a. In this scenario, the page size is 4KB as an example. The process of the RDMA network card performing authorization verification and memory address translation after receiving a packet includes, but is not limited to, the following steps:

[0073] 1. Extract information such as IOVA, Length, and Mkey from the message, and use MKey to index the corresponding MPT table entry.

[0074] Specifically, the network interface card (NIC) receives access packets (also known as access requests, access messages, access information, etc., which are not limited to this application and will not be repeated below) through the network interface. The packets include, but are not limited to, information such as IOVA, Length, Mkey, and opcode information. The NIC can look up the corresponding MPT entry based on the Mkey (e.g., the gray shaded part).

[0075] 2. Access permission verification, including address range verification, read and write permission verification, etc.

[0076] Specifically, the network interface card (NIC) performs access permission verification on this access operation based on the access permission information in the MPT table entry to verify whether it has access permission to the memory region. In this embodiment, the access operation includes, but is not limited to, reading and writing.

[0077] 3. After the permission verification is passed, the corresponding PA address is obtained by looking up the corresponding MTT table.

[0078] Specifically, after successful access verification, the network interface card (NIC) can locate the corresponding MTT table based on the MTT Pointer found in the MPT table entry. Then, based on IOVA and the Base in the MPT table entry, it locates the corresponding MTT table entry (e.g., the gray shaded area). The content of the MTT table entry indicates the physical address of the page (i.e., the starting physical address of the page). The NIC can obtain the corresponding PA based on the offset in IOVA and the starting physical address of the page. This address translation can be expressed as: PA = Function((IOVA - Base) >> 12) + IOVA & 0xfff. Here, IOVA & 0xfff is the offset in IOVA, and Function() queries the MTT table to obtain the page's physical address.

[0079] Figure 2b illustrates an exemplary MTT representation. For example, the MTT table supports one or more levels. The RDMA software stack selects the corresponding structure based on the size of the registered MR. Figure 2b illustrates a two-level MTT structure; this application does not limit the number of levels. Referring to Figure 2b, the specific MTT table creation process is as follows:

[0080] 1. The RDMA software stack allocates a memory page for MTT0.

[0081] 2. The RDMA software stack allocates a memory page for each of MTT1-0, MTT1-1, ..., MTT1-n, and saves the address of the memory page to the corresponding location in MTT0. For example, if the memory page table address of MTT1-0 is PA 0-0, the RDMA software stack saves PA 0-0 to MTT0.

[0082] 3. The RDMA software stack writes the PA address of the corresponding memory page of MR to the corresponding positions of MTT1-0, MTT1-1, ..., MTT1-n respectively.

[0083] For example, the RDMA software stack initializes MPT entries. Specifically, the RDMA software stack generates an MKEY. Based on this MKEY, an MPT entry is created, which includes, but is not limited to: the starting address (Base) of the memory location (MR), the size of the MR, the permission flag for open read / write access, and a pointer to the MTT table (MTT Pointer). The RDMA software stack saves the physical address of the memory page of MTT0 to the MTT Pointer field of the MPT entry.

[0084] In another possible implementation, the Unified Bus (UB) network interface card (NIC) uses the UMMU to implement the IOMMU function, providing a memory segment (MS) registration and deregistration mechanism (standard API interface). When registering an MS, corresponding TCT (Target Context Table) entries and MAPT (Memory Address Permission Table) tables are created; when deregistering an MS, the corresponding TCT entries and MAPT tables are destroyed. This example uses only the UB NIC; in other examples, it could be the host's bus interface or other UB I / O, which is not limited in this application and will not be repeated below.

[0085] For example, when registering a memory region specified by {VA, Length} as a MS, the UB software stack sets the Unified Bus Address (UBA) to equal the VA and generates a TID. Here, the IOVA used in the UB scenario is represented as UBA. A TCT entry is created based on this TID. The TCT entry includes, but is not limited to, a pointer to the MATT table and a pointer to the MAPT table. The UB software stack configures the open read / write permission identifier (Permission) in the MAPT table. The UMMU can use the MATT table for address translation based on UBA and the MAPT table for access permission verification. For example, the UMMU supports sharing page tables with the MMU; in this example, the MATT pointer points to the MMU page table of the corresponding process.

[0086] Figure 3a is an exemplary diagram of IOVA addressing. Please refer to Figure 3a. In this scenario, the page size is 4KB as an example. The process of the UB network card performing permission verification and memory address translation after receiving the packet includes, but is not limited to, the following steps:

[0087] 1. Extract information such as dstEID, UBA, Length, TID, and opcode from the message. Use dstEID to find the corresponding TCT pointer in the TECT table, and use TID to index the corresponding TCT table entry.

[0088] Specifically, the UB network card receives access packets through its network interface. These packets include, but are not limited to, information such as dstEID, UBA, Length, TID, and opcode. The network card looks up the TECT table based on dstEID and obtains a pointer to the TECT table from the corresponding entry. The network card can then find the corresponding TECT entry in the TECT table based on the TID.

[0089] 2. Simultaneously search the MATT table and the MAPT permission control table. If the permission verification passes, return the corresponding PA address.

[0090] Specifically, the network interface card (NIC) looks up the corresponding MAPT table (also known as the access control table) based on the MAPT Pointer in the TCT table entry. It then uses information such as IOVA, Length, and opcode to locate the corresponding MAPT table entry. Access control is then performed based on the content of the MAPT table entry.

[0091] Simultaneously, the network card looks up the corresponding MATT table based on the MATT Pointer in the TCT table entry. It then looks up the corresponding MATT table entry based on the IOVA (e.g., UBA). Finally, based on the IOVA and the address in the MATT table entry, it obtains the corresponding PA.

[0092] Figure 3b illustrates an exemplary MATT representation. For example, the MATT table structure is implemented based on a radix tree. During address translation, three types of tables may be encountered: Page Table, Block, and Page. When a Page Table is found, the search continues to the next level of the page table; when a Block or Page is found, the search ends, and the lookup result PA is obtained directly. Figure 3b uses a 4KB page size as an example. The address translation process includes, but is not limited to:

[0093] 1. Based on MATT Pointer and UBA[47:39], obtain the target entry address PA of the Level0 page table.

[0094] Specifically, MATT Pointer is the PA of the Level 0 page table. The network card calculates the target entry address PA of the Level 0 page table with UBA[47:39] as the offset.

[0095] 2. Read the target entry from the Level 0 page table and determine the type of the target entry.

[0096] In this example, the type is Page Table, and the network card obtains the target table entry address PA of the Level 1 page table by combining the Next-Level Table Address in the target table entry with UBA[38:30].

[0097] 3. Read the target entry from the Level 1 page table and determine the type of the target entry.

[0098] In this example, the type is Page Table, and the target table entry address PA of the Level 2 page table is obtained by combining the Next-Level Table Address in the target table entry with UBA[29:21].

[0099] Optionally, if it is a Block, the PA can be calculated directly from the Block to complete the address translation.

[0100] 4. Read the target entry from the Level 2 page table and determine the type of the target entry.

[0101] In this example, the type is Page Table, and the target table entry address PA of the Level 3 page table is obtained by combining the Next-Level Table Address in the target table entry with UBA[20:12].

[0102] Optionally, if it is a Block, the PA can be calculated directly from the Block to complete the address translation.

[0103] 5. Read the target entry from the Level 3 page table and complete the address translation.

[0104] For example, for a Page type, the network card obtains the PA based on the address indicated by the Page (used to indicate the address of the physical page) and UBA[11:0] (used to indicate the physical page offset).

[0105] (12) Collective Communication (COLL): Collective communication (COLL), also known as group communication or aggregate communication, is a collective communication behavior in which multiple processes running on multiple computing resources in a computing cluster participate in communication to form a process group in order to execute computing tasks. Collective communication defines a series of standard information exchange interfaces to solve the communication problem between different processes in parallel computing.

[0106] The term "collective communication operation," used in this article and also simply referred to as "collective communication" or "collective operation," refers to the concept of data being sent to or received from multiple compute nodes simultaneously in a computer cluster. In other words, performing each collective communication operation involves collaboration among compute nodes. Each collective communication operation consists of multiple point-to-point communications executed concurrently between compute nodes.

[0107] (13) Communication domain: Collective communication occurs on a set of communication objects. The communication domain is the context in which the collective communication operator is executed, and it manages the corresponding communication objects and the resources required for communication.

[0108] Figure 4 is a schematic diagram of an exemplary communication system. Referring to Figure 4, the computer cluster includes, but is not limited to, multiple computing nodes. Each computing node includes, but is not limited to, a processor, memory, and communication I / O interfaces. The communication I / O can be integrated I / O of the processor SoC, such as an RDMA network card, a Unified Bus (UB) network card, or a UB controller bus interface, etc., which is not limited in this application. Optionally, the communication I / O in the embodiments of this application can also be an external independent network card, such as an RDMA network card, a UB network card, etc., which is not limited in this application. Exemplarily, each computing node is managed by an OS and uses communication I / O to connect to an interconnected network structure.

[0109] As used herein, the term "computing node" is defined as a module that participates in a specific point-to-point (or node-to-node) communication as part of a collective communication operation. Accordingly, multiple computing nodes execute multiple point-to-point communications in parallel, thereby performing a collective communication operation as a whole. In the embodiments of this application, a computing node may be a standalone server or computer device, or it may be one of the computing units within a server or computer device; this application does not impose any limitations.

[0110] In applications such as artificial intelligence (AI), as AI models grow larger, the computing resources of a single computing device are insufficient to support their operation. Multiple computing devices need to be interconnected to form a computing cluster to collectively support the AI ​​model's operation. During AI model execution, multiple computing nodes within the cluster need to frequently exchange data, leading to a sharp increase in the demand for memory access bandwidth from processors and communication I / O. Therefore, in addition to increasing memory bandwidth, reducing inefficient memory accesses, minimizing memory bandwidth contention between computing and communication I / O, and improving the effective utilization of memory bandwidth have become urgent problems to be solved.

[0111] In existing technologies, memory access is typically achieved by setting up an additional Cluster Communication Library Buffer (CCL Buffer) or by exchanging virtual addresses of memory via communication messages. The former incurs additional data copying overhead, while the latter increases communication overhead.

[0112] The following is a brief explanation of the existing technology:

[0113] 1. CCL Buffer Method

[0114] Figure 5 is an exemplary schematic diagram of a collective communication process. The cluster communication library performs collective communication operations within a communication domain, and each communication domain includes at least two compute nodes. The scenario shown in Figure 5 illustrates the collective communication process between two compute nodes in a computer cluster, demonstrating the main communication and interaction processes. Depending on the communication algorithm used, one node can act as an initiator and the other as a target; the two compute nodes can also act as both initiator and target for each other. This application does not impose any limitations on this. Referring to Figure 5, the collective communication process between compute nodes specifically includes, but is not limited to, the following steps:

[0115] ① Create a communication domain, allocate a CCL buffer, and perform memory registration.

[0116] Specifically, each compute node (e.g., the Initiator and Target in the diagram) creates a communication domain in its communication library and allocates a CCL Buffer for that domain. The size of the CCL Buffer is specified by the system. After the CCL Buffer is allocated, the mapping relationship between the virtual address (VA) and physical address (PA) corresponding to the CCL Buffer can be obtained, and this mapping relationship can be saved to a mapping table (e.g., the MMU page table managed by the MMU).

[0117] For example, the compute node registers the CCL Buffer in memory. Optionally, if RDMA communication is used, the registered memory region is MR. If UB communication is used, the registered memory region is MS. After memory registration is completed, the mapping relationship between the CCL Buffer's IOVA and PA is obtained, that is, the mapping relationship between the IOVA address and the physical address, which can be stored in the IOMMU page table. A detailed description of the memory registration process can be found above and will not be repeated here. Optionally, in existing technologies, IOVA is equal to VA.

[0118] ② Computing nodes communicate CCL Buffer information to each other.

[0119] For example, the communication libraries of compute nodes (including Initiator and Target) mutually announce their CCL Buffer information. This CCL Buffer information includes, but is not limited to: a token (where the token is Mkey when using RDMA communication and TID when using UB communication), and the CCL Buffer's IOVA (the IOVA address can be VA when using RDMA and UBA when using UB communication). In this way, each compute node obtains the CCL Buffer information of other compute nodes within the communication domain and can access the corresponding compute node's CCL Buffer based on this information.

[0120] ③ Dynamically allocate the Data Buffer, perform memory registration, and start the computation.

[0121] For example, the computing framework of each computing node triggers a process of dynamically allocating Data Buffers, also known as dynamic memory allocation, generating corresponding MMU page tables. The entries in these page tables indicate the mapping relationship between the physical address (PA) and virtual address (VA) of the Data Buffer. Specifically, functions such as malloc and free can be used to perform dynamic memory allocation and deletion operations. Specific implementation details can be found in existing technologies and will not be elaborated here.

[0122] For example, the Data Buffer is registered in memory, a corresponding IOVA (i.e., a contiguous segment of IO virtual addresses) is allocated from the IOVA virtual address space, and a corresponding IOMMU page table is established. The IOMMU page table includes entries indicating the mapping relationship between the IOVA of the Data Buffer and the PA of the Data Buffer. Here, IOVA and VA are equal. In this way, communication IO can achieve memory address translation based on the IOMMU page table.

[0123] For example, the computing unit executes a computing operator, which calculates local data to generate new data. The specific steps can be found in existing technologies and will not be repeated here.

[0124] ④ Calculate the output data.

[0125] Specifically, Figure 5 illustrates this using the example of the Initiator accessing the memory of the Target node, where the access operation is a write operation. Specifically, after the computation operator completes its execution, the Initiator's computation unit writes the data to be sent to the Target into its local Data Buffer.

[0126] ⑤ The computing framework calls the communication operator.

[0127] For example, the computational frameworks of the Initiator and Target initiate a collection communication operation.

[0128] ⑥ The Initiator writes data to the Target's CCL Buffer.

[0129] Specifically, the Initiator's communication I / O reads data from the Data Buffer and sends a write request to the Target. This request includes, but is not limited to, a write operation, data, IOVA, Length, and Token. The Target responds to the received write request, writing the data to the physical memory corresponding to the CCL Buffer based on the IOVA and Token. Specifically, the communication I / O receives the write request and obtains the IOVA, Length, and Token information carried in the message. The communication I / O can then look up the IOMMU page table based on the Token and IOVA to obtain the corresponding PA address and write the data carried in the message to the physical memory located at the PA address. The specific addressing process can be referred to above and will not be repeated here.

[0130] ⑦ Local data copy.

[0131] Specifically, the Target reads data from the CCL Buffer and writes it to the Data Buffer. Specifically, the Target can use a DMA engine, load / store instructions, etc., to copy data from the CCL Buffer to the Data Buffer. The specific process can be found in existing technologies and will not be elaborated here.

[0132] ⑧ Perform the next iteration of calculation.

[0133] Specifically, the computation unit retrieves the data to be written (i.e., the data previously written to the Data Buffer) from the Data Buffer, performs the next iteration of computation, and writes the computation result back to the Data Buffer. This process is merely an illustrative example; the specific computation execution process can be set according to actual needs, and this application does not impose any limitations.

[0134] Optionally, when running a task, steps ① and ② above are executed only once, while steps ③ to ⑧ may be executed once or multiple times.

[0135] For example, Figure 5 illustrates the writing scenario. In the reading scenario, for instance, when the Initiator reads data from the Target, steps ① to ③ of the execution flow can be seen in Figure 5. Specifically, during the data reading process, the Target copies (i.e., reads and writes) the data (i.e., the data to be read) from the Data Buffer to the CCL Buffer. The Target responds to the read request from the Initiator, reads the data from the CCL Buffer, and sends it to the Initiator. The Initiator writes the data to the Data Buffer and performs calculations. Other details not described can be found in Figure 5 and will not be repeated here.

[0136] For example, a key reason why programs use dynamic memory allocation is that the size of certain data structures is only known during actual program execution. However, collection communication requires prior knowledge of the buffer information. The collection communication process in the above scenario uses a known CCL buffer for data communication, avoiding the need to announce dynamic Data Buffer information during each collection communication operation, thus avoiding additional communication overhead. However, this method involves local data copying, consuming additional memory bandwidth. Furthermore, due to system memory capacity limitations, the CCL buffer space is limited. When the CCL buffer space is smaller than the amount of communication data, the communication algorithm needs to slice the data and perform multiple rounds of communication, resulting in high algorithm complexity and potentially increased communication jitter.

[0137] 2. Message Passing Interface (MPI) standard: Rendezvous communication mechanism.

[0138] Figure 6 illustrates an exemplary aggregate communication process. Referring to Figure 6, when compute nodes communicate aggregatedly, they first send control information to the peer to announce their Data Buffer information (i.e., IOVA address and Token). Compute nodes can then perform access operations based on the acquired Data Buffer information, such as reading data. After completing the access, a data transmission completion notification (e.g., a FIN packet, indicating that the data transmission for this communication has been completed) is sent. This mechanism avoids local data copying. However, a single communication process between two nodes introduces two RDMA Send message exchanges of control information (i.e., control information and FIN message). Since an aggregated communication operation requires N data exchanges between M nodes, a total of 2N control information exchanges are introduced, making it difficult to control the time jitter of aggregated communication operations. Furthermore, the exchange, preprocessing, and post-processing of control information are best handled by CPU software; software involvement will introduce system noise.

[0139] This application provides an access control method that maps memory to a pre-configured Cluster Communication Library Window (CCL Window) to avoid the overhead of local data copying and improve memory access bandwidth utilization. Furthermore, it effectively reduces communication jitter by eliminating the need for dynamic exchange of control information.

[0140] Figure 7 is a flowchart illustrating an exemplary access control method. Referring to Figure 7, the specific steps include, but are not limited to, the following:

[0141] S701, the first compute node creates a cluster communication library window (CCL Window), wherein the CCL Window is allocated a subset of the IO virtual address space and the CCL Window is not allocated physical memory; the subset of the IO virtual address space is allocated from the input / output IO virtual address space and includes at least one IO virtual address.

[0142] For example, the first computing node creates a CCL Window for the corresponding communication domain. Specifically, the first computing node allocates a subset of the IO virtual address space (also referred to as an IO virtual address region or IO virtual memory, etc., which is not limited in this application) for the CCL Window from the IO virtual address space. The subset includes at least one IOVA. For example, the subset of the IO virtual address space may optionally be a contiguous segment of IOVAs (i.e., including multiple IOVAs with contiguous addresses). It can be understood that, as described above, the IO virtual address space may optionally be an ordered set of IOVAs, and the first computing node partitions a subset from the set for the CCL Window, the subset including multiple contiguous IOVAs.

[0143] In this embodiment, the first computing node does not allocate physical memory (which can also be understood as main memory, physical memory resources, etc., which are not limited in this application) to the CCL Window. Optionally, unlike the method of creating a Buffer described above, after the Buffer is created, the Buffer corresponds to physical memory, and this part of physical memory corresponds to a PA address, VA, and IOVA. In this embodiment, the CCL Window corresponds to an IO virtual address but is not allocated physical memory. That is, the CCL Window created by the first computing node corresponds to an IO virtual address but is not mapped to the PA. Optionally, some related parameters of the CCL Window (such as CCL Window information and IOMMU page tables, etc.) can be understood as system overhead.

[0144] For example, the size of the CCL Window is preset, and its size is greater than or equal to the length of the subsequent communication data (which can also be referred to as access data in this embodiment, such as write data or read data, which is not limited in this application). Optionally, in this embodiment, the CCL Window can be set to a maximum value allowed by the system to avoid the problem of the CCL Window being smaller than the communication data, which would require data to be segmented.

[0145] In one possible implementation, the first compute node creates a CCL Window and generates a corresponding IOMMU page table. The IOMMU page table may include, but is not limited to, address translation entries indicating the mapping relationship between the CCL Window's I / O virtual address (IOVA) and its physical address (PA). As mentioned above, the CCL Window has not yet been allocated physical memory; that is, the CCL Window's IOVA is not currently mapped to any PA. In this embodiment, the physical address in the address translation entry of the IOMMU page table is currently invalid, for example, it can be all 0s. This can be set according to actual needs, and this application does not impose any limitations.

[0146] This can be understood as the entry originally used to indicate the mapping relationship between IOVA and PA being invalid, meaning IOVA is not mapped to PA. In subsequent processing of this instance, the first compute node can fill the contents (i.e., the physical address) of the entry in the MMU page table indicating the mapping relationship between VA and PA into the IOMMU page table of the CCL Window, thus updating the invalid content in the aforementioned entry to make the mapping relationship between IOVA and PA of the CCL Window effective, thereby realizing the mapping between the CCL Window and physical memory. Details are provided below and will not be elaborated upon here.

[0147] In another possible implementation, the first compute node creates a CCL Window and generates a corresponding IOMMU page table. The IOMMU page table may include, but is not limited to, offset information, the current content of which is also invalid. In this scenario, during subsequent processing, the first compute node can update the contents of the offset information table entries, so that the updated offset information is used to indicate the offset (i.e., the difference) between the VA corresponding to the target physical memory mapped onto the CCL Window and the IOVA of the CCL Window, or, to indicate the offset (i.e., the difference) between the IOVA corresponding to the target physical memory mapped onto the CCL Window (e.g., UBA) and the IOVA of the CCL Window.

[0148] Optionally, the first computing node can be any node in the computer cluster, or a computing unit within any node. For example, each computing node in the computer cluster that needs to perform a collective communication operation can execute the operation shown in Figure 7 to achieve collective communication operations between different computing nodes.

[0149] S702, the first computing node sends the IO virtual address information of the CCL Window to the second computing node. The IO virtual address information is used to indicate a subset of the IO virtual address space of the CCL Window.

[0150] For example, the first computing node may send its CCL Window information (referred to as CCL Window information) to other computing nodes in the computing device cluster (also known as a computer cluster, computing node cluster, etc., which are not limited in this application). The CCL Window information includes, but is not limited to: the CCL Window's Token, the CCL Window's IO virtual address information, etc.

[0151] For example, the token of a CCL window can be an Mkey or a TID, etc. The concept can be referred to above, and will not be repeated here.

[0152] For example, the I / O virtual address information of a CCL Window is used to indicate a subset of the I / O virtual address space corresponding to the CCL Window. The I / O virtual address information of a CCL Window includes, but is not limited to: the starting address information of the CCL Window (which can also be understood as the starting address information of the subset of the I / O virtual address space corresponding to the CCL Window), and the size information of the CCL Window (which can also be understood as the size information of the subset of the I / O virtual address space corresponding to the CCL Window). The starting address information of the CCL Window may optionally be the starting IOVA (i.e., the first IOVA) of the subset of the I / O virtual address space corresponding to the CCL Window. The size of the CCL Window is used to indicate the size (or dimensions) of the CCL Window, which can also be understood as indicating the size of the subset of the I / O virtual address space.

[0153] In this way, other nodes in the cluster can perform memory access operations with the first compute node based on the CCL Window information.

[0154] Optionally, as described above, the first node can be any node in the cluster. That is, after each node in the cluster creates its own CCL Window, it can send the relevant information of its CCL Window to other nodes in the cluster. In this way, each node in the cluster can obtain the relevant information of the CCL Window of at least one other computing node in the cluster.

[0155] Optionally, the first computing node can send CCL Window related information to other computing nodes by proactively sending an instruction message after creating the CCL Window, the message including, but not limited to, CCL Window related information. Alternatively, any node requiring aggregate communication with the first computing node can send a CCL Window request to the first computing node, and the first computing node can respond to the request by sending a response message to the corresponding node, the message including, but not limited to, CCL Window related information. The interaction method for CCL Window related information can be set according to actual needs, and this application does not impose any limitations.

[0156] S703, the first computing node establishes a mapping relationship between the physical address of the target physical memory and a subset of the IO virtual address space of the CCL Window, so that the second computing node can access the target physical memory based on the IO virtual address information of the CCL Window.

[0157] For example, the first computing node can dynamically allocate target physical memory from physical memory. In this embodiment, the target physical memory can be referred to as a Data Buffer. The first computing node maps the dynamically allocated Data Buffer to a CCL Window so that other computing nodes (e.g., the second computing node) can access the target physical memory of the first computing node based on the IO virtual address information of the CCL Window, that is, access the Data Buffer mapped to the CCL Window.

[0158] In this embodiment, the mapping between the CCL Window and physical memory (e.g., the target physical memory in this embodiment) can also be understood as a mapping between a subset of the CCL Window's IO virtual address space and the PA of the Data Buffer. In this embodiment, the first computing node can establish a mapping relationship between the IOVA of the CCL Window (i.e., a continuous IOVA contained in the subset of the IO virtual address space) and the physical address (PA) of the Data Buffer.

[0159] In one possible implementation, the Data Buffer corresponds to a virtual address (VA), which comes from the virtual address space. The virtual address space and the IO virtual address space are two independent address spaces. Thus, in this embodiment, the IO virtual address corresponding to the target physical memory that can be accessed by other nodes is the IOVA of the CCL Window. The IOVA and the VA of the target physical memory can be different; that is, the IOVA and VA of bytes pointing to the same physical memory can be different.

[0160] During memory access, any other node in the cluster (such as the second compute node) can send an access request to the first compute node. The access request is used to indicate that the physical memory in the target physical memory is mapped to the target IOVA address.

[0161] Optionally, the access operation requested by the access request can be at least one of a read operation, a write operation, or an atomic operation.

[0162] Optionally, the access request may include, but is not limited to, a token, a target IOVA address, and a length. In one example, if the access request is a write request, the request may also include the data to be written; this application does not impose any limitations on this.

[0163] For example, the target IOVA address and Length are used to indicate a contiguous region of addresses in the CCL Window, where the target IOVA address is the starting address of the region and the Length is the size of the region.

[0164] For example, as shown in Figure 8, the target physical memory in the first compute node (e.g., the physical memory of the Data Buffer) is mapped to a subset of the CCL Window's IO virtual address space. That is, a mapping relationship is established between the CCL Window's IOVA and the target physical memory's PA. When the first compute node receives an access request, which includes, but is not limited to, information about the target IO virtual address, the request requests access to the physical memory mapped to the target IOVA address (e.g., the first physical memory in the figure). The target IO virtual address is a contiguous address region of the CCL Window. Based on the mapping relationship between the subset of the CCL Window's IO virtual address space and the target physical memory's PA, the first compute node can determine the mapping relationship between the target IO virtual address and the first physical memory within the target physical memory, and perform access operations on the first physical memory, such as writing received data into the first physical memory.

[0165] Optionally, the size of the target physical memory can be less than or equal to the size of the CCL Window, and can be set according to actual needs; this application does not impose any restrictions.

[0166] In one possible implementation, other nodes (which may be the same compute node or different compute nodes) may access the target physical memory of the first compute node multiple times, for example, by writing data to the target physical memory multiple times. In this example, the first compute node may optionally receive multiple access requests, each request message carrying IO virtual address information to indicate a contiguous region of the CCL Window. Optionally, if multiple access requests request access to the same target physical memory, the target IO virtual address indicated in each access request is different; that is, the starting IOVA of the CCL Window requested by the access requests is different, and the lengths may be the same or different, which is not limited in this application.

[0167] In one possible implementation, as described above, the first compute node creates a CCL Window and generates a corresponding IOMMU page table. The IOMMU page table may include, but is not limited to, address translation entries indicating the mapping relationship between the IOVA of the CCL Window and the PA of the target physical memory. The physical address in these address translation entries is currently invalid. During the mapping process between the CCL Window and physical memory, the first compute node can write the content (i.e., the physical address) from the entry in the MMU page table indicating the mapping relationship between physical memory addresses and virtual addresses to the corresponding position in the IOMMU page table of the CCL Window, thereby achieving the mapping between the CCL Window and physical memory. Specifically, the MMU page table includes, but is not limited to, entries indicating the mapping relationship between the VA of the target physical memory and the PA of the target physical memory, and the content of these entries includes, but is not limited to, the physical address of the Page corresponding to the target physical memory (e.g., the starting PA of the Page). Accordingly, the first compute node copies the contents of the entry in the MMU page table that indicates the mapping relationship between VA and PA (i.e., the physical address of the Page, the valid bits and other information do not need to be copied) to the corresponding position in the IOMMU page table of the CCL Window. This allows the first compute node to find the target PA (e.g., the PA corresponding to the first physical memory in Figure 8) based on the IOMMU page table during memory access.

[0168] In another possible implementation, the first compute node creates a CCL Window and generates a corresponding IOMMU page table. The IOMMU page table may include, but is not limited to, offset information, the current content of which is also invalid. During the mapping process between the CCL Window and physical memory, the first compute node can update the content in the offset information. In one example, the updated offset information indicates the offset between the VA of the target physical memory and the IOVA of the CCL Window, and can be expressed as: Offset = VA - IOVA. In another example, the updated offset information indicates the offset between the IOVA of the target physical memory (e.g., UBA) and the IOVA of the CCL Window, and can be expressed as: Offset = UBA - IOVA.

[0169] In this way, during memory access, after the first compute node obtains the target IOVA of the CCL Window that other nodes want to access, it can obtain the corresponding VA or UBA based on IOVA+Offset, and further obtain the corresponding PA based on VA or UBA by looking up the corresponding page table (MMU page table or UMMU page table, where the UMMU page table can also be understood as the IOMMU page table of physical memory).

[0170] In summary, the computing node in this embodiment can access other computing nodes through the CCL Window. Referring to Figure 9 for comparison and analysis, as shown in Figure 9, in existing technologies, the access operations received by the computing node's communication I / O include, but are not limited to, the IOVA of the CCL Buffer, used to request access to the CCL Buffer. The computing node can obtain the PA that has a mapping relationship with the IOVA by querying the IOMMU page table. The computing node's communication I / O can access the physical memory corresponding to the PA in the CCL Buffer, for example, writing data into that physical memory space. The computing node also needs to perform local copying of the data; for example, in a write scenario, it needs to read the data from the CCL Buffer and write it into the physical memory corresponding to the Data Buffer. Referring again to Figure 9, in the access control scheme provided in this application, computing nodes can allocate corresponding IOVAs for CCL Windows from the IO virtual address space. Other nodes can access the physical memory corresponding to the Data Buffer that is mapped to the CCL Window based on the CCL Window's IOVA. In essence, this application maps the physical memory corresponding to the Data Buffer to the CCL Window, enabling dynamically allocated Data Buffers to use the CCL Window's IOVA for access by other nodes. Other computing nodes can directly access the Data Buffer using the pre-obtained IOVA corresponding to the CCL Window, avoiding the additional overhead of data copying, saving system resources, and effectively improving memory access efficiency. Furthermore, there is no need to transmit additional control information to inform the Data Buffer of relevant information, reducing communication time. Furthermore, CCL Window does not allocate physical memory resources and is not limited by the system's physical memory capacity. By setting the size of CCL Window as large as possible, making the size of CCL Window greater than or equal to the size of the communication data (i.e., accessed data), computing nodes do not need to perform multiple rounds of communication by slicing data during the communication process, which can effectively reduce algorithm complexity and reduce communication jitter.

[0171] The process in Figure 7 will be explained in detail below with reference to specific access scenarios.

[0172] Figure 10 is an exemplary access control flow diagram. The cluster communication library performs aggregate communication operations within a communication domain, and each communication domain includes at least two compute nodes. The scenario shown in Figure 10 illustrates the aggregate communication flow between two compute nodes in a computer cluster, demonstrating the main communication and interaction processes. Depending on the communication algorithm used, one node can act as an Initiator and the other as a Target; the two compute nodes can also act as both Initiator and Target, which is not limited in this application. This scenario illustrates a write access operation performed by the Initiator node to the Target node, as shown in Figure 10, and includes, but is not limited to, the following steps:

[0173] ① Create a communication domain and create a CCL Window.

[0174] Specifically, each compute node (e.g., including the Initiator and Target in the diagram) creates a communication domain in its communication library and creates a CCL Window for that communication domain. Optionally, each compute node may create at least one communication domain, and each communication domain may contain one CCL Window. That is, access operations (which may be performed once or multiple times) by the compute node within that communication domain can be performed based on that CCL Window.

[0175] Specifically, a compute node can partition a subset of the I / O virtual address space from the I / O virtual address space based on a pre-set CCL Window size and allocate it to the CCL Window. The subset of the I / O virtual address space is a contiguous IOVA within the I / O virtual address space.

[0176] The compute node obtains CCL Window information. For example, a CCL Window corresponds to CCL Window information, which includes, but is not limited to: the CCL Window's token and its I / O virtual address information. For example, the CCL Window's token can be an Mkey or a TID, etc., the concepts of which are explained above and will not be repeated here.

[0177] For example, the I / O virtual address information of a CCL Window is used to indicate a subset of the I / O virtual address space corresponding to the CCL Window. The I / O virtual address information of a CCL Window includes, but is not limited to: the starting address information of the CCL Window (which can also be understood as the starting address information of the subset of the I / O virtual address space corresponding to the CCL Window), and the size information of the CCL Window (which can also be understood as the size information of the subset of the I / O virtual address space corresponding to the CCL Window). The starting address information of the CCL Window may optionally be the starting IOVA (i.e., the first IOVA) of the subset of the I / O virtual address space corresponding to the CCL Window. The size of the CCL Window is used to indicate the size (or dimensions) of the CCL Window, which can also be understood as indicating the size of the subset of the I / O virtual address space.

[0178] For example, a CCL Window is not allocated physical memory resources, which can also be understood as the CCL Window's IOVA not being mapped to any PA.

[0179] For example, the size of the CCL Window is greater than or equal to the size of the access data corresponding to the access operation in subsequent communication processes to avoid data segmentation. Data segmentation can be understood as the process where, when the size of the Data Buffer requested by the requesting end (e.g., the Initiator) is greater than the size of the CCL Window, the requesting end needs to segment the data so that the access data in a single communication is less than or equal to the size of a subset of the CCL Window's I / O virtual address space. Optionally, in this embodiment, the CCL Window can be set to a maximum value allowed by the system to avoid the problem of the CCL Window being smaller than the communication data (i.e., the access data), which would require data segmentation.

[0180] The following is a detailed explanation of the creation process of a CCL Window using a specific example.

[0181] In this example, multiple computing nodes in the computing cluster are interconnected using RDMA network cards, which can be standalone network cards attached to the processor or integrated network cards of the processor SoC.

[0182] The communication library calls the user-space network card driver to create a CCL Window. Specifically, the communication library calls the user-space network card driver's CCL Window creation interface, for example: (iova, mkey) = create_window(size). The parameter size specifies the size of the CCL Window.

[0183] In response to the call instruction, the user-mode network card driver switches to kernel mode to execute the CCL Window creation operation. Specifically, the user-mode network card driver triggers the kernel-mode network card driver to create the CCL Window. Optionally, the CCL Window can be destroyed after the current collection communication process is completed. The size of each recreated CCL Window can be the same or different; this application does not impose any limitations on this.

[0184] For example, the kernel-mode network interface card (NIC) driver begins creating a CCL Window. Specifically, the kernel-mode NIC driver allocates an Mkey to the CCL Window and allocates a subset of the I / O virtual address space for the CCL Window from the I / O virtual address space. In this embodiment, the I / O virtual address space is an independent address space, which is an ordered set of I / OVAs.

[0185] For example, taking a 4KB page size and a 2-level MTT table as an example, as shown in Figure 11. The number of MTT table levels is determined based on the size of the CCL window. Specifically, the kernel-mode network card driver requests the memory resources required for the MTT table, including, for example, the first-level page table MTT0 and the second-level page table MTT1 (including MTT1-0 to MTT1-n).

[0186] For example, the kernel-mode network card driver allocates and initializes the MTT table. Specifically, the kernel-mode network card driver allocates a memory page for MTT0. The kernel-mode network card driver allocates a memory page for MTT1-0 to MTT1-n respectively. The physical addresses in the entries of the MTT1-0 to MTT1-n page tables that indicate the mapping relationship between the CCL Window's IOVA and PA are currently invalid values, as shown in Figure 11 (each entry may also include other information, not shown in the figure). This can be understood as follows: the current CCL Window's IOVA is not mapped to PA, so there is no mapping relationship. Therefore, the content of the entry that should originally be written to PA is currently invalid, meaning the corresponding IOVA does not point to any PA. Optionally, invalid content can be all 0s or other invalid values; this application does not limit this.

[0187] The kernel-mode network interface card (NIC) driver initializes the MTT0 page table. Specifically, the kernel-mode NIC driver saves the physical addresses of the memory pages in the second-level page table MTT1 (i.e., the PA addresses corresponding to the memory pages, such as PA0-0 to PA0-n) to the corresponding locations in MTT0, as shown in Figure 11. Accordingly, during memory access, communication I / O (e.g., an RDMA NIC in this example) can look up the corresponding MTT second-level page table based on the PA addresses saved in MTT0.

[0188] The kernel-mode network card driver initializes the MPT table entry. Specifically, the kernel-mode network card driver creates the MPT table entry based on Mkey, saving the address of the memory page of MTT0 (i.e., the PA of the page where MTT0 is located) to the MTT Pointer field of the MPT table entry. Optionally, the MPT table entry may also include, but is not limited to: the starting address (Base) of MR is the starting IOVA of CCL Window, the size (Size) of MR is the size of CCL Window, and other permission information can be set according to actual needs, which is not limited in this application.

[0189] Optionally, the kernel-mode network card driver can record the correspondence between the starting IOVA and Mkey of the CCL Window.

[0190] For example, after the kernel-mode network card driver is created, it returns the execution result to the user-mode network card driver. The execution result includes, but is not limited to, the initial IOVA of the CCL Window and information such as Mkey (i.e., token).

[0191] For example, the user-mode network card driver reports the execution result to the communication library. For example, the communication library can obtain CCL Window information, including but not limited to: the CCL Window's initial IOVA, the CCL Window's Size, and Mkey, etc. The communication library stores the correspondence between the above information.

[0192] It should be noted that Figure 11 only shows the mapping content of the physical address; other information is not shown. Existing technologies can be referenced, and this application does not limit it.

[0193] ② Mutual notification of CCL Window information.

[0194] For example, the communication libraries of compute nodes (including Initiator and Target) mutually announce their CCL Window information. CCL Window information includes, but is not limited to: a token (where the token is Mkey when using RDMA communication and TID when using UB communication), and the CCL Window's I / O virtual address information. In this way, each compute node obtains the CCL Window information of other compute nodes within the communication domain and can access the corresponding compute node's Data Buffer based on the CCL Window information.

[0195] ③ Dynamically allocate the Data Buffer and start the calculation.

[0196] For example, the computing framework of each computing node triggers a process of dynamically allocating a Data Buffer, which can also be called dynamic memory allocation. Specifically, the details may differ between systems, but without loss of generality, the CPU software (specifically, a dynamic memory allocator) can call relevant functions (such as malloc) to create a Data Buffer based on the commands of the computing framework (e.g., instructions to indicate the required Buffer size) and perform operations such as pinning memory (i.e., locking memory to prevent it from being swapped). For specific implementation details, please refer to existing technologies, which will not be elaborated here.

[0197] For example, the compute node allocates corresponding physical memory for the Data Buffer and allocates a subset of its virtual address space from the virtual address space. This subset of the Data Buffer's virtual address space is a contiguous VA (Version Area) allocated from the virtual address space. The virtual address space and the I / O virtual address space are two independent address spaces.

[0198] For example, a compute node can access the MMU page table. The MMU page table includes entries indicating the mapping between a subset of the virtual address space of the Data Buffer and the physical memory of the Data Buffer; it can also be represented as a mapping between the Virtual Address Space (VA) and Physical Address Space (PA) of the Data Buffer. For instance, Figure 12 is an exemplary diagram of the MMU page table mapping. Referring to Figure 12, the MMU page table includes, but is not limited to, entries indicating the mapping between VA and PA. The content of the entries includes, but is not limited to, the starting PA of the corresponding physical memory page. For example, the physical address PA a-0-0 in the page table is the starting PA of page 0 in the Data Buffer. It can be understood that the corresponding entry in the MMU page table can be found based on the VA as an index, with each starting VA address of a virtual memory page corresponding to one entry. The corresponding PA is further determined by the content stored in the entry.

[0199] For example, the computing unit executes a computing operator, which calculates local data to generate new data. The specific steps can be found in existing technologies and will not be repeated here.

[0200] ④ Calculate the output data

[0201] Specifically, Figure 10 illustrates this using the example of the Initiator accessing the memory of the Target node, where the access operation is a write operation. Specifically, after the computation operator completes its execution, the Initiator's computation unit writes the data to be sent to the Target into the Initiator's local Data Buffer.

[0202] ⑤ The computing framework calls the communication operator to map the Data Buffer onto the CCL Window.

[0203] For example, the computational frameworks of the Initiator and Target initiate a collection communication operation.

[0204] The Initiator and Target map the physical memory corresponding to the Data Buffer to the CCL Window. Specifically, taking the Target compute node as an example, the Target's communication library calls the communication I / O driver to execute the memory mapping process, so as to map the allocated physical memory to the pre-created CCL Window.

[0205] For example, in the steps above, the compute node creates a CCL Window. This application embodiment provides multiple methods for creating and mapping CCL Windows. In this scenario, only one method for creating a CCL Window and its corresponding mapping method are described as an example. Other methods will be described in detail below.

[0206] Specifically, the communication library calls the mapping interface of the user-space network card driver to execute the mapping process between the CCL Window and the Data Buffer. For example, the communication library calls the user-space network card driver interface: mapping_window(va, length, mkey, iova), which includes, but is not limited to, information such as the starting VA of the Data Buffer and the length of the Data Buffer, to instruct that the physical memory of the Data Buffer specified by {VA, Length} be mapped to the CCL Window. This can be understood as a mapping relationship existing between the VA and PA of the Data Buffer. The mapping between the Data Buffer and the CCL Window can be understood as establishing a mapping relationship between the PA of the physical memory corresponding to a contiguous VA and a subset of the I / O virtual address space of the CCL Window.

[0207] As mentioned above, during the creation process, the driver can record the correspondence between IOVA and Mkey, with IOVA being an optional parameter.

[0208] The user-mode network interface card (NIC) driver switches to the kernel-mode NIC driver. Specifically, the kernel-mode NIC driver can find the corresponding MMU page table based on the process ID, VA, and Length (the specific lookup method can be found in existing technologies, and will not be elaborated here).

[0209] For example, the kernel-mode network card driver updates the MTT table based on the MMU page table, and the physical address in the updated MTT table is the physical address of the Data Buffer.

[0210] Specifically, as shown in Figure 13, the kernel-mode network card driver copies the physical address (i.e., the starting PA of the corresponding Page) from the MMU page table to the corresponding position in the entry of the MTT second-level page table. The copied content is the information indicating the mapping relationship between VA and PA, i.e., the PA of the page (i.e., the starting physical address of the page); other content (such as flag bits) does not need to be copied.

[0211] For example, referring to Figure 13, the kernel-mode network card driver sequentially reads the contents of each entry in the MMU page table and copies the read contents to the corresponding positions in the MTT table. For instance, the physical address "PA a-0-0" (e.g., the starting PA of page 0) in the first entry of the MMU page table is copied to the first entry of MTT 1-0. The copied position is the location where the content was previously invalid. Thus, the updated content in this entry is the physical address of page 0 (i.e., the starting PA), indicating the mapping relationship between the IOVA corresponding to this entry and the PA of Page 0 in the Data Buffer.

[0212] In this embodiment, taking page0 as an example, its corresponding virtual address (VA) comes from the virtual address space, and its corresponding IOVA is the IOVA of the CCL Window, which comes from the IO virtual address space. The virtual address space and the IO virtual address space are two independent spaces. Accordingly, in this embodiment, IOVA and VA can be different, that is, two different addresses (referring to IOVA and VA) can point to the same PA.

[0213] ⑥ The Initiator writes data to the Target's Data Buffer.

[0214] Specifically, the Initiator's communication I / O reads data from its local Data Buffer and sends a write request to the Target. The request includes, but is not limited to: the data to be written (referred to as "data"), the CCL Window's Mkey, and the target IOVA address information. The write request is used to request that the data be written to the physical memory corresponding to the PA that has a mapping relationship with the target I / O virtual address. The target IOVA address information indicates the target I / O virtual address and includes, but is not limited to, the starting IOVA and Length.

[0215] Figure 14a is an exemplary data writing diagram. Referring to Figure 14a, specifically, the Target responds to the received write request and writes data 1 into the first physical memory corresponding to the first IO virtual address. The first IO virtual address is the starting IOVA of a contiguous region within a subset of the CCL Window's IO virtual address space. The first physical memory is located in the physical memory of the Data Buffer. For example, the Target looks up the IOMMU page table (which could be the MPT table in this example) based on the Mkey and finds the corresponding MPT table entry. Further permission verification is performed based on information such as permissions in the MPT table entry, and the corresponding MTT table is then found.

[0216] Furthermore, Target, based on the first I / O virtual address, looks up the IOMMU page table (which in this example could be the MTT table) to determine the PA corresponding to the first I / O virtual address. As mentioned above, the physical address in the entry in the MTT table used to indicate the mapping relationship is the starting PA of the physical memory page corresponding to the Data Buffer. Accordingly, the PA obtained by Target from the MTT table is the PA corresponding to the physical memory page of the Data Buffer. The specific addressing method can be seen in Figure 2a, and will not be elaborated here.

[0217] For example, the Target writes data to the physical memory corresponding to the PA. For instance, as shown in Figure 14a, the Target writes data to the first physical memory corresponding to the first I / O virtual address of the CCL Window.

[0218] Optionally, the physical memory mapped to the first I / O virtual address can be contiguous or discrete. Figure 14a illustrates this only with the example of a contiguous block of physical memory, and this application does not impose any limitations.

[0219] Optionally, as described above, other compute nodes can access the same Data Buffer multiple times. As shown in Figure 14b, the Target receives a write request to write data 2. The target IOVA address information included in the request indicates the second IO virtual address. The Target can find the second physical memory corresponding to the second IO virtual address based on the mapping relationship between the CCL Window and the Data Buffer, and write the data to the second physical memory. It should be noted that the mapping relationship in Figure 14b is only an illustrative example and is not limited in this application. Furthermore, it should be noted that the target IO virtual address accessed multiple times can be set according to actual needs; Figure 14b is only an illustrative example and is not limited in this application.

[0220] In the embodiments of this application, the RDMA network card may cache the MPT and / or MTT in its implementation, which may lead to inconsistencies in page table data. To ensure data consistency, the compute node needs to disable the RDMA network card's caching function for the MTT, or the network card driver may optionally remove the MTT from the cache before updating the MTT table (i.e., before copying the MMU page table).

[0221] ⑦ Perform the next iteration of calculation.

[0222] For a detailed description, please refer to Figure 5, which will not be repeated here.

[0223] Optionally, when running a task, steps ① and ② above are executed only once, while steps ③ to ⑦ may be executed once or multiple times. For example, as shown in Figure 15, the compute node executes steps ③ to ⑦, dynamically allocating a new Data Buffer2 and mapping it to the CCL Window. During the mapping process, the mapping relationship recorded in the IOMMU page table is updated. For instance, the content in the first entry of MTT 1-0 in Figure 13 is updated (which can be understood as deleting the original content and writing new content) to the physical address in the new Data Buffer2's MMU page table entry, thus establishing the mapping relationship between the CCL Window's IOVA and the Data Buffer2's PA. The updated physical address in the first entry of MTT1-0 is used to indicate a physical memory page in Data Buffer2. In this example, the establishment of a mapping relationship between the CCL Window and Data Buffer2 can be understood as the CCL Window and Data Buffer1 having been de-mapped. The process of updating the mapping table can be understood as both updating the mapping relationship, i.e., deleting the original mapping relationship and writing a new one.

[0224] Referring again to Figure 15, the compute node can write data 3 to the third physical memory corresponding to the third I / O virtual address in the CCL Window. The third physical memory is located in the physical memory of Data Buffer 2. Other details not described above can be found above and will not be repeated here.

[0225] For example, Figure 10 illustrates only the writing scenario. In the reading scenario, for instance, when the Initiator reads data from the Target, steps ① to ③ of its execution flow can be seen in Figure 10. Specifically, during the data reading process, the Target writes the data (i.e., the data to be read) into the Data Buffer. The Initiator can read the data from the Target's Data Buffer based on the IOVA of the Target's CCL Window, write it into its own Data Buffer, and then perform calculations. Other details not described can be found in Figure 10 and will not be repeated here.

[0226] In one possible implementation, the scenario exemplified in the above embodiments is a scenario where the communication I / O is an RDMA network card. The technical solution in this application embodiment can also be applied to UB I / O interconnect scenarios, as shown in Figure 16. Each computing node includes a processor, memory, UB I / O, etc., where UB I / O can be integrated I / O (UB network card or UB Controller bus interface) of the processor SoC, or it can be an external, independent UB network card. In this scenario, IOVA can be UBA.

[0227] Specifically, in the CCL Window creation process, the communication library calls the UB driver's user-mode interface to create the CCL Window. The user-mode driver switches to the kernel-mode driver to execute the CCL Window creation operation. For example, the kernel-mode driver allocates a TID and a subset of the IO virtual address space (which in this example could be a contiguous UBA) for the CCL Window, and requests the memory resources required for the IOMMU page table (e.g., the MATT table).

[0228] Initialize the MATT table, MAPT table, and TCT table entries. The MATT table entries, used to indicate that the physical addresses in the CCL Window's IOVA and PA are currently invalid, are implemented similarly to the scenario described above and will not be repeated here. After creation, the communication IO sends the CCL Window's initial IOVA (which in this example can be the initial UBA) and TID back to the communication library.

[0229] For example, in the mapping process between CCL Window and Data Buffer, the communication library calls the UB driver user-mode interface to instruct the physical memory of the Data Buffer specified by (VA, length) to be mapped onto the CCL Window.

[0230] The user-mode driver switches to the kernel-mode driver and executes the mapping process. Specifically, the kernel-mode driver finds the corresponding MMU page table in the OS kernel based on the process ID, VA, and Length, and copies the contents of the MMU page table into the last-level page table of MATT, as shown in Figure 17. Figure 17 only shows the last-level page table of MATT; for example, the MATT table can be a four-level page table structure, the structure of which can be seen in Figure 3b, and will not be elaborated here. Other details not described herein can be found above and will not be repeated here.

[0231] Optionally, in this scenario, the compute node can disable the cache function corresponding to the MATT table, or the UB driver can optionally remove the MATT table from the cache before updating the MATT table (i.e. before copying the MMU page table) to ensure data consistency.

[0232] For example, in the above embodiment, the mapping between CCL Window and Data Buffer is achieved by copying the MMU page table. In this embodiment, the mapping between CCL Window and Data Buffer can also be achieved by setting the offset information in the IOMMU page table (for example, it can be an MPT entry in the MPT table or a TCT entry in the TCT table).

[0233] Specifically, the compute node creates a CCL Window and generates a corresponding IOMMU page table (e.g., an MPT entry in the MPT table or a TCT entry in the TCT table). The IOMMU page table may include, but is not limited to, offset information, the current content of which is invalid. In this scenario, during subsequent processing, the compute node can update the offset information so that the updated offset information is the offset between the VA of the Data Buffer and the IOVA of the CCL Window, or the offset between the IOVA of the Data Buffer (e.g., UBA) and the IOVA of the CCL Window, thereby achieving the mapping between the CCL Window and physical memory.

[0234] Optionally, this method can be applied to scenarios where the communication I / O is an RDMA network card and / or a UB network card, as shown in Figure 18.

[0235] In one example, if the communication I / O is an independent network card, such as one interconnected with the processor via a UB bus, the RDMA network card or UB network card exists as a UB bus device of the host, and the network card is in Shared Virtual Addressing mode, as shown in Figure 19.

[0236] Please refer to Figure 19. In this example, the network card can obtain the UBA corresponding to the IOVA based on the MPT table. It can be understood that in this example, the network card implements part of the IOMMU function (e.g., only looking up the MPT table), while the processor can implement all the IOMMU functions. That is, the processor can look up the UMMU page table based on the UBA to obtain the corresponding PA.

[0237] For example, during the creation of a CCL Window, MPT entries are initialized. These MPT entries include, but are not limited to, information such as Base, Size, Permission, TID, and offset (i.e., the offset information described in this embodiment). The offset value is currently invalid.

[0238] In other words, in this example, the MTT table is no longer requested (which can also be understood as the network card implementing part of the IOMMU function as described above). For example, the Data Buffer is dynamically allocated before the CCL Window establishes a mapping relationship with the Data Buffer. In this example, the compute node can register the MS corresponding to the Data Buffer with the UMMU, that is, perform memory registration on the Data Buffer, generate a UMMU page table, and the UMMU page table is used to indicate the mapping relationship between the Data Buffer's UBA and PA. Specifically, the compute node allocates a subset of the IO virtual address space for the Data Buffer from the IO virtual address space, for example, a continuous IOVA (represented as UBA in this example to avoid confusion with the CCL Window's IOVA), generates the corresponding TID, and the MATT table corresponding to the TID, and obtains the MATT table. The entries in the MATT table are used to indicate the mapping relationship between the Data Buffer's UBA and PA. The specific registration details can be found above and will not be repeated here. Optionally, the Data Buffer's UMMU page table and MMU page table can share an address translation table.

[0239] For example, the compute node updates the offset information in the MPT table of the network card based on the IOVA (which could also be UBA in this example) in the IOMMU page table of the Data Buffer, in order to achieve the mapping between the CCL Window and the Data Buffer.

[0240] Specifically, the communication library calls the user-space network card driver to execute the mapping process. The communication library can call the corresponding mapping interface, whose prototype can be: mapping_window(tid,uba,length,token,iova), where TID is used to indicate the TCT entry in the UMMU page table corresponding to the Data Buffer. UBA is the starting UBA (i.e., the starting IO virtual address) of the Data Buffer, and length is used to indicate the length of the Data Buffer. Token is used to indicate the MPT entry corresponding to the CCL Window. iova is an optional parameter because the driver can record the correspondence between IOVA and Token during the creation process.

[0241] For example, the user-mode network card driver updates the TID and offset field of the MPT table entry corresponding to the CCL Window (i.e., the offset information described in this embodiment), as shown in Figure 19. The MPT table entry also includes other information, which can be referred to above for details and will not be repeated here.

[0242] For example, the updated TID is the TID of the TCT entry in the UMMU page table, and the updated Offset is the offset between UBA in the UMMU page table (e.g., the MATT table) and IOVA of the CCL Window, which can be expressed as: Offset = UBA - IOVA.

[0243] This can be understood as follows: there is an offset between the UBA of the Data Buffer and the IOVA of the CCL Window. This offset is obtained and written to the Offset field of the MPT table entry of the network card. Then, during access, each time the IOVA of a CCL Window (i.e., the starting IOVA to be accessed) is obtained, the UBA corresponding to the physical memory of the Data Buffer can be obtained based on the offset information, and the corresponding PA can be further found based on the UBA.

[0244] For example, referring to Figure 19 again, in this example, when a compute node performs an access operation, the network interface card (NIC) adds the acquired IOVA to the Offset to obtain the corresponding UBA. The processor, responding to the acquired TID, UBA, and Length information, retrieves the corresponding PA by looking up the UMMU page table. Specifically, as shown in Figure 19, in the access scenario, the NIC receives access packets, which include, but are not limited to: target IO virtual address information (including but not limited to starting IOVA and Length), token information, and may also include access data.

[0245] Specifically, the network interface card (NIC) can look up the corresponding MPT entry based on the token and retrieve information such as TID and Offset stored in the MPT entry. The NIC can then obtain the corresponding UBA based on the Offset and IOVA, i.e., UBA = Offset + IOVA. After obtaining the TID, UBA, and Length information, the CPU (processor) looks up the corresponding UMMU page table and finds the starting PA of the corresponding page in the Data Buffer. The specific addressing method can be found above and will not be repeated here.

[0246] In another example, communication I / O can be integrated UB I / O. For instance, multiple compute nodes are interconnected using UB I / O, each compute node including a processor, memory, UB I / O, etc., as shown in Figure 16, where UB I / O can be the integrated UB network card or UB Controller bus interface of the processor SoC. In this scenario, during the creation of the CCL Window, the UMMU and MMU share the same address translation table. That is, the MATT table is no longer requested in this scenario. When initializing the TCT table entry, the pointer that originally pointed to the MATT table (i.e., the MATT Pointer) is pointed to the MMU page table of the corresponding process, and an Offset field is added to the TCT table entry, the content of which is currently invalid.

[0247] During the mapping process, the compute node can update the Offset field (i.e., the offset information described in the embodiments of this application) in the TCT table entry. The updated Offset is the offset between the VA of the Data Buffer and the IOVA of the CCL Window, which can be expressed as: Offset = VA - IOVA. For a detailed description, please refer to the above; it will not be repeated here.

[0248] In this way, during access, the compute node adds the Offset in the TCT table entry to the IOVA of the CCL Window to obtain the VA corresponding to the Data Buffer. The compute node (e.g., UB IO) can then use the VA to look up the UMMU page table, determine the corresponding PA, and access the corresponding physical memory. The specific implementation is similar to that described above and will not be repeated here.

[0249] In this way, by setting the Offset to the offset between IOVA and VA, the mapping relationship between CCL Window and Data Buffer is realized. During the memory mapping process, the software no longer needs to look up the MMU page table, which can shorten the command execution time, further reduce access time, and improve the efficiency of collection operation process.

[0250] The above mainly describes the solution provided by the embodiments of this application from the perspective of interaction between various network elements. It is understood that, in order to achieve the above functions, the access control device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0251] This application embodiment can divide the access control device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0252] When each functional module is divided according to its corresponding function, Figure 20 shows a possible structural diagram of the access control 2000 involved in the above embodiments. As shown in Figure 20, the access control device may include: a creation module 2001, used to create a cluster communication library window (CCL Window), wherein the CCL Window corresponds to a subset of the IO virtual address space allocated from the input / output IO virtual address space, and the CCL Window is not allocated physical memory; a communication module 2002, used to send the IO virtual address information of the CCL Window to other computing nodes in the computing device cluster, the IO virtual address information being used to indicate the subset of the IO virtual address space of the CCL Window; and a mapping module 2003, used to map the target physical memory to the CCL Window, so that other computing nodes can access the target physical memory based on the IO virtual address information of the CCL Window.

[0253] In another example, FIG21 shows a schematic block diagram of an access control device 2100 according to an embodiment of the present application. The access control device 2100 may include a processor 2101 and a transceiver / transceiver pin 2102, and optionally, a memory 2103. The processor 2101 can be used to execute the steps performed by the computing node in the methods of the foregoing embodiments, and control the receive pin to receive signals, and control the transmit pin to transmit signals.

[0254] The various components of the access control device 2100 are coupled together via a bus 2104, which includes a data bus, a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 2104 in the figure.

[0255] Optionally, the memory 2103 can be used for storage instructions in the foregoing method embodiments.

[0256] It should be understood that the access control device 2100 according to the embodiments of this application may correspond to the computing nodes in the methods of the foregoing embodiments, and the above and other management operations and / or functions of each element in the access control device 2100 are respectively for implementing the corresponding steps of the foregoing methods. For the sake of brevity, they will not be described in detail here.

[0257] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0258] Based on the same technical concept, embodiments of this application also provide a computer-readable storage medium storing a computer program containing at least one piece of code that can be executed by an access control device to control the access control device to implement the above-described method embodiments.

[0259] Based on the same technical concept, this application also provides a computer program, which, when executed by an access control device, is used to implement the above-described method embodiments.

[0260] The program may be stored, in whole or in part, on a storage medium packaged with the processor, or in part or in whole on a memory not packaged with the processor.

[0261] Based on the same technical concept, this application also provides a processor for implementing the above-described method embodiments. The processor can be a chip.

[0262] The steps of the methods or algorithms described in conjunction with the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor.

[0263] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0264] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0265] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0266] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0267] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0268] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An access control method, characterized in that, include: The first compute node creates a cluster communication library window (CCL Window), wherein the CCL Window is allocated a subset of the I / O virtual address space and the CCL Window is not allocated physical memory; the subset of the I / O virtual address space is allocated from the input / output I / O virtual address space, and the subset of the I / O virtual address space includes at least one I / O virtual address; The first computing node sends the IO virtual address information of the CCL Window to the second computing node. The IO virtual address information is used to indicate a subset of the IO virtual address space of the CCL Window. The first computing node establishes a mapping relationship between the physical address of the target physical memory and a subset of the IO virtual address space of the CCL Window, so that the second computing node can access the target physical memory based on the IO virtual address information of the CCL Window.

2. The method according to claim 1, characterized in that, The target physical memory is allocated with virtual addresses, which are allocated from a virtual address space. The virtual address space is independent of the I / O virtual address space.

3. The method according to claim 1, characterized in that, The method further includes: The first computing node receives an access request sent by the second computing node. The access request is used to request access to the physical memory in the target physical memory that is mapped to the target I / O virtual address. The target I / O virtual address belongs to a subset of the I / O virtual address space. The first computing node performs the corresponding access operation on the physical memory mapped to the target I / O virtual address based on the access request and the mapping relationship.

4. The method according to claim 3, characterized in that, The access operation includes at least one of a read operation, a write operation, and an atomic operation.

5. The method according to claim 1, characterized in that, The first compute node creates a CCL Window, including: The first computing node allocates a subset of the I / O virtual address space to the CCL Window from the I / O virtual address space based on the size of the CCL Window; wherein the size of the CCL Window is greater than or equal to the size of the access data corresponding to the access operation; The first computing node generates an Input / Output Memory Management Unit (IOMMU) page table corresponding to the CCL Window. The IOMMU page table includes address translation table entries for indicating the mapping relationship, wherein the physical addresses in the address translation table entries of the IOMMU page table are invalid.

6. The method according to claim 5, characterized in that, The first computing node establishes a mapping relationship between the physical address of the target physical memory and a subset of the CCL Window's I / O virtual address space, including: Copy the physical address from the address translation table entry of the MMU page table to the address translation table entry of the IOMMU page table; The address translation table entries in the MMU page table are used to indicate the mapping relationship between the physical address of the target physical memory and the virtual address of the target physical memory.

7. The method according to claim 1, characterized in that, The first compute node creates a CCL Window, including: The first computing node allocates a subset of the I / O virtual address space to the CCL Window from the I / O virtual address space based on the size of the CCL Window; wherein the size of the CCL Window is greater than or equal to the size of the access data corresponding to the access operation; The first computing node generates an IOMMU page table corresponding to the CCL Window, wherein the IOMMU page table includes offset information, and the current content of the offset information is invalid.

8. The method according to claim 7, characterized in that, The first computing node establishes a mapping relationship between the physical address of the target physical memory and a subset of the CCL Window's I / O virtual address space, including: Update the offset information, wherein the updated offset information is used to indicate the offset between the virtual address of the target physical memory and the IO virtual address of the CCL Window, or the offset between the IO virtual address of the target physical memory and the IO virtual address of the CCL Window.

9. An access control device, characterized in that, The device, applied to a first computing node, includes: A creation module is used to create a cluster communication library window (CCL Window), wherein the CCL Window is allocated a subset of the I / O virtual address space and the CCL Window is not allocated physical memory; the subset of the I / O virtual address space is allocated from the input / output I / O virtual address space, and the subset of the I / O virtual address space includes at least one I / O virtual address; A communication module is used to send the IO virtual address information of the CCL Window to the second computing node, wherein the IO virtual address information is used to indicate a subset of the IO virtual address space of the CCL Window; The mapping module is used to establish a mapping relationship between the physical address of the target physical memory and a subset of the IO virtual address space of the CCL Window, so that the second computing node can access the target physical memory based on the IO virtual address information of the CCL Window.

10. The apparatus according to claim 9, characterized in that, The target physical memory is allocated with virtual addresses, which are allocated from a virtual address space. The virtual address space is independent of the I / O virtual address space.

11. The apparatus according to claim 10, characterized in that, The device also includes an access module. The communication module is further configured to receive an access request sent by the second computing node, the access request being used to request access to the physical memory in the target physical memory that is mapped to the target IO virtual address, the target IOVA belonging to a subset of the IO virtual address space; The access module is used to perform corresponding access operations on the physical memory mapped to the target I / O virtual address based on the access request and the mapping relationship.

12. The apparatus according to claim 11, characterized in that, The access operation includes at least one of a read operation, a write operation, and an atomic operation.

13. The apparatus according to claim 9, characterized in that, The creation module is specifically used for: Based on the size of the CCLWindow, a subset of the I / O virtual address space is allocated to the CCLWindow from the I / O virtual address space; wherein the size of the CCLWindow is greater than or equal to the size of the access data corresponding to the access operation; Generate an Input / Output Memory Management Unit (IOMMU) page table corresponding to the CCL Window. The IOMMU page table includes address translation table entries for indicating the mapping relationship, wherein the physical addresses in the address translation table entries of the IOMMU page table are invalid.

14. The apparatus according to claim 13, characterized in that, The mapping module is specifically used for: Copy the physical address from the address translation table entry of the MMU page table to the address translation table entry of the IOMMU page table; The address translation table entries in the MMU page table are used to indicate the mapping relationship between the physical address of the target physical memory and the virtual address of the target physical memory.

15. The apparatus according to claim 9, characterized in that, The creation module is specifically used for: Based on the size of the CCL Window, a subset of the I / O virtual address space is allocated to the CCL Window from the I / O virtual address space; wherein the size of the CCL Window is greater than or equal to the size of the access data corresponding to the access operation; Generate an IOMMU page table corresponding to the CCL Window, wherein the IOMMU page table includes offset information, and the current content of the offset information is invalid.

16. The apparatus according to claim 15, characterized in that, The mapping module is specifically used for: Update the offset information, wherein the updated offset information is used to indicate the offset between the virtual address of the target physical memory and the IO virtual address of the CCL Window, or the offset between the IO virtual address of the target physical memory and the IO virtual address of the CCL Window.

17. A computer cluster, characterized in that, The computer cluster includes at least one computer device for performing the method according to any one of claims 1 to 8.

18. A computer storage medium, characterized in that, Includes computer instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1-8.

19. A computer program product, characterized in that, When the computer program product is run on a computer device, it causes the computer device to perform the method as described in any one of claims 1-8.

20. A chip, characterized in that, The device includes one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from the memory of the electronic device and send the signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the method according to any one of claims 1-8.

Images