Memory space configuration method and apparatus, and computing device cluster

Abstract

A memory space configuration method and apparatus, and a computing device cluster, relating to the field of computer technology. The method is applied to a computing cluster, the computing cluster comprises a plurality of computing nodes, and each computing node among the plurality of computing nodes separately provides a memory space. The method comprises: allocating a first memory space in a computing node to a service running on the computing node, wherein the first memory space is used for storing service data of the service; and creating a binding relationship between the first memory space and a second memory space, wherein the second memory space comprises a memory space in the computing node different from the first memory space or a memory space in a computing node different from the computing node among the plurality of computing nodes, and the second memory space is used for providing backup data of the service data. The solution can achieve data protection at lower cost.

Description

Memory space configuration methods, devices and computing device clusters

[0001] This application claims priority to Chinese Patent Application No. 202411614026.5, filed on November 12, 2024, entitled "Method, Apparatus and Computing Device Cluster for Memory Space Configuration", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to a memory space configuration method, apparatus, and computing device cluster. Background Technology

[0003] High-bandwidth memory (HBM) offers large storage capacity and is widely used as on-chip storage. However, HBM has a drawback: its error checking and correcting (ECC) technology has a high probability of multi-bit ECC failures. When a multi-bit ECC failure occurs, multiple bits of the service data stored in the HBM are corrupted and cannot be corrected by parity bits (ECC bits) to restore the service data. In other words, the service data is corrupted, and the service cannot continue to execute.

[0004] In related technologies, HBM is typically divided into two parts: the primary space and the mirror space. The primary space is used to store business data, while the mirror space is used to back up the business data stored in the primary space. This way, when a multi-bit ECC failure occurs in the primary space, backup data can be retrieved from the mirror space to ensure continued business operation.

[0005] However, with the development of computing technology and business needs, the amount of business data is increasing, and the additional HBM used as a mirror space is also increasing, especially in scenarios where computing device clusters execute business operations, where the additional HBM required by the entire computing device cluster increases exponentially. Therefore, how to achieve lower-cost data protection is an urgent problem to be solved. Summary of the Invention

[0006] To address the aforementioned technical problems, this application provides a memory space configuration method, apparatus, and computing device cluster. In this memory space configuration method, a binding relationship is created between a first memory space and a second memory space, providing backup data from the second memory space for business data stored in the first memory space. The second memory space is either local memory different from the first memory space or memory in a remote location, requiring no additional memory, thereby achieving lower-cost data protection.

[0007] In a first aspect, embodiments of this application provide a memory space configuration method applied to a computing cluster. The computing cluster includes multiple computing nodes, and each computing node provides its own memory space. The method includes: allocating a first memory space in the computing node for services running on the computing node, the first memory space being used to store service data; creating a binding relationship between the first memory space and a second memory space, the second memory space including a memory space in the computing node that is different from the first memory space or a memory space in one of the multiple computing nodes that is different from the first memory space, the second memory space being used to provide backup data for the service data.

[0008] In this embodiment, the second memory space is used to provide backup data for business data. Based on this, by creating a binding relationship between the first and second memory spaces, backup data can be obtained through this binding relationship to ensure business execution even if a memory failure occurs in the first memory space used by the business running on the computing node, such as a multi-bit ECC failure. This eliminates the need for the computing node to re-execute the business, significantly reducing time loss and costs associated with failures. The second memory space includes memory within the computing node that differs from the first memory space, or memory from remote computing nodes in the computing cluster. This eliminates the need for additional memory (e.g., HBM) specifically added to the computing node for memory anomaly handling, thus achieving lower-cost data protection.

[0009] According to the first aspect, after creating the binding relationship between the first memory space and the second memory space, the method further includes: in response to a service access request, accessing the first memory space, the access request being used to request first service data; upon receiving a fault notification returned by the first memory space, if the first service data and the faulty memory indicated by the fault notification satisfy the binding relationship, then accessing the second memory space according to the binding relationship to obtain backup data of the first service data, and performing fault handling of the faulty memory based on the backup data of the first service data.

[0010] In this embodiment, when a memory failure occurs, backed-up business data can be retrieved to continue business operations, rather than interrupting the business and retrieving historical data to restart it. This significantly improves business recovery time, execution efficiency, and user experience. Furthermore, by using the range of the faulty memory, the version number of the first business data, and the version number of the backup data, it can be determined whether memory failure handling is possible, thereby increasing the success rate of memory failure handling.

[0011] According to the first aspect, or any implementation of the first aspect above, a binding relationship between the first memory space and the second memory space is created, including: binding the first base address of the first memory space, the first version number of the business data, the second base address of the second memory space, and the second version number of the backup data of the business data to obtain the initial binding relationship between the first memory space and the second memory space; during the execution of the business, the initial binding relationship is updated to obtain the binding relationship between the first memory space and the second memory space.

[0012] In this embodiment, the consistency between backup data and business data is ensured by using data version numbers, thereby further improving the success rate of memory anomaly handling.

[0013] According to the first aspect, or any implementation of the first aspect above, the second memory space includes a memory space in the computing node that is different from the first memory space; updating the initial binding relationship includes: storing the current business data according to the first base address and updating the first version number; backing up the current business data according to the second base address to obtain the backup data of the current business data, and updating the second version number to correspond to the updated first version number.

[0014] In this embodiment, the correspondence between the first version number of the business data and the second version number of the backup data ensures the consistency between the backup data and the business data, further improving the success rate of memory anomaly handling. Furthermore, when the second memory space includes a memory space in the computing node that differs from the first memory space, backup data can be provided in the second memory space to ensure that the provision of backup data is less restricted and has a wider range of applications.

[0015] According to the first aspect, or any implementation of the first aspect above, each computing node includes a local agent HA, and each computing node corresponds to a processing unit; after creating the binding relationship between the first memory space and the second memory space, the method further includes: when the processing unit accesses the business data in the first memory space through the HA, if the HA receives a fault notification indicating a memory fault returned by the first memory space, the HA requests backup data of the business data from the processing unit; the processing unit instructs the second memory space to feed back the backup data of the business data to the HA; the HA processes the memory fault based on the received backup data of the business data.

[0016] In this embodiment of the application, each computing node includes a local agent HA, and each computing node has a corresponding processing unit. In this way, the processor can use HA to obtain backup data of business data in case of memory failure and to handle memory failure. The local agent HA is a logical hardware, which can further improve processing efficiency and reduce processor pressure.

[0017] According to the first aspect, or any implementation of the first aspect above, the second memory space includes the memory space of a computing node that is different from the computing node among multiple computing nodes; updating the initial binding relationship includes: storing the current business data in the first memory space of the computing node according to the first base address of the first memory space in each computing node, and updating the first version number; for the first memory space in each computing node, updating the second version number in the binding relationship to which the first memory space belongs to correspond to the updated first version number in the binding relationship.

[0018] In this embodiment, the correspondence between the first version number of the business data and the second version number of the backup data ensures the consistency between the backup data and the business data, further improving the success rate of memory anomaly handling. Furthermore, when the second memory space includes the memory spaces of computing nodes different from the original computing nodes, using the first memory space of the remote computing node as the second memory space allows for direct use of data identical to the business data in the remote computing node as backup data, resulting in greater efficiency and lower cost.

[0019] According to the first aspect, or any implementation of the first aspect above, after creating the binding relationship between the first memory space and the second memory space, the method further includes: if a fault notification is received, and the faulty memory indicated by the fault notification satisfies the binding relationship, and the first version number of the first service data satisfies the binding relationship within a first time duration threshold, then the second memory space is accessed according to the binding relationship to obtain the backup data of the first service data, and the fault notification is used to indicate that there is a memory fault in the first service data in the first memory space; and the service is executed based on the backup data of the first service data.

[0020] In this embodiment of the application, each computing node typically requires a certain delay to update its local business data. Correspondingly, there is also a certain delay in updating the second version number corresponding to each computing node. In this regard, when the computing node performs data protection, if the first version number and the second version number are inconsistent, it can wait for a certain period of time, such as a first time threshold, so that the second version number can be updated, thereby improving the success rate of obtaining backup data and thus improving the success rate of handling memory failures.

[0021] According to the first aspect, or any implementation of the first aspect above, each computing node includes a local agent HA and a communication module; after creating the binding relationship between the first memory space and the second memory space, the method further includes: when the first HA accesses the business data in the first memory space, if it receives a fault notification indicating a memory fault returned by the first memory space, the first HA requests backup data of the business data from the second communication module through the first communication module, and the first HA and the first communication module belong to the first computing node; the second communication module obtains the backup data of the business data from the second memory space through the second HA and feeds it back to the first communication module, and the second HA and the second communication module belong to the second computing node; the first HA processes the memory fault based on the backup data of the business data received from the first communication module.

[0022] For example, the first HA processes memory failures based on backup data of service data received from the first communication module. Specifically, this may include: the first HA feeding back the backup data to the first GPU core (i.e., computing unit core) to continue executing services, and the first HA using the backup data to update the faulty service data in the first memory space.

[0023] In this embodiment, the computing node includes a local agent HA and a communication module (e.g., an IB module and a corresponding IB bus). When the second memory is located on a remote computing node, the computing node acquires backup data through the HA and the communication module. This further improves processing efficiency through logical hardware HA and ensures communication with the remote second memory through the communication module.

[0024] According to the first aspect, or any of the above implementations of the first aspect, the update interval of the business data satisfies the second duration threshold, or each computing node runs the same business data.

[0025] In this embodiment, in a scenario where business data in the first memory space is periodically updated, the update interval of the business data satisfies a second duration threshold, which is also the update cycle of the business data. This reduces the frequency of business data updates, ensuring that the consistency between business data and backup data is not compromised by frequent updates. This improves the success rate of memory fault handling based on backup data, further enhancing the user experience. Alternatively, each computing node can run the same business data. In this case, each computing node can use the first memory space of other computing nodes in the computing cluster as its second memory space, and use the business data stored on other computing nodes as backup data for its own business data. In this scenario, the computing cluster does not need to perform dedicated backups of business data, improving the efficiency of data redundancy. This allows for faster memory fault handling based on backup data in the event of a memory failure, further improving the user experience.

[0026] According to the first aspect, or any implementation of the first aspect above, the size of the memory space in the computing node that is different from the first memory space is greater than the first memory space, and the supported read / write rate is less than the first memory space; the size of the memory space and the supported read / write rate of the computing nodes that are different from the computing node in the multiple computing nodes are both equal to the first memory space.

[0027] In this embodiment, when the second memory space and the first memory space are on the same computing node, the size of the second memory space is larger than that of the first memory space, and the supported read / write speed is lower than that of the first memory space, thereby further reducing memory costs and meeting greater backup requirements. When the second memory space and the first memory space are on different computing nodes, the size of the second memory space and the supported read / write speed are both equal to those of the first memory space, thereby providing higher read / write efficiency and reducing the impact of remote communication on efficiency.

[0028] Secondly, embodiments of this application provide a memory space configuration device applied to a computing cluster. The computing cluster includes multiple computing nodes, and each computing node provides its own memory space. The device includes: a memory allocation module, which allocates a first memory space in the computing node for services running on the computing node, the first memory space being used to store service data; and a data redundancy module, which creates a binding relationship between the first memory space and a second memory space, the second memory space including a memory space in the computing node that is different from the first memory space or a memory space in one of the multiple computing nodes that is different from the first memory space, the second memory space being used to provide backup data for the service data.

[0029] According to the second aspect, the device further includes a fault handling module, configured to: in response to a service access request, access a first memory space, the access request being used to request first service data; and upon receiving a fault notification returned by the first memory space, if the first service data and the faulty memory indicated by the fault notification satisfy a binding relationship, access a second memory space according to the binding relationship to obtain backup data of the first service data, and perform fault handling of the faulty memory based on the backup data of the first service data.

[0030] According to the second aspect, or any implementation of the second aspect above, the data redundancy module is specifically used to: bind the first base address of the first memory space, the first version number of the business data, the second base address of the second memory space, and the second version number of the backup data of the business data to obtain the initial binding relationship between the first memory space and the second memory space; during the execution of the business, update the initial binding relationship to obtain the binding relationship between the first memory space and the second memory space.

[0031] According to the second aspect, or any implementation of the second aspect above, the second memory space includes a memory space in the computing node that is different from the first memory space; the data redundancy module is specifically used for: storing the current business data according to the first base address and updating the first version number; backing up the current business data according to the second base address to obtain the backup data of the current business data, and updating the second version number to correspond to the updated first version number.

[0032] According to the second aspect, or any implementation of the second aspect above, the apparatus further includes a local agent HA distributed across each computing node, and each computing node has a corresponding processing unit; when the processing unit accesses service data in the first memory space through the HA, if the HA receives a fault notification indicating a memory fault returned by the first memory space, the HA requests backup data of the service data from the processing unit; the processing unit instructs the second memory space to feed back the backup data of the service data to the HA; the HA processes the memory fault based on the received backup data of the service data.

[0033] According to the second aspect, or any implementation of the second aspect above, the second memory space includes the memory space of a computing node that is different from the computing node among multiple computing nodes; the data redundancy module is specifically used to: store the current business data in the first memory space of the computing node according to the first base address of the first memory space in each computing node, and update the first version number; for the first memory space in each computing node, update the second version number in the binding relationship to which the first memory space belongs to correspond to the updated first version number in the binding relationship.

[0034] According to the second aspect, or any implementation of the second aspect above, the apparatus further includes a fault handling module, configured to: if a fault notification is received, and the faulty memory indicated by the fault notification satisfies the binding relationship, and the first version number of the first service data satisfies the binding relationship within a first time duration threshold, then access the second memory space according to the binding relationship to obtain backup data of the first service data, and the fault notification is used to indicate that there is a memory fault in the first service data in the first memory space; and execute the service based on the backup data of the first service data.

[0035] According to the second aspect, or any implementation of the second aspect above, the apparatus further includes a local agent HA and a communication module distributed on each computing node; when the first HA accesses service data in the first memory space, if it receives a fault notification indicating a memory fault returned by the first memory space, the first HA requests backup data of the service data from the second communication module through the first communication module, and the first HA and the first communication module belong to the first computing node; the second communication module obtains the backup data of the service data from the second memory space through the second HA and feeds it back to the first communication module, and the second HA and the second communication module belong to the second computing node; the first HA processes the memory fault based on the backup data of the service data received from the first communication module.

[0036] According to the second aspect, or any implementation of the second aspect above, the update interval of the business data meets the second duration threshold, or each computing node runs the same business data.

[0037] According to the second aspect, or any implementation of the second aspect above, the size of the memory space in the computing node that is different from the first memory space is greater than the first memory space, and the supported read / write rate is less than the first memory space; the size of the memory space and the supported read / write rate of the computing nodes that are different from the computing node in the multiple computing nodes are both equal to the first memory space.

[0038] The second aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the second aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0039] Thirdly, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs a method as described in the first aspect and any implementation thereof.

[0040] Fourthly, embodiments of this application provide a computer-readable medium for storing computer program instructions that, when executed by a cluster of computing devices, perform a method as described in the first aspect or any possible implementation thereof.

[0041] Fifthly, embodiments of this application provide a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to execute the instructions as described in the first aspect or any possible implementation thereof.

[0042] Sixthly, embodiments of this application provide a chip including a processing circuit and transceiver pins. The transceiver pins and the processing circuit communicate with each other via an internal connection path. When the processing circuit is applied to a computing cluster, it executes the method in the first aspect or any possible implementation of the first aspect to control the receiving pin to receive signals and to control the transmitting pin to transmit signals. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 is one of the structural block diagrams of a computing cluster provided in an embodiment of this application;

[0045] Figure 2 is one of the structural block diagrams of a computing cluster provided in an embodiment of this application;

[0046] Figure 3 is a flowchart illustrating a memory space configuration method provided in an embodiment of this application;

[0047] Figure 4 is one of the processing scenario examples of a memory space configuration method provided in the embodiments of this application;

[0048] Figure 5 is one of the processing scenario examples of a memory space configuration method provided in the embodiments of this application;

[0049] Figure 6a is one of the structural block diagrams of a memory space configuration device provided in an embodiment of this application;

[0050] Figure 6b is a structural block diagram of a memory space configuration device provided in an embodiment of this application;

[0051] Figure 7 is a structural block diagram of a computing device provided in an embodiment of this application;

[0052] Figure 8 is a structural block diagram of a computing cluster provided in an embodiment of this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0055] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0056] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0057] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0058] To facilitate understanding of this embodiment, some technical terms and background technologies involved in this embodiment will be introduced first:

[0059] Error checking and correcting (ECC) technology: By adding an extra ECC bit to the data, called a parity bit, if one bit in the data is wrong, it can be corrected to restore the data. If multiple bits are wrong, the erroneous data cannot be corrected.

[0060] PCIe (Peripheral Component Interconnect Express) bus: A bus that uses a high-speed serial computer expansion bus standard, enabling high-speed serial point-to-point dual-channel high-bandwidth transmission. Devices connected to this bus are allocated dedicated channel bandwidth and do not share bus bandwidth. It primarily supports active power management, error reporting, end-to-end reliable transmission, hot-plugging, and Quality of Service (QoS) functions.

[0061] PCIe endpoint (PCIe EP): This refers to the terminal device / PCIe endpoint connected on the PCIe bus. PCIe endpoint devices can connect directly to the root device, or they can connect to the root device through a switch, which functions similarly as an interface expansion device.

[0062] PCIe root complex (PCIe RC): The root complex, also known simply as the root bridge, connects the CPU and memory subsystem to the PCI Express switching structure, which consists of one or more PCIe or PCI devices. It is the "root" of the PCI reverse tree topology, representing the CPU in communication with other devices.

[0063] Home agent (HA): This usually refers to a memory manager that is responsible for managing and maintaining memory consistency.

[0064] Direct Memory Access (DMA): DMA is a memory access technology that provides high-speed data transfer between peripherals and memory, or between memory devices. In other words, it offers the most direct way for hardware to access memory. Data can be moved quickly without CPU intervention, thus freeing up CPU resources for other operations.

[0065] All-Reduce: A common communication operation in parallel and distributed computing used to aggregate data across multiple computing nodes. This aggregation involves, for example, adding the data from each node across all nodes or performing other reduction operations (e.g., summation, averaging, finding the maximum value). After aggregation, the results are broadcast back to each node, ensuring that each node has the same aggregated data for updating model parameters.

[0066] In scenarios where business operations, such as training artificial intelligence models, are performed through computing clusters, HBM (Hardware Storage) on the compute nodes of the cluster is typically used as on-chip high-capacity storage. During training, a large number of training parameters, such as optimizer parameters and weight parameters, are stored in HBM. However, HBM has a relatively high probability of multi-bit ECC (Error Correction). Multi-bit ECC can corrupt training parameters, also known as erroneous data, making it impossible to continue training and interrupting business operations.

[0067] In related technologies, during training, training weights, optimizer parameters, and other data are periodically saved to a checkpoint server. This periodic saving is called a checkpoint, and the timing of this periodic saving is called a checkpoint point. When an HBM multi-bit ECC failure occurs, the current training is stopped, and the previously saved training weights and optimizer parameters are retrieved from the checkpoint server, imported into the training server, and the training process is restarted. Training then begins again based on the checkpoint point.

[0068] However, the checkpoint server saves data every 20 minutes to 5 minutes. After each failure, the cluster needs to restart from the checkpoint, resulting in time loss. For Wanka clusters or larger clusters, the cost of each failure is very high.

[0069] To address this issue, this application provides a memory space configuration method. In this method, a second memory space is used to provide backup data for business data. Based on this, by creating a binding relationship between the first and second memory spaces, backup data can be obtained through this binding relationship to ensure business execution even if a memory failure occurs in the first memory space used by the business running on the computing node, such as a multi-bit ECC failure. This eliminates the need for the computing node to re-execute the business, significantly reducing time loss and costs associated with failures. The second memory space includes memory in the computing node that differs from the first memory space or memory in remote computing nodes within the computing cluster. This eliminates the need to add additional memory, such as HBM, specifically for memory anomaly handling in the computing node, thus achieving lower-cost data protection.

[0070] Before describing the technical solutions of the embodiments of this application, the operating platform of the memory space configuration method of the embodiments of this application will first be described with reference to the accompanying drawings. The embodiments of this application can be applied to a computing cluster. For example, the computing cluster can be in the form of a supernode device, that is, multiple computing nodes in the cluster are contained in the same device. Figure 1 is one of the structural block diagrams of a computing cluster provided by the embodiments of this application. As shown in Figure 1, the computing cluster may include: multiple computing nodes, such as computing nodes N1 to Nn, each computing node including a computing unit, for example, a graphics processing unit (GPU) or a neural network processing unit (NPU), etc. The computing unit manages the services running on the computing node through the computing unit core (such as GPU core, NPU core), and the service data of the service is stored in the first memory space of the computing node (HBM as shown in Figure 1). In specific applications, the number of computing units in each computing node can be one or more.

[0071] In one example, the computing cluster may further include processing units for managing each computing node within the cluster. These processing units may include one or more processors, such as a central processing unit (CPU), GPU, microprocessor (MP), or digital signal processor (DSP). Different processors may be independent devices or integrated within one or more processing units. The processor can manage the local memory of the computing node through its CPU core, for example, double data rate synchronous dynamic random access memory (DDR), which can provide a secondary memory space. The processor can communicate with the modules within the computing unit through some interconnect technology. For example, this interconnect technology may include a PCIe bus, other types of buses (such as a unified bus), NVLink (a high-speed interconnect technology proposed by NVIDIA), etc. If the interconnect technology is a PCIe bus, the PCIe root bridge on the PCIe bus represents the processor in communicating with the PCIe end nodes on the PCIe bus. The PCIe terminal node can be deployed in the computing unit to receive data sent from each module in the computing unit to the PCIe bus, and to store the data sent from the PCIe bus to each module in the computing unit. Each module in the computing unit can obtain the data requested from the PCIe bus from the PCIe terminal node.

[0072] In one example, the computing unit may also include a local agent (HA). In another example, the HA can be used to store the binding relationship between a first memory space and a second memory space created by the processor. In yet another example, the HA can access the second memory space, such as DDR, via PCIe terminal node direct memory access (DMA).

[0073] In one example, different computing nodes can exchange data via a network such as an Infiniband (IB) bus.

[0074] For example, Figure 2 is one of the structural block diagrams of a computing cluster provided in an embodiment of this application. As shown in Figure 2, the computing cluster may include multiple computing nodes, such as computing nodes M1 to Mn. The computing cluster shown in Figure 2 is similar to the computing cluster shown in Figure 1, except that each computing node in the computing cluster shown in Figure 2 contains a processor. The same parts will not be described again here, but are detailed in the description of the embodiment in Figure 1 above. In one example, each computing node may contain multiple processors, and the multiple processors in each computing node may form a processing unit. In specific applications, the computing nodes in the computing cluster shown in Figure 2 may be contained in the same device or different devices.

[0075] For the sake of logical layout and ease of description, the data protection process of the computing cluster, i.e., the functions of each module in the computing cluster, will be described in detail below with reference to Figures 4 and 5. Furthermore, the connection relationships illustrating the interaction between modules in this application's embodiments, such as circuit connections or communication connections, are merely illustrative and do not constitute a limitation on the structure of the computing cluster. In other embodiments of this application, the computing cluster may also employ different connection methods or combinations of multiple connection methods as described in the above embodiments.

[0076] It should be understood that the computing clusters shown in Figures 1 and 2 are merely one example of a computing cluster, and computing clusters may have more or fewer components than those shown in the figures, may combine two or more components, or may have different component configurations. The various components shown in Figures 1 and 2 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0077] The memory space configuration method provided in the embodiments of this application will be described in detail below with reference to Figures 3 to 5.

[0078] For example, Figure 3 is a flowchart illustrating a memory space configuration method provided in an embodiment of this application. As shown in Figure 3, the method is applied to a computing cluster (as shown in Figure 1 or Figure 2), which includes multiple computing nodes, each of which provides memory space. Specifically, the method may include:

[0079] S301 allocates the first memory space in the computing node for the services running on the computing node. The first memory space is used to store the service data of the services.

[0080] In practical applications, the processor managing the compute node can allocate a primary memory space within the compute node for the services running on that node. This primary memory space could be, for example, the HBM (Hardware Base) within the compute node.

[0081] Optionally, the update interval of the business data in the above-mentioned services meets the second duration threshold. For example, in the scenario where business data in HBM is updated periodically, the update interval of the business data meets the second duration threshold, which is also the update cycle of the business data. In this way, the update frequency of business data is relatively reduced, and the consistency between business data and backup data will not be damaged due to frequent updates of business data. This improves the success rate of memory fault handling of business data based on backup data, and further enhances the user experience.

[0082] Optionally, each compute node runs the same business data. Here, "same business data" means the data used in the same business process. For example, in an AI model training scenario where All-Reduce synchronizes gradient operations, each compute node runs the same business data. In this way, each compute node can use the first memory space of other compute nodes in the cluster as its second memory space, and use the business data stored on other compute nodes as backup data for its own business data. In this case, the compute cluster does not need to perform dedicated backups of business data, improving the efficiency of data redundancy. This allows for faster handling of memory failures based on backup data, further enhancing the user experience.

[0083] Optionally, when the first memory space and the second memory space are located on the same compute node, the size of the second memory space is larger than the size of the first memory space, and the read / write speed supported by the second memory space is lower than the read / write speed supported by the first memory space. For example, the first memory space is HBM in the compute node, and the second memory space is DDR in the compute node.

[0084] For example, Figure 4 is one of the processing scenario examples of a memory space configuration method provided in an embodiment of this application. As shown in Figure 4, when using memory (e.g., DDR) different from the first memory space in this computing node (i.e., this end) as the second memory space, the processor of this computing node is managed to run services for this computing node, such as model training tasks, and the first memory space is allocated to store service data. For example, the service data may be the training parameters of the model training task, and the first memory space may be the memory space with base address Base_address1 in HBM.

[0085] S302, create a binding relationship between the first memory space and the second memory space. The second memory space includes memory in the compute node that is different from the first memory space, or memory in multiple compute nodes that is different from the compute node. The second memory space is used to provide backup data for business data.

[0086] For example, the processor managing the compute node can use a memory allocation interface to synchronously allocate a second memory space of the same size as the first memory space in DDR, as a redundant backup of the HBM memory space. After the memory interface allocates memory space in DDR, it feeds back information about that memory space to the processor, which then configures a Mirror entry in the HA of the managed compute node, i.e., creates a binding relationship between the first and second memory spaces. This binding relationship indicates the correspondence between the address information of the first and second memory spaces. In one example, this binding relationship can also indicate the correspondence between service data and the first memory space, as well as the correspondence between the backup data of the service data and the second memory space, thereby facilitating faster retrieval of the backup data of the service data.

[0087] For example, the data structure of this binding relationship can be in the form of a table as shown in Table 1. The specific information contained in the binding relationship can be found in Table 1 below:

[0088] Table 1. Example 1 of the binding relationship between the first and second memory spaces.

[0089] It is understood that the above binding relationship is only an example. Any data structure that can be used to bind the first base address of the first memory space, the first version number of the business data, the second base address of the second memory space, and the second version number of the backup data of the business data can be used in this application. The embodiments of this application do not limit this.

[0090] In one example, the processor managing the compute node can synchronously configure the routing table entries for the bus between the DDR and the compute units (such as GPU / NPU) within the compute node through the aforementioned memory allocation interface. Access to addresses in the range base_address2 to base_address2+length based on these routing table entries accesses the redundancy-protected memory addresses in the local second memory space (i.e., DDR). In practical applications, if different memory spaces communicate via a PCIe bus, routing can be based on addresses defined by the PCIe protocol or on module IDs, for example...<Bus、Device、Function> If different memory spaces communicate via the IB bus, routing can be performed based on the addresses defined in the global route headers (GRH). Alternatively, if different memory spaces communicate through a unified central hub, routing can be performed via address or module ID.

[0091] Taking model training as an example, after the business script running on the compute node completes one step of computation in the compute cluster (each step can be, for example, one round of model training, lasting about 1 minute), the processor managing the compute node updates the training parameters in its local HBM through HA, and updates the local version number in the binding relationship, which is the first version number. For example, incrementing Idx1 as shown in Table 1: Idx1++. HA performs a backup task via DMA: the training parameters stored in the memory space of HBM starting from address Base_address_1 are backed up to the memory space of DDR starting from address Base_address_2 via the PCIe bus. After the backup is complete, DDR updates the version number of the backup data, which is the second version number, for example, incrementing: Idx2++, and feeds it back to HA. HA updates the second version number in the binding relationship, for example, Idx2 in Table 1. This process takes about 1 to 2 seconds, depending on the DDR speed.

[0092] Based on the above configuration, when a computing node experiences a memory failure, such as a multi-bit ECC failure, it can perform read operations on the second memory space to obtain backup data of the business data to execute the business, thereby ensuring uninterrupted business operations.

[0093] For ease of understanding, referring to Figure 4, taking the computing unit in computing node N1 as including GPU1, the computing unit core as GPU1Core, the processor as CPU, the CPU and computing units communicating via PCIe bus, and the multi-bit ECC as 2-bit ECC as an example, the above processing can include the following steps ① to ⑥:

[0094] ① When GPU1Core receives an access request for the first service data, it sends the access request to HA. This access request contains the storage address of the first service data in the first memory provided by HBM in compute node N1. This storage address can be a virtual address.

[0095] ② HA maps the virtual address to a hardware address and accesses the HBM in computing node N1 according to the hardware address to read the first service data.

[0096] ③ If a 2-bit ECC fault occurs in HBM, a fault notification is returned to HA. This fault notification is used to report a memory space fault indicated by the address ecc_address.

[0097] ④ Upon receiving the fault notification, if the HA determines, based on the binding relationship (e.g., the mirror entry shown in Table 1), that the ecc_address falls within the address range of the first memory space indicated by the mirror entry (i.e., the protection range), it compares whether Idx1 and Idx2 are consistent. If they are consistent, it calculates the backup address corresponding to ecc_address, which is also the storage address of the backup data of the first service data: address = ecc_address - base_address1 + Base_address2, and then initiates an access to the bus. For example, taking the PCIE bus as an example: After receiving the HA's access request for the backup data of the first service data from the PCIE bus connected to the computing unit in compute node N1, it searches the routing table entry of the bus between DDR and GPU1 according to the storage address indicated by the access request. Based on the search result, it encapsulates a message and sends the message to the PCIE bus between the CPU and DDR, accessing DDR through this PCIE bus. As shown in Figure 4, this PCIE bus is the bus connecting the PCIE root bridge and DDR.

[0098] ⑤ When the DDR receives an access request for the first service data from the PCIe bus, it sends back the backup data of the first service data in the memory space indicated by the access request to the PCIe bus, and then the PCIe bus returns the backup data of the first service data to the HA.

[0099] ⑥ After receiving the backup data of the first service data from the PCIe bus, HA returns it to GPU1Core and updates the erroneous data in HBM, i.e., the first service data, with the backup data. GPU1Core can then use the backup data to execute services, preventing service interruptions due to 2-bit ECC faults.

[0100] In one example, after a multi-bit ECC fault occurs, the computing cluster can also perform the following S1 to S3 processing:

[0101] S1, after receiving the fault notification, HA reads HBM according to the ECC fault address. If no more ECC alarms occur, the fault is repaired and no alarm is reported.

[0102] S2, if the ECC fault still exists, the processor needs to instruct the business script to stop the current step training and release the memory space occupied by the training parameters in the HBM that are memory faulty. The processor isolates the memory space indicated by the address of the memory faulty HBM, obtains a new HBM, and then reallocates the memory space in the new HBM that stores the training parameters, and updates the mirror table entry according to the result of the reallocation.

[0103] S3, HA retrieves the training parameters from DDR, restores them to the newly allocated HBM address, restarts training, and releases the memory occupied by the old DDR training parameters.

[0104] It is understandable that S2 to S3 above are equivalent to re-executing S301 for the new HBM obtained for the memory indicated by the HBM address that has been isolated from the fault.

[0105] Optionally, when the first memory space and the second memory space are located on different computing nodes, the size of the second memory space is equal to the size of the first memory space, and the read / write speed supported by the second memory space is equal to the read / write speed supported by the first memory space. For example, the first memory space is the memory space in the HBM of the local computing node, and the second memory space is the memory space in the HBM of the remote computing node.

[0106] For example, in scenarios where the business data is identical on each compute node—such as a scenario where a full All-Reduce gradient synchronization operation is performed after each step—each compute node updates its local training data based on the gradients after computation. This includes parameters that are identical across all compute nodes, such as model weights and optimizers—parameters that are not split. In other words, the weights on each relevant GPU or NPU are consistent in each step of this scenario. Therefore, HBM in remote compute nodes can be used to provide redundant memory space for training parameters.

[0107] For example, Figure 5 is one of the processing scenarios of a memory space configuration method provided in this application embodiment. As shown in Figure 5, memory in a computing node different from the local computing node, i.e., a remote computing node, is used. For example, when the HBM in the remote computing node is used as the second memory space, the processor managing each computing node allocates memory space in the HBM of the local computing node as the first memory space of that computing node, storing the training parameters used by the services running in that computing node. For example, the first memory space can be the memory space with base address Base_address1 in the HBM of the local computing node.

[0108] The processor managing the compute node can allocate a first memory space using the memory allocation interface. After the memory interface allocates the first memory space, it feeds back information about the memory space to the processor, which then configures a Mirror entry in the HA of the managed compute node, i.e., creates a binding relationship between the first and second memory spaces. A description of this binding relationship can be found in the existing description of the embodiment in Figure 4 above, and will not be repeated here.

[0109] For example, the data structure of this binding relationship can be in the form of a table as shown in Table 2. The specific information contained in the binding relationship can be found in Table 2 below:

[0110] Table 2. Example 2 of the binding relationship between the first and second memory spaces.

[0111] In one example, the processor managing the compute nodes can configure the routing table entries for the bus between each compute node and another compute node containing a second memory space bound to the first memory space of that compute node, through the aforementioned memory allocation interface. Taking the IB bus as an example: the processor managing the compute nodes can first convert the address range (base_address2~base_address2+length) to the IB global address range (global-address~global-address+length) through the aforementioned memory allocation interface, and then configure the IB GRH route for this global address range.

[0112] During business operations, after each compute node updates its local business data, such as weights and other parameters, it updates the corresponding data version number (idx1++) in the mirror table, which is the first version number. It then updates the data version number (idx2++) of the redundant memory node (the compute node containing the second memory space) via the aggregate communication interface; this data version number is the second version number. In practical applications, each compute node can update its business data using All-Reduce, in which case each compute node stores the same business data.

[0113] In one alternative implementation, after creating the binding relationship between the first memory space and the second memory space, the method further includes:

[0114] If, during the execution of a business, there is first business data in the first memory space that is affected by a memory failure, the faulty memory indicated by the fault notification satisfies the binding relationship, and the first version number of the first business data satisfies the binding relationship within the first time threshold, then the second memory space is accessed according to the binding relationship to obtain the backup data of the first business data.

[0115] The business is executed based on the backup data of the first business data.

[0116] For example, updating local business data on each computing node typically requires a certain delay; correspondingly, updating the second version number for each computing node also involves a certain delay. To address this, when a computing node is handling a fault, if the first and second version numbers are inconsistent, it can wait for a certain period, such as a first time threshold. This ensures that the second version number is updated successfully, thereby improving the success rate of fault handling.

[0117] Based on the above configuration, when a computing node experiences a memory failure, such as a multi-bit ECC failure, it can perform read operations on the second memory space to obtain backup data of the business data to execute the business, thereby ensuring uninterrupted business operations.

[0118] For ease of understanding, referring to Figure 5 again, taking the example that computing units in computing node N1 include GPU1 and computing unit cores include GPU1Core, computing units in computing node N2 include GPU2 and computing unit cores include GPU2Core, different computing units communicate via the IB bus, and multiple-bit ECC is 2-bit ECC, the above processing can include the following steps ① to ⑥:

[0119] ① When GPU1Core receives an access request for the first service data, it sends the access request to HA. This access request contains the storage address of the first service data in the first memory provided by HBM in compute node N1. This storage address can be a virtual address.

[0120] ② HA maps the virtual address to a hardware address and accesses the HBM in computing node N1 according to the hardware address to read the first service data.

[0121] ③ If a 2-bit ECC fault occurs in HBM, a fault notification is returned to HA. This fault notification is used to report a memory space fault indicated by the address ecc_address.

[0122] ④ Upon receiving the fault notification, if the HA determines, based on the binding relationship (e.g., the mirror entry shown in Table 1), that the ecc_address falls within the address range of the first memory space indicated by the mirror entry (i.e., the protection range), it compares whether Idx1 and Idx2 are consistent. If they are consistent, it calculates the backup address corresponding to ecc_address, which is also the storage address of the backup data of the first service data: address = ecc_address - base_address1 + Base_address2, and then initiates an access to the bus. For example, using the IB bus: the HA in compute node N1 sends an access request for the backup data of the first service data to the IB bus connected to compute node N1 via a communication module, such as the IB module. After receiving the access request, the IB bus first translates the accessed address to a global address, then searches the IB routing table entry, encapsulates the message based on the search result, and sends the message to the IB module in compute node N2 via the IB bus.

[0123] ⑤ The IB module or IB bus in compute node N2 sends the access request for the backup data of the first service data to the HA corresponding to GPU2 in compute node N2. The HA obtains the backup data of the first service data from the HBM corresponding to GPU2 in compute node N2 and returns it to the IB bus. Then, the IB bus returns the backup data to the HA corresponding to GPU1.

[0124] ⑥ After receiving the backup data of the first service data, the HA corresponding to GPU1 returns the backup data to GPU1Core and updates the erroneous data in the HBM corresponding to GPU1, i.e., the first service data, to the backup data of the first service data. GPU1Core can then use the backup data of the first service data to execute services, preventing service interruption due to a 2-bit ECC fault.

[0125] It is understandable that the process of computing node N2 handling memory failures is similar to that of computing node N1 in the embodiment shown in Figure 5. The difference lies in that when computing node N2 handles memory failures, the modules implementing the corresponding functions of each step are adaptively adjusted to the relevant modules of computing node N2. For example, the function implemented by GPU1Core in the embodiment of Figure 5 is implemented by GPU2Core. Assuming that when computing node N2 handles memory failures, the second memory space bound to the first memory space is located in computing node Nn, and the computing units in computing node Nn include GPUn, then the function implemented by the HA corresponding to GPU1 in the embodiment of Figure 5 is implemented by the HA corresponding to GPUn.

[0126] Optionally, if the second memory space is located on a remote computing node, the size of the second memory space can be larger than the size of the first memory space, and the read / write rate supported by the second memory space can be smaller than the read / write rate supported by the first memory space.

[0127] For example, the second memory space may include the DDR in the remote computing node. In this case, the HA of the computing node where the first memory resides can obtain the backup data in the DDR of the remote computing node through a communication method similar to steps ① to ⑥ in the embodiment of Figure 5 above. The difference is that, in this case, the IB bus corresponding to GPU2 obtains data from the DDR of the computing node where GPU2 resides through the HA corresponding to GPU2 and returns it to the HA of GPU1. Furthermore, the process by which the HA corresponding to GPU2 obtains data from the DDR of the computing node where GPU2 resides can be referred to the process of the HA obtaining data from the DDR in the embodiment of Figure 4, and will not be repeated here.

[0128] This application also provides a memory space configuration device applied to a computing cluster, the computing cluster including multiple computing nodes, each of the multiple computing nodes providing memory. For example, FIG6a is a structural block diagram of a memory space configuration device provided in an embodiment of this application. As shown in FIG6a, the device includes:

[0129] A memory allocation module is used to allocate a first memory space in the computing node for services running on the computing node, and the first memory space is used to store the service data of the service.

[0130] The data redundancy module is used to create a binding relationship between the first memory space and the second memory space. The second memory space includes memory in the computing node that is different from the first memory space or memory in the multiple computing nodes that is different from the computing node. The second memory space is used to provide backup data for the business data.

[0131] The functions of the memory allocation module and the data redundancy module can be implemented in software or hardware. For example, Figure 6b is a structural block diagram of a memory space configuration device provided in an embodiment of this application. As shown in Figure 6b, based on the device shown in Figure 6a, this device further includes:

[0132] The memory manager is used to respond to access requests from services and access the first memory space. The access request is used to request the first service data. If a fault notification is received from the first memory space, and the first service data and the faulty memory indicated by the fault notification have a binding relationship, the second memory space is accessed according to the binding relationship to obtain the backup data of the first service data. Based on the backup data of the first service data, the faulty memory is handled.

[0133] For example, the implementation of the memory allocation module will be described below. Similarly, the implementation of the data redundancy module can refer to the implementation of the memory allocation module.

[0134] As an example of a software functional unit, a memory allocation module may include code running on a compute instance. A compute instance may include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the aforementioned compute instance may be one or more. For example, a memory allocation module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0135] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0136] As an example of a hardware functional unit, a memory allocation module may include at least one computing device, such as a server. Alternatively, a memory allocation module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0137] The memory allocation module includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the memory allocation module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the memory allocation module can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0138] The memory manager can be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof. In one example, the memory manager can specifically be a local agent (HA).

[0139] This application also provides a computing device 100. Figure 7 is a structural block diagram of a computing device provided in an embodiment of this application. As shown in Figure 7, the computing device 100 includes: a bus 102, a processor 104, a first memory 106, and a communication interface 108. The processor 104, the first memory 106, and the communication interface 108 communicate with each other via the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 100.

[0140] Bus 102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 3, but this does not imply that there is only one bus or one type of bus. Bus 104 can include pathways for transmitting information between various components of computing device 100 (e.g., first memory 106, processor 104, communication interface 108).

[0141] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0142] In one example, processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0143] In one example, the first memory 106 may also include the aforementioned non-volatile memory. The first memory 106 may, for example, be HBM (Hardware Body Memory) used to provide a first memory space. In another example, the computing device 100 may also include a second memory 111, which may include the aforementioned non-volatile memory. The second memory 111 may, for example, be DDR (Dual Memory Memory) used to provide a second memory space.

[0144] In any of the above examples, the non-volatile memory can store executable program code. The processor 104 executes the executable program code to implement the functions of the aforementioned memory allocation module and data redundancy module, thereby realizing the memory space configuration method. That is, the first memory 106 stores instructions for executing the memory space configuration method.

[0145] Alternatively, in any of the above examples, the non-volatile memory stores executable code, and the processor 104 executes the executable code to implement the functions of the aforementioned memory space configuration device, thereby implementing the memory space configuration method. That is, the first memory 106 stores instructions for executing the memory space configuration method.

[0146] The communication interface 108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.

[0147] In one example, the computing device 100 may further include a memory manager 110. The memory manager 110 is used to manage and maintain memory consistency, for example, for handling memory faults based on the binding relationships provided in the embodiments of this application. The memory manager may be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0148] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0149] For example, FIG8 is a structural block diagram of a computing cluster provided in an embodiment of this application. As shown in FIG8, the computing device cluster includes at least one computing device 100. The first memory 106 or the second memory 111 or the processor 104 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing memory space configuration methods in their non-volatile memory.

[0150] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0151] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a memory space configuration method.

[0152] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform a memory space configuration method.

[0153] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement the memory space configuration method in the above embodiment.

[0154] In this embodiment, the electronic device, computer storage medium, or computer program product is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0155] Any content in the various embodiments of this application, as well as any content in the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0156] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0157] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A memory space configuration method, characterized by, Applied to a computing cluster, the computing cluster comprising multiple computing nodes, each of the multiple computing nodes providing memory space, the method includes: A first memory space in the computing node is allocated to the services running on the computing node, and the first memory space is used to store the service data of the services. A binding relationship is created between the first memory space and the second memory space. The second memory space includes a memory space in the computing node that is different from the first memory space, or a memory space in the multiple computing nodes that is different from the first computing node. The second memory space is used to provide backup data for the business data.

2. The method of claim 1, wherein, After establishing the binding relationship between the first memory space and the second memory space, the method further includes: In response to the access request of the service, the first memory space is accessed, and the access request is used to request the first service data; Upon receiving a fault notification from the first memory space, if the first service data and the faulty memory indicated by the fault notification satisfy the binding relationship, then the second memory space is accessed according to the binding relationship to obtain backup data of the first service data, and fault handling of the faulty memory is performed based on the backup data of the first service data.

3. The method according to claim 1 or 2, characterized in that, The creation of the binding relationship between the first memory space and the second memory space includes: The first base address of the first memory space, the first version number of the service data, the second base address of the second memory space, and the second version number of the backup data of the service data are bound together to obtain the initial binding relationship between the first memory space and the second memory space. During the execution of the service, the initial binding relationship is updated to obtain the binding relationship between the first memory space and the second memory space.

4. The method of claim 3, wherein, The second memory space includes a memory space in the computing node that is different from the first memory space; The updating of the initial binding relationship includes: Store the current business data according to the first base address and update the first version number; The current service data is backed up based on the second base address to obtain backup data of the current service data, and the second version number is updated to correspond to the updated first version number.

5. The method of claim 4, wherein, Each computing node includes a local agent HA, and each computing node has a corresponding processing unit; After establishing the binding relationship between the first memory space and the second memory space, the method further includes: When the processing unit accesses the service data in the first memory space through the HA, if the HA receives a fault notification indicating a memory fault returned by the first memory space, the HA requests backup data of the service data from the processing unit. The processing unit instructs the second memory space to feed back the backup data of the service data to the HA; The HA handles the memory failure based on the backup data of the received service data.

6. The method of claim 3, wherein, The second memory space includes the memory space of computing nodes that are different from the computing nodes among the plurality of computing nodes; The updating of the initial binding relationship includes: Based on the first base address of the first memory space in each computing node, the current business data is stored in the first memory space of the computing node, and the first version number is updated. For the first memory space in each computing node, update the second version number in the binding relationship to which the first memory space belongs to correspond to the updated first version number in the binding relationship.

7. The method of claim 6, wherein, After establishing the binding relationship between the first memory space and the second memory space, the method further includes: If a fault notification is received, and the faulty memory indicated by the fault notification satisfies the binding relationship, and the first version number of the first service data satisfies the binding relationship within a first time threshold, then the second memory space is accessed according to the binding relationship to obtain the backup data of the first service data. The fault notification is used to indicate that there is a memory fault in the first service data in the first memory space. The service is executed based on the backup data of the first service data.

8. The method of claim 6, wherein, Each computing node includes a local agent HA and a communication module; After establishing the binding relationship between the first memory space and the second memory space, the method further includes: When the first HA accesses the service data in the first memory space, if it receives a fault notification indicating a memory fault returned by the first memory space, the first HA requests backup data of the service data from the second communication module through the first communication module. The first HA and the first communication module belong to the first computing node. The second communication module obtains backup data of the business data from the second memory space through the second HA and feeds it back to the first communication module. The second HA and the second communication module belong to the second computing node. The first HA processes the memory failure based on backup data of the service data received from the first communication module.

9. The method according to any one of claims 1 to 8, characterized in that, The update interval of the business data of the service meets the second duration threshold, or each computing node runs the same business data.

10. The method according to any one of claims 1 to 9, characterized in that, The size of the memory space in the computing node that is different from the first memory space is larger than the first memory space, and the supported read and write speed is smaller than that of the first memory space. The memory space size and supported read / write speed of the computing node that is different from the first computing node are both equal to the first memory space.

11. A memory space configuration apparatus, characterized by comprising: Applied to a computing cluster, the computing cluster comprising multiple computing nodes, each of the multiple computing nodes providing memory space, the device includes: The memory allocation module allocates a first memory space in the computing node for the services running on the computing node, and the first memory space is used to store the service data of the services. The data redundancy module is used to create a binding relationship between the first memory space and the second memory space. The second memory space includes a memory space in the computing node that is different from the first memory space or a memory space in the multiple computing nodes that is different from the computing node. The second memory space is used to provide backup data for the business data.

12. The apparatus of claim 11, wherein, The device further includes a fault handling module for: In response to the access request of the service, the first memory space is accessed, and the access request is used to request the first service data; Upon receiving a fault notification from the first memory space, if the first service data and the faulty memory indicated by the fault notification satisfy the binding relationship, then the second memory space is accessed according to the binding relationship to obtain backup data of the first service data, and fault handling of the faulty memory is performed based on the backup data of the first service data.

13. The apparatus of claim 11 or 12, wherein, The data redundancy module is specifically used for: The first base address of the first memory space, the first version number of the service data, the second base address of the second memory space, and the second version number of the backup data of the service data are bound together to obtain the initial binding relationship between the first memory space and the second memory space. During the execution of the service, the initial binding relationship is updated to obtain the binding relationship between the first memory space and the second memory space.

14. The apparatus of claim 13, wherein, The second memory space includes a memory space in the computing node that is different from the first memory space; The data redundancy module is specifically used for: Store the current business data according to the first base address and update the first version number; The current service data is backed up based on the second base address to obtain backup data of the current service data, and the second version number is updated to correspond to the updated first version number.

15. The apparatus of claim 14, wherein, The device also includes a local agent HA distributed on each computing node, and each computing node has a corresponding processing unit; When the processing unit accesses the service data in the first memory space through the HA, if the HA receives a fault notification indicating a memory fault returned by the first memory space, the HA requests backup data of the service data from the processing unit. The processing unit instructs the second memory space to feed back the backup data of the service data to the HA; The HA handles the memory failure based on the backup data of the received service data.

16. The apparatus of claim 13, wherein, The second memory space includes the memory space of computing nodes that are different from the computing nodes among the plurality of computing nodes; The data redundancy module is specifically used for: Based on the first base address of the first memory space in each computing node, the current business data is stored in the first memory space of the computing node, and the first version number is updated. For the first memory space in each computing node, update the second version number in the binding relationship to which the first memory space belongs to correspond to the updated first version number in the binding relationship.

17. The apparatus of claim 16, wherein, The device further includes a fault handling module for: If a fault notification is received, and the faulty memory indicated by the fault notification satisfies the binding relationship, and the first version number of the first service data satisfies the binding relationship within a first time threshold, then the second memory space is accessed according to the binding relationship to obtain the backup data of the first service data. The fault notification is used to indicate that there is a memory fault in the first service data in the first memory space. The service is executed based on the backup data of the first service data.

18. The apparatus of claim 16, wherein, The device also includes a local agent HA and a communication module distributed on each computing node; When the first HA accesses the service data in the first memory space, if it receives a fault notification indicating a memory fault returned by the first memory space, the first HA requests backup data of the service data from the second communication module through the first communication module. The first HA and the first communication module belong to the first computing node. The second communication module obtains backup data of the business data from the second memory space through the second HA and feeds it back to the first communication module. The second HA and the second communication module belong to the second computing node. The first HA processes the memory failure based on backup data of the service data received from the first communication module.

19. The apparatus of any one of claims 11-18, wherein, The update interval of the business data of the service meets the second duration threshold, or each computing node runs the same business data.

20. The apparatus of any one of claims 11-19, wherein, The size of the memory space in the computing node that is different from the first memory space is larger than the first memory space, and the supported read and write speed is smaller than that of the first memory space. The memory space size and supported read / write speed of the computing node that is different from the first computing node are both equal to the first memory space.

21. A cluster of computing devices, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 10.

22. A computer program product comprising instructions, wherein: When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 10.

23. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 10.

24. A chip, characterized by The chip includes a processing circuit and transceiver pins; wherein the transceiver pins and the processing circuit communicate with each other through an internal connection path, and when the processing circuit is applied to a computing cluster, it executes the method as described in any one of claims 1 to 10.

Images