Memory sharing method, apparatus and device cluster

Abstract

A memory sharing method, an apparatus and a device cluster, relating to the technical field of computers. When a first node in the device cluster applies for a shared memory space, a management node can determine, on the basis of a memory ratio configuration of a plurality of nodes comprised in the device cluster and current idle memory spaces of the plurality of nodes, a target node for providing the shared memory space. Thus, there is no need to reserve and pre-configure a shared memory pool, but to pre-set the memory ratio configuration for the plurality of nodes in the device cluster instead. When no node sends a memory application request, memory resources of the plurality of nodes in the device cluster can be used to meet a plurality of different service requirements. When the first node sends the memory application request, the management node can allocate the shared memory space to the first node on the basis of the memory ratio configuration of the plurality of nodes and the idle memory space of each of the plurality of nodes, such that a waste of storage resources resulting from reserving the shared memory pool can be avoided, thereby improving the utilization rate of the memory resources.

Description

A memory sharing method, apparatus, and device cluster

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411755824.X, filed on December 2, 2024, entitled "A Memory Sharing Method and Device Cluster", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer technology, and in particular to a memory sharing method, apparatus, and device cluster. Background Technology

[0004] Memory sharing technology is an inter-process communication technique that allows multiple processes within a single node to communicate efficiently and process data by sharing memory. As data volumes continue to grow, a single node can no longer meet the business demands for data processing. Therefore, data processing tasks can be completed collaboratively by multiple processes running on multiple nodes within a device cluster.

[0005] In a device cluster, a separate storage node can be configured to provide shared memory for all nodes in the cluster. All nodes in the cluster can access this shared memory. To provide sufficient shared memory for the nodes, the storage node has a large memory capacity. When nodes in the cluster do not require shared memory or only some nodes use a small amount of shared memory, most of the reserved memory space in the storage node remains idle, resulting in wasted memory resources and low memory utilization. Summary of the Invention

[0006] This application provides a memory sharing method, apparatus, and device cluster, which can avoid wasting storage resources by reserving a shared memory pool and improve the utilization rate of memory resources.

[0007] Firstly, this application provides a device cluster comprising multiple nodes, wherein each node may be a computing device, computing chip, computing server, host, etc. A first node in the device cluster may send a memory request to a management node, requesting the management node to allocate the required shared memory space for the first node. The management node may, based on the memory ratio configuration of the multiple nodes in the device cluster and the current free memory space of the multiple nodes, determine a target node from among the multiple nodes to provide shared memory space to the first node, and send a memory allocation request to the target node, instructing the target node to provide shared memory space to the first node. The first node may be any one of the multiple nodes in the device cluster, and the management node may also be any one of the multiple nodes. The target node may include one or more nodes, and may include the first node.

[0008] The memory ratio configuration of any node in the device cluster indicates the maximum proportion of shared memory space that any node can provide relative to its total memory resources. For example, assuming the memory ratio configuration of the sixth node in the device cluster is 60%, it means that the sixth node can provide a maximum of 60% of its total memory resources as shared memory. When the sixth node's current free memory space accounts for 80% of its total memory resources, the sixth node can provide a maximum of 60% of its total memory resources as shared memory space.

[0009] In the device cluster provided in this application, when the first node requests shared memory space, the management node can determine the target node to provide the shared memory space based on the memory ratio configuration of multiple nodes in the device cluster and the current free memory space of multiple nodes, without needing to reserve and configure a shared memory pool in advance. It only needs to pre-set the memory ratio configuration for multiple nodes in the device cluster, indicating the maximum proportion of shared memory space that the corresponding node can provide relative to its total memory resources. When no node issues a memory request, the memory resources of multiple nodes in the device cluster can be used to meet various different business needs. When the first node issues a memory request, the management node can allocate shared memory space to the first node based on the memory ratio configuration of multiple nodes and the free memory space of each node, avoiding the waste of storage resources due to reserving a shared memory pool and improving the utilization rate of memory resources.

[0010] In one possible implementation, the first node and the target node belong to the same shared domain; the shared domain is determined based on the topological connections between multiple nodes in the device cluster. A shared domain is a set of nodes containing multiple nodes, and the transmission latency between any two nodes within the shared domain is within the allowable transmission latency. Exemplarily, in some embodiments, after the topological connections between nodes in the device cluster are determined, or after the topological connections between nodes in the device cluster change, for each node in the device cluster, the management node can determine at least one shared domain corresponding to that node based on the topological connections between that node and its surrounding nodes, and save the shared domain corresponding to each node. In other embodiments, after receiving a memory request sent by the first node, the management node can determine at least one shared domain corresponding to the first node based on the topological connections between the first node and its surrounding nodes, wherein the shared domain corresponding to the first node refers to the shared domain containing the first node. For example, a management node can determine, based on the topological connections between the first node and its surrounding nodes, a shared domain that includes the first node and can be directly connected (i.e., with a transmission delay of 1 hop), and a shared domain that includes the first node and has a transmission delay of at most 2 hops between any two nodes. In other words, the transmission delay between any two nodes in each shared domain is less than a set threshold. The management node can determine target nodes within the shared domain to which the first node belongs. Since determining a shared domain involves selecting multiple nodes with low transmission delays to form a shared domain, the transmission delay between the first node and any node within its shared domain is less than the transmission delay between the first node and any node outside its shared domain.

[0011] In the above implementation, the management node determines the target node that provides shared memory space for the first node within the shared domain to which the first node belongs, so that the first node and the target node belong to the same shared domain. The transmission latency between the first node and the target node is small, which can further improve the efficiency of the first node accessing the shared memory space in the target node.

[0012] In one possible implementation, the first node belongs to both a first shared domain and a second shared domain; the sharing type of the first shared domain differs from that of the second shared domain, or the maximum transmission latency between nodes in the first shared domain differs from that between nodes in the second shared domain. Exemplarily, in some embodiments, after the topological connections between nodes in the device cluster are determined, or after the topological connections between nodes in the device cluster change, or after receiving a memory request from the first node, the management node can determine multiple shared domains containing the first node based on the topological connections between the first node and surrounding nodes, and can also determine the sharing type of each shared domain based on the topological connections between nodes in each shared domain. For example, if all nodes in a shared domain except node A are connected to node A, node A can provide shared memory space for other nodes in the shared domain, and the sharing type of the shared domain can be single-node sharing; if multiple nodes in a shared domain are interconnected and can provide shared memory space to each other, the sharing type of the shared domain can be mutual sharing.

[0013] In other embodiments, after the topological connection relationship between nodes in the device cluster is determined, or after the topological connection relationship between nodes in the device cluster changes, or after receiving a memory request sent by the first node, for the first node, the management node can determine multiple shared domains containing the first node based on the topological connection relationship between the first node and surrounding nodes, and can also determine the maximum transmission latency between nodes in each shared domain based on the topological connection relationship between nodes in each shared domain.

[0014] In one possible implementation, the management node is used to determine the target node from multiple nodes based on the sharing type carried in the memory request, the memory ratio configuration of multiple nodes, and the current free memory space of multiple nodes, wherein the sharing type includes single-node sharing or mutual sharing.

[0015] In this context, a single-node shared domain refers to a shared domain where one node provides shared memory space for multiple nodes within that domain. The remaining nodes in the shared domain do not provide shared memory space. A mutually shared domain refers to a shared domain where multiple nodes support providing shared memory space to each other. For example, assuming a first node belongs to both a first and a second shared domain, the relationship type of the first shared domain can be single-node sharing, and the sharing type of the second shared domain can be mutual sharing. In this application, the first node can specify the sharing type of the shared domain based on the data storage requirements of the currently executed business or the required shared memory space capacity. The management node can determine the target node from the shared domains of the sharing type specified by the first node, thereby flexibly meeting the storage needs of various different businesses.

[0016] In one possible implementation, the management node is used to determine the target node from multiple nodes based on the latency requirements carried in the memory request, the memory ratio configuration of multiple nodes, and the current free memory space of multiple nodes.

[0017] The latency requirement limits the maximum latency of data transmission between the node using the shared memory space and the node providing the shared memory space. Latency can be represented by the length of the data transmission path, which can be expressed as the number of hops. For example, if the first node is directly connected to the fourth node, the data transmission path between the first node and the fourth node is 1 hop when the first node accesses the shared memory space provided by the fourth node, meaning the latency is 1 hop. If the first node is directly connected to the fourth node through a third node, the data transmission path between the first node and the fourth node needs to be forwarded through the third node when accessing the shared memory space provided by the fourth node, meaning the data transmission path between the first node and the fourth node is 2 hops, meaning the latency is 2 hops. In this application, the first node can determine its latency requirement based on the latency tolerance of the currently executed business. The management node can determine the target node for providing the shared memory space based on the latency requirement of the first node, ensuring the normal execution of the business.

[0018] In one possible implementation, the management node is also used to receive memory description information of the shared memory space sent by the target node, and send the memory description information to the first node. The first node creates a first virtual memory device based on the memory description information, and applications in the first node access the shared memory space by accessing the first virtual memory device.

[0019] The memory description information may include the identifier of the target node providing the shared memory space and the address of the shared memory space. The address of the shared memory space may be a global address, used to indicate the unique address of the shared memory space within the device cluster. For example, in some embodiments, after receiving a memory allocation request from the management node and allocating shared memory space based on the request, the target node can generate a global address for the shared memory space and send it to the management node. For instance, the global address of the shared memory space may include the identifier of the target node, the identifier of the application requesting the shared memory space, and the address of the shared memory space on the target node. The application requesting the shared memory space is an application running on the first node. When the first node sends a memory request to the management node, it may carry the identifier of the application requesting the shared memory space. Similarly, the memory allocation request sent by the management node to the target node may also carry the identifier of the application requesting the shared memory space. In other embodiments, the address of the shared memory space in the memory description information may also be the physical address of the shared memory space in the memory of the target node. When the management node receives a memory allocation request from the first node, it can allocate a global address for the shared memory space based on the required capacity of the shared memory space carried in the memory allocation request. This global address is then included in the memory allocation request and sent to the target node. The target node allocates shared memory space for the first node based on the global address carried in the memory allocation request and sends the physical address of the shared memory space in the target node's memory to the management node. This allows the management node to maintain the mapping between the physical address and the global address of the shared memory space in the target node's memory. Nodes in the device cluster can access the shared memory space through the global address, simplifying address translation and addressing, and improving data access efficiency. The first node creates a first virtual memory device based on memory description information, enabling applications on the first node to quickly access the shared memory space through this virtual memory device.

[0020] In one possible implementation, the device cluster may further include a second node. The memory request sent by the first node to the management node carries a shared node identifier, which indicates the second node permitted to access the shared memory space. The management node may also send memory description information of the shared memory space to the second node based on the first memory mapping request sent by the second node. The second node creates a second virtual memory device based on the memory description information, and applications within the second node access the shared memory space by accessing the second virtual memory device.

[0021] In the above implementation, before using the shared memory space, the second node can send a memory mapping request to the management node to obtain the memory description information of the shared memory space. Based on the memory description information, the second node creates a second virtual memory device, enabling applications on the second node to quickly access the shared memory space through the second virtual memory device.

[0022] In one possible implementation, the management node is also used to record the second node's reference to the shared memory space based on the first memory mapping request sent by the second node. The management node is also used to delete the second node's reference to the shared memory space based on the first memory unmapping request sent by the second node.

[0023] In the above implementation, the management node can record the second node's reference to the shared memory space based on the memory mapping request sent by the second node, and delete the second node's reference to the shared memory space based on the memory demapping request sent by the second node. Therefore, the management node can manage the shared memory space uniformly, without requiring each node in the device cluster to manage the shared memory space independently.

[0024] In one possible implementation, the management node is also used to record the first node's reference to the shared memory space based on the second memory mapping request sent by the first node. The management node is also used to delete the first node's reference to the shared memory space based on the second memory demapping request sent by the first node.

[0025] In the above implementation, the management node can record the first node's reference to the shared memory space based on the memory mapping request sent by the first node, and delete the first node's reference to the shared memory space based on the memory unmapping request sent by the first node. Therefore, the management node can manage the shared memory space uniformly, without requiring each node in the device cluster to manage the shared memory space independently.

[0026] In one possible implementation, the management node is also used to send a notification to the target node to delete the shared memory space based on the memory deletion request sent by the first node. The target node can then release the shared memory space based on the notification, and the released memory resources can be used to meet the storage needs of other services, avoiding long-term occupation of the shared memory space.

[0027] Through the above process, the management node can maintain and manage the lifecycle of the shared memory space, avoiding the resource consumption caused by each node independently maintaining the shared memory space.

[0028] In one possible implementation, the management node is also used to update the memory ratio configuration of multiple nodes based on received configuration information. The configuration information is user-inputted adjustment information regarding the memory ratio configuration of some or all of the multiple nodes.

[0029] In the above implementation, the management node provides users with the function of adjusting the memory ratio configuration of the nodes, allowing users to adjust the memory ratio configuration of each node in real time as needed.

[0030] In one possible implementation, multiple nodes in a device cluster can be connected via a memory interconnect bus, and the first node can access the shared memory space provided by the target node via the memory interconnect bus using memory semantics.

[0031] In the above implementation, the first node uses memory semantics to access the shared memory space provided by the target node, which allows for faster data reading, reduces latency, and improves data processing efficiency.

[0032] Secondly, this application provides a memory sharing method, which can be executed by a management node in a device cluster. The device cluster may include multiple nodes, and the management node can be any one of these nodes. The memory sharing method executed by the management node may include:

[0033] Receive memory request sent by the first node; the first node can be any of multiple nodes.

[0034] Based on the memory request, and according to the memory ratio configuration of multiple nodes and the current free memory space of multiple nodes, the target node for providing shared memory space to the first node is determined from multiple nodes.

[0035] Send a memory allocation request to the target node; the memory allocation request is used to instruct the target node to provide shared memory space for the first node.

[0036] In one possible implementation, the memory ratio configuration of any node in the device cluster is used to indicate the maximum proportion of shared memory space that any node can provide relative to the memory resources of any node.

[0037] In one possible implementation, the management node determines the target node within the shared domain to which the first node belongs. The shared domain is determined based on the topological connections between multiple nodes in the device cluster.

[0038] In one possible implementation, the memory request sent by the first node carries a latency requirement; the management node can determine the target node from multiple nodes based on the latency requirement carried in the memory request, the memory ratio configuration of multiple nodes, and the current free memory space of multiple nodes.

[0039] In one possible implementation, the memory request sent by the first node carries a shared type; the management node can determine the target node from multiple nodes based on the shared type carried in the memory request, the memory ratio configuration of multiple nodes, and the current free memory space of multiple nodes.

[0040] In one possible implementation, the management node can also receive memory description information of the shared memory space sent by the target node, and send the memory description information of the shared memory space to the first node.

[0041] In one possible implementation, the management node can also record the first node's reference to the shared memory space based on the memory mapping request sent by the first node, and delete the first node's reference to the shared memory space based on the memory demapping request sent by the first node.

[0042] In one possible implementation, the management node can also send memory description information of the shared memory space to the second node based on the memory mapping request sent by the second node.

[0043] In one possible implementation, the management node can also record the second node's reference to the shared memory space based on the memory mapping request sent by the second node, and delete the second node's reference to the shared memory space based on the memory demapping request sent by the second node.

[0044] In one possible implementation, the management node can also send a notification to the target node to delete the shared memory space based on the memory deletion request sent by the first node.

[0045] Thirdly, this application provides a memory sharing method, which can be executed by a first node in a device cluster. The device cluster may include multiple nodes, and the first node can be any one of these nodes. The memory sharing method executed by the first node may include:

[0046] Send a memory request to the management node; the memory request is used to instruct the management node to allocate shared memory space for the first node.

[0047] Receive memory description information of the shared memory space sent by the management node; the shared memory space is provided by the target node, which is determined by the management node from multiple nodes based on the memory ratio configuration of multiple nodes in the device cluster and the current free memory space of multiple nodes.

[0048] In one possible implementation, the first node can create a first virtual memory device based on the memory description information of the shared memory space, so that applications in the first node can access the shared memory space virtual device by accessing the first virtual memory device.

[0049] In one possible implementation, after sending a memory request to the management node, the first node can also send a memory mapping request to the management node so that the management node records the first node's reference to the shared memory space.

[0050] In one possible implementation, after determining that the first node no longer needs the shared memory space, it can also send a memory demapping request to the management node so that the management node can remove the first node's reference to the shared memory space.

[0051] In one possible implementation, the first node can also send a memory deletion request to the management node, so that the management node can notify the target node to delete the shared memory space.

[0052] Fourthly, this application provides a memory sharing method, which can be executed by a second node in a device cluster. This memory sharing method may include:

[0053] Send a memory mapping request to the management node so that the management node records the second node's reference to the shared memory space;

[0054] Receive memory description information of the shared memory space sent by the management node; the shared memory space is provided by the target node, which is determined by the management node from multiple nodes based on the memory ratio configuration of multiple nodes in the device cluster and the current free memory space of multiple nodes.

[0055] In one possible implementation, the second node can create a second virtual memory device based on the memory description information of the shared memory space, so that applications in the second node can access the shared memory space virtual device by accessing the second virtual memory device.

[0056] Fifthly, this application also provides a memory sharing device that can be applied to a management node in a device cluster. The device may include:

[0057] The receiving module is used to receive memory request requests sent by the first node; the first node can be any one of multiple nodes.

[0058] The cluster management module is used to determine the target node to provide shared memory space to the first node based on the memory request, the memory ratio configuration of multiple nodes in the device cluster, and the current free memory space of multiple nodes.

[0059] The sending module is used to send a memory allocation request to the target node; the memory allocation request is used to instruct the target node to provide shared memory space for the first node.

[0060] In one possible implementation, the memory ratio configuration of any node in the device cluster is used to indicate the maximum proportion of shared memory space that any node can provide relative to the memory resources of any node.

[0061] In one possible implementation, the cluster management module is specifically used to: determine the target node in the shared domain to which the first node belongs, wherein the shared domain is determined based on the topological connection relationship between multiple nodes contained in the device cluster.

[0062] In one possible implementation, the memory request sent by the first node carries a latency requirement; the cluster management module is specifically used to: determine the target node from multiple nodes based on the latency requirement carried in the memory request, the memory ratio configuration of multiple nodes, and the current free memory space of multiple nodes.

[0063] In one possible implementation, the memory request sent by the first node carries a shared type; the cluster management module is specifically used to: determine the target node from multiple nodes based on the shared type carried in the memory request, the memory ratio configuration of multiple nodes, and the current free memory space of multiple nodes.

[0064] In one possible implementation, the cluster management module can also be used to: record the first node's reference to the shared memory space based on the memory mapping request sent by the first node, and delete the first node's reference to the shared memory space based on the memory demapping request sent by the first node.

[0065] In one possible implementation, the cluster management module can also be used to send a notification to the target node to delete the shared memory space based on the memory deletion request sent by the first node.

[0066] Sixthly, this application also provides a memory sharing device, which can be applied to a first node in a device cluster. The device cluster may include multiple nodes, and the first node may be any one of the multiple nodes. The device may include:

[0067] The cluster management agent module is used to send memory request requests to the management node; the memory request is used to instruct the management node to allocate shared memory space for the first node;

[0068] The information receiving module is used to receive memory description information of the shared memory space sent by the management node. The shared memory space is provided by the target node, which is determined by the management node from multiple nodes based on the memory ratio configuration of multiple nodes in the device cluster and the current free memory space of multiple nodes.

[0069] In one possible implementation, the cluster management agent module can also be used to: create a first virtual memory device based on the memory description information of the shared memory space, so that applications in the first node can access the shared memory space virtual device by accessing the first virtual memory device.

[0070] In one possible implementation, the cluster management agent module can also be used to: send a memory mapping request to the management node so that the management node records the first node's reference to the shared memory space.

[0071] In one possible implementation, the cluster management agent module can also be used to: send a memory demapping request to the management node so that the management node removes the first node's reference to the shared memory space.

[0072] In one possible implementation, the cluster management agent module can also be used to send a memory deletion request to the management node so that the management node notifies the target node to delete the shared memory space.

[0073] Seventhly, this application also provides a memory sharing device that can be applied to a second node in a device cluster, and the device may include:

[0074] The cluster management agent module is used to send memory mapping requests to the management node so that the management node records the second node's reference to the shared memory space;

[0075] The information receiving module is used to receive memory description information of the shared memory space sent by the management node. The shared memory space is provided by the target node, which is determined by the management node from multiple nodes based on the memory ratio configuration of multiple nodes in the device cluster and the current free memory space of multiple nodes.

[0076] In one possible implementation, the cluster management agent module can also be used to: create a second virtual memory device based on the memory description information of the shared memory space, so that applications in the second node can access the shared memory space virtual device by accessing the second virtual memory device.

[0077] Eighthly, this application also provides a computing device including at least one processor and at least one memory. The at least one memory stores one or more computer programs, the one or more computer programs including instructions that, when executed by the at least one processor, cause the computing device to perform any of the methods described in the second aspect above.

[0078] Ninthly, this application also provides a computing device including at least one processor and at least one memory. The at least one memory stores one or more computer programs, the one or more computer programs including instructions that, when executed by the at least one processor, cause the computing device to perform any of the methods described in the third or fourth aspect above. For example, the computing device may be the first node or the second node described above.

[0079] In a tenth aspect, this application provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform any of the methods provided in the second, third, or fourth aspects described above.

[0080] In one aspect, this application provides a computer program product storing instructions that, when executed by a processor, implement any one of the methods provided in the second, third, or fourth aspects described above.

[0081] The technical effects that can be achieved by any of the technical solutions in the second to eleventh aspects mentioned above can be described with reference to the technical effects that can be achieved by the technical solution in the first aspect mentioned above, and the repeated parts will not be repeated. Attached Figure Description

[0082] Figure 1 is a schematic diagram of a device cluster provided in an embodiment of this application;

[0083] Figure 2 is a schematic diagram of the hardware structure of a node provided in an embodiment of this application;

[0084] Figure 3 is a flowchart illustrating the interaction between nodes during a shared memory allocation process according to an embodiment of this application.

[0085] Figure 4 is a logical diagram illustrating the interaction process between nodes in a device cluster according to an embodiment of this application;

[0086] Figure 5 is a schematic diagram of a shared domain provided in an embodiment of this application;

[0087] Figure 6 is a schematic diagram of another shared domain provided in an embodiment of this application;

[0088] Figure 7 is a flowchart illustrating the interaction between nodes during a shared memory mapping process according to an embodiment of this application.

[0089] Figure 8 is a flowchart of the interaction between nodes during a shared memory demapping process provided in an embodiment of this application.

[0090] Figure 9 is a flowchart illustrating the interaction between nodes during the deletion of shared memory according to an embodiment of this application.

[0091] Figure 10 is a schematic diagram of a memory sharing device provided in an embodiment of this application;

[0092] Figure 11 is a schematic diagram of another memory sharing device provided in an embodiment of this application. Detailed Implementation

[0093] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be described in detail below with reference to the accompanying drawings. The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application.

[0094] The following explanations of some terms used in this application are provided to facilitate understanding by those skilled in the art.

[0095] (1) Node: A node is a node with computing and data processing capabilities. A node can also be called a computing resource, computing device, computing unit, computing server, host, etc., which are not limited in this application. In the embodiments of this application, the device cluster may include multiple nodes. The form of a node can be a physical device with computing and data processing capabilities, or a module or unit with computing and data processing capabilities deployed on a cloud platform, such as a virtual machine, control unit, etc. For example, a node, as a computing device, can specifically be a computing server, a desktop computer, or a controller of a storage array, etc., which are not limited in this application.

[0096] (2) Shared memory space: This is a memory space that can be accessed by multiple nodes in the device cluster. Shared memory can be managed by the management node in the cluster. Each node can request and map the shared memory space to its local virtual memory device when it needs to use it.

[0097] (3) Memory ratio configuration: Each node in the device cluster has a memory ratio configuration. The memory ratio configuration of a node is used to indicate the maximum proportion of shared memory space that the node can provide to the node's memory resources. The memory ratio configuration of the nodes in the device cluster can be set by the user, and the user can adjust the memory ratio configuration of each node according to the actual needs of shared memory.

[0098] (4) Memory Description Information: After a node sends a memory allocation request to the management node, the management node can notify the target node to provide shared memory space and obtain the memory description information of that shared memory space. The memory description information of the shared memory space may include the identifier of the target node providing the shared memory space and the address of the shared memory space. The address of the shared memory space can be a global address, which indicates the unique address of the shared memory space within the device cluster.

[0099] In this application embodiment, "multiple" refers to two or more. Therefore, in this application embodiment, "multiple" can also be understood as "at least two". "At least one" can be understood as one or more, such as one, two, or more. For example, "including at least one" means including one, two, or more, and it does not limit which ones are included. For example, including at least one of A, B, and C, then it could include A, B, C, A and B, A and C, B and C, or A and B and C. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / ", unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0100] Unless otherwise stated, the ordinal numbers such as "first" and "second" mentioned in the embodiments of this application are used to distinguish multiple objects, and are not used to limit the order, sequence, priority or importance of multiple objects.

[0101] The memory sharing method provided in this application embodiment can be applied to the application scenario shown in Figure 1. As shown in Figure 1, the device cluster 1000 includes multiple nodes, such as node 1, node 2, node 3... node N, etc., and the nodes can transmit data to each other. Each node in the device cluster 1000 can be a computing device, such as a server, desktop computer, switch, or storage array controller. In terms of hardware, each node can include a processor, memory, and network card. The device cluster 1000 can include a management node, which can be any one of node 1, node 2, node 3... node N, or it can be a management server or computer independent of node 1, node 2, node 3... node N. In this application embodiment, node 1, node 2, node 3... node N can include computing nodes and memory nodes, and different nodes can be connected through bus 400.

[0102] For example, as shown in Figure 2, node 1 may include a processor 110, a memory 120, and a bus interface 130. The processor 110, memory 120, and bus interface 130 can be interconnected via an internal bus, which can be an address bus, control bus, or data bus. For example, the internal bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. Node 2 may include a processor 210, a memory 220, and a bus interface 230. The processor 210, memory 220, and bus interface 230 can be interconnected via an internal bus. The internal structure of nodes 3…N can be configured with reference to nodes 1 and 2, and will not be described further here.

[0103] Processor 110 in Node 1 and processor 210 in Node 2 can be used to execute applications (APPs), which can be simply referred to as applications. Processor 110 and processor 210 can include one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), application processor (AP), image signal processing unit (ISP), microprocessor (MP), controller, video codec, baseband processor, embedded neural network processing unit (NPU) in the field of AI, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), system-on-chip (SoC), or complex programmable logic device (CPLD). Different processors can be independent devices, such as independent chips, or they can be integrated into the same chip.

[0104] The memory 120 in node 1 may include RAM 121, and may also include hard disks, disks, etc. The memory 220 in node 2 may include RAM 221, and may also include hard disks, disks, etc. Hard disks and disks have slower read / write speeds and are typically used for persistent data storage. In one implementation, data, program instructions, etc., in the hard disks and disks need to be loaded into memory first, and then the processor 110 retrieves this data and / or program instructions from memory. Hard disks include, but are not limited to, non-volatile memory, such as read-only memory (ROM), hard disk drives (HDDs), solid-state drives (SSDs), or shingled magnetic recording hard disks, etc.

[0105] Memory refers to internal storage that directly exchanges data with the processor; it can also be called a memory module. The processor can read and write data to memory at any time, and the speed is very fast. It serves as temporary data storage for the operating system or other running applications running on the processor. Memory includes volatile memory, such as random access memory (RAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), and double data rate synchronous dynamic random access memory (DDR-SDRAM), etc. It can also include non-volatile memory, such as storage class memory (SCM), or combinations of volatile and non-volatile memory. DRAM is a semiconductor memory, and like most RAM, it is a type of volatile memory device. SCM is a composite storage technology that combines the characteristics of traditional storage devices and memory. Storage class memory can provide faster read and write speeds than hard drives, but its access speed is slower than DRAM, and it is also cheaper than DRAM. DRAM, SDRAM, DDR-SDRAM, and SCM are merely illustrative examples in this embodiment. Memory may also include other random access memories, such as read-only memories, for example, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), etc. Additionally, memory may also be a dual in-line memory module (DIMM), i.e., a module composed of DRAM.

[0106] There can be one or more memory modules. For example, multiple memory modules 121 can be configured in node 1, and these multiple memory modules 121 can be of different types. This application embodiment does not limit the number or type of memory in a node. Furthermore, the user can configure the memory to have a power-saving function. A power-saving function means that when the system experiences a power outage and is then powered on again, the data stored in the memory will not be lost. Memory with a power-saving function is called non-volatile memory. Some or all of the multiple memory modules 121 in node 1 can be used to provide shared memory space.

[0107] Processor 110 in Node 1 includes cache 111, and processor 210 in Node 2 includes cache 211. The cache can be static random access memory (SRAM), etc. In some embodiments, the cache can also be located externally to the processor; for example, the processor can be located on the motherboard, and the processor's cache can be located on the same motherboard as the processor. The cache can be located between the processor and memory, allowing the processor to read and write data to the cache at any time. Reading and writing data to the cache is faster than reading and writing data to memory. Therefore, the processor can store data in the cache before writing it to memory. The cache can include Level 1 (L1), Level 2 (L2), and Level 3 (L3) caches, with Level 1 cache being the fastest. The processor writes data to or reads data from the cache or memory at a cacheline granularity. A cacheline granularity can be 64 bytes, meaning the processor can write 64 bytes of data to or read 64 bytes of data from the cache or memory at a time.

[0108] Node 1 has a bus interface 130, and bus 400 is connected to bus interface 130. Node 2 has a bus interface 230, and bus 400 is connected to bus interface 230. For example, when node 1 provides shared memory space to node 2, node 2 can access the shared memory space provided by node 1 through bus 400. Bus 400 can be a memory interconnect bus, which is a set of communication lines connecting compute nodes. The memory interconnect bus is responsible for the transmission of data and control signals between compute nodes, and can provide high-bandwidth, low-latency, and consistent connections for multiple nodes. For example, the memory interconnect bus can be a bus based on the Compute Express Link (CXL) protocol or a unified bus (UB). Through the memory extension bus, nodes can access memory across nodes using memory semantics as if accessing local memory. Memory semantics can also be called Load / Store semantics, which refers to the read / write instructions used to access local memory.

[0109] In some application scenarios, Node 1 and Node 2 may also include network interface cards (NICs). The NIC in Node 1 can be used to communicate with other nodes (such as Node 2, Node N, etc.) via the network. The NIC in Node 2 can also be used to communicate with other nodes (such as Node 1, Node N, etc.) via the network. Considering that in related technologies, Node 1 and Node 2 may share data in units of data pages, requiring data sharing via network transmission. Transmitting data pages between nodes via the network is not only slow but also consumes a significant amount of network bandwidth.

[0110] Based on this, the device cluster provided in this application embodiment is configured with a shared memory space accessible to at least two nodes. For example, the device cluster includes multiple nodes, including a first node, a second node, a target node, and a management node. The management node can be any of the multiple nodes, such as the first node, the second node, or other nodes. The first node and the second node can access the shared memory space provided by the target node without needing to transmit data between nodes over a network, resulting in faster data transmission and saving bandwidth resources. In this application embodiment, before accessing the shared memory space provided by the target node, the first node can send a memory request to the management node. The memory request requests the management node to allocate the required shared memory space for the first node. The management node can determine the target node from among the multiple nodes to provide the shared memory space to the first node based on the memory ratio configuration of the multiple nodes in the device cluster and the current free memory space of the multiple nodes, and send a memory allocation request to the target node. The memory allocation request instructs the target node to provide the shared memory space to the first node. The first node can be any of the multiple nodes, and the target node can include one or more nodes, or the target node can include the first node. Through the above process, this embodiment of the application eliminates the need for pre-reserving and configuring a shared memory pool. It only requires pre-setting memory ratio configurations for multiple nodes in the device cluster. These memory ratio configurations indicate the maximum proportion of shared memory space that a given node can provide relative to its total memory resources. When no node issues a memory request, the memory resources of multiple nodes in the device cluster can be used to meet various different business needs. When the first node issues a memory request, the management node can allocate shared memory space to the first node based on the memory ratio configurations of the multiple nodes and the free memory space of each node. This avoids wasting storage resources by reserving a shared memory pool and improves the utilization rate of memory resources.

[0111] In this embodiment, when the first node needs to use shared memory space during business execution, it can first request shared memory space, perform shared memory mapping, and then access the shared memory space. When it is determined that the shared memory space is no longer needed, it can perform shared memory unmapping and delete the shared memory space.

[0112] Figure 3 illustrates, exemplarily, a flowchart of the interaction between nodes during a shared memory space application process according to an embodiment of this application. As shown in Figure 3, the shared memory space application process may include the following steps:

[0113] S301, the first node sends a memory request to the management node.

[0114] In some application scenarios, multiple nodes in a device cluster jointly execute the same service or collaborate on related services. For example, multiple nodes in a device cluster can collaborate on training a neural network model or using the neural network model for inference. For instance, when applications in the first node and the second node jointly execute the same service, the first node, upon starting or during application execution, determines that data shared with the second node needs to be saved to a shared memory space. The first node can request the shared memory space in advance. The first node and the second node can be any two nodes in the device cluster.

[0115] As shown in Figure 4, the first node is equipped with a cluster management agent module. The first node can generate a memory request through this module and send it to the management node. The memory request may include the capacity of the shared memory space required by the first node, which can determine the required capacity based on the data storage needs of the executed business. In some embodiments, the memory request may also include one or more of the following information: shared node identifier, sharing type, and latency requirements.

[0116] The shared node identifier is the identifier of the node that shares the same memory space as the first node. For example, when applications in the first node and applications in the second node jointly execute the same business, the shared node identifier may include the identifier of the second node. When more nodes jointly execute the same business with the first and second nodes, the shared node identifier may also include the identifiers of the additional nodes.

[0117] The sharing type indicates the type of shared domain to which the node providing the shared memory space belongs. A shared domain is determined based on the topological connections between multiple nodes in a device cluster. A device cluster can have one or more shared domains that include the first node; that is, the first node can belong to multiple shared domains. For example, the first node can belong to a first shared domain, and the first node can also belong to a second shared domain. The first and second shared domains can contain different nodes. For instance, the first shared domain can include a first node, a second node, a third node, and a fourth node, while the second shared domain can include a first node, a second node, and a fifth node.

[0118] The sharing type of the first shared domain and the sharing type of the second shared domain can be the same or different. In some embodiments, the sharing type can include single-node sharing and mutual sharing. A single-node shared domain refers to a shared domain in which one node provides shared memory space for multiple nodes within the domain. Apart from the node providing the shared memory space, the remaining nodes in the shared domain do not provide shared memory space. For example, in some embodiments, as shown in Figure 5(a), the first, second, and third nodes in the first shared domain are all connected to a fourth node via a bus. The fourth node can provide shared memory space for the first, second, and third nodes. The sharing type of the first shared domain is single-node sharing. A mutually shared domain refers to a shared domain in which multiple nodes support providing shared memory space to each other. For example, as shown in Figure 5(b), the first, second, and fifth nodes in the second shared domain are interconnected in pairs. The first node can provide shared memory space for the second and fifth nodes, the second node can provide shared memory space for the first and fifth nodes, and the fifth node can provide shared memory space for the second and first nodes. The sharing type of the second shared domain is mutual sharing.

[0119] In some embodiments, the sharing type may also include partial node sharing. For example, a shared domain includes a first node, a second node, a third node, a fourth node, and a fifth node, wherein the first node and the second node can provide shared memory space, and the first node, the second node, the third node, the fourth node, and the fifth node can all access the shared memory space provided by the first node or the second node; or, the first node, the second node, and the third node can provide shared memory space, and the first node, the second node, the third node, the fourth node, and the fifth node can all access the shared memory space provided by the first node, the second node, or the third node.

[0120] The first node can determine the sharing type in the memory request based on the data storage needs of the currently executed business or the required shared memory space capacity. For example, when the data storage needs or the required shared memory space capacity of the executed business are small, less than or equal to a set value, the sharing type in the memory request can be determined to be single-node sharing. When the data storage needs or the required shared memory space capacity of the executed business are large, greater than the set value, the sharing type in the memory request can be determined to be mutual sharing. The set value can be determined based on the maximum capacity of shared memory space that a single node can provide.

[0121] Latency requirements limit the maximum latency of data transmission between nodes using shared memory and nodes providing shared memory. Latency can be represented as the length of the data transmission path, which can be expressed as the number of hops. For example, if the first node is directly connected to the fourth node, the data transmission path between the first node and the fourth node is 1 hop long, meaning the latency is 1 hop. If the first node is directly connected to the fourth node through a third node, the data transmission path between the first node and the fourth node is 2 hops long, meaning the latency is 2 hops, and so on.

[0122] In some embodiments, the transmission delays corresponding to different shared domains may be different. The transmission delay corresponding to a shared domain refers to the maximum transmission delay between nodes contained in that shared domain. For example, the first node may belong to a first shared domain and a second shared domain. The first and second shared domains may contain different nodes. The transmission delay corresponding to the first shared domain and the transmission delay corresponding to the second shared domain may be the same or different. For example, as shown in Figure 6(a), when transmitting data between the second and third nodes, and between the second and fourth nodes, data needs to be forwarded through the first node. The transmission delay between the second and third nodes, and between the second and fourth nodes, is 2 hops, which is the maximum transmission delay. That is, the maximum transmission delay between nodes contained in the first shared domain is 2 hops, and the transmission delay of the first shared domain is 2 hops. As shown in Figure 6(b), among the multiple nodes contained in the second shared domain, every two nodes are directly connected, and the transmission delay between every two nodes is 1 hop. That is, the maximum transmission delay between the nodes contained in the second shared domain is 1 hop, and the transmission delay of the second shared domain is 1 hop.

[0123] The first node can determine the latency requirements included in the memory allocation request based on the latency tolerance of the currently executed business.

[0124] S302, the management node determines the target node from multiple nodes based on the memory ratio configuration of multiple nodes in the device cluster and the current free memory space of multiple nodes.

[0125] The target node is used to provide shared memory space to the first node, and the target node may include one or more nodes.

[0126] The management node stores the memory ratio configuration for each node in the device cluster. This configuration indicates the maximum proportion of shared memory space that node can provide relative to its total memory resources. For example, a 50% memory ratio configuration for the first node means that the maximum proportion of shared memory space that the first node can provide relative to its total memory resources is 50%. The memory ratio configuration for each node can be pre-set by the user and stored in the management node. Furthermore, the user can adjust the memory ratio configuration of some or all nodes in the device cluster as needed. The management node can receive configuration information input by the user, which may include adjustments to the memory ratio configuration of some or all nodes in the device cluster. The management node can update the memory ratio configuration of multiple nodes in the device cluster based on the received configuration information.

[0127] As shown in Figure 4, the management node is equipped with a cluster management module, which can be used to allocate shared memory space. When the management node receives a memory request from the first node, it can use the cluster management module to determine the target node to provide the shared memory space.

[0128] In some embodiments, the cluster management module can determine a target node from multiple nodes based on the shared memory space capacity required by the first node carried in the memory request, the memory ratio configuration of multiple nodes, and the current free memory space of multiple nodes. For example, if the memory ratio configuration of the sixth node is 60%, it indicates that the shared memory space that the sixth node can provide can account for up to 60% of the sixth node's memory resources. When the current free memory space of the sixth node accounts for 80% of the sixth node's memory resources, the sixth node can provide up to 60% of its memory resources as shared memory space; when the current free memory space of the sixth node accounts for 40% of the sixth node's memory resources, the sixth node can provide up to 40% of its memory resources as shared memory space. In one embodiment, if the memory ratio configuration of the sixth node in the device cluster is greater than or equal to the proportion of the sixth node's current free memory space in the sixth node's memory resources, and the current free memory space of the sixth node is greater than the capacity of the shared memory space required by the first node, the cluster management module can use the sixth node as the target node.

[0129] When the memory ratio configuration and current free memory space of multiple nodes in the device cluster meet the requirements, the cluster management module can select the node with the best performance as the target node from the multiple nodes that meet the requirements. The node with the best performance can be the node with the shortest data transmission path to the first node or the node with the largest current free memory space.

[0130] When each node in the device cluster cannot provide a sufficiently large shared memory space for the first node, multiple target nodes can be identified, and these target nodes can jointly provide the shared memory space for the first node. For example, assuming the fifth node has 1024MB of memory resources, a memory allocation of 40%, and 800MB of currently free memory, then the shared memory space that the fifth node can provide is 1024MB * 40% = 409MB. The sixth node has 1024MB of memory resources, a memory allocation of 60%, and 300MB of currently free memory. The first node requires 600MB of shared memory space. The cluster management module can use the fifth and sixth nodes as target nodes, and have them jointly provide the shared memory space for the first node. For example, the fifth node can provide 400MB, and the sixth node can provide 200MB, as the shared memory space.

[0131] In some embodiments, a device cluster may contain one or more shared domains including a first node. The first node can transmit data with each node in the shared domain via a bus. A management node includes a cluster management module. In some embodiments, after the topological connection relationship between nodes in the device cluster is determined, or after the topological connection relationship between nodes in the device cluster changes, for each node in the device cluster, the cluster management module can determine at least one shared domain corresponding to that node based on the topological connection relationship between that node and its surrounding nodes, and save the shared domain corresponding to each node. In other embodiments, after receiving a memory request from the first node, the cluster management module can determine at least one shared domain corresponding to the first node based on the topological connection relationship between the first node and its surrounding nodes. Here, the shared domain corresponding to the first node refers to a shared domain containing the first node. For example, the management node can determine, based on the topological connection relationship between the first node and its surrounding nodes, shared domains that can be directly connected (i.e., transmission latency of 1 hop) and contain the first node, and shared domains where the transmission latency between any two nodes is at most 2 hops and also contains the first node. That is, the transmission latency between any two nodes in each shared domain is less than a set threshold. The cluster management module can select target nodes that meet certain conditions from the shared domain containing the first node. The target node belongs to the same shared domain as the first node to ensure low transmission latency between them, allowing the first node to quickly access the shared memory space provided by the target node. The process of selecting a target node can be similar to the process described above, which determines the target node from multiple nodes based on the shared memory space capacity required by the first node in the memory request, the memory ratio configuration of multiple nodes, and the current free memory space of multiple nodes. This will not be elaborated further here.

[0132] In other embodiments, when the memory request includes a shared node identifier, the cluster management module can determine the shared node based on the shared node identifier in the memory request, and determine at least one shared domain including the first node and the shared node. From the determined shared domain, a target node that meets the conditions is selected. The target node and the first node and the shared node belong to the same shared domain, so as to ensure that the first node and the shared node can quickly access the shared memory space provided by the target node.

[0133] In other embodiments, when the memory request includes a shared type, the cluster management module can determine the target node from multiple nodes based on the shared type carried in the memory request, the memory ratio configuration of multiple nodes in the device cluster, and the current free memory space of the multiple nodes. The shared type indicates the type of the shared domain including the first node. For example, the cluster management module can determine at least one shared domain including the first node based on the topological connection relationship between multiple nodes in the device cluster, and select a target shared domain from the determined shared domains according to the shared type in the memory request. For instance, if the cluster management module determines that there is a first shared domain and a second shared domain including the first node, assuming the shared type of the first shared domain is single-node sharing and the shared type of the second shared domain is mutual sharing, and the shared type carried in the memory request is mutual sharing, then the cluster management module can use the second shared domain as the target shared domain and select a target node that meets the conditions from the target shared domain.

[0134] In other embodiments, when the memory request includes a shared node identifier and a sharing type, the cluster management module can determine the shared node based on the shared node identifier in the memory request, and determine at least one shared domain including the first node and the shared node, and select a target shared domain from the determined shared domains according to the sharing type in the memory request.

[0135] In other embodiments, when the memory request includes a latency requirement, the cluster management module can determine the target node from multiple nodes based on the latency requirement carried in the memory request, the memory ratio configuration of multiple nodes in the device cluster, and the current free memory space of the multiple nodes. For example, assuming the latency requirement carried in the memory request is 2 hops, the cluster management module can determine the target node from nodes whose data transmission path length to the first node is 1 hop or 2 hops, based on the memory ratio configuration of multiple nodes in the device cluster and the current free memory space of the multiple nodes.

[0136] In other embodiments, when the memory request includes a shared node identifier and latency requirements, the cluster management module can determine the shared node based on the shared node identifier in the memory request, and determine at least one shared domain including the first node and the shared node. From the determined shared domains, a target shared domain is selected based on the latency requirements in the memory request. For example, the cluster management module determines that there exists a first shared domain and a second shared domain including the first node and the shared node. Assuming the transmission latency of the first shared domain is 2 hops and the transmission latency of the second shared domain is 1 hop, when the latency requirement carried in the memory request is 2 hops, both the first and second shared domains can be used as target shared domains; when the latency requirement carried in the memory request is 1 hop, the second shared domain can be used as the target shared domain. A target node that meets the conditions is selected from the target shared domains to ensure that the data transmission latency between the node using the shared memory space and the node providing the shared memory space meets the requirements.

[0137] In other embodiments, when the memory request includes a shared node identifier, sharing type, and latency requirement, the cluster management module can determine the shared node based on the shared node identifier in the memory request, and determine at least one shared domain including the first node and the shared node. From the determined shared domains, a target shared domain is selected based on the sharing type and latency requirement in the memory request. For example, if the cluster management module determines that there exists a first shared domain and a second shared domain including the first node and the shared node, assuming the sharing type of the first shared domain is single-node sharing, the sharing type of the second shared domain is mutual sharing, the transmission latency of the first shared domain is 2 hops, the transmission latency of the second shared domain is 1 hop, the sharing type carried in the memory request is mutual sharing, and the latency requirement carried in the memory request is 2 hops, then the second shared domain can be used as the target shared domain, and a target node that meets the conditions can be selected from the target shared domain.

[0138] S303, the management node sends a memory allocation request to the target node.

[0139] After identifying a target node, the management node can send a memory allocation request to it. This request instructs the target node to provide shared memory space, and it carries the required capacity of that shared memory. When there is only one target node, the required capacity in the memory allocation request is equal to the capacity of the first target node. When there are multiple target nodes, the management node sends a memory allocation request to each node separately. The required capacity of the shared memory in the requests sent to different target nodes can be the same or different, but it will always be less than the capacity of the first target node.

[0140] S304, the target node allocates shared memory space based on the memory allocation request and determines the memory description information of the shared memory space.

[0141] Upon receiving a memory allocation request from the management node, the target node allocates shared memory space from its free memory space based on the required capacity of the shared memory space specified in the request. It then generates a memory description for the shared memory space, which may include the identifier of the target node providing the shared memory space and the address of the shared memory space. The address of the shared memory space can be its physical address within the target node's memory, or it can be a global address. The global address of the shared memory space uniquely indicates its location within the device cluster. Nodes in the cluster can access the shared memory space through this global address, simplifying address translation and addressing, and improving data access efficiency.

[0142] For example, in some embodiments, after receiving a memory allocation request from a management node, the target node allocates shared memory space based on the memory allocation request, and generates a global address for the shared memory space. This global address is then sent to the management node. For instance, the global address may include the target node's identifier, the identifier of the application requesting the shared memory space, and the address of the shared memory space on the target node. The application requesting the shared memory space is an application running on the first node. When the first node sends a memory request to the management node, it may carry the identifier of the application requesting the shared memory space. Similarly, the management node may also carry the identifier of the application requesting the shared memory space in the memory allocation request sent to the target node. In other embodiments, the address of the shared memory space in the memory description information may also be the physical address of the shared memory space in the target node's memory. When the management node receives a memory request from the first node, it can allocate a global address for the shared memory space based on the capacity of the shared memory space required by the first node carried in the memory request, and send the global address of the shared memory space in the memory allocation request to the target node. The target node allocates shared memory space to the first node based on the global address of the shared memory space carried in the memory allocation request, and sends the physical address of the shared memory space in the target node's memory to the management node so that the management node can save the correspondence between the physical address and the global address of the shared memory space in the target node's memory.

[0143] S305, the target node sends memory description information of the shared memory space to the management node.

[0144] The management node receives the memory description information of the shared memory space sent by the target node, and saves this information. This memory description information can be saved in correspondence with the shared memory space's identifier. The management node can also save the identifiers of nodes allowed to access the shared memory space, including the identifier of the first node and the shared node identifier carried in the memory request sent by the first node. The management node can also save the requesting node for this shared memory space as the first node; that is, the node requesting the creation of this shared memory space is designated as the first node.

[0145] The management node can also create a list of referencing nodes for the shared memory space. The initial list can be empty, used to record whether any nodes reference the shared memory space. A node referencing the shared memory space can be understood as the node about to use or currently using the shared memory space. As shown in Table 1, the shared memory space is identified as A1. When creating the list of referencing nodes for shared memory space A1, no nodes referenced shared memory space A1, and the table corresponding to the referencing nodes for shared memory space A1 is empty.

[0146] Table 1

[0147] The management node can maintain the usage status of the shared memory space through the list of reference nodes corresponding to the shared memory space. The identifier of the shared memory space can be generated by the management node or generated by the first node and sent to the management node. In some embodiments, the identifier of the shared memory space can be carried in the memory request sent by the first node to the management node.

[0148] S306, the management node sends the memory description information of the shared memory space to the first node.

[0149] S307, The first node creates the first virtual memory device based on the memory description information of the shared memory space.

[0150] When the first node receives the memory description information of the shared memory space sent by the management node, it can create a first virtual memory device based on the memory description information of the shared memory space. The first virtual memory device is used to point to the shared memory space so that applications in the first node can access the shared memory space by accessing the first virtual memory device.

[0151] Through the above process, the first node completes the allocation of shared memory space. Before using the shared memory space, the first node can also perform shared memory mapping. Shared memory mapping allows the management node to record the first node's reference to the shared memory space, facilitating its management. The shared memory mapping process may include:

[0152] S308, the first node sends a memory mapping request to the management node.

[0153] S309, the management node records the first node's reference to the shared memory space.

[0154] The first node can send a memory mapping request to the management node. This request may carry an identifier for the shared memory space, indicating that the first node will use that shared memory space. Upon receiving the memory mapping request from the first node, the management node can record the first node's reference to the shared memory space. For example, the management node can add the first node's identifier to the list of reference nodes corresponding to the identifier of the shared memory space to record the first node's reference to the shared memory space. As shown in Table 2, in the list of reference nodes for shared memory space A1, the reference nodes for shared memory space A1 may include the first node.

[0155] Table 2

[0156] After the first node sends a memory mapping request to the management node, it can also map the shared memory space to the virtual address space used by the application in the first node through the first virtual memory device, so that the application in the first node can access the shared memory space by accessing the first virtual memory device.

[0157] In some embodiments, a first node may share the shared memory space provided by a target node with multiple nodes. For example, a second node may share the shared memory space provided by the target node with the first node. Before using the shared memory space, the second node may perform shared memory mapping. Figure 7 exemplarily illustrates an interaction flowchart between the second node and the management node during a shared memory mapping process provided in an embodiment of this application. As shown in Figure 7, the shared memory mapping process may include the following steps:

[0158] S701, the second node sends a memory mapping request to the management node.

[0159] After the first node completes the shared memory space allocation process, it can send the shared memory space identifier to the sharing nodes. A sharing node refers to a node that shares the shared memory space provided by the target node with the first node. For example, when the second and third nodes jointly execute the same business or collaborate on related business with the first node, the second and third nodes can act as sharing nodes of the first node. Here, the second node can refer to any one of the sharing nodes.

[0160] As shown in Figure 4, the second node is equipped with a cluster management agent module. Before using the shared memory space, the second node can send a memory mapping request to the management node through the cluster management agent module. The memory mapping request carries the identifier of the shared memory space.

[0161] S702, the management node records the second node's reference to the shared memory space.

[0162] When the management node receives a memory mapping request from the second node, it can perform a validity check on the request through the cluster management module to determine whether the second node is allowed to use the shared memory space. For example, the management node can obtain the shared node identifier corresponding to the shared memory space identifier carried in the memory mapping request sent by the second node, and determine whether the shared node identifier contains the second node's identifier. If the shared node identifier does not contain the second node's identifier, the management node determines that the second node is not allowed to use the shared memory space, and the verification fails; if the shared node identifier contains the second node's identifier, the management node determines that the second node is allowed to use the shared memory space, and the verification passes. If the verification passes, the management node can add the second node's identifier to the list of reference nodes corresponding to the shared memory space identifier to record the second node's reference to the shared memory space. As shown in Table 3, in the list of reference nodes for shared memory space A1, the reference nodes for shared memory space A1 can include the first node and the second node.

[0163] Table 3

[0164] S703, the management node sends memory description information of the shared memory space to the second node.

[0165] If the above verification passes, the management node can also obtain the memory description information of the shared memory space corresponding to the identifier of the shared memory space, and send the memory description information of the shared memory space to the second node.

[0166] S704, the second node creates a second virtual memory device based on the memory description information of the shared memory space.

[0167] The second node receives the memory description information of the shared memory space sent by the management node. Based on this information, it can create a second virtual memory device. This second virtual memory device points to the shared memory space, mapping it to the virtual address space used by applications within the second node. This allows applications in the second node to access the shared memory space via the second virtual memory device. For example, when reading or writing data, applications in the second node use virtual addresses in the virtual address space. The second virtual memory device can translate these virtual addresses into global addresses in the shared memory space, enabling the application to write or read data from the shared memory space of the target node.

[0168] Before using the shared memory space, other shared nodes can also perform shared memory mapping in the same way as the second node, and then they can use the shared memory space.

[0169] After completing the shared memory mapping process, the first and second nodes can use the shared memory space. Taking the first node as an example, applications in the first node can write data to or read data from the shared memory space during business execution. For instance, when an application in the first node needs to write data to the shared memory space, it can use a virtual address to determine the virtual address where the data needs to be written. By accessing the first virtual memory device, the virtual address for writing data is transmitted to the first virtual memory device. The first virtual memory device can convert the virtual address for writing data into a global address in the shared memory space. Through the global address in the shared memory space, it can be determined that the global address is located in the memory of the target node. In some embodiments, the first virtual memory device can first convert the virtual address for writing data into a physical address in local memory, and then convert the physical address in local memory into a global address in the shared memory space. The physical address in local memory is also virtual and does not contain actual memory resources. After determining that the global address of the shared memory space is located in the target node, the first node can send a write data instruction to the target node to write data into the shared memory space located in the target node. When an application in the first node needs to read data from the shared memory space, it also uses virtual addresses to determine the virtual address of the data to be read. By accessing the first virtual memory device, the virtual address of the data is transmitted to the first virtual memory device. The first virtual memory device can translate the virtual address of the data into a global address in the shared memory space. Using the global address in the shared memory space, it can be determined that the global address is located in the memory of the target node. The first node can then send a read data command to the target node to read the required data from the shared memory space located on the target node. Because the first node and the target node are connected via a memory expansion bus, the first node can access the shared memory space using Load / Store semantics, just like accessing local memory. It can write data to or read data from the shared memory space of the target node at the cacheline granularity.

[0170] In some embodiments, when a first node writes data to the shared memory space, it can perform cache consistency either through software or hardware. Before writing data to the shared memory space, the first node first stores the data in its cache, and then writes the data stored in its cache to the shared memory space; this process constitutes cache consistency. If the nodes in the device cluster support both software and hardware-based cache consistency, the first node can perform cache consistency according to the user-defined cache consistency method. For example, the first node can call a cache consistency interface, which then performs cache consistency according to the user-defined method. If the user-defined cache consistency method is software-based, the cache consistency interface can perform cache consistency software-based; if the user-defined cache consistency method is hardware-based, the cache consistency interface can perform cache consistency hardware-based. In some embodiments, users can set the cache consistency mode according to the memory access type of the business executed by the application. In a write-multiple read scenario, where one node writes data to the shared memory space and multiple other nodes can read data from the shared memory space, users can set the cache consistency mode to software mode to avoid the problem of large access latency caused by performing cache consistency through hardware. In a write-multiple write scenario, where multiple nodes need to write data to the shared memory space, users can set the cache consistency mode to hardware mode to avoid increasing processor resource consumption by performing cache consistency through software.

[0171] The process of a shared node using the shared memory space can be performed in the same way as the process of the first node using the shared memory space, and will not be repeated here.

[0172] After determining that the shared memory space is no longer needed, the first node can initiate a shared memory demapping request and then delete the shared memory space. Similarly, after determining that the shared memory space is no longer needed, the second node can initiate a shared memory demapping request. Figure 8 exemplarily illustrates the interaction flowchart between nodes during a shared memory demapping process provided in an embodiment of this application. As shown in Figure 8, the shared memory demapping process may include the following steps:

[0173] S801, the second node sends the first memory demapping request to the management node.

[0174] When the second node determines that the application in the second node no longer needs to use the shared memory space, for example, when the application's task has been completed or the application has been closed, the second node sends a first memory demapping request to the management node.

[0175] In some embodiments, when multiple applications in the second node perform the same task, all applications can use the shared memory space. The second node can maintain a list of applications using the shared memory space. When an application no longer needs to use the shared memory space, it can call the demapping interface to send a demapping notification to the cluster management agent module in the second node. The cluster management agent module can then remove the application from the list of applications using the shared memory space based on this demapping notification. When the list of applications using the shared memory space is empty, the second node can determine that it no longer needs to use the shared memory space. In this case, the second node can send a first memory demapping request to the management node through the cluster management agent module. The first memory demapping request carries the identifier of the shared memory space.

[0176] S802, the management node removes the second node's reference to the shared memory space based on the first memory demapping request.

[0177] When the management node receives the first memory demapping request from the second node, it obtains the identifier of the shared memory space carried in the request. The cluster management module within the management node can then remove the second node's identifier from the list of referencing nodes of that shared memory space, thereby deleting the second node's reference to the shared memory space. For example, taking shared memory space A1 as an example, the second node's identifier can be removed from the list of referencing nodes of shared memory space A1 shown in Table 3, resulting in the list of referencing nodes of shared memory space A1 shown in Table 2.

[0178] S803, the first node sends a second memory demapping request to the management node.

[0179] When the first node determines that the application in the first node no longer needs to use the shared memory space, for example, when the application's task has been completed or the application has been closed, the first node can send a second memory demapping request to the management node, which carries the identifier of the shared memory space.

[0180] S804, the management node removes the first node's reference to the shared memory space based on the second memory demapping request.

[0181] When the management node receives the second memory demapping request from the first node, it obtains the identifier of the shared memory space carried in the second memory demapping request. The cluster management module in the management node can then remove the first node's identifier from the list of referencing nodes of the shared memory space based on this identifier, thereby deleting the first node's reference to the shared memory space. For example, taking shared memory space A1 as an example, the first node's identifier can be removed from the list of referencing nodes of shared memory space A1 shown in Table 2, resulting in an empty list of referencing nodes for shared memory space A1 shown in Table 1.

[0182] Once other shared nodes determine that they no longer need to use the shared memory space, they can also follow the same method as the second node initiating a shared memory unmapping request to perform the shared memory unmapping process, which will not be elaborated here.

[0183] After initiating a shared memory unmapping request, the first node can delete the shared memory space if it determines, based on business execution status, that no node needs to use the shared memory space. Figure 9 exemplarily illustrates a flowchart of the interaction between nodes during the deletion of shared memory space according to an embodiment of this application. As shown in Figure 9, the process of deleting shared memory space may include the following steps:

[0184] S901, the first node sends a memory deletion request to the management node.

[0185] After the first node sends a second memory demapping request to the management node, it can send a memory deletion request to the management node, which carries an identifier of the shared memory space.

[0186] S902, the management node determines that there are currently no nodes referencing the shared memory space.

[0187] When the management node receives a memory deletion request from the first node, it can first verify the validity of the memory deletion request through the cluster management module. For example, the management node can obtain the identifier of the shared memory space carried in the memory deletion request, and based on the identifier of the shared memory space and the saved information, determine whether the first node is the requesting node for that shared memory space. If the management node determines that the first node is the requesting node for that shared memory space, the verification of the memory deletion request passes.

[0188] The cluster management module in the management node can obtain the list of reference nodes for the shared memory space based on the identifier of the shared memory space. If the list of reference nodes for the shared memory space is empty, it can be determined that there are currently no nodes referencing the shared memory space.

[0189] S903, the management node sends a memory space deletion notification to the target node.

[0190] The target node is the node providing the shared memory space. When the cluster management module in the management node determines that no node currently references the shared memory space, it can send a memory space deletion notification to the target node, instructing the target node to delete the corresponding shared memory space. If there are multiple target nodes, the management node can send a memory space deletion notification to each target node individually.

[0191] S904, the target node deletes the shared memory space based on the received memory space deletion notification.

[0192] The target node receives a memory space deletion notification sent by the management node. The memory space deletion notification carries the identifier of the shared memory space. The target node deletes the shared memory space based on the identifier of the shared memory space, releasing the corresponding memory resources. The released memory resources can be used to meet other business needs and improve the utilization rate of memory resources.

[0193] The above process manages the lifecycle of the shared memory space through the management node, and this process is unknown to the user.

[0194] Based on the same design concept as the above embodiments, this application also provides a memory sharing device, which can be applied to the management node described above. The management node can be any node in the device cluster. In some embodiments, as shown in FIG10, the memory sharing device 1000 may include a receiving module 1001, a cluster management module 1002, and a sending module 1003. The memory sharing device 1000 can be used to implement the functions implemented by the management node in the above method embodiments, and therefore can achieve the beneficial effects of the above method embodiments.

[0195] The receiving module 1001 can be used to receive a memory request sent by a first node; the first node can be any node among multiple nodes; the cluster management module 1002 can be used to determine a target node to provide shared memory space to the first node based on the memory request, according to the memory ratio configuration of multiple nodes in the device cluster and the current free memory space of multiple nodes; the sending module 1003 can be used to send a memory allocation request to the target node; the memory allocation request is used to instruct the target node to provide shared memory space to the first node.

[0196] It should be noted that, in some embodiments, the receiving module 1001, the cluster management module 1002, and the sending module 1003 can all execute any step in the memory sharing method. The steps implemented by the receiving module 1001, the cluster management module 1002, and the sending module 1003 can be specified as needed. The receiving module 1001, the cluster management module 1002, and the sending module 1003 each implement different steps in the memory sharing method to achieve all the functions of the memory sharing device. The memory sharing device 1000 can also employ more or fewer functional modules to implement its functions.

[0197] In the embodiments of this application, the functional modules can be integrated into a single processor, or each module can exist physically separately, or two or more modules can be integrated into a single module. The integrated modules can be implemented in hardware or as software functional modules.

[0198] Based on the same design concept as the above embodiments, this application also provides a memory sharing device, which can be applied to the first node or the second node described above. The first node can be any node in the device cluster, and the second node can also be any node in the device cluster. As shown in FIG11, the memory sharing device 1100 may include a cluster management agent module 1101 and an information receiving module 1002. The memory sharing device 1100 can be used to implement the functions implemented by the first node or the second node in the above method embodiments, thus achieving the beneficial effects of the above method embodiments.

[0199] When the memory sharing device 1100 is applied to the first node, the cluster management agent module 1101 can be used to send a memory request to the management node; the memory request is used to instruct the management node to allocate shared memory space for the first node; the information receiving module 1102 can be used to receive the memory description information of the shared memory space sent by the management node; the shared memory space is provided by the target node, and the target node is determined by the management node from multiple nodes based on the memory ratio configuration of multiple nodes in the device cluster and the current free memory space of multiple nodes.

[0200] When the memory sharing device 1100 is applied to the second node, the cluster management agent module 1101 can be used to send a memory mapping request to the management node so that the management node records the second node's reference to the shared memory space; the information receiving module 1102 can be used to receive the memory description information of the shared memory space sent by the management node.

[0201] It should be noted that, in some embodiments, the cluster management agent module 1101 can be used to execute any step in the memory sharing method, and the information receiving module 1102 can also be used to execute any step in the memory sharing method. The steps implemented by the cluster management agent module 1101 and the information receiving module 1102 can be specified as needed. The cluster management agent module 1101 and the information receiving module 1102 respectively implement different steps in the memory sharing method to achieve all the functions of the memory sharing device. The memory sharing device 1100 can also use more or fewer functional modules to implement the functions of the memory sharing device 1100.

[0202] In the embodiments of this application, the functional modules can be integrated into a single processor, or each module can exist physically separately, or two or more modules can be integrated into a single module. The integrated modules can be implemented in hardware or as software functional modules.

[0203] This application also provides a device cluster. The device cluster includes multiple nodes, wherein a first node can send a memory request to a management node; the management node can determine a target node from among the multiple nodes to provide shared memory space to the first node, based on the memory ratio configuration of the multiple nodes and the current free memory space of the multiple nodes. This device cluster can be as shown in Figure 1, and will not be described further here.

[0204] This application also provides a computing device, which can be a management node in the aforementioned device cluster. The computing device may include at least one processor and at least one memory. The processor can execute a computer program stored in the memory to perform the memory-sharing method executed by the management node in the aforementioned process. In some embodiments, the computing device may adopt the node structure shown in FIG2, which will not be described further here.

[0205] This application also provides a computing node, which can be either the first node or the second node described above. The computing node may include at least one processor and at least one memory. The processor can execute a computer program stored in the memory to perform the memory-sharing method executed by the first node or the second node in the above process. In some embodiments, the computing node may adopt the node structure shown in FIG2, which will not be described again here.

[0206] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing a computer program or instructions. The computer program or instructions can constitute a computer program product.

[0207] This application also provides a computer program product comprising computer-executable instructions. In one embodiment, the computer-executable instructions are used to cause a computer to perform the functions described in the method embodiments above.

[0208] Computer-executable instructions can be stored in a computer-readable storage medium. This application also provides a computer-readable storage medium storing executable instructions. In one embodiment, the computer-executable instructions are used to cause a computer to perform the functions described in the method embodiments above.

[0209] The computer-readable storage medium provided in the embodiments of this application may be random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), register, hard disk, portable hard disk, CD-ROM, or any other form of computer-readable storage medium known in the art.

[0210] Computer-executable instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD).

[0211] One or more of the above modules or units can be implemented by software, hardware, or a combination of both. When any of the above modules or units is implemented by software, the software exists as computer program instructions and is stored in memory. The processor can be used to execute the program instructions and implement the above method flow. The processor can include, but is not limited to, at least one of the following: CPU, microprocessor, digital signal processor (DSP), microcontroller unit (MCU), or artificial intelligence processor, etc., various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor can be built into a SoC, DPU, or ASIC, or it can be a separate semiconductor chip. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as FPGAs, PLDs, or logic circuits that implement dedicated logic operations.

[0212] When the above modules or units are implemented in hardware, the hardware can be any one or any combination of CPU, microprocessor, DSP, MCU, artificial intelligence processor, ASIC, SoC, FPGA, PLD, special purpose digital circuit, hardware accelerator or non-integrated discrete device, which can run the necessary software or perform the above method flow independently of software.

[0213] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the scope of the technology disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application.

Claims

1. A device cluster, characterized in that, The device cluster includes multiple nodes; the multiple nodes include a first node and a management node; The management node is used to determine a target node from the plurality of nodes to provide shared memory space to the first node based on the memory request sent by the first node, according to the memory ratio configuration of the plurality of nodes and the current free memory space of the plurality of nodes, and to send a memory allocation request to the target node. The memory allocation request is used to instruct the target node to provide the shared memory space; The first node is any one of the plurality of nodes.

2. The equipment cluster according to claim 1, characterized in that, The memory ratio configuration of any one of the plurality of nodes is used to indicate the maximum proportion of shared memory space that any one node can provide to the memory resources of that node.

3. The equipment cluster according to claim 1 or 2, characterized in that, The first node and the target node belong to the same shared domain; the shared domain is determined based on the topological connection relationship between multiple nodes contained in the device cluster.

4. The equipment cluster according to claim 3, characterized in that, The first node belongs to a first shared domain and the first node belongs to a second shared domain; the sharing type of the first shared domain is different from the sharing type of the second shared domain, or the maximum transmission delay between nodes in the first shared domain is different from the maximum transmission delay between nodes in the second shared domain.

5. The equipment cluster according to any one of claims 1 to 4, characterized in that, The management node is used to determine the target node from the multiple nodes based on the sharing type carried in the memory request, the memory ratio configuration of the multiple nodes, and the current free memory space of the multiple nodes; the sharing type includes single-node sharing or mutual sharing.

6. The equipment cluster according to any one of claims 1 to 5, characterized in that, The management node is used to determine the target node from the multiple nodes based on the latency requirements carried in the memory request, the memory ratio configuration of the multiple nodes, and the current free memory space of the multiple nodes.

7. The equipment cluster according to any one of claims 1 to 6, characterized in that, The management node is also configured to receive memory description information of the shared memory space sent by the target node, and send the memory description information to the first node; The first node creates a first virtual memory device based on the memory description information, and the application in the first node accesses the shared memory space by accessing the first virtual memory device.

8. The equipment cluster according to any one of claims 1 to 7, characterized in that, The device cluster also includes a second node; the memory request carries a shared node identifier; the shared node identifier is used to indicate the second node that is allowed to access the shared memory space; The management node is also used to send memory description information of the shared memory space to the second node based on the first memory mapping request sent by the second node; The second node creates a second virtual memory device based on the memory description information, and the application in the second node accesses the shared memory space by accessing the second virtual memory device.

9. The equipment cluster according to claim 8, characterized in that, The management node is also used to record the second node's reference to the shared memory space based on the first memory mapping request; The management node is also used to delete the second node's reference to the shared memory space based on the first memory demapping request sent by the second node.

10. The equipment cluster according to any one of claims 1 to 9, characterized in that, The management node is also used to record the first node's reference to the shared memory space based on the second memory mapping request sent by the first node; The management node is also used to delete the first node's reference to the shared memory space based on the second memory demapping request sent by the first node.

11. The equipment cluster according to any one of claims 1 to 10, characterized in that, The management node is also used to update the memory ratio configuration of the plurality of nodes based on the received configuration information; the configuration information is adjustment information for the memory ratio configuration of some or all of the plurality of nodes input by the user.

12. A memory sharing method, characterized in that, The method is applied to a management node in a device cluster; the device cluster includes multiple nodes; the method includes: Receive a memory request sent by a first node; the first node can be any one of the plurality of nodes; Based on the memory request, and according to the memory ratio configuration of the plurality of nodes and the current free memory space of the plurality of nodes, a target node for providing shared memory space to the first node is determined from the plurality of nodes; A memory allocation request is sent to the target node; the memory allocation request is used to instruct the target node to provide shared memory space for the first node.

13. The method according to claim 12, characterized in that, The memory ratio configuration of any one of the plurality of nodes is used to indicate the maximum proportion of shared memory space that any one node can provide to the memory resources of that node.

14. The method according to claim 12 or 13, characterized in that, The memory request carries a latency requirement; the step of determining a target node from the plurality of nodes to provide shared memory space to the first node based on the memory request, according to the memory ratio configuration of the plurality of nodes and the current free memory space of the plurality of nodes, includes: The target node is determined from the plurality of nodes based on the latency requirements carried in the memory request, the memory ratio configuration of the plurality of nodes, and the current free memory space of the plurality of nodes.

15. A memory sharing device, characterized in that, Used as a management node in a device cluster; The device cluster includes multiple nodes; the device includes: A receiving module is used to receive a memory request sent by a first node; the first node is any one of the plurality of nodes. The cluster management module is used to determine, based on the memory request, the memory ratio configuration of the multiple nodes, and the current free memory space of the multiple nodes, the target node for providing shared memory space to the first node. The sending module is used to send a memory allocation request to the target node; the memory allocation request is used to instruct the target node to provide shared memory space for the first node.

16. The apparatus according to claim 15, characterized in that, The memory ratio configuration of any one of the plurality of nodes is used to indicate the maximum proportion of shared memory space that any one node can provide to the memory resources of that node.

17. A computing device, characterized in that, It includes at least one processor and at least one memory; wherein the at least one memory stores a computer program, and when the at least one processor executes the computer program, it causes the computing device to perform the method as described in any one of claims 12 to 14.

18. A computer program product, characterized in that, It includes computer-executable instructions for causing a computer to perform the method as described in any one of claims 12 to 14.

Images