Model parameter acquisition method and related device

Abstract

Embodiments of the present application provide a model parameter acquisition method and a related device, which can be applied to the field of artificial intelligence. The method comprises: a first node obtains first request information, and obtains first response information on the basis of the first request information, wherein the first request information indicates that a first accelerator card requests first model parameters, the first response information indicates that the first accelerator card obtains the first model parameters from a memory of a second accelerator card, and the memory of the second accelerator card stores the first model parameters. Since the first accelerator card and the second accelerator card belong to the same node cluster, and a high-speed communication network is used between different accelerator cards in the same node cluster, the acquisition speed of model parameters is greatly improved.

Description

A method for obtaining model parameters and related equipment

[0001] This application claims priority to Chinese Patent Application No. 202510081800.9, filed on January 17, 2025, entitled "A Method for Obtaining Model Parameters and Related Equipment", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to artificial intelligence technology, and more particularly to a method for obtaining model parameters and related equipment. Background Technology

[0003] With the development of artificial intelligence (AI) technology, cloud servers can provide users with a large number of model inference services. For example, the process of a cloud server providing model inference services to a user may include: after receiving input data from the user, the cloud server performs an inference operation through a model to obtain a prediction result, and then returns the prediction result to the user. The cloud server can provide a large number of model inference services to users by deploying multiple model instances according to the peak demand for model inference services. These multiple model instances can be multiple instances of the same model, or they can include model instances of each of several different models. Since the number of model inference services required by users fluctuates, some of the aforementioned multiple model instances will not be needed most of the time, resulting in a waste of computing resources.

[0004] To flexibly deploy model instances according to user needs, a remote storage system and a local node cluster can be deployed. The remote storage system stores all model parameters for the various models mentioned above. When user demand for model inference services increases, new model instances can be added to the local node cluster according to user needs. The remote storage system can send the required model parameters to the nodes in the aforementioned node cluster to deploy a new model instance on the node. When user demand for model inference services decreases, some model instances on the local node cluster can be terminated, which can release computer resources in a timely manner to avoid wasting computer resources.

[0005] However, since the required model parameters need to be retrieved from the remote storage system every time a new model instance is added, the retrieval speed of model parameters is relatively slow. Summary of the Invention

[0006] This application provides a method and related equipment for obtaining model parameters, which instructs a first accelerator card to obtain model parameters from the memory of a second accelerator card within the same node cluster. Since different accelerator cards within the same node cluster use a high-speed communication network, the speed of obtaining model parameters is greatly improved.

[0007] This application provides the following technical solution:

[0008] In one aspect, this application provides a method for obtaining model parameters. In this method, a first node obtains first request information, which instructs a first accelerator card in the node cluster to request first model parameters. The first node can obtain first response information based on the first request information. The first response information instructs the first accelerator card to obtain the first model parameters from the memory of a second accelerator card in the node cluster. The first model parameters are stored in the memory of the second accelerator card. The first accelerator card and the second accelerator card belong to the same node cluster.

[0009] For example, in one scenario, the first model parameters may include all the parameters of a model (hereinafter referred to as the "first model" for convenience). Alternatively, in another scenario, the first model parameters may also include some of the model parameters of the first model. In other words, all the model parameters of the first model can be divided into at least two subsets, each of which may include some of the model parameters of the first model. The first model parameters may include the model parameters within one of the aforementioned at least two subsets. For instance, when a model has a large number of model parameters, it may be necessary to split the model parameters to distribute all the model's parameters across at least two different accelerator cards. The at least two different accelerator cards can then perform inference operations in parallel, thus dividing all the model's parameters into at least two subsets. This splitting of the model's model parameters can also be called model parameter sharding.

[0010] For example, the accelerator card in this application can also be referred to as an accelerator or accelerator chip, etc. "The second accelerator card has the first model parameters stored in its memory" can be understood as the second accelerator card being an accelerator card that is working, in other words, the second accelerator card is performing inference operations based on the first model parameters stored in its memory.

[0011] In this implementation, since the first accelerator card and the second accelerator card belong to the same node cluster, and different accelerator cards within the same node cluster use a high-speed communication network, the speed at which the first accelerator card obtains model parameters from the memory of the second accelerator card is much greater than the speed at which it obtains model parameters from a remote storage system, thus greatly improving the speed of obtaining model parameters.

[0012] In one possible implementation, the first node may store first information, and the first node obtains first response information based on the first request information, including: the first node obtains the first response information based on the first information and the first request information. The first information indicates at least one accelerator card where each set of model parameters in at least one set of model parameters resides; the first model parameters are one set of at least one set of model parameters; and the at least one accelerator card where the first model parameters reside includes a second accelerator card.

[0013] For example, any one of the above at least one set of model parameters may include all the model parameters of a model, or any one of the at least one set of model parameters may include some of the model parameters of a model, wherein the first model parameter is one of the at least one set of model parameters, and the meaning of the information of each set of model parameters is similar to the meaning of the first model parameter.

[0014] For example, the first information may include: information about each group of model parameters in at least one set of model parameters, and the identifier of each accelerator card in at least one accelerator card where each group of model parameters is located. Any group of model parameters in at least one set of model parameters is referred to as the target group model parameters. The at least one accelerator card where the target group model parameters are located can be understood as each of the aforementioned at least one accelerator card storing the target group model parameters in its memory. The information of the target group model parameters may include the identifier of the model to which the target group model parameters belong. Optionally, if the target group model parameters include part of the model parameters of a model, the information of the target group model parameters may also include a group identifier.

[0015] In this implementation, the first node can obtain the first response information based on the first information. The first information indicates at least one accelerator card where each set of model parameters is located in at least one set of model parameters. The first model parameter is one set of at least one set of model parameters. The at least one accelerator card where the first model parameter is located includes the second accelerator card. This clarifies how to obtain the first response information based on the first request information and improves the feasibility of this solution.

[0016] In one possible implementation, the first node receiving the first request information may include: the first node receiving at least two request messages, including a first request message and a second request message, wherein the second request message instructs a third accelerator card in the node cluster to request the first model parameters. The first request message originates from the third node in the node cluster, and the second request message originates from a fifth node in the node cluster. The third and fifth nodes can be the same node in the node cluster or different nodes. After obtaining the first response information based on the first request information, the first node may also send the first response information to the third node; the first node may also obtain the second response information based on the second request information, wherein the second response information instructs the third accelerator card to retrieve the first model parameters from the memory of the first accelerator card and send the second response information to the fifth node.

[0017] When the first node determines that multiple accelerator cards have requested the first model parameters, since multiple accelerator cards would need to share the bandwidth of the second accelerator card if they were to copy the first model parameters from the second accelerator card that is currently in operation, this implementation adopts a pipelined copying strategy. The first accelerator card copies the first model parameters from the second accelerator card that is currently in operation, and the third accelerator card copies the first model parameters from the first accelerator card that is currently copying the first model parameters. This is beneficial to improving the efficiency of the copying process, reducing the impact on the second accelerator card that is currently in operation, and ensuring the smoothness of the second accelerator card in performing inference operations.

[0018] In one possible implementation, before the first node receives the first response information based on the first request information, it may also receive second information indicating that the first model parameters are stored in the memory of the second accelerator card. The second information may come from the second accelerator card in the fourth node, or it may come from the central processing unit in the fourth node.

[0019] The second information is used to update the first information. For example, after receiving the second information, the first node can update the first information based on the second information. If the first information already includes information about the first model parameters, the first node can add an identifier for a second accelerator card to the at least one accelerator card containing the first model parameters to update the at least one accelerator card containing the first model parameters. If the first information does not include information about the first model parameters, the first node can add an identifier for the first model parameters and an identifier for the second accelerator card to the first information. The identifier for the second accelerator card indicates that the at least one accelerator card containing the first model parameters includes the second accelerator card.

[0020] In this implementation, after the first model parameters are stored in the memory of the second accelerator card, the first node receives second information indicating that the first model parameters are stored in the memory of the second accelerator card. The first node obtains the first information based on the second information. In other words, after the accelerator cards in the node cluster store the model parameters, they will promptly inform the first node to ensure that the first information can reflect the storage status of the model parameters in the memory of the accelerator cards in the node cluster. This is beneficial for obtaining as many model parameters as possible from the memory of the accelerator cards in the node cluster, and for improving the efficiency of the process of obtaining model parameters.

[0021] In one possible implementation, the first node may also send a second instruction to a second node in the node cluster, instructing that the second model parameters be stored in the shared memory of the second node. The second model parameters are a set of model parameters, which may include all model parameters of a given model, or may include only a portion of the model parameters of a given model. The meaning of the second model parameters is similar to that of the first model parameters.

[0022] For example, when the model instance to which the second model parameter belongs needs to be deleted, the second model parameter stored in the memory of a certain accelerator card in the second node also needs to be deleted. Before deleting the second model parameter stored in the memory of a certain accelerator card in the second node, the second model management module in the second node can send a second query message to the model scheduling module in the first node, inquiring whether the second model parameter needs to be copied to shared memory. If the model scheduling module in the first node determines that the second model parameter needs to be copied, it can send a second instruction message to the second model management module in the second node; correspondingly, after receiving the second instruction message, the second node can copy the second model parameter to the shared memory of the second node.

[0023] In this implementation, the second model parameter that meets the first condition is stored in the shared memory of the second node in the node cluster. So even if the memory of all the accelerator cards in the node cluster does not store the second model parameter, when the second model parameter is used, it can be copied from the shared memory of the second node. Compared with obtaining the second model parameter from a remote storage system, the efficiency of obtaining the second model parameter is improved.

[0024] In one possible implementation, the second model parameter satisfies a first condition, which includes: the usage frequency of the model parameter is greater than or equal to a frequency threshold, i.e., the usage frequency of the second model parameter is greater than or equal to a frequency threshold. For example, the usage frequency of the model parameter can be understood as the loading frequency of the model parameter; for instance, the first node can determine the loading frequency of the second model parameter based on the frequency at which the second model parameter is requested. Alternatively, the usage frequency of the model parameter can also be understood as the frequency of performing inference operations based on the model parameter; for instance, the first node can determine the usage frequency of the second model parameter based on the frequency of performing inference operations based on the second model parameter. Or, the usage frequency of the model parameter is the sum of the loading frequency of the model parameter and the frequency of performing inference operations based on the model parameter. Alternatively, the usage frequency of the model parameter can be the usage frequency of the model to which the model parameter belongs; the model usage frequency can be understood as the frequency of providing model inference services based on that model. Alternatively, the usage frequency of the model parameter can also be defined from other dimensions, which can be determined in conjunction with the actual application scenario.

[0025] In this implementation, the first condition includes that the usage frequency of the model parameters is greater than or equal to the frequency threshold. In other words, only the second model parameters with high usage frequency will be stored in the shared memory of the second node, so that the operation of storing the second model parameters is an effective use of the shared memory of the second node, avoiding the waste of the storage resources of the shared memory of the second node.

[0026] Secondly, this application provides a method for obtaining model parameters, the method comprising: a third node sending a first request message to a first node, the first request message instructing a first accelerator card to request a first model parameter; receiving a first response message from the first node, the first response message instructing the first accelerator card to obtain the first model parameter from the memory of a second accelerator card, the first accelerator card and the second accelerator card belonging to the same node cluster; and copying the first model parameter from the memory of the second accelerator card to the memory of the first accelerator card based on the first response message.

[0027] For example, a communication connection can be established between the first accelerator card and the second accelerator card in the third node. In other words, the first accelerator card in the third node can establish a connection with the second accelerator card. Furthermore, the first accelerator card in the third node can copy the first model parameters from the memory of the second accelerator card to the memory of the first accelerator card. In other words, after the connection is established between the first accelerator card and the second accelerator card in the third node, the first accelerator card in the third node can call the communication interface to transfer the first model parameters from the memory of the second accelerator card to the memory of the first accelerator card.

[0028] In one possible implementation, after the third node copies the first model parameters into the memory of the first accelerator card, it can also send third information to the first node, indicating that the first model parameters are stored in the memory of the first accelerator card.

[0029] In the second aspect of this application, the third node may also perform the steps in the first aspect and the various possible implementations of the first aspect. The meanings of the terms in the second aspect of this application and the various possible implementations of the second aspect, as well as the beneficial effects brought about by each possible implementation, can be referred to the descriptions in the various possible implementations of the first aspect, and will not be repeated here.

[0030] Thirdly, this application provides a model parameter acquisition device, which includes: a processing unit for obtaining first request information, the first request information instructing a first accelerator card to request first model parameters; the processing unit is further configured to obtain first response information based on the first request information, the first response information instructing the first accelerator card to obtain the first model parameters from the memory of a second accelerator card, the first model parameters being stored in the memory of the second accelerator card, and the first accelerator card and the second accelerator card belonging to the same node cluster.

[0031] In the third aspect of this application, the model parameter acquisition device is also used to perform the steps executed by the first node in the first aspect and various possible implementations of the first aspect. The meanings of the terms in the third aspect of this application and various possible implementations of the third aspect, as well as the beneficial effects brought about by each possible implementation, can be referred to the descriptions in the various possible implementations of the first aspect, and will not be repeated here.

[0032] Fourthly, this application provides a model management module, specifically used for: sending a first request message to a first node, the first request message instructing a first accelerator card to request a first model parameter; receiving a first response message from the first node, the first response message instructing the first accelerator card to obtain the first model parameter from the memory of a second accelerator card, the first accelerator card and the second accelerator card belonging to the same node cluster; and copying the first model parameter from the memory of the second accelerator card to the memory of the first accelerator card based on the first response message.

[0033] In the fourth aspect of this application, the model parameter acquisition device is also used to perform the steps executed by the third node in the second aspect and various possible implementations of the second aspect. The meanings of the terms in the fourth aspect of this application and various possible implementations of the fourth aspect, as well as the beneficial effects brought about by each possible implementation, can be referred to the descriptions in the various possible implementations of the second aspect, and will not be repeated here.

[0034] Fifthly, embodiments of this application provide an apparatus including a processor and a memory, the processor being coupled to the memory, the memory being used to store a program; the processor being used to execute the program in the memory, causing the apparatus to perform the methods described in the first or second aspect above.

[0035] In a sixth aspect, embodiments of this application provide a system comprising a first node and a third node, wherein the first node executes the method described in the first aspect, and the third node executes the method described in the second aspect.

[0036] In a seventh aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first or second aspect above.

[0037] Eighthly, embodiments of this application provide a computer program product, which includes a program that, when run on a computer, causes the computer to perform the methods described in the first or second aspect above.

[0038] Ninthly, this application provides a chip system including a processor and a communication interface. The communication interface is used to communicate with modules outside the chip system, and the processor is used to support the functions involved in the foregoing aspects. In one possible design, the chip system further includes a memory for storing necessary program instructions and data. The chip system may be composed of chips or may include chips and other discrete devices.

[0039] The second to ninth aspects of this application correspond to the first aspect or multiple possible ways of the first aspect, and have corresponding beneficial effects. Attached Figure Description

[0040] Figure 1 is a structural diagram of an artificial intelligence main framework provided in this application;

[0041] Figure 2 is a schematic diagram of an architecture of a node cluster provided in an embodiment of this application;

[0042] Figure 3 is a schematic diagram of another node cluster architecture provided in an embodiment of this application;

[0043] Figure 4 is a schematic diagram of a communication method used by nodes in a node cluster provided in an embodiment of this application;

[0044] Figure 5 is a flowchart illustrating a method for obtaining model parameters provided in an embodiment of this application;

[0045] Figure 6 is a schematic diagram of a first accelerator card obtaining first model parameters from the memory of a second accelerator card according to an embodiment of this application;

[0046] Figure 7 is another schematic diagram of the method for obtaining model parameters provided in the embodiments of this application;

[0047] Figure 8 is a schematic diagram of another method for obtaining model parameters provided in the embodiments of this application;

[0048] Figure 9 is another schematic diagram of the method for obtaining model parameters provided in the embodiments of this application;

[0049] Figure 10 is a schematic diagram of a pipelined replication strategy provided in an embodiment of this application;

[0050] Figure 11 is another schematic diagram of the method for obtaining model parameters provided in the embodiments of this application;

[0051] Figure 12 is a schematic diagram of a model parameter acquisition device provided in an embodiment of this application;

[0052] Figure 13 is a schematic diagram of a device provided in an embodiment of this application. Detailed Implementation

[0053] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0054] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0055] In the embodiments of this application, "send" and "receive" refer to the direction of signal transmission. For example, "send information to device XX" can be understood as the destination of the information being device XX, which may include direct transmission via the air interface or indirect transmission by other units or modules via the air interface. "Receive information from device YY" can be understood as the source of the information being device YY, which may include direct reception from device YY via the air interface or indirect reception from device YY via other units or modules via the air interface. "Send" can also be understood as the "output" of the chip interface, and "receive" can also be understood as the "input" of the chip interface. In other words, sending and receiving can occur between devices or within devices, for example, through buses, traces, or interfaces between components, modules, chips, software modules, or hardware modules within a device. It is understood that information may undergo necessary processing, such as encoding and modulation, between the source and destination of information transmission, but the destination can understand the valid information from the source. Similar expressions in this application can be understood in a similar way and will not be elaborated further.

[0056] In the embodiments of this application, "instruction" can include direct and indirect instructions, as well as explicit and implicit instructions. The information indicated by a certain piece of information (hereinafter referred to as instruction information) is called the information to be instructed. In specific implementation, there are many ways to indicate the information to be instructed, such as, but not limited to, directly indicating the information to be instructed, such as the information to be instructed itself or its index. It can also indirectly indicate the information to be instructed by indicating other information, where there is an association between the other information and the information to be instructed; or it can indicate only a part of the information to be instructed, while the other parts are known or pre-agreed upon. For example, the instruction can be implemented by using a pre-agreed (e.g., protocol predefined) arrangement of various information, thereby reducing the instruction overhead to a certain extent. This application does not limit the specific method of instruction. It is understood that for the sender of the instruction information, the instruction information can be used to indicate the information to be instructed; for the receiver of the instruction information, the instruction information can be used to determine the information to be instructed.

[0057] First, the overall workflow of the artificial intelligence system is described, as shown in Figure 1. Figure 1 is a structural diagram of one aspect of the artificial intelligence framework provided in this application. The framework is then elaborated on from two dimensions: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of artificial intelligence, information (provided and processed by technical means), to the industrial ecosystem of the system.

[0058] (1) Infrastructure

[0059] The infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. Communication with the outside world is achieved through sensors; computing power is provided by intelligent chips, which can specifically employ hardware acceleration chips such as central processing units (CPUs), embedded neural network processing units (NPUs), graphics processing units (GPUs), tensor processing units (TPUs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). The basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0060] (2) Data

[0061] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0062] (3) Data processing

[0063] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0064] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training of data by symbolizing and formalizing it.

[0065] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0066] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0067] (4) General ability

[0068] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0069] (5) Smart Products and Industry Applications

[0070] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent terminals, intelligent manufacturing, intelligent transportation, smart homes, intelligent healthcare, intelligent security, intelligent driving, and smart cities.

[0071] With the development of artificial intelligence technology, cloud servers can provide users with a large number of model inference services. For example, the process of cloud servers providing model inference services to users may include: after receiving the user's input data, the cloud server performs inference operations through a model to obtain a prediction result and returns the prediction result to the user.

[0072] Cloud servers can provide a large number of model inference services to users by deploying multiple model instances based on peak demand. These multiple model instances can include multiple instances of the same model or instances of each of several different models. Since the number of model inference services needed by users fluctuates, some of these multiple model instances will be unnecessary most of the time, resulting in wasted computing resources. To flexibly deploy model instances according to user demand, a remote storage system and a local node cluster can be deployed. The remote storage system stores all model parameters for the various models. When user demand for model inference services increases, new model instances can be added to the local node cluster. The remote storage system can then send the required model parameters to the nodes in the cluster to deploy a new model instance. When user demand for model inference services decreases, some model instances on the local node cluster can be terminated, releasing computing resources promptly and avoiding waste. However, since each time a new model instance is added, the required model parameters need to be retrieved from the remote storage system, the parameter retrieval speed is relatively slow.

[0073] To improve the speed of model parameter acquisition, this application discloses that: a first node obtains a first request message, which instructs a first accelerator card in the node cluster to request the first model parameters. The first node can generate a first response message for the first request message, which instructs the first accelerator card to obtain the first model parameters from the memory of a second accelerator card in the node cluster. The first model parameters are stored in the memory of the second accelerator card. Since the first and second accelerator cards belong to the same node cluster, and different accelerator cards in the same node cluster use a high-speed communication network, the speed at which the first accelerator card obtains the model parameters from the memory of the second accelerator card is much greater than the speed at which it obtains the model parameters from a remote storage system, thus greatly improving the speed of model parameter acquisition.

[0074] Before detailing the method for obtaining model parameters provided in this application, please refer to Figure 2. Figure 2 is a schematic diagram of the architecture of a node cluster provided in an embodiment of this application. As shown in Figure 2, the node cluster may include multiple nodes. Figure 2 uses the example of multiple nodes in the node cluster including a master node and multiple non-master nodes. For example, each node in the multiple nodes can be a server. Each node in the node cluster includes at least one CPU and at least one accelerator card in terms of hardware. The master node can be understood as the management node of the node cluster. At the software level, the master node may run an auto scaler, a deployment controller, and a resource scheduler; optionally, a model scheduler may also run on the master node. The accelerator cards of the non-master nodes may run a first model management module. Exemplarily, the accelerator card in this application can also be replaced by an accelerator or accelerator chip, etc. An accelerator card can be understood as a chip used to perform computationally intensive operations. In other words, deploying accelerator cards in a node can improve the node's computing speed. For example, accelerator cards can be NPUs, GPUs, or TPUs, etc., and we will not list them all here.

[0075] The scaling management module, deployment control module, and resource scheduling module in the master node are used to dynamically add and / or delete model instances to meet users' elastic demands for model inference services. For example, to meet users' elastic demands for model inference services, the scaling module determines which types of model instances need to be deployed in the node cluster and the total number of each type. Based on the types of model instances to be deployed and the total number of each type determined by the scaling module, and all currently deployed model instances in the node cluster, the deployment control module determines which model instances need to be added and / or deleted. If a new model instance needs to be added, it can request computer resources from the resource scheduling module to deploy the new model instance; if a model instance needs to be deleted, it can send an instruction to the resource scheduling module to delete the model instance. The resource scheduling module is responsible for deploying new model instances and deleting model instances. "Deleting a model instance" can also be replaced with "model instance dies." The resource scheduling module can determine which non-master nodes and which accelerator cards will deploy the new model instance. For example, the resource scheduling module can send an instruction to the corresponding accelerator card to trigger it to begin deploying the new model instance. The corresponding accelerator card will need to obtain model parameters; for example, the first accelerator card in the non-master nodes of the cluster needs to obtain the first model parameter. When determining which model instances need to be deleted, the resource scheduling module can send an instruction to the corresponding accelerator card to trigger it to delete the model parameters stored in memory.

[0076] For example, after determining the first model parameter to be obtained, the first model management module on one of the accelerator cards in the non-master node (hereinafter referred to as the "first accelerator card" for ease of distinction) can send a first request message to the model scheduling module in the master node. The first request message instructs the first accelerator card to request the first model parameter. If the first model parameter is stored in the memory of at least one accelerator card in the node cluster, the model scheduling module of the master node is used to determine whether to obtain the first model parameter from another accelerator card (hereinafter referred to as the "second accelerator card") in the node cluster other than the first accelerator card. The master node then sends a first response message to the first model management module on the first accelerator card in the non-master node. The first response message instructs the first accelerator card to obtain the first model parameter from the memory of the second accelerator card. It should be understood that the example in Figure 2 is only for the convenience of understanding this scheme and is not intended to limit this scheme.

[0077] Please refer to Figure 3. Figure 3 is another schematic diagram of the node cluster architecture provided in the embodiment of this application. Figure 3 also takes the node cluster including master nodes and non-master nodes as an example. The master node can run a scaling management module (auto scaler), a deployment control module (deployment controller), a resource scheduling module (resource scheduler), and a model scheduling module (model scheduler). Figure 3 can be understood with reference to the above description of Figure 2. The repeated parts will not be described here.

[0078] The difference lies in the architecture diagram shown in Figure 3. Not only can a first model management module (model-manager) run on each accelerator card of a non-master node, but a second model management module can also run on each non-master node. The second model management module is responsible for managing the model parameters across the entire non-master node. This second model management module can run on the CPU of the non-master node, or it can run on an accelerator card of the non-master node. As shown in Figure 3, after determining the first model parameter to be obtained, the first model management module on the first accelerator card in the non-master node can send a first request message to the second model management module in the non-master node. This first request message instructs the first accelerator card to request the first model parameter, which is then sent by the second model management module in the non-master node to the model scheduling module of the master node. If the first model parameter is stored in the memory of at least one accelerator card in the node cluster, the model scheduling module of the master node determines to obtain the first model parameter from the second accelerator card in the node cluster and sends a first response message to the second model management module in the non-master node. This first response message instructs the first accelerator card to obtain the first model parameter from the memory of the second accelerator card. The second model management module in the non-master node sends the first response information to the first model management module in the non-master node. It should be understood that the example in Figure 3 is only for the convenience of understanding this scheme and is not intended to limit this scheme.

[0079] It should be noted that Figures 2 and 3 above only show two schematic diagrams of the node cluster architecture. The node cluster can also adopt other architectures. For example, the master node of the node cluster may not include a model scheduling module. After determining the first model parameters to be obtained, the first model management module on the first accelerator card in the non-master node may also independently determine to obtain the first model parameters from the second accelerator card if the first model parameters are stored in the memory of at least one accelerator card in the node cluster. The specific architecture of the node cluster can be flexibly determined in combination with the actual application scenario.

[0080] Referring to Figures 2 and 3 above, a node cluster can include multiple nodes, and each node can include one or more accelerator cards. Please refer to Figure 4, which is a schematic diagram of a communication method used in a node cluster according to an embodiment of this application. Figure 4 shows an example where each node includes multiple accelerator cards. As shown in Figure 4, the multiple accelerator cards within the same node in this node cluster are fully meshed. Multiple accelerator cards within the same node can communicate using the Huawei Cache Coherence System (HCCS) protocol. The communication bandwidth between different accelerator cards within the same node can be 28GB / s unidirectionally. Accelerator cards in different nodes within this node cluster can communicate using Ethernet (ETH). The bandwidth for ETH communication between accelerator cards in different nodes can be 28GB / s unidirectionally. For example, when the accelerator card is an NPU, the aforementioned ETH communication uses the NPU's built-in SerDes interface. SerDes is short for SE Rializer / DE Serializer. In this node cluster, different nodes can communicate via a data communication network (DCN). The bandwidth for communication between different nodes using the DCN can be 3GB / s unidirectionally. For example, communication between different nodes can be understood as communication between the CPUs of different nodes. As shown in Figure 4, a high-speed communication network is used for communication between accelerator cards on different nodes in the cluster, as well as between different accelerator cards within the same node. It should be understood that the example in Figure 4 is only for illustrative purposes and is not intended to limit this solution.

[0081] It should be noted that the method provided in this application can be applied not only to scenarios that provide model inference services to users, but also to scenarios that provide model inference services to users and perform hybrid deployment of model training, as well as to model training scenarios. The method provided in this application can be used in any scenario that requires flexible deployment of model instances.

[0082] Based on the above description, the method for obtaining model parameters provided in this application will be described in detail below. Please refer to Figure 5, which is a schematic flowchart of a method for obtaining model parameters provided in an embodiment of this application. The method for obtaining model parameters provided in an embodiment of this application may include:

[0083] 501. The first node receives the first request information, which instructs the first accelerator card to request the first model parameters.

[0084] For example, in one scenario, the first model parameters may include all the parameters of a model (hereinafter referred to as the "first model" for convenience). Alternatively, in another scenario, the first model parameters may also include some of the model parameters of the first model. In other words, all the model parameters of the first model can be divided into at least two subsets, each of which may include some of the model parameters of the first model. The first model parameters may include the model parameters within one of the aforementioned at least two subsets. For instance, when a model has a large number of model parameters, it may be necessary to split the model parameters to distribute all the model's parameters across at least two different accelerator cards. The at least two different accelerator cards can then perform inference operations in parallel, thus dividing all the model's parameters into at least two subsets. This splitting of the model's model parameters can also be called model parameter sharding.

[0085] For example, in one scenario, the first node can be the master node in a node cluster. Step 501 may include: the third node sending a first request message to the first node; correspondingly, the first node receives the first request message from the third node, the first request message instructing the first accelerator card to request the first model parameters. Here, the first accelerator card is an accelerator card within the third node, the first request message originates from the first accelerator card, the first node and the third node belong to the same node cluster, and the third node can be a non-master node in this node cluster. The architecture of this node cluster can be understood in conjunction with the above descriptions of Figures 2 and 3, and will not be elaborated further here.

[0086] Furthermore, in one implementation, the first accelerator card in the third node directly sends the first request information to the first node. For example, the first model management module (or second model management module) running on the first accelerator card in the third node sends the first request information to the first node; correspondingly, the first node receives the first request information from the first accelerator card in the third node. For example, the model scheduling module running in the first node receives the first request information from the first accelerator card in the third node.

[0087] In another implementation, after the first accelerator card in the third node generates the first request information, it sends the first request information to the first node through the central processing unit in the third node. For example, the second model management module running on the central processing unit in the third node sends the first request information to the first node; correspondingly, the first node receives the first request information from the central processing unit in the third node. For example, the model scheduling module running on the first node receives the first request information from the central processing unit in the third node.

[0088] For example, the first request information may include information about the first model parameters and the identifier of the first accelerator card; optionally, the first request information may also include the identifier of the third node to which the first accelerator card belongs.

[0089] For example, if the first model parameter includes all parameters of a model, the information of the first model parameter may include the identifier (ID) of the first model, whereby the ID of the first model indicates that the first accelerator card requests all model parameters of the first model. If the first model parameter includes only some parameters of the first model, the information of the first model parameter may include the identifier and group identifier of the first model, whereby the identifier and group identifier indicate which of the at least two groups of model parameters the first accelerator card requests. Alternatively, the ID of the first model in the information of the first model parameter may be replaced with the name of the first model, the function of the first model, or other information that can uniquely identify the first model, as determined based on the actual application scenario.

[0090] Furthermore, the identifier of the first model can be represented as a string of numbers, a string composed of numbers and letters, a string composed of numbers and text, or other forms of representation. The group identifier can also be represented as a string of numbers, a string composed of numbers and letters, a string composed of numbers and text, or other forms of representation. The specific representation can be determined based on the actual application scenario.

[0091] The identifier of the first accelerator card is used to uniquely point to the first accelerator card among the multiple accelerator cards included in the node cluster, and the identifiers of different accelerator cards in the node cluster are different; the identifier of the third node is used to uniquely point to the third node among the multiple nodes included in the node cluster, and the identifiers of different nodes in the node cluster are different.

[0092] To further understand this solution, an example of the first request information in code form is provided below:

[0093] Wherein, model_id represents the model identifier, rank_index represents the group identifier, device_id represents the accelerator card identifier, and node represents the node identifier. The information of the first model parameter may include model_id and rank_index. In the example above, the first request information indicates that the first accelerator card device_0 in node 0 requests the model parameters of the Llama13B model in the rank_0 group. It should be understood that the example here is only for the convenience of understanding this scheme and is not intended to limit this scheme.

[0094] In another scenario, the first node can also be a non-master node in the node cluster. This non-master node can independently determine where to obtain the first model parameter. Step 501 may include: the first node generating first request information, which instructs the first accelerator card to request the first model parameter. The first accelerator card is an accelerator card within the first node. The information types contained in the first request information can be found in the above description, except that the identifier of the third node is replaced with the identifier of the first node; this will not be repeated here.

[0095] Furthermore, in one implementation, step 501 may include: the first accelerator card in the first node generating first request information, for example, the first model management module running on the first accelerator card in the first node generating the first request information.

[0096] In another implementation, if a second model management module is also deployed on the first node, then after the first model management module running on the first accelerator card in the first node generates the first request information, step 501 may include: the second model management module in the first node receives the first request information sent by the first model management module on the first accelerator card in the first node.

[0097] 502. The first node obtains the first response information based on the first request information. The first response information instructs the first accelerator card to obtain the first model parameters from the memory of the second accelerator card. The first model parameters are stored in the memory of the second accelerator card. The first accelerator card and the second accelerator card belong to the same node cluster.

[0098] It should be noted that, in one case, the execution subject of step 502 is the model scheduling module in the first node; in another case, the execution subject of step 502 is the first model management module running on the first accelerator card in the first node; and in yet another case, the execution subject of step 502 is the second model management module in the first node. The subsequent description of step 502 will be based on the execution subject as the first node, and the aforementioned three cases will not be described in detail.

[0099] For example, after obtaining the first request information, the first node can determine that the first accelerator card requests the first model parameter. The first node can determine whether at least one accelerator card in the node cluster has the first model parameter stored in its memory. If at least one accelerator card in the node cluster has the first model parameter stored in its memory, the first node can generate a first response message, which instructs the first accelerator card to obtain the first model parameter from the memory of the second accelerator card. If none of the accelerator cards in the node cluster store the first model parameter, the first node can generate a third response message, which instructs the first accelerator card to obtain the first model parameter from a remote storage system.

[0100] For example, the first response information may include an identifier of the second accelerator card, which is used to uniquely point to the second accelerator card among multiple accelerator cards included in the node cluster. The identifier of the second accelerator card is used to indicate that the first model parameters are obtained from the memory of the second accelerator card. The second accelerator card may be an accelerator card that is performing inference operations; more specifically, the second accelerator card may be an accelerator card that is performing inference operations based on the first model parameters stored in memory.

[0101] Optionally, the first response information may also include information about the first model parameters. The meaning of the first model parameters can be found in the explanation in step 501 above, and will not be repeated here. Optionally, the first response information may also include the identifier of the node to which the second accelerator card belongs.

[0102] To further understand this solution, an example of the first response message in code form is provided below:

[0103] Where model_id represents the model identifier, rank_index represents the group identifier, and result indicates the second accelerator card. In the example above, the first response information indicates that LLama13B and the first model parameter pointed to by rank_0 are obtained from the memory of the second accelerator card device_5 in node 1. It should be understood that the example here is only for the convenience of understanding this scheme and is not intended to limit this scheme.

[0104] For example, the memory of multiple accelerator cards included in the multiple nodes of the node cluster can store at least one set of model parameters. Optionally, the first node can store first information indicating at least one accelerator card where each set of model parameters stored in the multiple accelerator cards of the node cluster is located. Any one of the aforementioned at least one set of model parameters can include all the model parameters of a model, or any one of the at least one set of model parameters can include some of the model parameters of a model. The specific situation can be determined according to the actual application scenario.

[0105] Furthermore, the first information may include: information about each set of model parameters in at least one set of model parameters, and the identifier of each accelerator card in at least one accelerator card containing each set of model parameters. The first model parameters are one set of at least one set of model parameters, and the meaning of the information for each set of model parameters can be understood by referring to the above description of the information for the first model parameters.

[0106] For example, in this application, any one set of model parameters in at least one set of model parameters is referred to as the target group model parameters. The at least one accelerator card where the target group model parameters are located can be understood as the memory of each of the aforementioned at least one accelerator card storing the target group model parameters. The information of the target group model parameters may include the identifier of the model to which the target group model parameters belong. Optionally, if the target group model parameters include part of the model parameters of a model, the information of the target group model parameters may also include a group identifier.

[0107] For example, the first information can be stored in the form of a table, array, text description, or other data format. To better understand this solution, an example of the first information in code form is disclosed below:

[0108] Here, node-devicelist represents a list of accelerator cards containing a set of model parameters. “model_id”:“LLama13B”, “rank_index”:“rank_0”, “node-devicelist”:“node1-device_4”, and “node1-device_6” represent at least one accelerator card containing a set of model parameters with the model identifier LLama13B and the group identifier rank_0. This includes the accelerator card device_4 in node1 and the accelerator card device_6 in node1. In other words, both node1-device_4 and node1-device_6 store a set of model parameters with the model identifier LLama13B and the group identifier rank_0 in their memory.

[0109] “model_id”:“LLama13B”, “rank_index”:“rank_1”, “node-devicelist”:“node1-device_5”, “node1-device_7” represent that at least one accelerator card containing a set of model parameters with model identifier LLama13B and group identifier rank_1 is located in node1. This includes accelerator card device_5 in node1 and accelerator card device_7 in node1. In other words, both node1-device_5 and node1-device_7 store a set of model parameters with model identifier LLama13B and group identifier rank_1 in their memory.

[0110] The string "model_id": "TaBA", "rank_index": "NULL", and "node-devicelist": "node2-device_9", "node3-device_12", and "node3-device_13" represent a set of model parameters for the model identified as TaBA. This set of parameters resides on the following accelerator cards: device_9 in node2, device_12 in node3, and device_13 in node3. It should be understood that this example is only for illustrative purposes and is not intended to limit the scope of this solution.

[0111] Optionally, step 502 may include: the first node obtaining first response information based on the first information and the first request information, wherein at least one accelerator card containing the first model parameters includes a second accelerator card.

[0112] For example, the first node can determine the information of the first model parameter based on the first request information, and perform a query operation on the first information according to the information of the first model parameter to determine whether the first information contains the information of the first model parameter, that is, to determine whether at least one accelerator card in the node cluster has the first model parameter stored in its memory. If the first information contains the information of the first model parameter, it means that at least one accelerator card in the node cluster has the first model parameter stored in its memory. Then the first node can obtain the at least one accelerator card where the first model parameter is located from the first information, determine the second accelerator card from the at least one accelerator card where the first model parameter is located, and then generate the first response information. The first response information instructs the first accelerator card to obtain the first model parameter from the memory of the second accelerator card. If the first information does not contain the information of the first model parameter, the first node can determine that the first model parameter is not stored in the memory of any of the accelerator cards in the cluster node, and then generate the third response information. The third response information instructs the first accelerator card to obtain the first model parameter from the remote storage system.

[0113] Optionally, after determining that the first model parameter is not stored in the memory of any of the accelerator cards included in the cluster, the first node can further determine whether the first model parameter is stored in the shared memory of all the nodes included in the cluster. If the first model parameter is not stored in the shared memory of any of the nodes included in the cluster, a third response message can be generated. If the first model parameter is stored in the shared memory of the sixth node in the cluster, a fourth response message can be generated. The fourth response message instructs the first accelerator card to obtain the first model parameter from the sixth node. The sixth node and the third node can be the same node or different nodes.

[0114] Regarding the specific implementation of the first node determining the second accelerator card from at least one accelerator card containing the first model parameters, for example, the first node may randomly select one accelerator card from at least one accelerator card containing the first model parameters as the second accelerator card, or the first node may select the accelerator card with the least available communication resources from at least one accelerator card containing the first model parameters as the second accelerator card, or the first node may select the accelerator card with the most remaining computer resources from at least one accelerator card containing the first model parameters as the second accelerator card, etc. The specific selection strategy for the second accelerator card can be determined based on the actual application scenario.

[0115] In this embodiment, the first node can obtain the first response information based on the first information. The first information indicates at least one accelerator card where each set of model parameters is located in at least one set of model parameters. The first model parameter is one set of at least one set of model parameters. The at least one accelerator card where the first model parameter is located includes a second accelerator card. This clarifies how to obtain the first response information based on the first request information and improves the feasibility of this solution.

[0116] Alternatively, the first node may send a first query message to each node in the node cluster, which is used to inquire whether the first model parameters are stored in the memory of the accelerator card in the node; correspondingly, each node in the node cluster can receive the first query message.

[0117] For example, if the accelerator cards in at least one node in the node cluster (hereinafter referred to as "target nodes" for convenience) store the first model parameters in their memory, each target node can send a fourth indication message to the first node. This fourth indication message indicates that the accelerator cards in the target node store the first model parameters in their memory. Correspondingly, the first node can receive the fourth indication message sent by each target node, thus obtaining at least one fourth indication message. The first node can determine a second accelerator card from the at least one accelerator card indicated by the at least one fourth indication message, and then generate a first response message. This first response message indicates that the first accelerator card obtains the first model parameters from the second accelerator card. If none of the accelerator cards in the cluster node store the first model parameters in their memory, then no node in the node cluster sends a fourth indication message to the first node. The first node can determine that none of the accelerator cards in the cluster node store the first model parameters in their memory, and then the first node can generate a third response message. This third response message indicates that the first accelerator card obtains the first model parameters from a remote storage system.

[0118] 503. The first node sends the first response information.

[0119] Step 503 is optional. If the first node is the master node in the node cluster, then step 503 can be executed. Step 503 may include: the first node sending a first response message (or a third response message, or a fourth response message) to the third node, and correspondingly, the third node receiving the first response message (or a third response message, or a fourth response message) from the first node.

[0120] Furthermore, in one implementation, the first node directly sends a first response message (or a third response message, or a fourth response message) to the first accelerator card in the third node; exemplarily, the model scheduling module running in the first node directly sends the first response message (or a third response message, or a fourth response message) to the first accelerator card in the third node. Correspondingly, the first accelerator card in the third node receives the first response message (or the third response message, or the fourth response message) from the first node; exemplarily, the first model management module (or the second model management module) running on the first accelerator card in the third node receives the first response message (or the third response message, or the fourth response message) from the first node.

[0121] In another implementation, the first node sends a first response message (or a third response message, or a fourth response message) to the central processing unit (CPU) in the third node. For example, the model scheduling module running on the first node sends the first response message (or the third response message, or the fourth response message) to the CPU in the third node. Correspondingly, the CPU in the third node receives the first response message (or the third response message, or the fourth response message) from the first node and can then forward it to the first accelerator card in the third node. For example, the second model management module running on the CPU in the third node receives the first response message (or the third response message, or the fourth response message) from the first node.

[0122] If the third node receives the first response information, a communication connection can be established between the first accelerator card and the second accelerator card in the third node. In other words, the first accelerator card in the third node can establish a connection with the second accelerator card. Then, the first accelerator card in the third node can copy the first model parameters from the memory of the second accelerator card to the memory of the first accelerator card. In other words, after the connection is established between the first accelerator card and the second accelerator card in the third node, the first accelerator card in the third node can call the communication interface to transfer the first model parameters from the memory of the second accelerator card to the memory of the first accelerator card.

[0123] For example, if the first and second accelerator cards are located within the same node, they can communicate using HCCS. Alternatively, if they are located on different nodes within the same node cluster, they can communicate using ETH. Optionally, the first and second accelerator cards can establish aggregated communication, such as send-receive communication or broadcast communication.

[0124] To understand this solution more intuitively, please refer to Figure 6. Figure 6 is a schematic diagram of a first accelerator card obtaining the first model parameters from the memory of the second accelerator card according to an embodiment of this application. As shown in Figure 6, after a communication connection is established between the first accelerator card and the second accelerator card, the first accelerator card in the third node can call the communication interface to transfer the first model parameters in the memory of the second accelerator card to the memory of the first accelerator card. It should be understood that the example in Figure 6 is only for the convenience of understanding this solution and is not intended to limit this solution.

[0125] If the third node receives the third response information, it can send a third request information to the remote storage system. This third request information is used to request the acquisition of the first model parameters. For example, the third request information may include information about the first model parameters, the meaning of which can be found in the above description and will not be elaborated here. The third node receives the first model parameters from the remote storage system and stores them in the memory of the first accelerator card.

[0126] To understand this solution more intuitively, please refer to Figure 7. Figure 7 is another schematic diagram of the method for obtaining model parameters provided in the embodiment of this application. As shown in Figure 7, the first model management module running on the first accelerator card of the third node sends a first request message to the model scheduling module on the first node. The first request message instructs the first accelerator card to request the first model parameter. The model scheduling module on the first node can send a response message to the first model management module running on the first accelerator card of the third node. If the first model management module running on the first accelerator card of the third node obtains the first response message, the first accelerator card of the third node obtains the first model parameter from the memory of the second accelerator card. If the first model management module running on the first accelerator card of the third node obtains the third response message, the third node obtains the first model parameter from the remote storage system and stores the first model parameter in the memory of the first accelerator card. It should be understood that the example in Figure 7 is only for the convenience of understanding this solution and is not intended to limit this solution.

[0127] If the third node receives the fourth response information, the fourth response information instructs the first accelerator card to obtain the first model parameters from the shared memory of the sixth node. If the third node and the sixth node are the same node in the node cluster, the third node can directly obtain the first model parameters from the shared memory and store them in the memory of the first accelerator card. If the third node and the sixth node are different nodes, the third node can send a fourth request information to the sixth node to request the first model parameters. The third node receives the first model parameters from the sixth node and stores them in the memory of the first accelerator card.

[0128] If the first node is a non-master node in the node cluster and the execution subject of steps 501 and 502 is the second model management module running on the non-master node, step 503 can also be executed. Step 503 may include: the second model management module in the first node sends a first response message (or a third response message, or a fourth response message) to the first model management module running on the first accelerator card of the first node.

[0129] If the first model management module running on the first accelerator card of the first node receives the first response information, a communication connection can be established between the first accelerator card and the second accelerator card in the first node. Then, the first accelerator card in the first node can copy the first model parameters from the memory of the second accelerator card to the memory of the first accelerator card.

[0130] If the first model management module running on the first accelerator card of the first node receives the third response information, the first node can send the third request information to the remote storage system. The first node receives the first model parameters sent from the remote storage system and stores the first model parameters in the memory of the first accelerator card.

[0131] If the first model management module running on the first accelerator card of the first node receives the fourth response information, and if the first node and the sixth node are the same node in the node cluster, the first node can directly obtain the first model parameters from the shared memory and store them in the memory of the first accelerator card. If the first node and the sixth node are different nodes, the first node can send a fourth request information to the sixth node to request the first model parameters. The first node receives the first model parameters from the sixth node and stores them in the memory of the first accelerator card.

[0132] The specific implementation of the above steps can be found in the above description. The only difference is that the first acceleration card in the above description is replaced with the first acceleration card. It will not be described in detail here.

[0133] If the first node is a non-master node in the node cluster, and the execution entity of steps 501 and 502 is the first model management module running on the first accelerator card of the non-master node, then step 503 can be omitted. If the first model management module running on the first accelerator card of the first node generates first response information, a communication connection can be established between the first accelerator card and the second accelerator card in the first node. Then, the first accelerator card in the first node can copy the first model parameters from the memory of the second accelerator card to the memory of the first accelerator card. If the first model management module running on the first accelerator card of the first node generates first response information, the first node can send a first request to the remote storage system. The first node receives the first model parameters sent from the remote storage system and stores the first model parameters in the memory of the first accelerator card.

[0134] If the first model management module running on the first accelerator card of the first node generates a fourth response message, and if the first node and the sixth node are the same node in the node cluster, the first node can directly obtain the first model parameters from the shared memory and store them in the memory of the first accelerator card. If the first node and the sixth node are different nodes, the first node can send a fourth request message to the sixth node to request the first model parameters. The first node receives the first model parameters from the sixth node and stores them in the memory of the first accelerator card.

[0135] In this embodiment, the first node obtains a first request message, which instructs the first accelerator card in the node cluster to request the first model parameter. The first node can generate a first response message for the first request message, which instructs the first accelerator card to obtain the first model parameter from the memory of the second accelerator card in the node cluster. The first model parameter is stored in the memory of the second accelerator card. Since the first node, the first accelerator card, and the second accelerator card belong to the same node cluster, and different accelerator cards in the same node cluster use a high-speed communication network, the speed at which the first accelerator card obtains the model parameter from the memory of the second accelerator card is much greater than the speed at which it obtains the model parameter from a remote storage system, greatly improving the speed of obtaining the model parameter.

[0136] Optionally, based on the embodiment corresponding to Figure 5, please refer to Figure 8. Figure 8 is another flowchart illustrating the method for obtaining model parameters provided in this application embodiment. In Figure 8, the first node is taken as the master node in the node cluster. The method for obtaining model parameters provided in this application embodiment may include:

[0137] 801. The first node receives the second information, which comes from the second accelerator card in the fourth node. The second information indicates that the first model parameters are stored in the memory of the second accelerator card.

[0138] Step 801 is optional. For example, after the fourth node in the node cluster stores the first model parameters in the memory of the second accelerator card, it can send the second information to the first node; correspondingly, the first node receives the second information from the fourth node. For example, the model scheduling module in the first node receives the second information from the fourth node.

[0139] It should be noted that in the embodiment shown in Figure 8, all steps executed by the first node can be executed by the model scheduling module in the first node, and the following description will focus on the first node.

[0140] In one implementation, step 801 may include: the second accelerator card in the fourth node directly sending the second information to the first node; exemplarily, the first model management module (or second model management module) running on the second accelerator card in the fourth node sends the second information to the first node. In another implementation, after the second accelerator card in the fourth node generates the second information, step 801 may include: sending the second information to the first node via the central processing unit in the fourth node; exemplarily, the second model management module running on the central processing unit in the fourth node sends the second information to the first node.

[0141] For example, the second information may include information about the first model parameters and the identifier of the second accelerator card; optionally, the second information may also include the identifier of the fourth node to which the second accelerator card belongs; it should be noted that the fourth node and the third node may be the same node in the node cluster or different nodes in the node cluster, which can be determined in combination with the actual application scenario.

[0142] For example, the first node may store first information. The meaning of the first information can be referred to the description in the embodiment corresponding to Figure 5 above, and will not be repeated here. After receiving the second information, the first node may update the first information based on the second information. If the first information already includes the information of the first model parameters, the first node may add the identifier of the second accelerator card to the at least one accelerator card where the first model parameters are located, so as to realize the update of the at least one accelerator card where the first model parameters are located; if the first information does not include the information of the first model parameters, the first node may add the identifier of the first model parameters and the identifier of the second accelerator card to the first information. The identifier of the second accelerator card indicates that the at least one accelerator card where the first model parameters are located includes the second accelerator card.

[0143] To more intuitively understand this solution, please refer to Figure 9. Figure 9 is another schematic diagram of the method for obtaining model parameters provided in this application embodiment. As shown in Figure 9, after the first model parameters are stored in the memory of the second accelerator card in the fourth node, the first model management module running on the second accelerator card in the fourth node can send second information to the first node. The second information indicates that the first model parameters are stored in the memory of the second accelerator card, and the second information is used to obtain the first information. Then, after receiving the first request information sent by the first model management module in the first accelerator card running in the third node, the first node can send first response information to the first model management module in the first accelerator card running in the third node. The first response information indicates that the first accelerator card obtains the first model parameters from the memory of the second accelerator card. It should be understood that the example in Figure 9 is only for the convenience of understanding this solution and is not intended to limit this solution.

[0144] In this embodiment, after the first model parameters are stored in the memory of the second accelerator card, the first node receives second information indicating that the first model parameters are stored in the memory of the second accelerator card. The first node obtains the first information based on the second information. In other words, after the accelerator cards in the node cluster store the model parameters, they will promptly inform the first node to ensure that the first information can reflect the storage status of the model parameters in the memory of the accelerator cards in the node cluster. This is beneficial for obtaining as many model parameters as possible from the memory of the accelerator cards in the node cluster, and for improving the efficiency of the process of obtaining model parameters.

[0145] Optionally, after a set of model parameters is deleted from the memory of an accelerator card in a node of the node cluster, the node can also send a deregistration message to the first node. The deregistration message indicates that a set of model parameters stored in the aforementioned accelerator card has been deleted. This deregistration message is used to update the first information, so that the first information can correctly reflect the storage status of model parameters in the memory of the accelerator cards included in the node cluster.

[0146] For example, after the first model parameter in the memory of the second accelerator card in the fourth node is deleted, the fourth node can also send deregistration information to the first node, indicating that the first model parameter stored in the second accelerator card has been deleted. If the first information indicates that the at least one accelerator card containing the first model parameter includes not only the second accelerator card but also other accelerator cards, then the first node can delete the identifier of the second accelerator card from the at least one accelerator card containing the first model parameter; if the first information indicates that the at least one accelerator card containing the first model parameter includes only the second accelerator card, then the first node can delete not only the identifier of the second accelerator card but also the information of the first model parameter.

[0147] 802. The first node receives at least one request message, the at least one request message including a first request message, the first request message instructing the first accelerator card to request the first model parameters.

[0148] Optionally, step 802 includes: the first node receiving at least two request messages, the at least two request messages including a first request message and a second request message, wherein the first request message comes from the first accelerator card in the third node, and the second request message comes from the third accelerator card in the fifth node; for example, step 802 may include: the first node receiving the first request message from the third node, the first node receiving the second request message from the fifth node, the specific implementation of "the first node receiving the second request message from the fifth node" is similar to the specific implementation of "the first node receiving the first request message from the third node", the difference being that the third node is replaced by the fifth node, and the first request message is replaced by the second request message. The specific implementation of the aforementioned steps can be referred to the description in the embodiment corresponding to Figure 5 above, and will not be described in detail here.

[0149] The second request information instructs the third accelerator card to request the first model parameters. The second request information may include information about the first model parameters and the identifier of the third accelerator card. Optionally, the second request information may also include the identifier of the fifth node. The third accelerator card in the fifth node belongs to the node cluster. It should be noted that the second accelerator card and the third accelerator card are two different accelerator cards in the node cluster. The fifth node is a non-master node in the node cluster. The fifth node and the third node can be the same node in the node cluster or two different nodes in the node cluster. The specific situation can be determined based on the actual application scenario.

[0150] 803. The first node obtains the first response information based on the first request information. The first response information instructs the first accelerator card to obtain the first model parameters from the memory of the second accelerator card. The first model parameters are stored in the memory of the second accelerator card. The first accelerator card and the second accelerator card belong to the same node cluster.

[0151] 804. The first node sends the first response information to the third node.

[0152] 805. Based on the first response information, the third node copies the first model parameters from the memory of the second accelerator card to the memory of the first accelerator card.

[0153] For example, the first node can send a first response message to the first accelerator card in the third node. Based on the first response message, the first accelerator card in the third node copies the first model parameters from the memory of the second accelerator card to the memory of the first accelerator card. The specific implementation methods of steps 803 to 805 and the meanings of the terms used in steps 803 to 805 can be found in the descriptions of the embodiments corresponding to Figure 5 above, and will not be repeated here.

[0154] Optionally, after the first model parameters are stored in the first accelerator card of the third node, the third node can also send third information to the first node. The third information indicates that the first model parameters are stored in the memory of the first accelerator card. The third information is used to update the first information.

[0155] The specific implementation of the above steps can be found in the description of step 801 above. The difference is that the fourth node in the above description is replaced with the third node, the second information in the above description is replaced with the third information, and the second acceleration card in the above description is replaced with the first acceleration card. The details will not be elaborated here.

[0156] 806. The first node obtains the second response information based on the second request information. The second response information instructs the third accelerator card to obtain the first model parameters from the memory of the first accelerator card.

[0157] Step 806 is optional. The first node can receive multiple requests for the first model parameters. After determining that at least one accelerator card in the node cluster stores the first model parameters in its memory, the first node can determine the response information for each of the multiple request messages based on a replication strategy. Optionally, the first node can adopt a pipeline replication strategy, whereby the first accelerator card currently replicating the first model parameters is used as the data source for the third accelerator card to obtain the first model parameters. For example, both the first and second request messages request the first model parameters. The first response message instructs the first accelerator card to obtain the first model parameters from the memory of the second accelerator card (i.e., the second accelerator card currently in operation) that stores the first model parameters, and the second response message instructs the third accelerator card to obtain the first model parameters from the memory of the first accelerator card.

[0158] For example, the second response information may include an identifier of the first accelerator card, which indicates that the first model parameters are being retrieved from the memory of the first accelerator card, wherein the first accelerator card is the accelerator card that is copying the first model parameters. Optionally, the second response information may also include information about the first model parameters. Optionally, the first response information may also include an identifier of the node to which the second accelerator card belongs.

[0159] For example, when the first node receives multiple requests to obtain the first model parameters, the scheduling scenario can be abstracted into a 1->N replication scenario and an M->S replication scenario. The M->S replication scenario can be divided into multiple 1->N replication scenarios. The 1->N replication scenario represents using a working accelerator card as the data source when replicating the first model parameters, and N represents N accelerator cards that request the first model parameters. To understand this solution more intuitively, please refer to Figure 10. Figure 10 is a schematic diagram of a pipelined replication strategy provided by an embodiment of this application. Figure 10 shows an M->S replication scenario. In Figure 10, taking the value of M as 2 and the value of S as 6 as an example, the 2->6 replication scenario is divided into two 1->3 replication scenarios. As shown in Figure 10, accelerator card 1 and accelerator card 2 are both working accelerator cards. In other words, the memory of accelerator card 1 and accelerator card 2 both store the first model parameters, and accelerator card 1 and accelerator card 2 both perform inference operations based on the stored first model parameters. Accelerator cards 3, 4, 5, 6, 7, and 8 all request the first model parameters, as shown in Figure 10. When a pipelined replication strategy is adopted, accelerator card 3 obtains the first model parameters from accelerator card 1, accelerator card 4 obtains the first model parameters from accelerator card 3, accelerator card 5 obtains the first model parameters from accelerator card 4; accelerator card 6 obtains the first model parameters from accelerator card 2, accelerator card 7 obtains the first model parameters from accelerator card 6, and accelerator card 8 obtains the first model parameters from accelerator card 7. It should be understood that the example in Figure 10 is only for the convenience of understanding this scheme and is not intended to limit this scheme.

[0160] 807. The first node sends the second response information to the fifth node.

[0161] Step 807 is optional. For example, the first node can send a second response message to the third accelerator card in the fifth node. The specific implementation of step 807 can be found in the description of step 503 in the embodiment corresponding to Figure 5 above. The difference is that the third node in the above description is replaced by the fifth node in step 807. This will not be elaborated further here. It should be understood that the fifth node and the third node can be the same node or different nodes, depending on the actual application scenario.

[0162] 808. Based on the second response information, the fifth node copies the first model parameters from the memory of the first accelerator card to the memory of the third accelerator card.

[0163] Step 808 is optional. For example, after receiving the second response information, the fifth node can establish a communication connection between the third accelerator card in the fifth node and the first accelerator card in the third node based on the second response information. In other words, the third accelerator card in the fifth node and the first accelerator card in the third node establish a connection. Furthermore, the third accelerator card in the fifth node can copy the first model parameters from the memory of the first accelerator card to the memory of the third accelerator card. In other words, after establishing a connection between the third accelerator card in the fifth node and the first accelerator card in the third node, the third accelerator card in the fifth node can call the communication interface to transfer the first model parameters from the memory of the second accelerator card to the memory of the first accelerator card.

[0164] For example, if the first and third accelerator cards are located within the same node, they can communicate using HCCS. Alternatively, if they are located on different nodes within the same node cluster, they can communicate using ETH. Optionally, the first and third accelerator cards can establish aggregated communication, such as send-receive communication or broadcast communication.

[0165] To better understand this solution, please refer to Figure 11. Figure 11 is another schematic diagram of the model parameter acquisition method provided in this application embodiment. It should be noted that, for the sake of simplicity, only one first model management module is shown on each node in Figure 11. In actual applications, one first model management module runs on each NPU. As shown in Figure 11, 1. Loading model parameters includes: Non-master node 1 loads model parameter 1 and model parameter 2 from the remote storage system, where model parameter 1 and model parameter 2 are two sets of model parameters included in the same model. 2. Registering model parameters includes: Non-master node 1 can register model parameters with the model scheduling module through the first model management module running on the NPU. To elaborate, the first model management module running on NPU0 sends second information 1 to the model scheduling module, indicating that model parameter 1 is stored in the memory of NPU0; the first model management module running on NPU1 sends second information 2 to the model scheduling module, indicating that model parameter 2 is stored in the memory of NPU0; thus, the first information stored in the model scheduling module indicates that model parameter 1 is stored in the memory of NPU0 and model parameter 2 is stored in the memory of NPU1.

[0166] 3. Obtaining the copy location of model parameters includes: the first model management module running on NPU2 obtains the copy location of model parameter 1 from the model scheduling module, and the first model management module running on NPU3 obtains the copy location of model parameter 2 from the model scheduling module. More specifically, this can be achieved by the first model management module running on the NPU sending a request to the model scheduling module, and the model scheduling module responding with a response. Further details are omitted here. The first model management module running on NPU4 obtains the copy location of model parameter 1 from the model scheduling module, and the first model management module running on NPU5 obtains the copy location of model parameter 2 from the model scheduling module.

[0167] 4. Link establishment includes: NPU2 can establish a link with the currently operating NPU0 to copy model parameter 1 from NPU0's memory to NPU2's memory; NPU3 can establish a link with the currently operating NPU1 to copy model parameter 2 from NPU1's memory to NPU3's memory; NPU4 can establish a link with NPU2, which is copying model parameters, to copy model parameter 1 from NPU2's memory to NPU4's memory; and NPU5 can establish a link with NPU3, which is copying model parameters, to copy model parameter 2 from NPU3's memory to NPU5's memory. This achieves a pipelined copying strategy for copying model parameters 1 and 2. It should be understood that the example in Figure 11 is only for the convenience of understanding this scheme and is not intended to limit this scheme.

[0168] When the first node determines that multiple accelerator cards have requested the first model parameters, since multiple accelerator cards would need to share the bandwidth of the second accelerator card if they were to copy the first model parameters from the second accelerator card that is currently in operation, this embodiment adopts a pipelined copying strategy. The first accelerator card copies the first model parameters from the second accelerator card that is currently in operation, and the third accelerator card copies the first model parameters from the first accelerator card that is copying the first model parameters. This is beneficial to improving the efficiency of the copying process, reducing the impact on the second accelerator card that is currently in operation, and ensuring the smoothness of the second accelerator card in performing inference operations.

[0169] Optionally, after the first model parameters are stored in the third accelerator card of the fifth node, the third node can also send a fourth message to the first node. The fourth message indicates that the first model parameters are stored in the memory of the third accelerator card, and the fourth message is used to update the first message.

[0170] The specific implementation of the above steps can be found in the description of step 801 above. The difference is that the fourth node in the above description is replaced with the fifth node, the second information in the above description is replaced with the fourth information, and the second acceleration card in the above description is replaced with the third acceleration card. The details will not be elaborated here.

[0171] It should be understood that steps 806 to 808 are optional steps. If steps 806 to 808 are not executed, the first node may also use steps 803 to 805 to process each of the at least two request messages if the first node receives at least two request messages.

[0172] 809. The first node sends a second instruction message to the second node, which instructs that the second model parameters be stored in the shared memory of the second node. The second node belongs to the node cluster.

[0173] Step 809 is optional. The first node may also send a second instruction to the second node. For example, the first node sends the second instruction to the second model management module in the second node; the second instruction instructs that the second model parameters be stored in the shared memory of the second node. Optionally, the second model parameters are a set of model parameters that satisfy the first condition. The second model parameters may include all model parameters of a certain model, or they may include some model parameters of a certain model. The meaning of the second model parameters is similar to that of the first model parameters, and can be understood by referring to the above description of the first model parameters.

[0174] Optionally, the first condition includes: the usage frequency of the model parameter is greater than or equal to a frequency threshold, that is, the usage frequency of the second model parameter is greater than or equal to a frequency threshold. For example, the usage frequency of the model parameter can be understood as the loading frequency of the model parameter. For example, the first node can determine the loading frequency of the second model parameter based on the frequency of requesting the second model parameter. Alternatively, the usage frequency of the model parameter can also be understood as the frequency of performing inference operations based on the model parameter. For example, the first node can determine the usage frequency of the second model parameter based on the frequency of performing inference operations based on the second model parameter. Alternatively, the usage frequency of the model parameter is the sum of the loading frequency of the model parameter and the frequency of performing inference operations based on the model parameter. Alternatively, the usage frequency of the model parameter can be the usage frequency of the model to which the model parameter belongs. The usage frequency of the model can be understood as the frequency of providing model inference services based on that model. Alternatively, the usage frequency of the model parameter can also be defined from other dimensions, which can be determined in combination with the actual application scenario.

[0175] For example, when the model instance to which the second model parameter belongs needs to be deleted, the second model parameter stored in the memory of a certain accelerator card in the second node also needs to be deleted. Before deleting the second model parameter stored in the memory of a certain accelerator card in the second node, the second model management module in the second node can send a second query message to the model scheduling module in the first node, inquiring whether the second model parameter needs to be copied to shared memory. If the model scheduling module in the first node determines that the second model parameter needs to be copied, it can send a second instruction message to the second model management module in the second node; correspondingly, after receiving the second instruction message, the second node can copy the second model parameter to the shared memory of the second node. If the model scheduling module in the first node determines that it does not need to copy the second model parameters, in one implementation, the second node may not send an instruction to the second model management module in the second node. After waiting for a preset time, the second model management module in the second node will delete the second model parameters stored in the memory of a certain accelerator card in the second node. Alternatively, in another implementation, the second node may send a third instruction to the second model management module in the second node. Correspondingly, after receiving the third instruction, the second node may delete the second model parameters stored in the memory of a certain accelerator card in the second node.

[0176] Optionally, after receiving the query information, the first node can determine whether the second model parameter meets the first condition. If the model scheduling module in the first node determines that the second model parameter meets the first condition, it determines that the second model parameter needs to be copied. If the model scheduling module in the first node determines that the second model parameter does not meet the first condition, it determines that the second model parameter does not need to be copied.

[0177] Optionally, the first condition further includes: after deleting the second model parameter stored in the memory of a certain accelerator card in the second node, the second model parameter is no longer stored in the memory of any accelerator card in the node cluster; in other words, if the usage frequency of the second model parameter is greater than or equal to the frequency threshold, and the second model parameter is about to be completely removed from the memory of any accelerator card in the node cluster, the second model parameter can be copied to the shared memory of the second node.

[0178] Alternatively, the first node can determine whether to copy the second model parameters based on the remaining space in the second node's shared memory. For example, if the remaining space in the second node's shared memory is greater than or equal to a preset threshold, then it is determined that the second model parameters need to be copied; if the remaining space in the second node's shared memory is less than the preset threshold, then it is determined that the second model parameters do not need to be copied. Alternatively, other strategies can be used to determine whether to copy the second model parameters.

[0179] In this embodiment, the second model parameter that meets the first condition is stored in the shared memory of the second node in the node cluster. So even if the memory of all the accelerator cards in the node cluster does not store the second model parameter, when the second model parameter is used, it can be copied from the shared memory of the second node. Compared with obtaining the second model parameter from a remote storage system, the efficiency of obtaining the second model parameter is improved.

[0180] In addition, the first condition includes that the usage frequency of the model parameters is greater than or equal to the frequency threshold. In other words, only the second model parameters with high usage frequency will be stored in the shared memory of the second node, so that the operation of storing the second model parameters is an effective use of the shared memory of the second node, avoiding the waste of the storage resources of the shared memory of the second node.

[0181] Based on the embodiments corresponding to Figures 1 to 11, in order to better implement the above-described solutions of this application, related equipment for implementing the above-described solutions is also provided below. Specifically, refer to Figure 12, which is a schematic diagram of a model parameter acquisition device provided in this application embodiment. The model parameter acquisition device 1200 is deployed in a first node. In one case, if the first node is the master node in the node cluster, the model parameter acquisition device 1200 can be a model scheduling module in the master node. The network architecture of the node cluster can be referred to in Figures 2 and 3 above. In another case, if the first node is a non-master node in the node cluster, the non-master node in the node cluster can independently determine where to obtain the first model parameter. The model parameter acquisition device 1200 can also be a second model management module in the non-master node, or it can also be a first model management module in the non-master node. The specific product form of the model parameter acquisition device 1200 can be determined in conjunction with the actual application scenario.

[0182] For example, the model parameter acquisition device 1200 includes: a processing unit 1201, configured to obtain first request information, the first request information instructing a first accelerator card to request first model parameters; and a processing unit 1201, configured to obtain first response information based on the first request information, the first response information instructing the first accelerator card to obtain the first model parameters from the memory of a second accelerator card, the memory of the second accelerator card storing the first model parameters, and the first accelerator card and the second accelerator card belonging to the same node cluster.

[0183] Optionally, the processing unit 1201 is specifically used to obtain first response information based on first information and first request information. The first information indicates at least one accelerator card where each set of model parameters is located in at least one set of model parameters. The first model parameter is one set of at least one set of model parameters. The at least one accelerator card where the first model parameter is located includes a second accelerator card.

[0184] Optionally, the processing unit 1201 is specifically configured to receive at least two request messages, including a first request message and a second request message, wherein the first request message comes from a first accelerator card, the second request message comes from a third accelerator card, the second request message instructs the third accelerator card to request the first model parameter, and the third accelerator card belongs to the node cluster; the processing unit 1201 is further configured to obtain a second response message based on the second request message, the second response message instructing the third accelerator card to obtain the first model parameter from the memory of the first accelerator card.

[0185] Optionally, the model parameter acquisition device 1200 further includes: a receiving unit 1202, used to receive second information, the second information indicating that the memory of the second accelerator card stores the first model parameters, and the second information is used to update the first information.

[0186] Optionally, the model parameter acquisition device 1200 further includes a sending unit 1203, used to send second indication information to the second node, the second indication information indicating that the second model parameters are stored in the shared memory of the second node, the second node belonging to the node cluster.

[0187] Optionally, the second model parameter satisfies the first condition, which includes: the usage frequency of the model parameter is greater than or equal to the frequency threshold.

[0188] It should be noted that the information interaction and execution process between the modules / units in the model parameter acquisition device 1200 are based on the same concept as the various method embodiments corresponding to Figures 1 to 11 in this application. For details, please refer to the description in the method embodiments shown above in this application, and it will not be repeated here.

[0189] This application embodiment also provides a model management module, which includes a first model management module and / or a second model management module in a non-master node. The model management module is specifically used for: sending a first request message to the first node, the first request message instructing the first accelerator card to request the first model parameters; receiving a first response message from the first node, the first response message instructing the first accelerator card to obtain the first model parameters from the memory of the second accelerator card, the first accelerator card and the second accelerator card belonging to the same node cluster; and copying the first model parameters from the memory of the second accelerator card to the memory of the first accelerator card based on the first response message.

[0190] Optionally, the model management module is also used to send third information to the first node, the third information indicating that the first model parameters are stored in the memory of the first accelerator card.

[0191] It should be noted that the information interaction and execution process between the modules / units in the model management module are based on the same concept as the various method embodiments corresponding to Figures 1 to 11 in this application. For details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0192] The following describes a device provided by an embodiment of this application. Please refer to Figure 13, which is a structural schematic diagram of a device provided by an embodiment of this application. Optionally, the device 1300 performs the functions of the first node, second node, third node, fourth node, fifth node, or sixth node in the various method embodiments corresponding to Figures 1 to 11.

[0193] Device 1300 includes a memory 1302, at least one processor 1301, and at least one accelerator card 1303. Exemplarily, the processor 1301 reads program instructions stored in the memory 1302 and implements the methods in the above embodiments through the accelerator card 1303; or, the processor 1301 may also read program instructions stored internally and implement the methods in the above embodiments through the accelerator card 1303.

[0194] When the processor 1301 reads the program instructions stored in the memory 1302 to implement the method in the above embodiments, the memory 1302 stores the program instructions that implement the method provided in the above embodiments of this application.

[0195] Optionally, at least one processor 1301 is one or more CPUs, either a single-core CPU or a multi-core CPU. For example, the memory 1302 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, or optical memory. The memory 1302 stores program instructions for the operating system. For example, at least one accelerator card 1303 may include at least one of the following: GPU, NPU, TPU, ASIC, FPGA, or other types of accelerators. After the program instructions stored in the memory 1302 are read by the at least one processor 1301, the device 1300 executes the corresponding operations in the foregoing embodiments.

[0196] Optionally, the device 1300 also includes a network interface 1304, which can be a wired interface or a wireless interface. The network interface 1304 is used to send and receive data in the various method embodiments corresponding to Figures 1 to 11.

[0197] It should be understood that network interface 1304 has the functions of receiving and sending data. The functions of "receiving data" and "sending data" can be integrated into the same transceiver interface, or the functions of "receiving data" and "sending data" can be implemented in different interfaces, which is not limited here. In other words, network interface 1304 may include one or more interfaces for implementing the functions of "receiving data" and "sending data".

[0198] After the processor 1301 reads the program instructions from the memory 1302, other functions that the device 1300 can perform are described in the preceding method embodiments.

[0199] Optionally, the device 1300 also includes a bus 1305, through which the processor 1301 and memory 1302 are typically interconnected, or in other ways.

[0200] The device 1300 provided in this application embodiment is used to execute the methods executed by the first node, second node, third node, fourth node, fifth node, or sixth node in the above-described method embodiments, and to achieve the corresponding beneficial effects. The specific implementation of the device 1300 shown in Figure 13 can be referred to the descriptions in the foregoing method embodiments, and will not be repeated here.

[0201] This application also provides a system comprising a first node and a first accelerator card in a third node. The first node executes the steps performed by the first node in the method described in the embodiments shown in Figures 1 to 11 above, and the first accelerator card in the third node executes the steps performed by the first accelerator card in the third node in the method described in the embodiments shown in Figures 1 to 11 above.

[0202] Optionally, the system may further include a second node, which performs the steps executed by the second node in the methods described in the embodiments shown in Figures 1 to 11. Optionally, the system may further include a fourth node, which performs the steps executed by the fourth node in the methods described in the embodiments shown in Figures 1 to 11. Optionally, the system may further include a fifth node, which performs the steps executed by the fifth node in the methods described in the embodiments shown in Figures 1 to 11.

[0203] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the steps executed by the first node, second node, third node, fourth node, fifth node, or sixth node in the methods described in the embodiments shown in Figures 1 to 11 above.

[0204] This application also provides a computer program product, which includes a program that, when run on a computer, causes the computer to perform the steps executed by the first node, second node, third node, fourth node, fifth node, or sixth node in the methods described in the embodiments shown in Figures 1 to 11 above.

[0205] This application also provides a circuit system including a processing circuit configured to execute the steps performed by the first node, second node, third node, fourth node, fifth node, or sixth node in the method described in the embodiments shown in Figures 1 to 11 above.

[0206] This application also provides a chip system including a processor and a communication interface. The communication interface is used to communicate with modules outside the chip system, and the processor is used to support the implementation of the steps executed by the first node, second node, third node, fourth node, fifth node, or sixth node in the methods described in the embodiments shown in Figures 1 to 11. In one possible design, the chip system further includes a memory for storing necessary program instructions and data. The chip system may be composed of chips or may include chips and other discrete devices.

[0207] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0208] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CLUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0209] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0210] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. A method for obtaining model parameters, characterized in that, The method includes: The first node receives the first request information, which instructs the first accelerator card to request the first model parameters. The first node obtains a first response based on the first request information. The first response information instructs the first accelerator card to obtain the first model parameters from the memory of the second accelerator card. The first model parameters are stored in the memory of the second accelerator card. The first accelerator card and the second accelerator card belong to the same node cluster.

2. The method according to claim 1, characterized in that, The first node obtains the first response information based on the first request information, including: the first node obtains the first response information based on the first information and the first request information, wherein the first information indicates at least one accelerator card in which each set of model parameters is located in at least one set of model parameters, the first model parameter is one set of the at least one set of model parameters, and the at least one accelerator card in which the first model parameter is located includes the second accelerator card.

3. The method according to claim 2, characterized in that, The first node receives the first request information, including: the first node receives at least two request information, the at least two request information including the first request information and the second request information, wherein the first request information comes from the first accelerator card, the second request information comes from the third accelerator card, the second request information instructs the third accelerator card to request the first model parameters, and the third accelerator card belongs to the node cluster; The method further includes: the first node obtaining second response information based on the second request information, wherein the second response information instructs the third accelerator card to obtain the first model parameters from the memory of the first accelerator card.

4. The method according to claim 2, characterized in that, Before the first node obtains the first response information based on the first information and the first request information, the method further includes: The first node receives second information, which indicates that the first model parameters are stored in the memory of the second accelerator card, and the second information is used to update the first information.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The first node sends a second instruction to the second node, which instructs that the second model parameters be stored in the shared memory of the second node, which belongs to the node cluster.

6. The method according to claim 5, characterized in that, The second model parameter satisfies the first condition, which includes: the usage frequency of the model parameter is greater than or equal to the frequency threshold.

7. A method for obtaining model parameters, characterized in that, The method is used for a first accelerator card, and the method includes: Send a first request message to the first node, the first request message instructing the first accelerator card to request the first model parameters; The system receives a first response message from the first node, which instructs the first accelerator card to obtain the first model parameters from the memory of the second accelerator card. The first accelerator card and the second accelerator card belong to the same node cluster.

8. The method according to claim 7, characterized in that, The method further includes: copying the first model parameters from the memory of the second accelerator card to the memory of the first accelerator card based on the first response information; A third message is sent to the first node, indicating that the first model parameters are stored in the memory of the first accelerator card.

9. A device for acquiring model parameters, characterized in that, The device for acquiring model parameters is deployed in the first node, and the device for acquiring model parameters includes: Processing unit, configured to obtain first request information, wherein the first request information indicates that the first accelerator card requests first model parameters; The processing unit is further configured to obtain first response information based on the first request information, wherein the first response information indicates that the first accelerator card obtains the first model parameters from the memory of the second accelerator card, the first model parameters are stored in the memory of the second accelerator card, and the first accelerator card and the second accelerator card belong to the same node cluster.

10. The apparatus according to claim 9, characterized in that, The processing unit is specifically configured to obtain the first response information based on the first information and the first request information. The first information indicates at least one accelerator card where each set of model parameters is located in at least one set of model parameters. The first model parameters are one set of the at least one set of model parameters. The at least one accelerator card where the first model parameters are located includes the second accelerator card.

11. The apparatus according to claim 10, characterized in that, The processing unit is specifically configured to receive at least two request messages, including a first request message and a second request message, wherein the first request message comes from the first accelerator card, the second request message comes from the third accelerator card, the second request message instructs the third accelerator card to request the first model parameters, and the third accelerator card belongs to the node cluster. The processing unit is further configured to obtain second response information based on the second request information, wherein the second response information instructs the third accelerator card to obtain the first model parameters from the memory of the first accelerator card.

12. The apparatus according to claim 9, characterized in that, The device further includes: The receiving unit is also configured to receive second information, the second information indicating that the first model parameters are stored in the memory of the second accelerator card, and the second information is used to update the first information.

13. The apparatus according to any one of claims 9 to 12, characterized in that, The device for acquiring model parameters further includes: The sending unit is used to send a second indication message to a second node, the second indication message indicating that the second model parameters be stored in the shared memory of the second node, the second node belonging to the node cluster.

14. The apparatus according to claim 13, characterized in that, The second model parameter satisfies the first condition, which includes: the usage frequency of the model parameter is greater than or equal to the frequency threshold.

15. A model management module, characterized in that, The model management module is deployed in the first accelerator card, and the model management module is specifically used for: Send a first request message to the first node, the first request message instructing the first accelerator card to request the first model parameters; The system receives a first response message from the first node, which instructs the first accelerator card to obtain the first model parameters from the memory of the second accelerator card. The first accelerator card and the second accelerator card belong to the same node cluster.

16. The module according to claim 15, characterized in that, The model management module is also used for: Based on the first response information, the first model parameters are copied from the memory of the second accelerator card to the memory of the first accelerator card. A third message is sent to the first node, indicating that the first model parameters are stored in the memory of the first accelerator card.

17. A device, characterized in that, It includes a processor and a memory, wherein the processor is coupled to the memory. The memory is used to store programs; The processor is configured to execute a program in the memory, causing the device to perform the method as described in any one of claims 1 to 8.

18. A system, characterized in that, The system includes a first node and a first accelerator card, wherein the first node performs the method as described in any one of claims 1 to 6, and the first accelerator card performs the method as described in claim 7 or 8.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 8.

20. A computer program product, characterized in that, The computer program product includes a program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 8.

Images