Communication system, and data processing method and apparatus

Abstract

A communication system, comprising a first computing unit and a second computing unit. The first computing unit has a first model slice of an attention layer deployed thereon. The first computing unit is used to acquire a first feature, the first feature being obtained by means of the first model slice processing input data, and the first feature comprising a first sub-feature corresponding to a token. The first computing unit is used to send the first sub-feature to the second computing unit when a mapping relationship indicates that the token corresponds to the second computing unit. The second computing unit is used to process the first sub-feature via an addition and normalization layer to obtain a first processing result. The present application can reduce the overall communication volume.

Description

A communication system, data processing method and apparatus

[0001] This application claims priority to Chinese Patent Application No. 202411644905.2, filed on November 15, 2024, entitled “A Communication System, Data Processing Method and Apparatus Thereof”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence, and in particular to a communication system, a data processing method and an apparatus thereof. Background Technology

[0003] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0004] With the increasing computing power of computing devices, the improvement of communication network bandwidth, and the accumulation of data in big data networks, research is underway to deploy large-scale neural network models based on neural networks on distributed networks for computation and processing, in order to realize applications such as large-scale parallel data processing, distributed storage, elastic topology, high redundancy, and nonlinear operations in cloud computing.

[0005] Distributed model training (e.g., MoE (Mixure-of-Experts)) is one of the technical approaches to achieving ultra-large-scale model training. The idea behind this model is to train multiple neural networks (distributed across multiple computing nodes), with each node training a different portion of the dataset. Input data from different devices passes through their respective routing networks (e.g., Gating networks), selecting different expert nodes; thus, communication occurs between different computing units to distribute the data.

[0006] In existing technologies, when performing expert parallelism, a gating network is needed to identify which expert module on the computing unit should process the features of each token. Then, all-to-all communication is used to send the features to the corresponding computing node. For features processed by the attention layer, an all-reduce communication process is required to ensure that each computing unit obtains all features. The computing unit can then process the obtained features through subsequent summation and normalization layers. After processing by the expert module, different computing units still need to synchronize data through an all-reduce communication process, resulting in significant communication overhead from these two all-reduce communication processes. Summary of the Invention

[0007] In a first aspect, this application provides a communication system comprising a first computing unit and a second computing unit. The first computing unit is deployed with a first model slice containing an attention layer. The first computing unit is configured to acquire a first sub-feature, which is obtained by processing input data through the first model slice. The first computing unit is configured to send the first sub-feature to the second computing unit when a mapping relationship indicates that the input data corresponds to the second computing unit. The second computing unit is configured to process the first sub-feature through a summation and normalization layer to obtain a first processing result, and to process the first processing result through a first expert module.

[0008] The input data can be one or more tokens.

[0009] In this embodiment, for the data processed by the attention layer, a mapping relationship is used to determine which computing unit the input data corresponds to (e.g., the input data corresponds to the second computing unit). Then, the features (first sub-features) of the input data are sent to the second computing unit. The second computing unit can then process the first sub-features through a summation and normalization layer. When the input data needs to be processed locally (by the second computing unit), the result obtained from the summation and normalization layer (i.e., the first processing result) is directly processed by the first expert module. Compared to the prior art, where data is transmitted after identifying which expert module each input data corresponds to through a gating network, this application determines which computing unit the input data is likely to be computed in before the summation and normalization layer. If the mapping relationship is accurate, the transmitted data can be directly processed locally by the expert module, eliminating the need for the data transmission process with significant overhead required by the prior art.

[0010] In one possible implementation, the input data is a first token; the first computing unit is specifically used to obtain a first feature, the first feature including a first sub-feature and a second sub-feature, the second sub-feature being a feature of the second token; the first computing unit is used to process the second sub-feature through a summation and normalization layer when the mapping relationship indicates that the second token corresponds to the first computing unit.

[0011] In this embodiment, since only a portion of the data (the features of the first token, i.e., the first sub-feature) is transmitted, the other portion of the data (the features of the second token, i.e., the second sub-feature) can be computed locally and then subjected to the summation and normalization layer operation. Even if the expert module determines through the gating network that the token is not processed by the computing unit it belongs to, it only needs to be transmitted to the corresponding computing unit (e.g., through the all-to-all communication process). Since the communication overhead of the all-to-all communication process is much smaller than that of the all-reduce communication process, the overall communication volume can be reduced.

[0012] Furthermore, in existing implementations, the slices of the attention layer are distributed across different computing units. After obtaining data through the model slices of the attention layer, each computing unit synchronizes the data via all-to-all communication. Each computing unit can obtain the data of all tokens and process it through the summation and normalization layer. In this embodiment, since each computing unit only needs to obtain the data of the corresponding token indicated in the mapping relationship, the summation and normalization layer only needs to process the data of a portion of the tokens, thereby reducing computational overhead.

[0013] In this process, the features of each token tend to be processed by one of the multiple expert modules. The idea behind constructing the mapping relationship is to first predict which expert module's computing unit is best suited to deliver the token's features, then deliver the token's features to the corresponding computing unit, and perform summation and normalization layer operations. Afterward, the gating network will determine that the token is more likely to be processed by the expert module located in its current computing unit. Even if it is determined that the token is not processed by the expert module located in its current computing unit, it only needs to be delivered to the corresponding computing unit (e.g., through an all-to-all communication process). Since the communication overhead of the all-to-all communication process is much smaller than that of the all-reduce communication process, the overall communication volume can be reduced.

[0014] The above approach can be called a semantically aware Token-Expert joint rearrangement algorithm. Based on the endogeneity correlation between Token and Expert during the activation process of the MoE model, the placement of Expert is rearranged. At the same time, Token is shuffled before reduce-scatter to enhance Token-Expert affinity, avoid unnecessary remote Expert activation, and reduce the communication volume of the MoE layer.

[0015] In this embodiment, the parallel scheme of hybrid TP-SP-EP splits the all-reduce after the attention layer into reduce-scatter combined with allgather through lossless communication splitting. Under the premise that the communication and computation of the attention layer remain basically unchanged, it realizes a smooth transition from TP to SP and then to EP, reducing the computation and communication complexity of the MoE layer.

[0016] In one possible implementation, the second computing unit is deployed with a second model slice of the attention layer;

[0017] The second calculation unit is used to obtain a third sub-feature, which is obtained by processing the input data through the second model slice; the second calculation unit is used to process the third sub-feature through the summation and normalization layer when the mapping relationship indicates that the input data corresponds to the second calculation unit, to obtain a second processing result.

[0018] In one possible implementation, sending the first sub-feature to the second computing unit includes:

[0019] The first sub-feature is sent to the second computing unit through a reduce-scatter communication process.

[0020] In one possible implementation, the first computing unit is further deployed with a second expert module, which is used to determine, based on the second processing result, via a gating network, that the input data corresponds to the first expert module.

[0021] The second computing unit is used to send the second processing result to the first computing unit;

[0022] The first calculation unit is used to process the second processing result through the second expert module to obtain a third processing result.

[0023] In one possible implementation, sending the second processing result to the first computing unit includes:

[0024] The second processing result is sent to the first computing unit through an all-to-all communication process.

[0025] In one possible implementation, the first computing unit is used to send the third processing result to the second computing unit through an Allgather communication process.

[0026] In one possible implementation, the first computing unit is configured to perform a shuffle operation on the sub-features included in the first feature to aggregate multiple sub-features corresponding to the mapping relationship indication in the first feature and the second computing unit, thereby obtaining an aggregation result, wherein the multiple sub-features include the first sub-feature; the step of sending the first sub-feature to the second computing unit includes:

[0027] The aggregation result is sent to the second computing unit.

[0028] In one possible implementation, the first computing unit is a graphics processing unit (GPU), a network processing unit (NPU), or a tensor processing unit (TPU), and the second computing unit is a GPU, an NPU, or a TPU. The first computing unit and the second computing unit are deployed on the same server or on different servers.

[0029] Secondly, this application provides a data processing method applied to a first computing unit, the first computing unit communicating with a second computing unit, the second computing unit being equipped with a first expert module; the method includes:

[0030] Obtain the first sub-feature, which is obtained by processing the input data through the first model slice;

[0031] If the mapping relationship indicates that the input data corresponds to the second computing unit, the first sub-feature is sent to the second computing unit.

[0032] In one possible implementation, sending the first sub-feature to the second computing unit includes:

[0033] The first sub-feature is sent to the second computing unit through a reduce-scatter communication process.

[0034] In one possible implementation, the method further includes:

[0035] The sub-features included in the first feature are shuffled to aggregate multiple sub-features corresponding to the mapping relationship indication in the first feature and the second computing unit, thereby obtaining an aggregation result, wherein the multiple sub-features include the first sub-feature; the step of sending the first sub-feature to the second computing unit includes:

[0036] The aggregation result is sent to the second computing unit.

[0037] In one possible implementation, the first computing unit is a graphics processing unit (GPU), a network processing unit (NPU), or a tensor processing unit (TPU), and the second computing unit is a GPU, an NPU, or a TPU. The first computing unit and the second computing unit are deployed on the same server or on different servers.

[0038] Thirdly, this application provides a data processing apparatus applied to a first computing unit, the first computing unit communicating with a second computing unit, the second computing unit being equipped with a first expert module; the apparatus includes:

[0039] The acquisition module is used to acquire a first sub-feature, which is obtained by processing the input data through the first model slice;

[0040] The transceiver module is used to send the first sub-feature to the second computing unit when the mapping relationship indicates that the input data corresponds to the second computing unit.

[0041] In one possible implementation, the transceiver module is specifically used to send the first sub-feature to the second computing unit through a reduce-scatter communication process.

[0042] In one possible implementation, the device further includes:

[0043] The rearrangement module is used to perform a rearrangement (shuffle) operation on the sub-features included in the first feature, so as to aggregate multiple sub-features corresponding to the mapping relationship indication in the first feature to the second calculation unit to obtain an aggregation result, wherein the multiple sub-features include the first sub-feature;

[0044] The transceiver module is specifically used to send the aggregation result to the second computing unit.

[0045] In one possible implementation, the first computing unit is a graphics processing unit (GPU), a network processing unit (NPU), or a tensor processing unit (TPU), and the second computing unit is a GPU, an NPU, or a TPU. The first computing unit and the second computing unit are deployed on the same server or on different servers.

[0046] Fourthly, this application provides a data processing apparatus, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform the methods described in the second aspect above and any optional methods described in the second aspect above.

[0047] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the second aspect and any optional methods thereof, or the methods described in the third aspect and any optional methods thereof.

[0048] In a sixth aspect, embodiments of this application provide a computer program that, when run on a computer, causes the computer to perform the methods described in the second aspect above and any of its alternatives.

[0049] Seventhly, this application provides a chip system including a processor for supporting the implementation of the functions involved in the foregoing aspects, such as transmitting or processing data involved in the foregoing methods; or, information. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for execution or training devices. This chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0050] Figure 1 is a schematic diagram of a structural framework for artificial intelligence.

[0051] Figure 2 is a schematic diagram of the application system framework of the present invention;

[0052] Figure 3 is a schematic diagram of the application system framework of an embodiment of this application;

[0053] Figure 4 is a schematic diagram of the application system framework of an embodiment of this application;

[0054] Figures 5A to 5I are schematic diagrams illustrating some operations of embodiments of this application;

[0055] Figure 6 is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0056] Figures 7A and 7B are schematic diagrams of the network structure of an embodiment of this application;

[0057] Figures 8A and 8B are schematic flowcharts of a data processing method provided in an embodiment of this application;

[0058] Figure 9 is a schematic diagram of the structure of the data processing device according to an embodiment of this application;

[0059] Figure 10 is a schematic diagram of a chip structure provided in an embodiment of this application;

[0060] Figures 11 to 13 are schematic diagrams of a computing unit provided in an embodiment of this application. Detailed Implementation

[0061] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0062] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0063] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0064] The terms “substantially,” “about,” and similar terms used herein are used as approximations rather than as terms of degree, and are intended to take into account the inherent biases of measurements or calculations known to those skilled in the art. Furthermore, the use of “may” in describing embodiments of the invention refers to “one or more possible embodiments.” The terms “use,” “using,” and “used” used herein are to be considered synonymous with the terms “utilize,” “utilizing,” and “utilized,” respectively. Additionally, the term “exemplary” is intended to refer to an instance or illustration.

[0065] In some scenarios, large-scale neural network models based on neural networks can be deployed on distributed networks for computation and processing, enabling applications such as massive parallel processing, distributed storage, elastic topology, high redundancy, and nonlinear operations in cloud computing. Specifically, from the initial AlexNet network and VGG (visual geometry group network) to ResNet (residual neural network), GPT-3 (generative pre-trained transformer), and the Pangu model, large-scale neural network models based on deep learning have become increasingly deeper, wider in scope, and have a larger number of parameters. Through these large-scale neural network models, massive computing and data resources can be utilized to achieve multi-task concurrent processing and large-scale computation.

[0066] However, when deploying large-scale neural network models on a system for computation, the rapid increase in model parameter size and data makes it difficult to perform complete training, inference, and other data processing on a single system. While deploying large-scale neural network models on a distributed system theoretically improves scalability and computational efficiency, the communication overhead—including data communication between nodes and communication between models—significantly increases when multiple models are processed in parallel across multiple nodes. This leads to reduced data transmission efficiency, resulting in decreased data processing efficiency, increased consumption of storage and computing resources, and higher application and maintenance costs. Furthermore, current large-scale neural network model architectures only consider deployment in densely connected networks (DenseNet), neglecting model sparsity and failing to support sparse architectures. In other words, it is impossible to activate only a portion of the large-scale neural network architecture to achieve computational processing for a given task, sample, or label.

[0067] It should be understood that the nodes in the embodiments of this application can be devices with the same granularity as the computing units, such as chips or AI cards. Alternatively, the computing units can be devices with finer granularity than the nodes. For example, a node can include multiple computing units. Specifically, a node can be a server, and the computing units can be chips or AI cards included in the server.

[0068] A mixture of experts (MoE) system, consisting of one or more expert modules, can be deployed across multiple nodes to build large-scale neural network models. When multiple expert modules of a MoE system are deployed at the same layer on multiple nodes, the input data of each node is independent and generally different. Further, the input data of each node is segmented at each node into data corresponding to each of the multiple expert modules, and each node sends the segmented data to the corresponding other nodes. Thus, each expert module on a node processes the data segmented from its own node and the data sent from other nodes. After all expert modules on each node have completed their data processing, the processed data are aggregated to generate the overall output of the neural network model.

[0069] Figure 1 illustrates a schematic diagram of an example system architecture 100 according to an embodiment of the present disclosure. As shown in Figure 1, system architecture 100 may include at least a node cluster 110, a node server 120, a client device 130, and a network 140. The node cluster 110, node server 120, and client device 130 in system architecture 100 can be connected via network 140. It should be understood that system architecture 100 in Figure 1 may include more or fewer functional devices, and is not limited thereto.

[0070] In one example embodiment, client device 130 sends data to be processed to node cluster 110 and node server 120 via network 140 as input data for node cluster 110. Node server 120 controls node cluster 110 to process the data based on the data received from client device 130. Node cluster 110 processes the data received from client device 130 into a predetermined form through parallel processing and feeds it back to node server 120 or returns it to client device 130. In some embodiments, only a portion of the structure in system architecture 100 may participate in data processing. In one example embodiment, node server 120 may directly provide data to be processed to node cluster 110, while client device 130 does not participate in data processing. In one example embodiment, only a portion of the nodes in node cluster 110 may participate in data processing, while other nodes do not. It should be understood that the data processing operations between node cluster 110, node server 120, client device 130, and network 140 in system architecture 100 are not limited to these methods, and any operational approach can be implemented.

[0071] In some embodiments, node cluster 110 may include one or more nodes. For example, Figure 1 illustrates a scenario where node cluster 110 includes multiple nodes (node ​​110-1, node 110-2, node 110-3, ..., node 110-N). In the case of multiple nodes, node cluster 110 can be distributed to form a neural network model, that is, the neural network model is deployed on multiple nodes of node cluster 110 to process data in parallel.

[0072] It should be understood that the node cluster 110 may include multiple nodes, each node may include multiple computing units, or the node cluster 110 may include multiple computing units without any intermediate granularity device such as "node". For example, the node cluster 110 may be a server cluster, which may include multiple servers, each server may include multiple computing units. Or, for example, the node cluster 110 may be a server, and the server may include multiple computing units.

[0073] In some embodiments, the node server 120 can be used to manage and control the data processing of the node cluster 110, so as to determine the data processing algorithm or method of the node cluster 110 according to different data, or to use the node cluster 110 to perform training, derivation or other processing based on neural network models. In some embodiments, the node server 120 can be connected to all or a portion of the nodes of the node cluster 110, or it can be distributed across all or a portion of the nodes of the node cluster 110.

[0074] In some embodiments, client device 130 may be one or more suitable mobile or non-mobile computing devices for providing data input or receiving data feedback. Client device 130 may have data collection, processing, and output functions, such as input / output (I / O) devices like a recorder, camera, camcorder, mouse, keyboard, and monitor. Client device 130 may run various software applications, such as productivity or office support software, web browsing software, camera software, software supporting voice calls, video conferencing, and email. The computing device is connected to network 140 via wired and / or wireless communication links to communicate with node cluster 110 or node server 120.

[0075] In some embodiments, network 140 can be used to enable communication between various structures in system architecture 100, including node cluster 110, node server 120, and client device 130. For example, network 140 may include, for instance, a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a Layer 3 virtual private network (VPN), an intranet, or any combination thereof. It should be understood that the various structures in system architecture 100 can communicate directly with each other via network 140, or indirectly with each other via any structures connected to network 140, or they may not communicate with each other at all.

[0076] In this document, the term "input" refers to data sent to a device / unit / module / part, etc. The following example illustrates the use of image data as input data to the neural network model of node cluster 110. It should be understood that input data includes, but is not limited to, at least one of image data, text data, speech data, and classification data, as long as it is a data type suitable for processing by the neural network model. According to embodiments of this disclosure, it can be effectively applied to model training, inference, and processing of various types of data, including structured and unstructured data (e.g., image data, text data, speech data).

[0077] In this paper, the term "computing unit" refers to computing resources used for data processing, such as graphics processing units (GPUs), central processing units (CPUs), tensor processing units (TPUs), deep learning processing units (DPUs), neural network processing units (NPUs), and brain processing units (BPUs). It should be understood that the type and implementation of computing units are not limited to these; any computing unit suitable for processing neural network models is acceptable.

[0078] In this document, the term "parallel processing" means that data processing is performed in batches or stages at independent and asynchronous speeds, and the batches or stages of data processing may overlap in time. For example, the execution of one batch of data processing may not have finished on all nodes while the execution of another batch of data processing has already begun. It should be understood that parallel processing is not limited to what is listed herein, and any data processing of neural network models that can be understood by those skilled in the art is within the scope of parallel processing defined in the embodiments of this disclosure.

[0079] Figure 2 illustrates a schematic diagram of an example node cluster 110 according to an embodiment of the present disclosure. For example, Figure 2 exemplarily illustrates a scenario where the node cluster 110 shown in Figure 1 includes multiple nodes (node ​​210-1, node 210-2, node 210-3, ..., node 210-N), wherein a neural network model 200 is deployed across the multiple nodes (node ​​210-1, node 210-2, node 210-3, ..., node 210-N), and the neural network model 200 includes P expert modules, where P is greater than or equal to 2. As an illustrative embodiment, the neural network model 200 may include 100 expert modules, which are distributed across the multiple nodes (node ​​210-1, node 210-2, node 210-3, ..., node 210-N). Each node may deploy a portion of the 100 expert modules, and the expert modules included in each node collectively constitute the expert module whole of the neural network model 200. As shown in Figure 2, in some embodiments, nodes 210-1, 210-2, 210-3, ..., 210-N are connected sequentially to perform data processing in a pipelined parallel manner. As an exemplary implementation of the pipelined parallel approach, for example, node 210-1 processes the input data of node cluster 110 and sends its output data (i.e., the intermediate data of node cluster 110 obtained after processing by node 210-1) to node 210-2. Then, node 210-2 processes the output data of node 210-1 and sends its output data (i.e., another intermediate data of node cluster 110 obtained after processing by node 210-2) to node 210-3. And so on, node 210-N processes the output data of the node connected to it and uses its output data (i.e., the data obtained after processing by node 210-N) as the output data of node cluster 110. It should be understood that the node connection method and data transmission method of multiple nodes in the node cluster 110 are not limited to this. The pipeline-based parallel method can be modified or extended on this basis as an additional or alternative method.

[0080] Figure 2 also illustrates the node structure in node cluster 110. For example, node 210-1 shows a node comprising one computing unit 20 and one computing unit 50, and node 210-3 shows a node comprising one computing unit 20 and two computing units 50, wherein computing unit 20 is a first computing unit based on expert parallelism, and computing unit 50 is a second computing unit based on tensor parallelism. It should be understood that the number of computing units 20 and 50 in each node can be arbitrary; each node may include only computing unit 20, only computing unit 50, or both computing units 20 and 50, and in some cases, may not include either computing unit 20 or computing unit 50.

[0081] Figure 3 illustrates a schematic diagram of a computing unit (e.g., a first computing unit or a second computing unit in an embodiment of this application) according to an embodiment of the present disclosure. For example, Figure 3 exemplarily illustrates a computing unit 20 (i.e., a first computing unit) included in a node in a node cluster 110 shown in Figures 1 and 2. It should be understood that although a single node is described herein as an example, the node structure described herein can be applied to any node in the node cluster 110, and in some embodiments, it can also be applied to other structures in the system architecture 100 (e.g., node server 120, client device 130, etc.).

[0082] As shown in Figure 3, in some embodiments, each node includes one or more computing units, each of which can physically independently perform data processing. The node also includes one or more computing units, each of which may include one or more expert modules (e.g., fragments of expert modules) constituting a mixture of experts (MoE). Here, the mixture of experts system can be implemented by a single neural network or a combination of multiple neural networks, with each expert in the mixture of experts system implementing part or all of the neural network of the mixture of experts system. In some embodiments, the mixture of experts system can be deployed on one or more nodes or computing units. In some embodiments, each expert module can be used to perform a specific task and process a specific type of data in the node's input data. In some embodiments, the node also includes one or more storage resources (not shown) for storing intermediate processing data of the node, etc.

[0083] In some embodiments, a node may include multiple computing units, each computing unit including multiple expert modules E, wherein each expert module E may be a fragment of a complete expert module (i.e. a partial expert module), and expert modules on different computing units (e.g., different computing units on the same node, or computing units on different nodes) perform data processing in an expert parallel manner.

[0084] As an exemplary implementation of an expert-parallel approach, for example, as shown in Figure 3, the node includes computing unit 11 and computing unit 12 (described as computing resources in the figure). Computing unit 11 and computing unit 12 perform data processing independently. For example, computing unit 11 includes expert modules E11 and E12, where E11 is a fragment of expert module A and E12 is a fragment of expert module B. Computing unit 12 includes expert modules E21 and E22, where E21 is another fragment of expert module A and E22 is another fragment of expert module B.

[0085] When data processing is required through expert module A, computing unit 11 can call E11 for data processing, and computing unit 12 can call E21 for data processing, and the two calculations are performed in parallel.

[0086] It should be understood that the number and relationship of computing units, computing units and expert modules E in this example are not limited thereto and do not constitute a limitation on the embodiments of this disclosure.

[0087] In some embodiments, each computing unit may include a gate unit. For example, as shown in FIG3, computing unit 11 includes a gate unit 31, and computing unit 12 includes a gate unit 32. The gate unit is used to route the input data for the computing unit to determine which expert module (e.g., expert module A or expert module B) should execute the data.

[0088] For example, input data can also be segmented and processed in parallel by different computing units, as shown in Figure 4. This includes two nodes, each containing four computing units. The input features can be split into eight parts, each processed by a computing unit on a node. In this case, after obtaining the input features, each computing unit can calculate which expert module needs to process the features of each token in the input features. If the expert module is deployed on its own node, it can be retained (and subsequently synchronized through communication between computing units within the node, such as through the Allgather communication process). If the expert module is deployed on other nodes (i.e., not on its own node), data synchronization can be achieved through communication between computing units between nodes, such as through the AlltoAll communication process.

[0089] The following are some terms that will be used in the embodiments of this application:

[0090] 1) Reduce-scatter

[0091] This set communication primitive combines the reduce and scatter operations. As shown in Figure 5A, it performs a global reduce operation (such as summation, maximum value, etc.) on the data of an input array, and then distributes the results to various nodes.

[0092] 2) Allgather

[0093] AllGather communication is a many-to-many communication primitive, as shown in Figure 5B. It allows each node to send its data to all other nodes and collect data from all other nodes. After the AllGather operation, each node holds the complete dataset of all nodes.

[0094] 3) Allreduce

[0095] AllReduce is a commonly used aggregation communication primitive in distributed computing, playing a crucial role, especially in distributed training in deep learning. This primitive is primarily used to synchronize data across multiple nodes or devices (such as an NPU), as shown in Figure 5C. Typically, it aggregates data from multiple nodes, performs some reduction operation (such as summation, maximum value calculation, average value calculation), and then broadcasts the result back to all nodes. In the scenario of distributed training of large models, this primitive is mainly applied to the synchronization of activation values ​​in tensor-parallel forward and backward propagation, and the synchronization of gradient data in data-parallel systems. It is easy to see that the AllReduce primitive is equivalent to the superposition of the Reduce-scatter and Allgather communication primitives.

[0096] 4) AlltoAll

[0097] The AlltoAll operation allows each node to send its data to all other nodes and receive data from all other nodes. This is a many-to-many communication pattern that ensures each node receives data from all other nodes. This set communication primitive comprises two phases: data distribution and data aggregation. In the data distribution phase, on each node, the input data is split into `split_count` blocks according to the specified `split_dim` (split dimension). These blocks are then sent to other nodes in a certain order (e.g., block index); for example, the i-th block might be sent to the i-th node. In the data aggregation phase, each node receives data blocks from all other nodes. The received data blocks are then reassembled into a complete Tensor according to `concat_dim` (connection dimension). Tensors on all nodes must have the same shape and format to ensure correct data aggregation. As shown in Figure 5D, AlltoAll performs the transpose operation of data between distributed nodes.

[0098] 5) Tensor Parallelism

[0099] Tensor parallelism (TP) in distributed training of large models is a technique that divides the model parameters into multiple tensors and assigns each tensor to a different computing unit (such as an NPU) for computation.

[0100] For the Transformer architecture, tensor parallelism has specific applications in the MLP layer and the Attention layer. As shown on the right side of Figure 5E, taking the MLP layer as an example, the parameter matrix W1 is first "column-cut," and then the parameter matrix W2 is "row-cut" to perform parallel computation on two NPUs. During the forward propagation phase of training, according to the matrix operation rules, the function... The activation values ​​calculated on different devices need to be synchronized using an Allreduce operation to ensure the correctness of the forward computation. Similarly, during the backpropagation phase of training, the function f needs to be synchronized using an Allreduce operation on the gradients of the backpropagated activations to ensure the correctness of matrix operations.

[0101] As shown on the left side of Figure 5E, the Attention layer is similar to the MLP layer, and the parameter matrix W is also first processed. Q W K and W V First, use "column cutting", then apply it to matrix W. LThe "row-by-row" approach is used, therefore, Allreduce synchronization operations are introduced in both the forward and backward training phases. When using tensor parallelism, the overall computation and communication flow of the Transformer Block is shown in Figure 5F, with 2 Allreduce operations in the forward direction and 2 Allreduce operations in the backward direction, for a total of 4 Allreduce operations.

[0102] 6) Megatron sequence parallelism

[0103] Megatron's Sequence Parallelism (SP) is an important technique in distributed training of large models, primarily to address the memory consumption issues in large Transformer models.

[0104] As shown in Figure 5G, taking the MLP module as an example, is the sequence splitting consistent with the tensor parallelism of model parameters? Why? That is, the complete sequence is split into TP parts and placed on TP NPU cards respectively. During forward propagation, since each NPU card only has a portion of the sequence, before matrix operations, each NPU card in the TP domain needs to initiate Allgather communication to retrieve the sequences from other cards and concatenate them into a complete sequence for computation. After computation, the reduction operation required by tensor parallelism needs to be performed, and the data is distributed to each NPU card along the sequence dimension to restore the "original" sequence state. This involves calling Reduce-scatter communication. It's easy to understand that when tensor parallelism is further superimposed with Megatron's sequence parallelism, the original Allreduce communication is split into Allgather and Reduce-scatter. As shown in Figure 5G, in the forward propagation phase of model training, function g represents the Allgather operation required for computation by the Attention module and the MLP module, while function... This refers to the Reduce-scatter operation; similarly, in the backpropagation phase, This indicates a Reduce-scatter operation, while the function g represents an Allgather operation. Normally, there are 2 Allgather operations and 2 Reduce-scatter operations in the forward direction, and 2 Allgather operations and 2 Reduce-scatter operations in the reverse direction, for a total of 4 Allgather operations and 4 Reduce-scatter operations.

[0105] 7) Parallel Expertise

[0106] Expert parallelism (EP) in distributed training of large models is a training strategy that allows the model to dynamically select a set of "experts" to perform computations based on the input data. This parallelization approach aims to optimize the use of computational resources by introducing sparsity, while maintaining or improving the model's performance.

[0107] The input data first passes through a gating network, which assigns weights or probabilities to different experts based on the input features. Then, based on these weights, the model selectively activates one or more experts to process the input data. As shown in Figure 5H, since different experts may be placed on different devices (such as an NPU), and each device may contain data belonging to all experts, from the perspective of each device, it is necessary to send the data needed by other experts on the current device to the corresponding device, and retrieve the data needed for computation by the experts on the current device stored on other devices. This operation can be achieved by initiating an All-to-All ensemble communication. Each expert independently processes its received data and produces an output. These outputs are then returned to the original device and merged to produce the final output of the model; therefore, All-to-All communication needs to be initiated again here. As shown in Figure 5I, during the forward propagation phase of model training, the function h before and after expert parallelism represents the sum. These represent dispatch and combine AlltoAll communication, respectively, and the reverse propagation phase works similarly.

[0108] In this embodiment of the application, the first computing unit and the second computing unit can be cards with the same number on different nodes.

[0109] The Scaling Law states that the intelligence of a model increases to some extent with its size. Mixture-of-Experts (MoE) architecture is a common model sparsity technique, often used to significantly expand the model's size. Typically, a dense model consists of multiple stacked Transformer Blocks, each containing a Self-Attention module and a Feedforward Neural Network (FFN) module. In the MoE architecture, a single Feedforward Neural Network is replaced by multiple FFNs (i.e., E in the right figure). 21 ,…,E 2k Each FFN is activated by a gating unit (G2 in the right figure) during actual operation. In the MoE architecture, each FFN is called an Expert.

[0110] Clearly, MoE significantly increases the number of model parameters by horizontally expanding a single FFN into multiple models, while maintaining the same computational cost, achieving sub-linear model scaling. The number of parameters in a large MoE model is often tens or even hundreds of times greater than that of a large model with equivalent FLOPs. Therefore, large MoE models often involve parallel inference across multiple GPUs. Common parallelism methods include data parallelism (DP), tensor parallelism (TP), expert parallelism (EP), and sequence parallelism (SP). Among these, TP is the most widely used in inference.

[0111] Existing large-scale model inference engines (such as vLLM, TensorRT-LLM, etc.) often use full-TP (such as Mixtral, Deepseek, Arctic, PanGu-Σ, etc.) to complete the inference of the MoE model. That is, all layers adopt TP parallelism, the parameters of each expert in the MoE layer are evenly distributed to different computing devices, each computing device receives the same input data, and the results are calculated using all-reduce reduction.

[0112] Existing technologies do not take into account the semantic characteristics of Experts and the affinity between Tokens and Experts, which may lead to unnecessary remote activation and thus additional communication overhead.

[0113] This application provides a data processing method. The data processing method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0114] Referring to Figure 6, which is a flowchart of a data processing method provided in an embodiment of this application, the data processing method provided in this application may include steps 601 to 603, which will be described in detail below.

[0115] 601. The first calculation unit obtains a first sub-feature, which is obtained by processing the input data through the first model slice.

[0116] The first computing unit in this embodiment may belong to a communication system. The communication system includes a first computing unit and a second computing unit, and the first computing unit communicates with the second computing unit.

[0117] In one scenario, the communication system can be a collection of computing units, which can be chip-level, such as an NPU. In another scenario, the communication system can be a collection of nodes, each containing computing units. Nodes can be servers, and computing units can be chip-level, such as an NPU.

[0118] The first feature can be a feature representation obtained by the first computing unit during the processing of the input sequence by a machine learning model (e.g., a language model). For example, the first feature can be obtained by processing the input data through the first model slice. The first model slice can be a model slice of the attention layer, that is, it contains some parameters of the attention layer.

[0119] For example, a transformer network may include an embedding layer and at least one transformer layer. The at least one transformer layer may be N transformer layers (N being an integer greater than 0), where each transformer layer includes sequentially adjacent attention layers, an add-and-normalize layer, a feed-forward layer, and another add-and-normalize layer. In the embedding layer, the current input is embedded to obtain multiple feature vectors. In the attention layer, P input vectors are obtained from the layer above the transformer layer. Using any one of the P input vectors as the center, based on the correlation between each input vector within a preset attention window and that input vector, an intermediate vector corresponding to that input vector is obtained. This process is repeated to determine P intermediate vectors corresponding to the P input vectors.

[0120] The first feature may include sub-features corresponding to each of the multiple tokens. Specifically, it may include the first sub-feature corresponding to the input data. It should be understood that the input data may include one or more tokens.

[0121] 602. When the mapping relationship indicates that the input data corresponds to the second computing unit, the first computing unit sends the first sub-feature to the second computing unit.

[0122] The mapping relationship can include the correspondence between each token and the computing unit among multiple tokens.

[0123] In this embodiment, for the data processed by the attention layer, a mapping relationship is used to determine which computing unit the input data corresponds to (e.g., the input data corresponds to the second computing unit). Then, the features (first sub-features) of the input data are sent to the second computing unit. The second computing unit can then process the first sub-features through a summation and normalization layer. When the input data needs to be processed locally (by the second computing unit), the result obtained from the summation and normalization layer (i.e., the first processing result) is directly processed by the first expert module. Compared to the prior art, where data is transmitted after identifying which expert module each input data corresponds to through a gating network, this application determines which computing unit the input data is likely to be computed in before the summation and normalization layer. If the mapping relationship is accurate, the transmitted data can be directly processed locally by the expert module, eliminating the need for the data transmission process with significant overhead required by the prior art.

[0124] In one possible implementation, the input data is a first token; the first computing unit is specifically used to obtain a first feature, the first feature including a first sub-feature and a second sub-feature, the second sub-feature being a feature of the second token; the first computing unit is used to process the second sub-feature through a summation and normalization layer when the mapping relationship indicates that the second token corresponds to the first computing unit.

[0125] In this embodiment, since only a portion of the data (the features of the first token, i.e., the first sub-feature) is transmitted, the other portion of the data (the features of the second token, i.e., the second sub-feature) can be computed locally and then subjected to the summation and normalization layer operation. Even if the expert module determines through the gating network that the token is not processed by the computing unit it belongs to, it only needs to be transmitted to the corresponding computing unit (e.g., through the all-to-all communication process). Since the communication overhead of the all-to-all communication process is much smaller than that of the all-reduce communication process, the overall communication volume can be reduced.

[0126] The mapping relationship can be constructed based on the endogeneity and correlation between tokens and expert modules during the activation process of the MoE model.

[0127] In one possible implementation, sending the first sub-feature to the second computing unit includes sending the first sub-feature to the second computing unit via a reduce-scatter communication process.

[0128] For example, the first sub-feature may include sub-features of multiple tokens, which can be internally reduced by the first computing unit and then sent to the second computing unit.

[0129] In one possible implementation, the first computing unit is configured to perform a shuffle operation on the sub-features included in the first feature (e.g., rearrange the storage locations of the sub-features) to aggregate multiple sub-features corresponding to the second computing unit in the mapping relationship indicated in the first feature (e.g., store them in a contiguous space) to obtain an aggregation result, wherein the multiple sub-features include the first sub-feature; and then, the aggregation result can be sent to the second computing unit.

[0130] The above approach can be called a semantically aware Token-Expert joint rearrangement algorithm. Based on the endogeneity correlation between Token and Expert during the activation process of the MoE model, the placement of Expert is rearranged. At the same time, Token is shuffled before reduce-scatter to enhance Token-Expert affinity, avoid unnecessary remote Expert activation, and reduce the communication volume of the MoE layer.

[0131] In this embodiment, the parallel scheme of hybrid TP-SP-EP splits the all-reduce after the attention layer into reduce-scatter combined with allgather through lossless communication splitting. Under the premise that the communication and computation of the attention layer remain basically unchanged, it realizes a smooth transition from TP to SP and then to EP, reducing the computation and communication complexity of the MoE layer.

[0132] In one possible implementation, the second computing unit is deployed with a second model slice of the attention layer; the second computing unit is configured to obtain a third sub-feature, which is obtained by processing the input data through the second model slice; the second computing unit is configured to process the third sub-feature through the summation and normalization layer to obtain a second processing result when the mapping relationship indicates that the input data corresponds to the second computing unit.

[0133] 603. The second calculation unit processes the first sub-feature through a summation and normalization layer to obtain a first processing result, and processes the first processing result through a first expert module.

[0134] In existing implementations, the slices of the attention layer are distributed across different computing units. After obtaining data through the model slices of the attention layer, each computing unit synchronizes the data via all-to-all communication. Each computing unit can obtain the data of all tokens and process it through the summation and normalization layer. In this embodiment, since each computing unit only needs to obtain the data of the corresponding token indicated in the mapping relationship, the summation and normalization layer only needs to process the data of a portion of the tokens, thereby reducing computational overhead.

[0135] In one possible implementation, the first computing unit is further configured with a second expert module, the second computing unit being configured to determine, based on the second processing result and through a gating network, that the input data corresponds to the first expert module; the second computing unit being configured to send the second processing result to the first computing unit; and the first computing unit being configured to process the second processing result through the second expert module to obtain a third processing result.

[0136] In this context, the sub-features corresponding to the expert module can be understood as the sub-features that need to be processed by the expert module. For example, the first sub-feature corresponding to the first expert module can be understood as: the first sub-feature is the feature that needs to be processed by the first expert module.

[0137] The first processing result can be the input of the gating module (that is, the gating module), which can determine which expert module needs to execute the feature representation of the token by inputting the feature representation of each token.

[0138] The expert module in this application embodiment can be part of a machine learning model, such as part of a Transformer Block (e.g., a network in an MLP used to perform linear transformations), or of other network types.

[0139] For example, the Transformer Block mainly consists of two modules: Attention and MLP.

[0140] The Attention mechanism plays a crucial role in the Transformer model, allowing the model to pay attention to other positions in the sequence while processing each position. The Attention mechanism works by roughly the following steps:

[0141] 1. Calculate Query(Q), Key(K), and Value(V):

[0142] For each position in the input sequence, compute its corresponding Q, K, and V vectors. These vectors are typically obtained by embedding the input into W respectively. Q W K and W V The weights are obtained by multiplying the weight matrices. The Q vector represents the information that needs attention at the current position, the K vector is used to match the Q vector, and the V vector contains the actual information content.

[0143] 2. Calculate the Attention weights:

[0144] By calculating the scaled dot-product attention (Q-vector and all K-vectors), the degree of "attention" each position gives to the current position is obtained, which is the attention score. This dot-product operation measures the similarity or matching degree between Q and K. Since the dot-product result can be very large, especially with high input dimensions, this can cause the softmax function to saturate when calculating attention weights. Therefore, the dot-product result is usually scaled, typically by the square root of the input dimension. The scaled dot-product result is then fed into the softmax function to obtain the attention weights for each key relative to the query. These weights represent the degree of attention the model gives to different positions when generating the output at the current position.

[0145] 3. Weighted summation:

[0146] The attention weights are summed with their corresponding value vectors to obtain the Attention output for the current position. This output is a weighted sum that considers information from all positions, thus achieving attention to global information.

[0147] In the Transformer model, the MLP part is mainly used to perform further nonlinear transformations on the Attention output. Its working principle can be summarized in the following steps:

[0148] 1. Linear transformation:

[0149] First, the output of the Attention mechanism (usually a vector) is fed into one or more linear layers. These linear layers perform linear transformations on the input using weight matrices and bias terms, thereby extracting different features.

[0150] 2. Nonlinear activation:

[0151] After a linear transformation, a nonlinear activation function (such as ReLU) is typically applied to increase the nonlinearity of the model. Nonlinear activation functions can map the output after the linear transformation to a new space, allowing the model to learn more complex functional relationships.

[0152] 3. Stacking multiple layers:

[0153] In Transformer models, the MLP part is typically composed of multiple such linear layers and non-linear activation functions stacked together. This stacked structure allows the model to learn richer feature representations and improves its generalization ability.

[0154] In one possible implementation, sending the second processing result to the first computing unit includes: sending the second processing result to the first computing unit through an All-to-All communication process.

[0155] In one possible implementation, the first computing unit is used to send the third processing result to the second computing unit through an Allgather communication process.

[0156] The baseline for comparison in this application's embodiments is primarily the Full TP MoE inference scheme, which is currently the most commonly used MoE model inference technology in the industry. As shown in Figure 7A, the left side shows an example of implementing distributed MoE inference using Full TP, where all layers employ TP parallelism. The parameters of each Expert in the MoE layer are evenly distributed across different computing devices, each receiving the same input data, and the results are computed using all-reduce. In the figure, TP represents tensor parallelism, REP represents independent replicas (i.e., data and model parameters are copied), SP represents sequence parallelism, and EP represents expert parallelism.

[0157] As shown on the right side of Figure 7A, this embodiment of the application splits the all-reduce layer after the attention layer in the prior art into reduce-scatter and all-gather after the MoE layer. After the split, the communication volume of the attention layer remains unchanged (the communication volume of all-reduce is theoretically equal to the sum of the communication volumes of reduce-scatter and all-gather), and the LayerNorm&Add layer changes from REP to SP, reducing the computational cost to 1 / D of the original (D is the number of TP devices). The MoE layer changes from TP to EP, and all-reduce is replaced by all2all with lower communication cost. The system performance does not degrade in any way.

[0158] Building upon the aforementioned modifications, this function enhances Expert-Token affinity by jointly rearranging Expert and Token in the MoE layer, reducing remote activations and optimizing communication overhead. As shown in Figure 7B, this function first splits the all-reduce of the attention layer into reduce-scatter + all-gather. While maintaining the same communication / computational load in the attention layer, it achieves a smooth transition from TP to SP to EP, reducing the computational / communication complexity of the MoE layer. Furthermore, based on the endogeneity correlation between Token and Expert during the MoE model activation process, this invention rearranges the placement of Experts; simultaneously, it shuffles Tokens before reduce-scatter and reshuffles them after all-gather, enhancing Token-Expert affinity, avoiding unnecessary remote Expert activations, and reducing the all-to-all communication load of the MoE layer.

[0159] In Figure 7B, the dashed lines represent the offline process, and the solid lines represent the online process. The offline process mainly involves profile collection. This invention first performs profiling on the distributed inference of the MoE model using a publicly available dialogue dataset, collecting Token ID data and Expert activation data during the token computation process. Based on the offline collected profiles, this invention uses the STEC algorithm to obtain a Token-device mapping table and an Expert-device mapping table, and implements joint Token-Expert rearrangement. The Expert rearrangement is performed offline (completed only once before the inference engine deployment), while the Token rearrangement is performed at runtime (during inference engine operation, the online portion of the STEC algorithm rearranges the Token based on the user-inputted Token).

[0160] The following is a specific illustration of how to construct a mapping relationship:

[0161] Figure 8A illustrates the implementation process of this embodiment, covering both offline and online steps. Figure 8B details the flow of the STEC algorithm, which consists of an offline part and an online part. The offline part is responsible for constructing the token-device scheduling table and the expert-device scheduling table; the online part is responsible for calculating token shuffle indices based on the user-input token, thereby performing shuffle and reshuffle before and after the MoE, enhancing the affinity between the token and the expert. Algorithm details are shown in Figure 8B.

[0162] The offline part of the algorithm involves the SpectralCoClustering algorithm, also known as spectral bidirectional clustering. This section describes the details of this algorithm. As shown in Figure 8B, the offline part of the STEC algorithm is illustrated. Let T be the expert activation frequency table of the profile token at layer l. l ∈R t×E The spectral clustering algorithm clusters the frequency table into n_clusters classes through the following process. After the clustering is completed, each row and each column belongs to one and only one class label.

[0163] First, for matrix T l Perform normalization: Where R and C are diagonal matrices, R ii =Σ j T l [i,j],C ii =Σ i T l [i,j].

[0164] The second step is to process matrix T. ln Perform eigenvalue decomposition: Starting from the second eigenvector, take The row eigenvectors and column eigenvectors constitute matrix Z, as described below:

[0165] Among them, U=[u2,…,u l+1 ],V=[v2,…,v l+1 After constructing matrix Z, the binary clustering problem converges to a univariate clustering problem. The algorithm clusters the rows of Z using the k-means algorithm to find the cluster to which each row and column belongs. The objective function for clustering is as follows:

[0166] After clustering is completed, the labels in the first t rows of matrix Z define the category of each token, and the labels in the last E rows define the category of each expert.

[0167] Referring to Figure 9, which is a schematic diagram of the structure of a data processing device 900 provided in an embodiment of this application, the device is applied to a first computing unit, the first computing unit communicates with a second computing unit, and the second computing unit is equipped with a first expert module; the device includes:

[0168] The acquisition module 901 is used to acquire a first sub-feature, which is obtained by processing the input data through the first model slice;

[0169] The specific details of the acquisition module 901 can be found in the description of step 601 in the above embodiments, and the similarities will not be repeated here.

[0170] The transceiver module 902 is used to send the first sub-feature to the second computing unit when the mapping relationship indicates that the input data corresponds to the second computing unit.

[0171] For a detailed description of the transceiver module 902, please refer to the description of step 602 in the above embodiment. The similarities will not be repeated here.

[0172] In one possible implementation, the transceiver module 902 is specifically used to send the first sub-feature to the second computing unit through a reduce-scatter communication process.

[0173] In one possible implementation, the device further includes:

[0174] The rearrangement module is used to perform a rearrangement (shuffle) operation on the sub-features included in the first feature, so as to aggregate multiple sub-features corresponding to the mapping relationship indication in the first feature to the second calculation unit to obtain an aggregation result, wherein the multiple sub-features include the first sub-feature;

[0175] The transceiver module is specifically used to send the aggregation result to the second computing unit.

[0176] In one possible implementation, the first computing unit is a graphics processing unit (GPU), a network processing unit (NPU), or a tensor processing unit (TPU), and the second computing unit is a GPU, an NPU, or a TPU. The first computing unit and the second computing unit are deployed on the same server or on different servers.

[0177] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0178] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0179] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0180] Specifically, please refer to Figure 10, which is a schematic diagram of a chip structure provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1000. The NPU 1000 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1003, which is controlled by the controller 1004 to extract matrix data from the memory and perform multiplication operations. In this embodiment, the first and second computing units can be the neural network processor (NPU) 1000. In the embodiment corresponding to Figure 6, the process of neural network calculation in the first and second computing units can be implemented by the controller 1004 controlling the arithmetic circuit 1003, while the routing process based on mapping relationships can be performed by the host CPU.

[0181] For example, in step 601 of the embodiment corresponding to Figure 6, the process of processing the input data through the first model slice can be implemented by the controller 1004 controlling the arithmetic circuit 1003.

[0182] For example, in step 602 of the embodiment corresponding to Figure 6, the routing process based on the mapping relationship for sub-features can be performed by the Host CPU.

[0183] In some implementations, the arithmetic circuit 1003 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1003 is a two-dimensional pulsating array. The arithmetic circuit 1003 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1003 is a general-purpose matrix processor.

[0184] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1002 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1001 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1008.

[0185] Unified memory 1006 is used to store input and output data. Weight data is directly transferred to weight memory 1002 via Direct Memory Access Controller (DMAC) 1005. Input data is also transferred to unified memory 1006 via DMAC.

[0186] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1009.

[0187] The Bus Interface Unit (BIU) 1010 is used by the instruction fetch memory 1009 to fetch instructions from external memory, and also by the memory access controller 1005 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0188] The DMAC is mainly used to move input data from external memory DDR to unified memory 1006, or to weight data to weight memory 1002, or to input data to input memory 1001.

[0189] The vector computation unit 1007 includes multiple processing units that further process the output of the computation circuit 1003 when needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0190] In some implementations, the vector computation unit 1007 can store the processed output vector in the unified memory 1006. For example, the vector computation unit 1007 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1003, such as performing linear interpolation on the feature planes extracted by the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1007 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1003, for example, for use in subsequent layers of the neural network.

[0191] The instruction fetch buffer 1009 connected to the controller 1004 is used to store the instructions used by the controller 1004;

[0192] Unified memory 1006, input memory 1001, weight memory 1002, and instruction fetch memory 1009 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0193] This application also provides a computing device 1100, which can be a computing unit (e.g., a first computing unit or a second computing unit) as described in the above embodiments. As shown in FIG11, the computing device 1100 includes: a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other via the bus 102. The computing device 1100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1100.

[0194] Bus 102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 11, but this does not imply that there is only one bus or one type of bus. Bus 102 can include pathways for transmitting information between various components of the computing device 1100 (e.g., memory 106, processor 104, communication interface 108).

[0195] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0196] Memory 106 may include volatile memory, such as random access memory (RAM). Processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0197] The memory 106 stores executable program code, which the processor 104 executes to implement the functions of the aforementioned acquisition module 901, gating module 903, and transceiver module 902, thereby realizing the data processing method. In other words, the memory 106 stores instructions for executing the data processing method.

[0198] The communication interface 108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1100 and other devices or communication networks.

[0199] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0200] As shown in Figure 12, the computing device cluster includes at least one computing device 1100. The memory 106 of one or more computing devices 1100 in the computing device cluster may store the same instructions for performing data processing methods.

[0201] In some possible implementations, the memory 106 of one or more computing devices 1100 in the computing device cluster may also store partial instructions for executing data processing methods. In other words, a combination of one or more computing devices 1100 can jointly execute instructions for executing data processing methods.

[0202] It should be noted that the memory 106 in different computing devices 1100 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the computing unit. That is, the instructions stored in the memory 106 of different computing devices 1100 can implement the functions of one or more modules among the acquisition module 901, the gating module 903, and the transceiver module 902.

[0203] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 13 illustrates one possible implementation. As shown in Figure 13, two computing devices 1100A and 1100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 106 in computing device 1100A stores instructions for executing the functions of the acquisition module 901. Simultaneously, the memory 106 in computing device 1100B stores instructions for executing the functions of the gating module 903 and the transceiver module 902.

[0204] It should be understood that the functions of computing device 1100A shown in Figure 12 can also be performed by multiple computing devices 1100. Similarly, the functions of computing device 1100B can also be performed by multiple computing devices 1100.

[0205] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device clusters in Figures 11 and 12. The difference is that the memory 106 of one or more computing devices 1100 in this computing device cluster can store the same instructions for executing data processing methods.

[0206] In some possible implementations, the memory 106 of one or more computing devices 1100 in the computing device cluster may also store partial instructions for executing data processing methods. In other words, a combination of one or more computing devices 1100 can jointly execute instructions for executing data processing methods.

[0207] It should be noted that the memory 106 in different computing devices 1100 within the computing device cluster can store different instructions for executing some functions of the communication system. That is, the instructions stored in the memory 106 of different computing devices 1100 can implement the functions of one or more modules among the acquisition module 901, the gating module 903, and the transceiver module 902.

[0208] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0209] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0210] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0211] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0212] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. A communication system, characterized in that, The communication system includes a first computing unit and a second computing unit, wherein the first computing unit is deployed with a first model slice of an attention layer. The first calculation unit is used to obtain a first sub-feature, which is obtained by processing the input data through the first model slice; The first computing unit is configured to send the first sub-feature to the second computing unit when the mapping relationship indicates that the input data corresponds to the second computing unit; The second calculation unit is used to process the first sub-feature through a summation and normalization layer to obtain a first processing result, and to process the first processing result through a first expert module.

2. The system according to claim 1, characterized in that, Sending the first sub-feature to the second computing unit includes: The first sub-feature is sent to the second computing unit through a reduce-scatter communication process.

3. The system according to claim 1 or 2, characterized in that, The input data is the first token; The first calculation unit is specifically used to obtain a first feature, which includes a first sub-feature and a second sub-feature, wherein the second sub-feature is a feature of the second token; The first computing unit is configured to process the second sub-feature through a summation and normalization layer when the mapping relationship indicates that the second token corresponds to the first computing unit.

4. The system according to any one of claims 1 to 3, characterized in that, The second computing unit is equipped with a second model slice of the attention layer; The second calculation unit is used to obtain a third sub-feature, which is obtained by processing the input data through the second model slice; The second calculation unit is used to process the third sub-feature through the summation and normalization layer to obtain a second processing result when the mapping relationship indicates that the input data corresponds to the second calculation unit.

5. The system according to any one of claims 1 to 4, characterized in that, The first computing unit is further equipped with a second expert module, which is used to determine, based on the second processing result, through a gating network, that the input data corresponds to the first expert module. The second computing unit is used to send the second processing result to the first computing unit; The first calculation unit is used to process the second processing result through the second expert module to obtain a third processing result.

6. The system according to claim 5, characterized in that, Sending the second processing result to the first computing unit includes: The second processing result is sent to the first computing unit through an all-to-all communication process.

7. The system according to claim 5 or 6, characterized in that, The first computing unit is used to send the third processing result to the second computing unit through an Allgather communication process.

8. The system according to any one of claims 3 to 7, characterized in that, The first computing unit is configured to perform a shuffle operation on the sub-features included in the first feature, so as to aggregate multiple sub-features corresponding to the mapping relationship indication in the first feature and the second computing unit to obtain an aggregation result, wherein the multiple sub-features include the first sub-feature; the step of sending the first sub-feature to the second computing unit includes: The aggregation result is sent to the second computing unit.

9. The system according to any one of claims 1 to 8, characterized in that, The first computing unit is a graphics processing unit (GPU), a network processing unit (NPU), or a tensor processing unit (TPU), and the second computing unit is a GPU, an NPU, or a TPU. The first computing unit and the second computing unit are deployed on the same server or on different servers.

10. A data processing method, characterized in that, The method is applied to a first computing unit, which communicates with a second computing unit, the second computing unit having a first expert module deployed thereon; the method includes: Obtain the first sub-feature, which is obtained by processing the input data through the first model slice; When the mapping relationship indicates that the input data corresponds to the second computing unit, the first sub-feature is sent to the second computing unit so that the second computing unit processes the first sub-feature through a summation and normalization layer to obtain a first processing result, and processes the first processing result through a first expert module.

11. The method according to claim 10, characterized in that, Sending the first sub-feature to the second computing unit includes: The first sub-feature is sent to the second computing unit through a reduce-scatter communication process.

12. The method according to claim 10 or 11, characterized in that, The method further includes: The sub-features included in the first feature are shuffled to aggregate multiple sub-features corresponding to the mapping relationship indication in the first feature and the second computing unit, thereby obtaining an aggregation result, wherein the multiple sub-features include the first sub-feature; the step of sending the first sub-feature to the second computing unit includes: The aggregation result is sent to the second computing unit.

13. The method according to any one of claims 10 to 12, characterized in that, The first computing unit and the second computing unit are graphics processing units (GPUs), network processing units (NPUs), or tensor processing units (TPUs). The first computing unit is deployed on a first node, and the second computing unit is deployed on a second node. The first node and the second node are different servers.

14. The method according to any one of claims 10 to 13, characterized in that, The first computing unit is a graphics processing unit (GPU), a network processing unit (NPU), or a tensor processing unit (TPU), and the second computing unit is a GPU, an NPU, or a TPU. The first computing unit and the second computing unit are deployed on the same server or on different servers.

15. A data processing apparatus, characterized in that, The device is applied to a first computing unit, which communicates with a second computing unit, the second computing unit having a first expert module deployed thereon; the device includes: The acquisition module is used to acquire a first sub-feature, which is obtained by processing the input data through the first model slice; The transceiver module is configured to send the first sub-feature to the second computing unit when the mapping relationship indicates that the input data corresponds to the second computing unit, so that the second computing unit processes the first sub-feature through a summation and normalization layer to obtain a first processing result, and processes the first processing result through a first expert module.

16. The apparatus according to claim 15, characterized in that, The transceiver module is specifically used to send the first sub-feature to the second computing unit through a reduce-scatter communication process.

17. The apparatus according to claim 15 or 16, characterized in that, The device further includes: The rearrangement module is used to perform a rearrangement (shuffle) operation on the sub-features included in the first feature, so as to aggregate multiple sub-features corresponding to the mapping relationship indication in the first feature to the second calculation unit to obtain an aggregation result, wherein the multiple sub-features include the first sub-feature; The transceiver module is specifically used to send the aggregation result to the second computing unit.

18. The apparatus according to any one of claims 15 to 17, characterized in that, The first computing unit is a graphics processing unit (GPU), a network processing unit (NPU), or a tensor processing unit (TPU), and the second computing unit is a GPU, an NPU, or a TPU. The first computing unit and the second computing unit are deployed on the same server or on different servers.

19. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the operation of the method according to any one of claims 10 to 14.

20. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 10 to 14.

21. A system, characterized in that, It includes at least one processor and at least one memory; the processor and the memory are connected via a communication bus and communicate with each other. The at least one memory is used to store code; The at least one processor is used to execute the code to perform the method as described in any one of claims 10 to 14.

22. A chip, characterized in that, It includes at least one processing unit and an interface circuit, the interface circuit being used to provide program instructions or data to the at least one processing unit, the at least one processing unit being used to execute the program instructions to implement the method of any one of claims 10 to 14.

Images