36 results about "Collective communication" patented technology

The embodiment of the invention discloses a set communication acceleration method and device for GPUs, equipment, a medium and a program product. The method comprises the following steps: receiving input data sent by a source end GPU in a plurality of GPUs; performing expert routing calculation according to the input data, and determining a target expert for processing the input data in the hybrid expert model; the input data is sent to a GPU corresponding to the target expert, and the GPU corresponding to the target expert is used for processing the input data through the target expert to obtain an output result; obtaining an output result corresponding to the activated expert in the hybrid expert model; and carrying out specification on the output result corresponding to the activated expert in the hybrid expert model.

A collective communication processing method, comprising: acquiring a node orchestration request for a target collective operation in collective communication, wherein an original node list carried in the node orchestration request comprises server nodes selected for executing the target collective operation (202); acquiring topology information of the server nodes, wherein the topology information is used for describing switching nodes connected to the server nodes in hierarchies of a node communication network, and the server nodes communicate with each other on the basis of the connected switching nodes in the hierarchies (204); on the basis of the topology information, orchestrating and updating the arrangement order of the server nodes hierarchy by hierarchy according to the hierarchies in the node communication network so as to obtain an orchestrated node list (206); and according to the arrangement order of the server nodes in the orchestrated node list, performing collective communication for the target collective operation by means of the server nodes (208).

A method for emulating a distributed computing scenario using a graph-based representation of AI / ML workload execution with an expanded collective communication operation includes receiving a graph-based representation of AI / ML workload execution comprising a collective communication node and expanding the collective communication node by replacing a collective communication operation of the collective communication node with low-level processing instructions. A modified graph-based representation of AI / ML workload execution comprising the low-level processing instructions is generated. The modified graph-based representation of AI / ML workload execution is implemented in an emulated test case using an emulation engine.

Provided are a collective communication method and apparatus. The method is applied to a distributed computing system, and comprises: acquiring a task to be computed, wherein the task to be computed comprises a first-type parallel computing task and a second-type parallel computing task that have no dependency relationship; with respect to the first-type parallel computing task and the second-type parallel computing task, respectively creating a first-type communication domain and a second-type communication domain, wherein the first-type communication domain and the second-type communication domain at least comprise two identical computing devices, and each of the first-type communication domain and the second-type communication domain has an independent communication buffer and an independent communication connection; and concurrently executing a collective communication operation for the first-type communication domain and a collective communication operation for the second-type communication domain. The collective communication method provided in the present application can create an independent communication buffer and an independent communication connection for each of a plurality of communication domains that can be concurrently executed, thereby realizing concurrent execution of the plurality of communication domains, avoiding communication connection preemption, and improving the execution efficiency of a distributed computing task.

The invention discloses a heterogeneous GPU cluster-oriented efficient collective communication method and device, and the method comprises the steps: obtaining a target communication operator set and a physical topology corresponding to a heterogeneous GPU cluster, and determining an initial state constraint and a target state constraint; the scheduling time cost of each communication scheme is determined, and the communication scheme corresponding to the minimum scheduling time cost in the multiple scheduling time costs is selected as the efficient collective communication mode of the heterogeneous GPU cluster. According to the method, the communication scheme corresponding to the minimum scheduling time cost of the heterogeneous GPU cluster is determined to serve as the efficient collective communication mode of the heterogeneous GPU cluster based on the actual conditions of topological structure difference, link bandwidth imbalance and the like in the heterogeneous GPU cluster environment; it is ensured that the communication process is not limited by the bandwidth bottleneck, and system resource utilization efficiency and parallel training performance are prevented from being limited.

A method of performing all-to-all collective communication scheduling includes scaling a max concurrent multi-commodity flow (MCF) framework by decomposing a MCF problem and parallelizing the MCF problem to perform a fast link-based all-to-all schedule computation. The method further includes computing a time-stepped version of the MCF problem for a host-based forwarding network topology, utilizing the time-stepped version of the MCF problem to create a direct-connect graph, and then using the direct-connect graph to compute time-stepped MCF schedules to manage a mixed topology. The method further includes identifying a direct-connect topology to perform all-to-all collective communication based on the time-stepped MCF schedules.

The embodiment of the invention discloses a collective communication method, the method is applied to a collective communication system, the collective communication system comprises a management node and N computing nodes, the N computing nodes are used for processing a first communication task in parallel, N is an integer greater than 1, and the method comprises the following steps: the management node receives first information, the first information indicates that at least one first link breaks down, and each first link is a communication link used for processing the first communication task among the N computing nodes; the management node receives N pieces of second information from the N computing nodes, wherein each piece of second information is used for indicating whether the original data of the first communication task locally stored by one computing node is changed or not; and if the original data of the N computing nodes are not changed, the management node sends third information to the N computing nodes to instruct the N computing nodes to reprocess the first communication task based on the original data. In this way, the service recovery time can be shortened after the communication fault occurs.

The invention discloses a set communication unloading method, system and device and a medium, and is applied to the technical field of computers, and the method comprises the steps: in a model deployment stage, a distributed reasoning controller generates a communication primitive blueprint based on a computational graph description file of a tensor parallel reasoning model and issues the communication primitive blueprint to a DPU; the DPU establishes a hardware-level communication context semantic environment based on the communication primitive blueprint; in the model reasoning stage, the GPU / NPU sends a trigger signal to the DPU when calculating to a communication boundary; and the DPU executes a DMA data pulling assembly line and an RDMA data sending assembly line in parallel based on a hardware-level communication context semantic environment, performs aggregation calculation on all tensor fragment data to be synchronized, and writes an aggregation calculation result back to the GPU / NPU. A hardware-level communication context semantic environment is established in advance to a model deployment stage, and double assembly lines are executed in the DPU in parallel, so that end-to-end communication delay is remarkably reduced, and zero participation of a host CPU is realized.

The invention discloses a set communication-oriented model training fault diagnosis method and system, and belongs to the field of artificial intelligence and distributed computing. The method comprises the following steps: implanting a trigger count in a torchplug-in to count the triggering times of a set communication primitive; the method comprises the following steps: adding completion information for processing a set task into a communication library, including counting of various set communication primitives and transceiving data volume, and registering a query interface to torchplug; and finally, pre-starting an independent fault diagnosis analysis thread in different RANK processes of model training, collecting task counts of a set communication primitive issued by upper-layer training on different nodes, task counts actually completed by a set communication layer and the amount of received and transmitted data completed by the set communication layer according to a set sampling interval, and carrying out fault diagnosis analysis on the received and transmitted data. And analyzing and diagnosing fault information when model training is abnormal. The method is suitable for a large-scale distributed training scene, and the troubleshooting efficiency of the model training task can be improved.

The present application relates to the field of computer technology, and discloses a collective communication method and a computing device cluster. The method comprises: during the process of N computing units in a computing device cluster executing a collective communication operation, a first computing unit receives first data from a second computing unit and / or sends second data to a third computing unit, wherein when data in each computing unit among the N computing units is greater than a first threshold, the data in each computing unit is segmented into N pieces of slice data, the number of pieces of slice data in the second computing unit comprised in the first data is inversely proportional to a physical distance between the first computing unit and the second computing unit, and the number of pieces of slice data in the first computing unit comprised in the second data is inversely proportional to a physical distance between the first computing unit and the third computing unit. Thus, cross‑switch traffic conflicts during the process of N computing units executing a collective communication operation can be reduced, and the efficiency of collective communication can be improved.

The present disclosure provides a Mixture-of-Experts model based collective communication method, system and device. The collective communication method is applied to a communication receiving end, and includes: receiving a data write-in instruction, where the data write-in instruction includes to-be-processed data and first address information; accessing a virtual expert address in a pre-created virtual address space according to the first address information; applying for a corresponding actual physical space for the virtual expert address based on a size of the to-be-processed data; and writing the to-be-processed data into the actual physical space.

The application discloses a kind of collection communication method and device, computing cluster, belong to communication technical field.The computing cluster includes first computing group and second computing group, first computing node in first computing group obtains first data, and after first part data in first data is sent to second computing node in second computing group, second part data in first data is sent to third computing node in second computing group.The first data is the data obtained by executing first collection communication operation in first computing group, the first part data corresponds to second computing node, and the second part data corresponds to third computing node, and the first part data and the second part data are used to realize second collection communication operation executed between different computing groups in the computing cluster.This application can realize hierarchical all-reduce algorithm in asymmetric computing cluster, and application scene is wide.

Systems, methods, and devices that perform computing operations are provided. In one example, a system includes a least one node, the at least one node having one or more processors, each having associated memory, a clock, a scheduler, the scheduler monitoring one or more of rates, rates of lanes, rates at which packets are sent, times, latencies of packets, topology, communication states, nodes, and packets in the system, an attribute monitor that measures counters for one or more of congestion state, line rate, and communication attributes. A packet scheduler determines a destination node based on information from the scheduler and the attribute monitor, and sends at least a portion of a packet to the destination node.

The application discloses a kind of collection communication library parameter automatic tuning method, storage medium, equipment and computer program product, comprising: in network communication process, collection communication library generates a large number of collection communication library parameter combinations, obtains high-dimensional collection communication library parameter space;High-dimensional collection communication library parameter space is mapped to low-dimensional space, and the collection communication library parameter characteristics of low-dimensional space are obtained;The collection communication library parameter characteristics of low-dimensional space are stored in the corresponding bucket memory according to the set step, and the collection communication library parameter characteristics of low-dimensional space are reflected in the bucket memory Mapping to high-dimensional collection communication library parameter space;In the process of optimizing collection communication library parameter using Bayesian algorithm, the collection communication library parameters in different bucket memories are sequentially selected for iterative training, and the abnormal collection communication library parameters are pruned through online pruning strategy, to realize the optimization of collection communication library parameter.The application improves network communication efficiency and expansibility.

A networking device and system are described, among other things. An illustrative system is disclosed to include a congestion controller that manages traffic across a network fabric using receiver-based packet scheduling and a networking device that employs the congestion controller for data flows qualified as a large data flow but bypasses the congestion controller for data flows qualified as a small data flow. For example, the networking device may receive information describing a data flow directed toward a processing network; determine, based on the information describing the data flow, a size of the data flow; determine the size of the data flow is below a predetermined flow threshold; and in response to determining that the size of the data flow is below a predetermined threshold, bypass the congestion controller.

The invention provides an RDMA network performance abnormity diagnosis method under set communication. In a collective communication process, firstly, a monitor located at a server side executes algorithm disassembly in advance, monitors related performance information of a collective communication flow in real time according to steps, and reports the related performance information to an analysis server; the monitors inform the servers of waiting relations by sending Notification packets among the monitors, adaptively execute an appropriate anomaly detection triggering process when performance anomaly occurs, send a polling query data packet to the switches through the servers, trigger telemetry information collection on all the switches related to the anomaly, and report the telemetry information collection to an analysis server in sequence; and finally, the analysis server constructs a waiting relation graph and a network traceability graph, and comprehensively analyzes the waiting relation graph and the network traceability graph to give a diagnosis result. The problems that in a set communication scene, existing analysis modes such as a network traceability graph cannot give the overall performance bottleneck of set communication and cannot give contribution evaluation of flow to overall set communication performance abnormity and the like are solved.

The disclosed device includes a collectives engine that can offload collectives communications of multiple nodes and perform collective operations. The collectives engine can manage a direct mapping scheme of local memories of the nodes for access by the collectives engine. Various other methods, systems, and computer-readable media are also disclosed.

The present invention discloses a model training fault diagnosis method and system for collective communication, which belongs to the field of artificial intelligence and distributed computing. The method comprises: implanting a trigger count to count the number of triggers of the collective communication primitive in the torch_plugin plug-in; adding completion information of processing collective tasks to the communication library, including the counts of various collective communication primitives and the amount of data sent and received, and registering a query interface with the torch_plugin; finally, pre-starting a separate fault diagnosis and analysis thread in different RANK processes of model training, collecting the task counts of the collective communication primitives issued by the upper layer training on different nodes and the task counts actually completed by the collective communication layer, and the amount of data sent and received by the collective communication layer according to the set sampling interval, and analyzing and diagnosing the fault information when the model training is abnormal. The present invention is suitable for large-scale distributed training scenarios and can improve the troubleshooting efficiency of model training tasks.

The invention relates to the technical field of communication, and provides a synchronous optimization method and device for aggregate communication, equipment and a storage medium, and the method comprises the steps: reading a first step number in a first memory, the first step number being used for representing a step number processed by a transmitting end, the first step number is remotely updated by the sending end after first data of each step of the to-be-processed task is processed; and when it is detected that the number of the first step meets a data ready condition, processing received second data of each step, the second data of each step being generated and sent by the sending end after processing the first data of each step. According to the method, through a step-based synchronization mode, synchronization control can be refined to each specific processing step, fine granularity control of a set communication synchronization process is realized, and the degree of parallelism is remarkably improved.

The present disclosure generally relates to optimizing load-balancing of network endpoints using tree collectives representing a logical network communication topology for the network endpoints. Systems and methods described herein eliminate the previously restrictive conditions imposed on tree-based communication collectives by generating collective trees with any arity and representing any number of physical network endpoints. The resulting collective trees ensure that each represented network endpoint has a number of outgoing flows and a number of incoming flows that are no more than the arity of the collective tree. In this way, the described systems and methods inject significant efficiencies into communication collectives within networked compute nodes by eliminating communication bandwidth latencies and bottlenecks.

The present application provides a collective communication method and a communication device, which relate to the field of communication technology, can realize short-range communication, and save communication resources. The method includes: a first terminal device receives at least one second message from a first network device. The second message includes information about a first process and information about a network device corresponding to the first process. The first process is used to perform a first task. The information about the network device corresponding to the first process is information about the network device to which the terminal device where the first process is located belongs. The first terminal device determines a third message based on at least one second message. The third message includes information about a target network device and information about all first processes that perform the first task corresponding to the target network device. The target network device is at least one of the network devices corresponding to the first task. The first terminal device sends a third message to the first network device.

An NCCL set communication optimization method oriented to a large model dynamic training environment comprises the following steps of (1) integrating open-source NCCL codes into PyTorch so as to be applied to actual distributed training; (2) a multi-node consistency guarantee mechanism is used, and system crash caused by strategy asynchronization is avoided; (3) respectively realizing a self-adaptive Channel allocation strategy on three levels of task scheduling, CUDAkernel and Proxy of the NCCL communication library; and (4) collecting, storing and visualizing multi-dimensional indexes. The method is compatible with mainstream deep learning ecology, and the communication efficiency and stability in a large model dynamic training environment are remarkably improved on the premise that hardware is not modified.

The present invention provides a processor chip, a collective communication method, and an electronic device. The processor chip includes: a plurality of processor cores arranged in an array, hierarchically divided according to a first communication hierarchical division rule to obtain a plurality of first split units; the plurality of processor cores included in each first split unit form a corresponding first loop according to a first ring formation rule; the plurality of processor cores are hierarchically divided according to a second communication hierarchical division rule to obtain a plurality of second split units; the plurality of processor cores included in each second split unit form a corresponding second loop according to a second ring formation rule; each first split unit communicates in parallel, and the corresponding processor core communicates based on the first loop; each second split unit communicates in parallel, and the corresponding processor core communicates based on the second loop. The processor chip, collective communication method, and electronic device provided by the embodiments of the present invention improve the collective communication efficiency of the processor chip.

The present disclosure generally relates to optimizing load-balancing of network endpoints using tree collectives representing a logical network communication topology for the network endpoints. Systems and methods described herein eliminate the previously restrictive conditions imposed on tree-based communication collectives by generating collective trees with any arity and representing any number of physical network endpoints. The resulting collective trees ensure that each represented network endpoint has a number of outgoing flows and a number of incoming flows that are no more than the arity of the collective tree. In this way, the described systems and methods inject significant efficiencies into communication collectives within networked compute nodes by eliminating communication bandwidth latencies and bottlenecks.

Embodiments of this application provide a chip system and a collective communication method, and relate to the field of chip technologies. A specific solution is: The chip system includes a processor core, a task scheduler, and a first accelerator. The processor core is configured to: orchestrate all tasks of collective communication in an expected execution sequence, to obtain an orchestrated execution sequence of the tasks, and deliver the orchestrated execution sequence of the tasks and a synchronization task to the task scheduler, where the synchronization task is used to control the orchestrated execution sequence of the tasks. The task scheduler is configured to: schedule the tasks to the first accelerator in the orchestrated execution sequence of the tasks, and control the orchestrated execution sequence of the tasks based on the synchronization task. The first accelerator is configured to execute the tasks in the orchestrated execution sequence of the tasks.

The application discloses a data processing method and an exchange in asymmetric set communication. The method applied to the exchange comprises the following steps: receiving a first data packet sent by a first processor; the first data packet comprises at least a packet type field, a processor identifier corresponding to the first processor and a virtual address; the packet type field is located in a packet header of the first data packet; the virtual address corresponds to a memory access physical address in a processor; the packet type field represents a data processing stage of the set communication; determining a second processor corresponding to the first processor according to the processor identifier corresponding to the first processor; and sending a second data packet to the second processor.

A processing system dynamically and selectively quantizes or compresses variable-length input data for a collective operation to fit within a predetermined limit, executes the collective operation on the compressed data, and converts the data back to variable length results by dequantizing or decompressing the results of the collective operation.

The invention relates to the technical field of communication, and provides a deadlock avoiding method and device in set communication and a storage medium, and the method comprises the steps: receiving a data request from a transmitting end, the data request being obtained and transmitted from a request queue when the transmitting end detects that a receiving end has corresponding resources; when it is detected that the number of the data requests exceeds the depth of a receiving buffer area, determining a first request and a second request from the data requests, putting the first request into the receiving buffer area, and putting the second request into a target memory; wherein the target memory is used for sending the second request to the receiving buffer area under the condition that the receiving buffer area has the free space. According to the method, the data request is stored at the sending end by using the request queue, and the overflow request is temporarily stored at the receiving end by using the target memory, so that the deadlock risk in set communication can be remarkably reduced, and the resource utilization rate and the system stability are improved.