49 results about "Collective operation" patented technology

A method for data processing by a data clean room orchestration system is described. The method includes receiving an indication of mutually attested code for a data clean room between two or more partners. The method further includes configuring a trusted execution environment (TEE), including one or more virtual machines (VMs) that are individually or collectively operable to execute the mutually attested code. The method further includes transmitting, to endpoints associated with the partners, an attestation report including at least an encrypted token and a host public key of a host machine associated with the one or more VMs. The method further includes receiving respective partner secret keys wrapped with the host public key. The method further includes executing the mutually attested code on respective partner datasets in the TEE based on using a host private key of the host machine to unwrap the respective partner secret keys.

Provided are systems, methods, and apparatuses for distributed, in-switch collective processing. In one or more examples, the systems, devices, and methods include configuring the group of collective engines as one or more logical collective engines. The systems, devices, and methods include distributing a collective message from the source node to a master engine of all, or some portion of, the group of collective engines, and the collective engines exchanging one or more collective messages with participating engines and end nodes of the collective to satisfy the collective operation. Multiple collective contexts may be supported simultaneously with either the same or a different collective engine instance serving as the master instance for a given collective context.

Aspects of the disclosure are directed to a tree-based network architecture for serving and / or training machine learning models. The architecture includes one or more multi-chip packages having a plurality of compute-memory stacks connected via an input / output (I / O) die. The I / O die includes an aggregator to aggregate computations from the compute-memory stacks. The architecture can further include a plurality of the multi-chip packages connected on a server via a server level aggregator and a plurality of the servers connected on a rack via a rack level aggregator for further aggregation of the computations from the compute-memory stacks. The tree-based network architecture allows for fewer hops, resulting in lower latency and savings in bandwidth when serving and / or training machine learning models.

Examples described herein relate to a network interface device that includes: a host interface; a direct memory access (DMA) circuitry; a network interface to receive, in at least one packet, time data associated with at least one of multiple layers, wherein the multiple layers provide inputs to a collective operation associated with a large language model (LLM); and circuitry. The circuitry is to based, at least in part, on the time data associated with the multiple layers, identify a first operation of a first layer of the multiple layers as a late completing process relative to times to completion of multiple first operations of other layers and based on the first operation being identified as a late completing process, perform a remedial action to adjust at least one configuration of a first device to execute a second operation of the first layer.

A collective communication processing method, comprising: acquiring a node orchestration request for a target collective operation in collective communication, wherein an original node list carried in the node orchestration request comprises server nodes selected for executing the target collective operation (202); acquiring topology information of the server nodes, wherein the topology information is used for describing switching nodes connected to the server nodes in hierarchies of a node communication network, and the server nodes communicate with each other on the basis of the connected switching nodes in the hierarchies (204); on the basis of the topology information, orchestrating and updating the arrangement order of the server nodes hierarchy by hierarchy according to the hierarchies in the node communication network so as to obtain an orchestrated node list (206); and according to the arrangement order of the server nodes in the orchestrated node list, performing collective communication for the target collective operation by means of the server nodes (208).

The disclosed device can perform a collective operation on received datasets, and split the result into chunks in accordance with a chunking scheme. The device can also forward the chunks in accordance with a routing scheme that can direct chunks to appropriate nodes of a collective network. Various other methods, systems, and computer-readable media are also disclosed.

Systems, methods, and machine-readable media may facilitate programmable data trimming. Metadata associated with a collective operation may be determined by an application of a server. The metadata may specify a job identifier corresponding to a unit of work to be completed in conjunction with the collective operation, a collective type of the collective operation, and / or an ordering mode for packets corresponding to the collective operation. The metadata associated with the collective operation may be sent by the application to a network interface card (NIC). The NIC may be caused by the application to transmit a data packet with the metadata embedded in a cookie of the data packet to a switch of a network fabric to cause the switch to use a selected network path and / or selected load-balancing for the collective operation based on one or more of the job identifier, the collective type, and / or the ordering mode.

Automatic run-time skew-aware optimization of rooted collectives includes determining skews amongst compute nodes of a distributed computing system, as an application executes on the compute nodes, and determining implementations for collective operations of the application based at least in part on the skews. Skews may be determined based on timestamps of operations of the application program. Time stamps of one or more of the compute nodes may be estimated or inferred from timestamps of other compute nodes. Timestamps may be aggregated to determine global skews. Collective implementations may be determined for a sequence of collective operations based on skew impacts amongst the sequence of collective operations. Subsequent collective operations may be predicted based on current collective operations and a history of persistent collective operations, and implementations may be determined for the predicted collective operations prior to receipt of calls for the predicted collective operations.

A method for performing a message passing interface (MPI) collective operation in a network, wherein the network comprises a plurality of interconnected nodes, the method comprising: receiving, at a node of the plurality of interconnected nodes, MPI collective operation information identifying the MPI collective operation to be performed, and a graph of the network; determining a number of algorithmic steps of the MPI collective operation based on the MPI collective operation and the graph of the network; determining an initialisation process for the algorithmic steps; determining a finalisation process for the algorithmic steps; determining, for each of the algorithmic steps: a subset of nodes of the plurality of interconnected nodes for the node to communicate with; one or more portions of data for the node to send to and receive from the nodes within the subset of nodes; and initialising the MPI collective operation based on the determined subset, initialisation process and finalisation process, and the one or more portions of data.

Examples described herein relate to a network interface device that includes: a host interface; a direct memory access (DMA) circuitry; a network interface to receive, in at least one packet, time data associated with at least one of multiple layers, wherein the multiple layers provide inputs to a collective operation associated with a large language model (LLM); and circuitry. The circuitry is to based, at least in part, on the time data associated with the multiple layers, identify a first operation of a first layer of the multiple layers as a late completing process relative to times to completion of multiple first operations of other layers and based on the first operation being identified as a late completing process, perform a remedial action to adjust at least one configuration of a first device to execute a second operation of the first layer.

A system includes one or more processors to trace one or more packets transmitted by an application distributed among a plurality of computing nodes. The one or more processors are to generate tracing data based at least in part on tracing the one or more packets. The tracing data includes temporal information associated with transmission of the one or more packets. The one or more processors are to manage a data allocation associated with the application based on the tracing data.

Systems, methods, apparatuses, and computer program products for zero-copy symmetric multicast communication buffers for offload operations. A method may include receiving an instruction for performing a collective operation across a plurality of processing elements. The method may also include determining an offset of the first virtual address from a first base address associated with the first set of contiguous virtual addresses. The method may further include translating the first virtual address to a corresponding multicast virtual address based on the offset. Further, the method may include causing the collective operation to be performed based at least on the multicast virtual address.

Implementations for compressed transaction layer encoding of collective operations in accelerator networks. An accelerator for an accelerator interconnect network may encode both collective operation requests and unicast requests into a compressed request field of a transaction layer flit using a shared multi-bit command field, enabling collective operations to achieve the same bandwidth efficiency as unicast operations at the transaction layer. The accelerator may further process compressed responses by extracting a single-bit indicator that distinguishes collective operation responses from unicast responses without reference to the original request, enabling stateless response differentiation at the receiver. The compressed encoding may overload existing identifier fields to carry group identifiers for collective operations while retaining physical accelerator identifiers for unicast operations. Implementations may further map the response type to protocol layer interface signals and may support dedicated compressed response field types for block collective operations.

Implementations for virtual partition lifecycle management with collective operation resource allocation in an accelerator interconnect. In some implementations, a system for a UALink-based network comprises switches with group tables and queues for collective operations, and a centralized controller that, upon creating a virtual partition (which may be a virtual pod), programs forwarding entries and then programs group table entries and allocates queue entries for accelerators within the virtual partition. The centralized controller may deactivate group table entries and deallocate queue entries before removing forwarding entries on teardown. In some implementations, the centralized controller maintains per-virtual-partition resource accounting of group table entries and queue entries allocated to each virtual partition, and enforces per-virtual-partition quotas by rejecting allocation requests that would exceed a configured quota.

Implementations for switch-offloaded block collective operations with autonomous memory access in an accelerator interconnect. In some implementations, a switch for a UALink-based network comprises a queue for storing collective operation parameters and a circuit configured to receive an invocation request carrying a control block specifying collective parameters, issue read requests to a group of destination accelerators, perform a reduction operation on received data, issue write requests to write reduced data, and write a completion status to a status buffer in the accelerator. The switch may constrain generated addresses to a bounded memory region using an address mask register. In some implementations, the switch manages queue resources through dynamic allocation of queue partitions, concurrent processing of collective operations from different partitions, and completion of outstanding operations before releasing allocated entries upon deallocation.

Systems, methods, and devices for performing computational operations and managing network congestion are provided. An example describes a device comprising a processing unit that collects multiple messages and performs an operation as part of a collective operation on the data contained within the multiple messages. It then generates an output message containing the result of the operation performed on the data contained within the multiple messages. The processing unit can further embed a congestion notification in the output message if at least one of the multiple messages also contains a corresponding congestion notification.

The disclosed apparatus includes a collective engine that may select a communication cost model for a collective operation from a plurality of communication cost models and configure a topology of a collective network for performing the collective operation using the selected communication cost model. Various other methods, systems, and computer readable media are also disclosed.

Implementations for collective operation resilience in a chiplet-based accelerator interconnect. In some implementations, a system for a UALink-based network comprises a chiplet die with stations coupled to an accelerator die via a die-to-die interface, switches with group tables identifying groups of accelerators for collective operations, and a centralized controller. Upon detection of a station fault on the chiplet die, the centralized controller identifies group table entries on the switches that reference accelerators coupled to the faulted station and updates the entries to exclude those accelerators, enabling collective operations to continue with reduced membership. In some implementations, an accelerator die coupled to chiplet dies redistributes transaction layer flits from a faulted chiplet die to operational chiplet dies, and the centralized controller updates forwarding entries on switches to reflect the changed connectivity.

Systems and methods for performing collective operations associated with artificial intelligence (AI) using a property of electromagnetic radiation are described. An example method for processing an artificial intelligence (AI) model includes a first set of neurons associated with a first layer of the AI model communicating via electromagnetic radiation: (1) reference signals for facilitating a collective operation associated with the AI model, and (2) input signals for use with a second layer of the AI model for performing the collective operation associated with the AI model. The method further includes a second set of neurons associated with the second layer receiving, as a result of processing of a property of the electromagnetic radiation, the reference signals, and the input signals for performing the collective operation associated with the AI model.

A system includes one or more processors to trace one or more packets transmitted by an application distributed among a plurality of computing nodes. The one or more processors are to generate tracing data based at least in part on tracing the one or more packets. The tracing data includes temporal information associated with transmission of the one or more packets. The one or more processors are to manage a data allocation associated with the application based on the tracing data.

The invention relates to efficient iterative collective operation using network attached memory. A system receives a first request to perform a collective operation. The system stores a mapping of a first virtual address to a descriptor of a physical location of the allocated memory region. The system performs the collective operation by writing data to a first section of the memory region and accessing data from other sections of the memory region. The system receives a second request to perform an update operation, the second request indicating the first virtual address, one or more portions of a memory region segment to be updated, and corresponding data units to be written to the portions. The system updates only the indicated portion by writing the corresponding data unit based on the mapping. The system performs a subsequent iteration of the collective operation based on the mapping by avoiding writing any data to the memory region and accessing data units only from the memory region.

A system receives a first request to perform a collective operation. The system stores a mapping of a first virtual address to a descriptor for a physical location of an allocated memory region. The system performs the collective operation, by writing data to a first segment of the memory region and accessing data from other segments of the memory region. The system receives a second request to perform an update operation, the second request indicating the first virtual address, one or more portions of a memory region segment to be updated, and corresponding data units to write to the portions. The system updates, based on the mapping, only the indicated portions by writing the corresponding data units. The system performs a subsequent iteration of the collective operation, based on the mapping, by bypassing writing any data to the memory region and only accessing data units from the memory region.

The disclosed device includes a collectives engine that can offload collectives communications of multiple nodes and perform collective operations. The collectives engine can manage a direct mapping scheme of local memories of the nodes for access by the collectives engine. Various other methods, systems, and computer-readable media are also disclosed.

In some examples, a processor receives a first request to allocate a memory region for a collective operation by process entities in a plurality of computer nodes. In response to the first request, the processor creates a virtual address for the memory region and allocates the memory region in a network-attached memory coupled to the plurality of computer nodes over a network. The processor correlates the virtual address to an address of the memory region in mapping information. The processor identifies the memory region in the network-attached memory by obtaining the address of the memory region from the mapping information using the virtual address in a second request. In response to the second request, the processor performs the collective operation.

A system comprises one or more processors for tracking one or more packets transmitted by an application distributed across multiple compute nodes. The one or more processors are used to generate tracking data, at least in part, based on the tracking of the one or more packets. This tracking data includes temporal information associated with the transmission of the one or more packets. The one or more processors are also used to manage application-related data allocation based on this tracking data.

A system receives a first request to perform a collective operation. The system stores a mapping of a first virtual address to a descriptor for a physical location of an allocated memory space. The system performs the collective operation by writing data to a first segment of the memory space and accessing data from other segments of the memory space. The system receives a second request to perform an update operation, the second request specifying the first virtual address, one or more portions of a memory space segment to be updated, and corresponding data units to be written to the portions. Based on the mapping, the system updates only the specified portions by writing the corresponding data units.The system performs a subsequent iteration of the collective operation based on the mapping by bypassing writing data to the memory area and accessing only data units from the memory area.

An apparatus facilitating efficient key refresh in a node is provided. During operation, the apparatus can determine a collective operation initiated by the node. The node can include a processor and can be in a distributed system comprising a plurality of nodes. The collective operation can be performed by a subset of the plurality of nodes in conjunction with each other. The apparatus can generate a new key based on a previous key maintained at the apparatus. Here, a respective key can be used for encrypting an inter-node packet in the distributed system. The apparatus can maintain the new and previous keys for the duration of the collective operation. Either of the new and previous keys can be used for decrypting messages received at the apparatus from other nodes of the distributed system. Upon determining a threshold point of the collective operation, the apparatus can discard the previous key.

Implementations for selective cryptographic processing of collective and unicast traffic at a switch in an accelerator network. As AI training and inference workloads increasingly rely on in-network collective operations to accelerate gradient synchronization, broadcast, and reduction across large-scale accelerator pods, some implementations include a switch comprising a circuit that determines, for each encrypted transaction, whether it is a collective transaction or a unicast transaction. For collective transactions, the circuit decrypts transaction data for processing such as arithmetic reduction at the switch. For unicast transactions, the circuit bypasses decryption and forwards the transaction with data remaining encrypted between source and destination accelerators. Some implementations further include selective trust elevation of the switch into a trusted computing base for collective operations via a security manager, secure session establishment, attestation verification, and encryption key programming, while maintaining the switch outside the trusted computing base for unicast operations.

An AI-accelerating processor system may include an acyclic subset of hardware processing nodes. The acyclic subset includes a plurality of end nodes that are disconnected from other end nodes in the acyclic subset. The acyclic subset of hardware processing nodes is configured to perform, according to schedules, computations that are part of a collective operation. A first hardware processing node in the subset has a first scheduling pattern and a second hardware processing node in the subset has a second scheduling pattern that is different from the first scheduling pattern to account for the subset being acyclic. The acyclic subset of hardware processing nodes is also configured to transmit computation outputs to neighboring hardware processing nodes among the acyclic subset through the bi-directional links to generate a result that is part of the collective operation. The result is contributed by each of the hardware processing nodes in the acyclic subset.

An AI-accelerating processor system may include an acyclic subset of hardware processing nodes. The acyclic subset includes a plurality of end nodes that are disconnected from other end nodes in the acyclic subset. The acyclic subset of hardware processing nodes is configured to perform, according to schedules, computations that are part of a collective operation. A first hardware processing node in the subset has a first scheduling pattern and a second hardware processing node in the subset has a second scheduling pattern that is different from the first scheduling pattern to account for the subset being acyclic. The acyclic subset of hardware processing nodes is also configured to transmit computation outputs to neighboring hardware processing nodes among the acyclic subset through the bi-directional links to generate a result that is part of the collective operation. The result is contributed by each of the hardware processing nodes in the acyclic subset.