Data Shuffle Offload via Network Interface Controller
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data shuffle operations in networking devices, particularly at CPUs, are inefficient due to resource utilization bottlenecks, memory bandwidth consumption, and the need for complex element location descriptions, especially when handling small data elements or non-homogenous data sizes.
Innovation Solution
Offloading data shuffle operations from CPUs to network devices, such as switches or Network Interface Controllers (NICs), which include a data shuffle unit capable of receiving data and a descriptor for the shuffle pattern, performing the shuffle using dedicated circuitry, and outputting the shuffled data for memory storage or network transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data shuffle operations are performed at a CPU or GPU of a host device, then the data shuffle operation can be completed, but the resources of the CPU or GPU are not free to perform other tasks and memory bandwidth is consumed
Solution Approach 1:
The patent extracts the data shuffle operation from the CPU/GPU and relocates it to a network device (NIC). The NIC includes a data shuffle unit with dedicated circuitry that performs the shuffle operation independently, freeing the CPU/GPU resources to perform other computational tasks while the shuffle operation completes in parallel at the network device.
Solution Approach 2:
The network device acts as an intermediary between the host device and the memory system. Instead of the CPU/GPU directly performing the shuffle operation and accessing memory, the request is offloaded to the network device which handles the shuffle operation and memory interaction, reducing the computational burden on the CPU/GPU.
2Adaptability or versatility
If a large data structure is shuffled using existing NIC capabilities, then arbitrary physical memory locations can be mapped, but complex element location description is required with multiple {offset, size} elements
Solution Approach 1:
The patent changes the parameter representation from complex multi-element descriptors to a simplified single-descriptor format. Instead of using multiple {offset, size} elements to describe data structure locations, the system uses a single descriptor with base address and stride parameters that can mathematically define the same memory locations, reducing complexity while maintaining mapping capability.
3Productivity
If small data elements are shuffled using existing NIC capabilities, then the shuffle operation can be performed, but multiple PCIe reads are required which wastes PCIe bandwidth and encounters multiple bottlenecks
Solution Approach 1:
The patent applies preliminary action by performing the data shuffle operation before the PCIe transfer. Instead of reading small data elements multiple times through PCIe (which would waste bandwidth), the shuffle is completed in advance at the network device using local memory and circuitry, and only the final shuffled result is transferred through PCIe in a single operation.
4Productivity
If data is read and written again to memory to support a data shuffle operation, then the shuffle operation can be performed, but memory bandwidth is consumed
Solution Approach 1:
The patent extracts the data shuffle operation from the memory subsystem and relocates it to the network device. The NIC includes its own local memory and dedicated shuffle circuitry, allowing it to perform the shuffle operation using local resources without requiring additional read/write operations to the host device's main memory, thereby conserving memory bandwidth.
Data Source
AI summary
Devices, methods, and systems are provided. In one example, a device is described to include a device interface that receives data from at least one data source; a data shuffle unit that collects the data received from the at least one data source, receives a descriptor that describes a data shuffle operation to perform on the data received from the at least one data source, performs the data shuffle operation on the collected data to produce shuffled data, and provides the shuffled data to at least one data target.