Graph Neural Network Execution on NPUs with CPU–DPU–DSP Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deploying graph neural networks (GNNs) on resource-constrained devices like NPUs faces challenges due to irregular memory access patterns, dynamic computation workloads, and the need for effective parallelism, leading to inefficiencies and high energy consumption, particularly in sparse and dynamic graph structures.
Innovation Solution
The GraNNite methodology optimizes GNN deployment on NPUs through model-specific graph partitioning, dynamic node and edge updates, node padding, replacing control-heavy DSP operations with data-parallel DPU operations, and techniques like INT8 quantization and vertical fusion to minimize memory usage and computation costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If GNNs are deployed on NPUs to achieve high accuracy AI applications, then computational performance is improved, but energy consumption increases significantly
Solution Approach 1:
The GNN computation is segmented into two distinct phases: control-intensive operations (graph preprocessing, adjacency matrix construction, dynamic graph updates) executed on the CPU, and data-parallel operations (matrix multiplications, convolution operations, activation functions) executed on the NPU. This segmentation allows each processor to operate in its optimal efficiency zone, reducing overall energy consumption while maintaining high computational performance.
Solution Approach 2:
The system dynamically changes operational parameters by adjusting the boundary between CPU and NPU tasks based on graph characteristics (sparsity, diameter, node degree distribution). For sparse graphs with large diameters, more preprocessing is offloaded to the CPU, while for dense graphs with small diameters, more computation is moved to the NPU, optimizing the energy-performance tradeoff for different graph types.
2Adaptability or versatility
If control-heavy DSP operations are used for GNN execution, then flexibility in handling dynamic graphs is improved, but resource efficiency deteriorates
Solution Approach 1:
The system substitutes control-intensive DSP operations with data-parallel DPU operations where feasible. Specifically, graph preprocessing tasks that involve regular patterns (adjacency matrix construction, node feature initialization) are replaced with optimized DPU kernels that leverage parallel processing capabilities, thereby improving resource efficiency while maintaining flexibility through the CPU-DPU hybrid architecture.
3Adaptability or versatility
If graph preprocessing is performed on CPU to handle irregular memory access patterns, then adaptability to dynamic graphs is improved, but computation time increases
Solution Approach 1:
The system performs preliminary graph preprocessing operations on the CPU during idle periods or in parallel with NPU computation. Adjacency matrices, node feature vectors, and edge lists are preprocessed and stored in optimized formats before NPU execution begins. This preliminary action reduces the computation time during actual GNN inference by having data ready in NPU-friendly formats.
Solution Approach 2:
The CPU and NPU operate in a continuous pipeline where the CPU performs preprocessing tasks while the NPU executes computation tasks, and vice versa. This overlapping execution ensures that neither processor remains idle, maintaining continuous useful action and reducing overall computation time while preserving adaptability to dynamic graphs through CPU-based graph updates.
Data Source
AI summary
Workloads for executing a graph neural network (GNN) may be divided among various processing units, such as a central processing unit (CPU) and a neural processing unit (NPU). The NPU may include a data processing unit (DPU) and a digital signal processor (DSP). The CPU may perform precomputation, model optimization, hardware optimization, and compilation. For example, the CPU may precompute a parameter matrix and use the parameter matrix as internal parameters of a GNN. The CPU may also perform node padding, approximation computation, or transfer of DSP operations to DPU to optimize the GNN. The CPU may also perform sparsity data compute and storage, vertical fusion of DSP operations and DPU operations, or data quantization to optimize performance of the NPU. The compiled GNN may be provided to the NPU, and the DPU and DSP may perform the operations in the compiled GNN to produce a prediction of the GNN.