Neural Processing Unit Training for Real-Time Edge Adaptation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for training and fine-tuning deep neural networks (DNNs) on neural processing units (NPUs) face challenges due to architectural differences with GPUs and TPUs, leading to inefficiencies in handling complex architectures and high computational demands, particularly in edge devices where real-time adaptation, privacy, and cost-effective scalability are essential.
Innovation Solution
The method involves offloading forward and backward passes of the training process to NPUs, integrating an automatic differentiation module for seamless gradient computation, and using mixed-precision formats like BF16 and FP16 to enhance computational efficiency and reduce memory bandwidth, enabling direct on-device training and fine-tuning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If DNNs are trained on NPUs to achieve real-time adaptation and enhanced privacy, then model adaptation capability and data security are improved, but computational efficiency and training speed deteriorate due to architectural differences with GPUs and TPUs
Solution Approach 1:
The patent introduces a hybrid training architecture where the NPU serves as an intermediary component working in conjunction with a CPU or GPU. The NPU handles specific neural network operations locally on the edge device, while the CPU/GPU manages complex training tasks and coordinate the overall training process. This intermediary approach allows the system to leverage the NPU's real-time processing capabilities while maintaining compatibility with standard training frameworks, thus resolving the contradiction between real-time adaptation and training speed.
2Quantity of substance
If mixed-precision formats (BF16, FP16) are used to reduce memory bandwidth and enhance computational efficiency, then memory usage is reduced, but precision and accuracy of gradient computation may deteriorate
Solution Approach 1:
The patent applies mixed-precision training by using different precision formats for different parts of the computation process. Specifically, BF16 (Brain Floating Point 16-bit) format is used for weight storage and forward propagation to reduce memory bandwidth requirements, while FP32 (32-bit floating point) is retained for gradient computation and parameter updates to maintain numerical stability and precision. This local quality approach allows the system to optimize memory usage in non-critical paths while preserving accuracy in critical gradient calculation operations.
3Extent of automation
If forward and backward passes are offloaded to NPU to enable on-device training, then device autonomy and privacy are improved, but device complexity and implementation difficulty increase due to architectural differences
Solution Approach 1:
The patent implements a universal training framework that enables NPUs to perform multiple functions including forward propagation, backward propagation, and parameter updates within a single unified architecture. The system designs the NPU to handle diverse neural network operations (convolution, fully-connected layers, normalization) through a common computational engine, eliminating the need for separate specialized hardware for each operation type. This multi-functionality reduces implementation complexity while maintaining the capability for complete on-device training.
Data Source
AI summary
A kernel on a neural processing unit may perform matrix multiplications (MatMuls) on tensors of various dimensions. A neural network may be trained through a forward operation and backward operation, both of which may be offloaded to the kernel. For the forward operation, the kernel may execute a layer by performing a MatMul on an input tensor and weight tensor and produce an output tensor. A loss may be computed. For the backward operation, the kernel may compute a weight gradient of the loss by performing a MatMul on the input tensor and a gradient of the output tensor and compute an input gradient of the loss by performing a MatMul on the gradient of the output tensor and the weight tensor. The gradient of the output tensor may be computed using an automatic differentiation module. The weight tensor may be updated based on the input gradient and weight gradient.