On-Device Neural Network Training With Forward Gradients
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for adapting pre-trained models to personalized data on memory-constrained devices, such as edge devices, are prohibitive due to the large memory footprint required for traditional backpropagation and the lack of training capabilities in existing low power engines and microcontrollers.
Innovation Solution
A processor-implemented method for a fixed-point, forward-forward on-device model training that includes running a first forward call according to positive perturbation parameters sampled from a random perturbation vector that follows standard, normal distribution, with positive perturbation parameters sampled from a random perturbation vector that follows standard, normal distribution, and negative perturbation parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional backpropagation is used for model training, then model adaptation to personalized data is achieved, but memory footprint becomes prohibitively large for edge devices
Solution Approach 1:
The patent extracts and eliminates the backward pass computation from the training process, keeping only the forward pass. This removes the need to store intermediate activations for gradient computation, thereby dramatically reducing memory footprint while maintaining model adaptation capability through forward-gradient estimation
Solution Approach 2:
The patent introduces forward gradients as an intermediary concept that approximates the true gradients without requiring exact backward propagation. This mediator enables gradient-based optimization on memory-constrained devices by providing sufficient signal for weight updates without the full computational overhead of traditional backpropagation
2Adaptability or versatility
If traditional backpropagation is used for model training, then model adaptation to personalized data is achieved, but existing low power neural processing engines and microcontrollers cannot perform training due to lack of training capabilities
Solution Approach 1:
The patent extracts the training capability from complex dedicated training hardware and implements it using only the forward pass computation that existing inference accelerators already possess. This extraction enables training functionality on devices designed only for inference by removing the requirement for backward pass computation
Solution Approach 2:
The patent makes existing fixed-point inference accelerators multi-functional by enabling them to perform both inference and training operations. The same forward pass computation engine used for inference is repurposed for training, eliminating the need for separate training hardware on edge devices
3Use of energy by moving object
If fixed-point computation is used on edge devices, then power consumption is reduced and inference is accelerated, but training capabilities are lost
Solution Approach 1:
The patent enables fixed-point inference accelerators to perform both inference and training functions using the same hardware. The forward pass computation, which is essential for both inference and the proposed training method, is leveraged to provide training capability without adding dedicated training hardware
Solution Approach 2:
The patent changes the computational paradigm from requiring floating-point precision for both forward and backward passes to using fixed-point arithmetic for forward passes with gradient estimation. This parameter change in precision requirements enables training on low-power fixed-point hardware while maintaining adequate training effectiveness
Data Source
AI summary
A processor-implemented method for a fixed-point, forward-forward on-device model training/adaptation is described. The processor-implemented method includes running a first forward call according to positive perturbation parameters sampled from a random perturbation vector that follows standard, normal distribution. The processor-implemented method also includes running a second forward call according to negative perturbation parameters sampled from the random perturbation vector. The processor-implemented method further includes computing forward gradients according to the random perturbation vector and a directional derivative based on the first forward call and the second forward call. The processor-implemented method also includes updating weights of the on-device model according to the forward gradients.