Local Forward Gradient Scaling for Large Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Standard forward gradient algorithms suffer from the curse of dimensionality, leading to prohibitively high computational requirements for training larger machine-learned models, making them inefficient.
Innovation Solution
Implementing localized optimization functions for discrete portions of machine-learned models to scale forward gradients efficiently, using techniques such as blockwise, patchwise, and channel-wise loss criteria, and applying perturbations to model units for gradient determination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If standard forward gradient algorithms are used for training machine-learned models, then gradient computation can be performed in a biologically plausible manner, but computational resources required become prohibitively high for larger models
Solution Approach 1:
The patent divides the machine-learned model into discrete model portions (e.g., layers or groups of layers) and applies localized optimization functions to each portion. This segmentation allows gradient computation to be performed locally rather than globally, significantly reducing computational resources while maintaining forward gradient learning
Solution Approach 2:
The patent implements localized optimization functions that operate on specific model portions rather than the entire model. Each local optimization function computes gradients for its associated model portion independently, enabling efficient resource utilization while preserving the biological plausibility of forward gradient computation
2Use of energy by moving object
If localized optimization functions are implemented for model portions, then computational resources are reduced, but model training efficiency may be affected
Solution Approach 1:
The patent applies optimization functions to discrete model portions rather than attempting to optimize the entire model simultaneously. This partial action approach reduces computational burden while still achieving effective model training through iterative optimization of local portions
Solution Approach 2:
The patent transitions from global model optimization to local portion optimization, adding a spatial dimension to the optimization process. By organizing optimization operations across multiple model portions in sequence or parallel, the system achieves both resource efficiency and training effectiveness
Data Source
AI summary
A plurality of model portions are determined from a machine-learned model based on at least one criterion. A plurality of local optimization functions are respectively determined for the plurality of model portions. Forward-mode differentiation is performed for each model portion of the plurality of model portions. Performing forward-mode differentiation includes applying a perturbation to outputs of one or more model units of the model portion. Performing forward-mode differentiation includes, based at least in part on the perturbation, determining a gradient of the local optimization function for the model portion. Performing forward-mode differentiation includes modifying one or more parameters of the model portion based on the gradient.