Layer-Wise Learning Rate Braking for Gradient Skip Training

The patent applies dynamics by making the learning rate adjustment timing adaptive rather than fixed. The system dynamically determines when to adjust the learning rate based on the actual convergence state of each layer, using the error gradient threshold and convergence determination unit to detect when a layer has sufficiently converged. This allows the learning rate to be adjusted at optimal moments for each specific layer, preventing both premature skipping that would hurt accuracy and delayed adjustment that would waste computation.

The patent changes the parameter of learning rate adjustment timing from a fixed schedule to a dynamic parameter based on convergence detection. By monitoring error gradients and determining convergence states, the system adjusts the learning rate parameter at appropriate times for each layer. This parameter change enables the system to skip layers more aggressively when appropriate (improving speed) while maintaining accuracy by adjusting rates only when convergence is confirmed.

The patent extracts the learning rate adjustment operation from the regular training loop and applies it selectively only when needed. The convergence determination unit identifies which layers have sufficiently converged based on error gradient thresholds, and the learning rate is adjusted only for those specific layers at specific moments. This extraction prevents unnecessary aggregation and reflection operations for layers that have already converged, reducing processing time while maintaining accuracy for layers that still need training.

The patent implements skipping by allowing layers to be bypassed in the learning rate adjustment process once they converge. The Gradient Skip determination unit identifies converged layers, and the system rushes through these layers by skipping further learning rate adjustments and aggregation operations for them. This skipping mechanism eliminates wasted processing time on converged layers while continuing to invest computational resources in layers that have not yet converged.

The patent applies local quality by allowing different learning rates for different layers based on their individual convergence states. Instead of using a uniform high learning rate across all layers (which causes oscillations), the system identifies layers that have converged and applies lower or zero learning rates to them, while maintaining higher learning rates for layers that have not yet converged. This localized adaptation of learning rates enables fast convergence for active layers while ensuring stability for converged layers.

The patent makes the learning rate dynamic and adaptive rather than fixed. The learning rate for each layer changes over time based on its convergence state, detected through error gradient monitoring. Layers transition from high learning rates during active training to low or zero learning rates after convergence, creating a dynamic system that achieves both fast convergence and stability.