Neural Network Memory Management for Larger Batch Training

The patent divides memory into two distinct segments: local memory for storing forward propagation results and global memory for storing gradients. This segmentation allows each memory type to be optimized for its specific function, with local memory providing fast access for intermediate results and global memory providing sufficient capacity for gradient storage, thereby resolving the contradiction between access speed and memory occupancy.

The patent extracts gradient storage from local memory to global memory. By removing the gradient storage function from local memory, the available local memory capacity is increased, enabling larger batch sizes to be processed in parallel operations. This extraction resolves the contradiction by allowing larger batches (higher throughput) without proportionally increasing local memory requirements.

The patent introduces global memory as an intermediary storage mechanism for gradients. Checkpoints are set in multiple layers to improve gradient calculation accuracy, and the gradients from these checkpoints are stored in global memory rather than consuming local memory. This intermediary approach allows multiple checkpoints to coexist without proportionally increasing local memory usage, resolving the contradiction between accuracy and memory consumption.