GPU Cluster Convolutional Neural Network Update via Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional GPUs with limited memory capacity struggle to update convolutional neural networks due to the high memory requirements for storing parameters and output results of fully connected and convolutional layers, making it difficult to train these networks effectively.
Innovation Solution
A method utilizing a GPU cluster where parameters of each layer are distributed across multiple GPUs, allowing for efficient training by performing operations in parallel and sharing memory resources, with one GPU acting as a coordinator to manage the process and combine results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a single GPU is used to update the convolutional neural network, then the memory capacity is sufficient to store all parameters and output results, but the device complexity increases and training time becomes excessively long
Solution Approach 1:
The patent divides the convolutional neural network into front-end network and back-end network, distributing parameters across multiple GPUs. Each GPU stores only a portion of the network parameters, reducing individual memory requirements while enabling training of larger models through parallel processing.
2Device complexity
If a single GPU is used to update the convolutional neural network, then the system structure is simple, but the training time becomes excessively long due to sequential processing
Solution Approach 1:
The patent segments the computational workload across multiple GPUs, with each GPU processing specific layers or batches of data in parallel. This segmentation enables simultaneous computation of different network portions, dramatically reducing training time while maintaining manageable system complexity through coordinated processing.
Solution Approach 2:
The patent transitions from single-GPU sequential processing to multi-GPU parallel processing by adding the dimension of spatial distribution across multiple devices. This dimensional expansion allows simultaneous execution of computational tasks that would otherwise be sequential, overcoming the time limitation while keeping individual GPU operations relatively simple.
3Quantity of substance
If parameters are distributed across multiple GPUs, then the memory constraint of individual GPUs is overcome, but the device complexity and coordination overhead increase
Solution Approach 1:
The patent introduces a parameter distribution mechanism that acts as an intermediary, managing how parameters are allocated and synchronized across multiple GPUs. This intermediary layer handles the complexity of coordination by establishing clear rules for parameter storage, retrieval, and updates, reducing the overhead burden on individual GPUs while maintaining efficient parallel processing.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Implementations of the present specification provide a method, an apparatus, and a device for updating a convolutional neural network by using a GPU cluster. The GPU cluster includes a first GPU and several other GPUs. The method is performed by a first GPU and includes: obtaining a sample with a classification label; performing a first operation on the sample based on parameters of each layer of front-end network, to obtain a first operation result; performing a second operation on the sample based on the first operation result and the parameters of each layer of back-end network that the first GPU has, to obtain a second operation result; separately sending the first operation result to the other GPUs; receiving a third operation result obtained after each other GPU performs a third operation on the sample based on their parameters of each layer of back-end network and the first operation result; combining the second operation result and the third operation result to obtain a classification result; determining a prediction error based on the classification result and the classification label; and updating the convolutional neural network based on the prediction error.