Budgeted Deep Neural Network Training with GPU Parallel Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current training techniques for deep neural networks (DNNs) are computationally intensive and require large amounts of data, necessitating improved methods for efficient training and learning.
Innovation Solution
Implementing a graphics processing unit (GPU) communicatively coupled to host processors to accelerate DNN training through dedicated circuitry for efficient command processing, utilizing parallel processing techniques such as SIMT architectures and pipelining to optimize training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional CPU-based training methods are used for deep neural networks, then programming flexibility is maintained, but training speed and computational efficiency deteriorate
Solution Approach 1:
The patent replaces traditional CPU-based sequential processing with GPU-based parallel processing architecture. The system uses multiple graphics processing units with thousands of cores to perform parallel computations for neural network training, substituting the mechanical sequential execution model with a parallel computing model that dramatically improves training speed while distributing computational complexity across multiple devices.
Solution Approach 2:
The patent divides the neural network training workload into multiple segments that can be processed in parallel across different GPU cores and processing units. The training data and computational tasks are segmented into batches and distributed across the parallel processing architecture, allowing simultaneous execution of multiple training operations that would otherwise be sequential.
2Measurement precision
If larger datasets are used to improve model accuracy, then learning quality improves, but computational resource requirements and training time worsen
Solution Approach 1:
The patent segments large datasets into smaller batches that can be processed in parallel across multiple GPU cores. Each processing unit works on a portion of the data simultaneously, maintaining model accuracy through comprehensive data coverage while reducing overall training time through parallel execution of batch processing operations.
Solution Approach 2:
The patent implements techniques where not all parameters need to be computed to full precision at each iteration, or where approximate methods are used that provide sufficient accuracy for the application. This allows processing of larger datasets with reduced computational overhead per data point, maintaining acceptable model accuracy while significantly reducing training time.
3Productivity
If more GPU resources are allocated for parallel processing, then training efficiency improves, but hardware cost and system complexity worsen
Solution Approach 1:
The patent designs the GPU-based system to perform multiple functions: training neural networks, processing graphics data, and executing general-purpose computations. This multi-functionality allows the same hardware resources to be utilized for different tasks, improving training efficiency without requiring dedicated specialized hardware for each function, thereby controlling system complexity.
Solution Approach 2:
The patent combines multiple processing functions and data handling operations into a unified GPU-based architecture. By merging data preprocessing, feature extraction, and model training operations into the same parallel processing system, the patent improves overall training efficiency while reducing the complexity that would arise from multiple separate hardware systems.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Methods and systems for budgeted and simplified training of deep neural networks (DNNs) are disclosed. In one example， a trainer is to train a DNN using a plurality of training sub-images derived from a down-sampled training image. A tester is to test the trained DNN using a plurality of testing sub-images derived from a down-sampled testing image. In another example， in a recurrent deep Q-network (RDQN) having a local attention mechanism located between a convolutional neural network (CNN) and a long-short time memory (LSTM), a plurality of feature maps are generated by the CNN from an input image. Hard-attention is applied by the local attention mechanism to the generated plurality of feature maps by selecting a subset of the generated feature maps. Soft attention is applied by the local attention mechanism to the selected subset of generated feature maps by providing weights to the selected subset of generated feature maps in obtaining weighted feature maps. The weighted feature maps are stored in the LSTM. A Q value is calculated for different actions based on the weighted feature maps stored in the LSTM.