FPGA Memory Bandwidth Management for Deep Learning Batches
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning applications in data centers face challenges with memory bandwidth limitations, leading to inefficiencies and inability to meet performance requirements, especially when using general-purpose processors for neural network evaluations.
Innovation Solution
Incorporating a parallel processor, such as a field-programmable gate array (FPGA), that supports parallel processing to reduce memory bandwidth requirements by queuing input data from multiple streams and loading weight datasets in batches, allowing for efficient processing of multiple streams simultaneously.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If general-purpose processors are used for neural network evaluations, then flexibility and ease of operation are maintained, but memory bandwidth limitations cause performance inefficiency and inability to meet throughput requirements
Solution Approach 1:
The patent segments the neural network evaluation process into multiple parallel streams that can be processed simultaneously. By dividing the workload into N parallel streams and processing them concurrently, the system achieves higher throughput while reducing the memory bandwidth requirement for each individual stream, as the weight dataset is loaded once and shared across all streams.
Solution Approach 2:
The patent employs a field-programmable gate array (FPGA) that provides universal processing capability for neural network evaluations. The FPGA can be configured to handle different neural network architectures and workloads, offering both the flexibility of software and the performance of dedicated hardware, thereby resolving the contradiction between productivity and resource consumption.
2Productivity
If parallel processing is implemented to increase throughput, then productivity improves, but device complexity increases due to additional hardware requirements
Solution Approach 1:
The patent introduces an FPGA as an intermediary between the CPU and the neural network evaluation process. The FPGA acts as a dedicated co-processor that handles the computationally intensive parallel processing tasks, while the CPU maintains its general-purpose functionality. This intermediary approach enables high-throughput parallel processing without significantly increasing the complexity of the existing system architecture.
3Measurement precision
If weight datasets are loaded for each individual evaluation, then processing accuracy is maintained, but memory bandwidth consumption increases
Solution Approach 1:
The patent merges multiple neural network evaluation operations by loading the weight dataset once and using it across N parallel streams simultaneously. This combining approach maintains full processing accuracy for each evaluation while reducing memory bandwidth consumption by a factor of N, as the same weight dataset is shared across all parallel processing streams.
Data Source
Figure 1
Figure 2
Figure 3A~3b
AI summary
In a data center, neural network evaluations can be included for services involving image or speech recognition by using a field programmable gate array (FPGA) or other parallel processor. The memory bandwidth limitations of providing weighted data sets from an external memory to the FPGA (or other parallel processor) can be managed by queuing up input data from the plurality of cores executing the services at the FPGA (or other parallel processor) in batches of at least two feature vectors. The at least two feature vectors can be at least two observation vectors from a same data stream or from different data streams. The FPGA (or other parallel processor) can then act on the batch of data for each loading of the weighted datasets.