Deep Learning Accelerator Input Buffering for Higher Inference Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware accelerator devices for deep learning inference face throughput deterioration due to data movement time, which is a significant factor in evaluating inference throughput.
Innovation Solution
A hardware accelerator device with input storage, an accelerator, output storage, and a status register that allows for proactive replacement of query data when it is no longer required during the deep learning operation, enabling simultaneous processing of multiple data sets without buffers, thus improving throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is moved from host processor to hardware accelerator device, then deep learning operation can be performed, but data movement time acts as a throughput deterioration factor
Solution Approach 1:
The patent implements a buffer memory that pre-stores query data before it is needed by the accelerator. The host processor can write new query data into the buffer in advance, and the accelerator retrieves data from the buffer without requiring real-time data movement from the host processor, thereby eliminating data movement time as a throughput bottleneck
Solution Approach 2:
The buffer memory acts as an intermediary between the host processor and the accelerator. It decouples the data supply from the processing, allowing the accelerator to continuously access pre-stored data without waiting for the host processor, thus improving inference throughput by removing the data movement time constraint
Data Source
AI summary
Disclosed are a computing system, hardware accelerator device, and method for deep learning inference. The accelerator device includes input storage configured to store input query data, an accelerator configured to output inference data, which is the result of a deep learning operation on the input query data, and output storage configured to store the inference data. The input storage stores subsequent query data, input from a host processor, in advance during the deep learning operation on the input query data in the accelerator.