Multistage Memory Acceleration for PCIe Neural Network Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing semiconductor memory devices, such as DRAM, face limitations in data transfer speed and performance due to bottlenecks in communication protocols like PCIe, especially when handling neural network operations, leading to inefficient use of resources and increased power consumption.
Innovation Solution
Implementing a multistage acceleration approach within the memory device, where a first accelerator inside the DRAM package performs coarse acceleration to convert sparse data into dense data, and a second accelerator in the device controller performs fine acceleration to generate neural network outputs, optimizing data processing and reducing power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is transferred through PCIe protocol between memory device and device controller, then data transfer can be performed, but data transfer speed is limited and becomes a performance bottleneck
Solution Approach 1:
The patent merges the memory device and accelerator functions by integrating a first accelerator inside the memory package and a second accelerator in the device controller. This allows data processing to be performed closer to the data storage location, reducing reliance on PCIe data transfer and eliminating the performance bottleneck caused by limited transfer speeds.
Solution Approach 2:
The patent introduces a new dimension of computation by adding accelerator units at multiple levels (inside memory package and in device controller). This transforms the system from a simple data transfer architecture to a multi-dimensional processing architecture where computation occurs at the edge of data storage, bypassing the PCIe speed limitation.
2Productivity
If sparse data is processed directly without conversion, then data can be stored efficiently, but neural network operations are inefficient and power consumption increases
Solution Approach 1:
The first accelerator inside the memory package performs preliminary conversion of sparse data to dense data before the data is transferred to the device controller. This preliminary action prepares the data in a format optimized for neural network operations, improving efficiency and reducing power consumption during subsequent processing.
Solution Approach 2:
The first accelerator acts as an intermediary between the memory storage and the device controller, performing sparse-to-dense conversion at this intermediate stage. This mediator function ensures that only necessary data transformations occur, optimizing the balance between storage efficiency and processing performance.
3Productivity
If all data processing is performed in the device controller, then processing capability is sufficient, but data transfer volume increases and consumes more power
Solution Approach 1:
The patent segments the data processing function into two parts: the first accelerator inside the memory package handles sparse-to-dense conversion, while the second accelerator in the device controller handles neural network operations. This segmentation reduces the volume of data that needs to be transferred over PCIe, thereby reducing power consumption.
Solution Approach 2:
The patent applies local quality by placing the first accelerator specifically inside the memory package to handle data conversion locally, rather than transferring all raw data to the device controller. This localized processing optimizes the system by performing specific functions at the most appropriate location, reducing overall energy consumption.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An electronic device is provided. The electronic host includes: a host; a memory package including a plurality of memory devices and a first accelerator circuit configured to receive first data from the plurality of memory devices and perform a coarse acceleration operation based on the first data to obtain second data; and a memory controller including a second accelerator circuit configured to receive the second data from the first accelerator circuit and perform a fine acceleration operation based on a neural network and the second data to obtain an inference result.