DIMM-Form Neural Network Accelerator for Low-Latency Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing DNN accelerators, such as TPUs, face challenges in sharing, high latency, and software overhead due to PCIe or USB deployments, making them inadequate for low-latency, fine-grained interactions with application components.
Innovation Solution
A neural network dual in-line memory module (NN-DIMM) with a DNN accelerator and bridge chip, configured to mimic a standard DRAM interface, allowing direct memory access and integration into standard DIMM slots, enabling low-latency model inference and easy sharing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If DNN accelerators are deployed via PCIe or USB interfaces, then they can be effectively used for batch processing and dedicated scenarios, but they suffer from high latency, difficulty in sharing, and software overhead that make them inadequate for low-latency fine-grained interactions
Solution Approach 1:
The patent introduces a bridge chip as an intermediary component that translates between the PCIe/USB interface protocol and the memory controller protocol. This bridge enables the DNN accelerator to be accessed through standard memory interfaces while maintaining compatibility with existing PCIe/USB connection architectures, thereby reducing latency for fine-grained interactions while preserving batch processing capabilities
Solution Approach 2:
The DNN accelerator is designed to support multiple access modes through the bridge interface, allowing it to function both as a high-throughput batch processing device and as a low-latency response device. The bridge enables the same hardware to serve different operational requirements without needing separate dedicated interfaces
2Power
If DNN accelerators are deployed via PCIe or USB interfaces, then they can provide dedicated processing power, but they create software overhead and complexity in accessing the devices
Solution Approach 1:
The bridge chip acts as an intermediary that translates complex PCIe/USB access protocols into simple memory read/write operations. This allows software to access the DNN accelerator using standard memory interface protocols, eliminating the need for specialized drivers and reducing software overhead while preserving full processing power
Solution Approach 2:
The bridge interface is designed to be automatically recognized and configured by the system, allowing the DNN accelerator to be accessed through standard memory operations without requiring specialized software layers. The system self-adapts to the new device type through existing memory interface protocols
3Adaptability or versatility
If PCIe devices are shared among different applications, then resource utilization improves, but it becomes challenging to manage and coordinate access
Solution Approach 1:
By implementing the DNN accelerator through a standard memory interface bridge, the device can be simultaneously accessed by multiple applications using conventional memory access operations. This universal interface approach enables seamless sharing among different applications without requiring complex coordination protocols or specialized access management software
Data Source
AI summary
The technology relates to a neural network dual in-line memory module (NN-DIMM), a microelectronic system comprising a CPU and a plurality of the NN-DIMMs, and a method of transferring information between the CPU and the plurality of the NN-DIMMS. The NN-DIMM may include a module card having a plurality of parallel edge contacts adjacent to an edge of a slot connector thereof and configured to have the same command and signal interface as a standard dual in-line memory module (DIMM). The NN-DIMM may also include a deep neural network (DNN) accelerator affixed to the module card, and a bridge configured to transfer information between the DNN accelerator and the plurality of parallel edge contacts via a DIMM external interface.