Multi-die Dot-Product Engine for Large-Scale ML Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deep learning hardware accelerators face limitations due to on-die weight storage, which restricts the size and complexity of deep neural networks (DNNs), especially when implemented on a single silicon device, leading to inefficiencies in power usage and die area, and challenges in coordinating thousands of neural network inference cores.
Innovation Solution
A multi-die dot-product engine architecture is introduced, leveraging a multi-chip design to support large-scale machine learning inference applications, enabling power-efficient processing and accommodating varying neural network sizes by using multiple silicon devices and a scalable interface that connects inference computation units efficiently across multiple chips.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If on-die weight storage is used in deep learning hardware accelerators, then the architecture is simplified and integration is improved, but the size and complexity of deep neural networks are restricted
Solution Approach 1:
The patent divides the weight storage function across multiple separate memory devices rather than consolidating all weights on a single die. Each memory device stores a portion of the neural network weights, and multiple inference computation units are distributed across these devices. This segmentation allows the system to accommodate larger and more complex DNNs by distributing the weight storage capacity across multiple components, directly resolving the contradiction between integration simplicity and DNN size adaptability.
2Productivity
If multiple inference computation units are coordinated on a single device, then processing capacity increases, but power usage and die area efficiency deteriorate
Solution Approach 1:
The patent segments the inference computation units across multiple separate memory devices rather than concentrating them on a single die. Each device contains a subset of inference computation units that operate independently on their assigned neural network layers. This distribution reduces the power density and die area requirements of each individual device while maintaining high overall processing capacity through parallel operation across the multi-device system.
3Productivity
If multiple inference computation units are coordinated on a single device, then processing capacity increases, but coordination complexity and die area efficiency deteriorate
Solution Approach 1:
The patent divides the neural network processing into distinct layers, with each inference computation unit specialized for specific layer types. This segmentation creates a modular architecture where each device handles a defined portion of the computational graph, reducing the coordination complexity compared to a monolithic design. The system achieves high processing capacity through this modular, distributed approach while maintaining manageable coordination overhead.
4Device complexity
If on-die weight storage is used, then hardware architecture is simplified, but power efficiency for large-scale ML applications deteriorates
Solution Approach 1:
The patent implements a distributed weight storage architecture where neural network weights are partitioned across multiple separate memory devices. Each device stores and processes a subset of weights locally, eliminating the need for frequent data movement between components. This segmentation reduces communication overhead and energy consumption associated with data transfer, improving overall power efficiency for large-scale ML applications while maintaining architectural simplicity through standardized device interfaces.
Data Source
AI summary
Systems and methods are provided for a multi-die dot-product engine (DPE) to provision large-scale machine learning inference applications. The multi-die DPE leverages a multi-chip architecture. For example, a multi-chip interface can include a plurality of DPE chips, where each DPE chip performs inference computations for performing deep learning operations. A hardware interface between a memory of a host computer and the plurality of DPE chips communicatively connects the plurality of DPE chips to the memory of the host computer system during an inference operation such that the deep learning operations are spanned across the plurality of DPE chips. Due to the multi-die architecture, multiple silicon devices are allowed to be used for inference, thereby enabling power-efficient inference for large-scale machine learning applications and complex deep neural networks. The multi-die DPE can be used to build a multi-device DNN inference system performing specific applications, such as object recognition, with high accuracy.