Full-Mesh AI Accelerator Chiplets With DIMC for Transformers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer-based neural network models are computationally intensive and have high memory requirements, making it difficult to serve them at scale, especially with the rapid growth in model size and compute requirements.
Innovation Solution
An AI accelerator apparatus is designed with chiplet devices configured for full mesh connectivity and in-memory compute, utilizing digital in-memory compute (DIMC) devices and non-diagonal die-to-die interconnects to accelerate transformer computations, integrated with computational functions and memory fabric, and optimized for modular scalability and efficient mapping.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional GPU-based computing is used for transformer workloads, then model training can be performed, but computational efficiency is low and training time is excessive (e.g., 4 months for GPT-3 with 1024 GPUs)
Solution Approach 1:
The system segments the computational workload into multiple processing tiles distributed across multiple chiplets. Each chiplet contains multiple tiles that can independently process different portions of the transformer computations, enabling parallel processing and significantly reducing training time for large models like GPT-3
Solution Approach 2:
The patent transitions from traditional von Neumann architecture to in-memory computing architecture, adding a spatial dimension to computation by placing compute units directly within the memory fabric. This eliminates the memory wall bottleneck and enables simultaneous data access and processing, dramatically improving computational efficiency
2Adaptability or versatility
If model size increases to handle more complex NLP tasks, then model capability improves, but memory requirements and compute requirements grow by a thousand times
Solution Approach 1:
The patent merges memory and compute functions into a unified in-memory computing fabric where storage and processing occur in the same physical space. This integration allows large transformer models to be executed directly in memory without frequent data movement, scaling memory capacity to accommodate trillion-parameter models while maintaining computational efficiency
Solution Approach 2:
The chiplet architecture provides a universal platform that can accommodate transformer models of varying sizes through configurable tile arrangements and scalable interconnects. The same hardware infrastructure can efficiently handle both smaller models and trillion-parameter models by adjusting the activation of different chiplet groups
3Speed
If full mesh connectivity is implemented across multiple chiplets, then communication efficiency between processing units improves, but device complexity increases
Solution Approach 1:
The full mesh interconnect is segmented into hierarchical levels: intra-chiplet connections for tiles within the same chiplet, and inter-chiplet connections for communication between chiplets. This segmentation manages complexity by organizing numerous connections into structured layers rather than requiring a single monolithic interconnect
4Productivity
If in-memory compute is used to accelerate computations, then throughput increases (512 multiply accumulates per clock cycle), but power consumption becomes a critical concern
Solution Approach 1:
The patent extracts computational functions from traditional CPU/GPU processing units and places them directly within the memory arrays. This extraction allows computations to occur where data already resides, eliminating energy-intensive data movement operations and reducing overall power consumption while maintaining high throughput
Data Source
AI summary
An AI accelerator apparatus using in-memory compute chiplet devices. The apparatus includes a first semiconductor substrate having a plurality of chiplets, each of which includes a plurality of tiles. Each tile includes a plurality of slices, a central processing unit (CPU), and a hardware dispatch device. Each slice can include a digital in-memory compute (DIMC) device configured to perform high throughput computations. In particular, the DIMC device can be configured to accelerate the computations of attention functions for transformer-based models (a.k.a. transformers) applied to machine learning applications. The chiplets are in a full mesh connectivity configuration such that at least one of the die-to-die (D2D) interconnects of each chiplet is coupled to one of the D2D interconnects of each other chiplet using a non-diagonal link. The chiplets can also include other interfaces to facilitate communication between the chiplets, memory and a server or host system.