Full-Mesh AI Chiplets With In-Memory Compute for Transformer Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer-based neural network models, such as BERT and GPT, face significant challenges with high computational intensity and memory requirements, making it difficult to efficiently serve large-scale NLP models.
Innovation Solution
An AI accelerator apparatus utilizing chiplet devices with full mesh connectivity and in-memory compute (DIMC) functionality, coupled via D2D interconnects, to perform high-throughput transformer computations, optimizing computational performance and reducing power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If transformer model size and compute requirements are increased to improve NLP performance, then model accuracy and capability are improved, but training time and resource requirements increase significantly
Solution Approach 1:
The system segments the transformer model processing into multiple chiplet devices, each handling specific computational tasks. The model is divided across multiple tiles and slices on different chiplets, allowing parallel processing of different model layers or operations, thereby reducing overall training time while maintaining model accuracy.
Solution Approach 2:
The patent introduces a new dimension of parallelism through full mesh connectivity between chiplets, enabling simultaneous data and weight transfers across multiple processing units. This multi-dimensional parallel processing architecture allows the system to handle larger models faster by utilizing spatial distribution of computational tasks.
2Productivity
If more computational resources are allocated to handle larger transformer models, then processing capability is improved, but power consumption increases
Solution Approach 1:
The patent merges computation and memory functions within each chiplet tile, creating integrated processing units that reduce data transfer overhead. By combining weight storage and computation in close proximity within the same chiplet, the system reduces energy consumption associated with data movement while maintaining high processing capability.
Solution Approach 2:
The system segments computational workloads across multiple chiplets with full mesh connectivity, allowing efficient load distribution and parallel processing. Each chiplet handles specific computational tasks independently, reducing overall system power consumption through distributed processing rather than concentrating all computational demands in a single high-power unit.
3Adaptability or versatility
If traditional GPU architectures are used for transformer workloads, then general-purpose computing is achieved, but computational efficiency and throughput are limited
Solution Approach 1:
The patent implements local quality optimization by designing specialized computation units within each chiplet tile that are optimized for specific transformer operations such as matrix multiplications and activation functions. Each tile contains dedicated hardware accelerators for common neural network operations, providing high computational efficiency for transformer workloads while maintaining adaptability through programmable elements.
Solution Approach 2:
The chiplet architecture provides universality through a combination of specialized computation units and programmable processing elements. Each chiplet can be configured to handle different transformer model architectures and operations, allowing the same hardware platform to efficiently process various NLP tasks while maintaining high computational efficiency through architecture-specific optimizations.
4Measurement precision
If model parameters are increased to trillion-scale for advanced NLP tasks, then model capability is improved, but memory requirements and serve difficulty increase
Solution Approach 1:
The patent segments the storage of model parameters across multiple chiplets, with each chiplet holding a portion of the weight matrices. The full mesh interconnect enables efficient access to distributed parameters through coordinated data transfer between chiplets, allowing the system to handle trillion-parameter models by distributing the memory burden across multiple devices rather than requiring a single large memory system.
Data Source
AI summary
An AI accelerator apparatus using in-memory compute chiplet devices. The apparatus includes a first semiconductor substrate having a plurality of chiplets, each of which includes a plurality of tiles. Each tile includes a plurality of slices, a central processing unit (CPU), and a hardware dispatch device. Each slice can include a digital in-memory compute (DIMC) device configured to perform high throughput computations. In particular, the DIMC device can be configured to accelerate the computations of attention functions for transformer-based models (a.k.a. transformers) applied to machine learning applications. The chiplets are in a full mesh connectivity configuration such that at least one of the die-to-die (D2D) interconnects of each chiplet is coupled to one of the D2D interconnects of each other chiplet using a non-diagonal link. The chiplets can also include other interfaces to facilitate communication between the chiplets, memory and a server or host system.