In-Memory Compute Chiplets for Scalable Transformer Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer-based neural network models require significant computational resources and are time-intensive, making it difficult to serve them at scale due to high compute and memory demands, especially with the rapid growth of model sizes.
Innovation Solution
A server system with AI accelerator apparatuses using in-memory compute chiplet devices, featuring modular chiplets with digital in-memory compute (DIMC) and single instruction multiple data (SIMD) devices, coupled via die-to-die interconnects and memory interfaces, to perform high-throughput transformer computations efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional GPU-based computing is used for transformer workloads, then computational capability is provided, but power consumption is high and computational efficiency is low
Solution Approach 1:
The system segments the computing architecture into separate compute chiplets and memory chiplets that can be independently optimized and scaled. Compute chiplets handle transformer computations while memory chiplets provide high-speed memory access, allowing the system to achieve high computational efficiency without the excessive power consumption of traditional GPU architectures.
Solution Approach 2:
The patent introduces an intermediary high-speed interconnect architecture that bridges compute chiplets and memory chiplets. This intermediary structure enables efficient data transfer between computation and memory resources, improving computational efficiency while reducing the power consumption associated with data movement in traditional architectures.
2Measurement precision
If model size increases to improve AI performance, then inference accuracy improves, but compute requirements and time intensity increase significantly
Solution Approach 1:
The system segments large transformer models into smaller computation blocks that can be distributed across multiple compute chiplets. This segmentation allows the system to handle larger model sizes with higher inference accuracy while maintaining compute throughput through parallel processing across the chiplet array.
Solution Approach 2:
The patent adds a spatial dimension to computation by utilizing a two-dimensional array of compute chiplets connected through a mesh network. This dimensional expansion allows the system to scale computational capacity to handle larger models without proportionally increasing time intensity, as computations can be distributed across the spatial array of chiplets.
3Quantity of substance
If more memory resources are allocated to handle larger models, then model capacity increases, but memory access time and power consumption increase
Solution Approach 1:
The memory system is segmented into multiple independent memory chiplets, each providing high-speed access to specific data blocks. This segmentation allows the system to provide large total memory capacity while maintaining fast access times through localized memory access patterns and parallel memory operations across multiple chiplets.
Solution Approach 2:
The patent merges compute chiplets and memory chiplets into a unified heterogeneous computing system with tight integration through high-speed interconnects. This merging reduces the effective memory access time by placing memory resources in close proximity to compute resources, eliminating the memory wall bottleneck present in traditional architectures.
4Productivity
If system complexity increases to provide modular chiplet architecture, then scalability and efficiency improve, but device complexity increases
Solution Approach 1:
The system is segmented into standardized compute chiplets and memory chiplets with defined interfaces and communication protocols. This segmentation enables scalability as new chiplets can be added to the array without redesigning the entire system, while the modular nature actually reduces overall complexity by allowing independent optimization of each chiplet type.
Solution Approach 2:
The patent implements universal interconnect interfaces and communication protocols that work across all compute and memory chiplets regardless of their specific function or capacity. This universality simplifies the system architecture by providing a single standardized method for chiplet communication, reducing the complexity that would otherwise arise from multiple specialized interfaces.
Data Source
AI summary
A server system with AI accelerator apparatuses using in-memory compute chiplet devices. The system includes a plurality of multiprocessors each having at least a first server central processing unit (CPU) and a second server CPU, both of which are coupled to a plurality of switch devices. Each switch device is coupled to a plurality of AI accelerator apparatuses. The apparatus includes one or more chiplets, each of which includes a plurality of tiles. Each tile includes a plurality of slices, a CPU, and a hardware dispatch device. Each slice can include a digital in-memory compute (DIMC) device configured to perform high throughput computations. In particular, the DIMC device can be configured to accelerate the computations of attention functions for transformer-based models (a.k.a. transformers) applied to machine learning applications. A single input multiple data (SIMD) device configured to further process the DIMC output and compute softmax functions for the attention functions.