Immersion Cooling AI Server with In-Memory Compute Chiplets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer-based neural network models require significant computational resources and memory, leading to inefficiencies and long training times, especially with the rapid growth of model sizes exceeding billions and soon trillions of parameters, making it challenging to serve NLP models at scale.
Innovation Solution
The implementation of an AI accelerator apparatus with chiplet devices configured for in-memory compute (IMC) functionality, utilizing a modular architecture with digital in-memory-compute (DIMC) devices and single input multiple data (SIMD) processors to accelerate transformer computations, along with an immersion cooling system to manage heat and improve performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional CPU/GPU architectures are used for transformer workloads, then general-purpose computing is achieved, but computational performance is insufficient and training time is excessive
Solution Approach 1:
The system segments the monolithic processor architecture into heterogeneous compute units (CPUs, GPUs, AI accelerators, FPGAs) that can be independently configured and optimized for specific transformer workload types, enabling parallel processing and reducing overall training time
Solution Approach 2:
The system dynamically changes operational parameters including precision formats (FP32, FP16, INT8, BF16), parallelization degrees, and memory allocation based on the specific transformer model being trained, allowing optimization of computational performance for different workload characteristics
2Adaptability or versatility
If model size is increased to improve AI capabilities, then model accuracy and intelligence are enhanced, but computational resource requirements and power consumption increase significantly
Solution Approach 1:
The system applies different precision formats and computational strategies to different parts of the transformer model based on their specific requirements, allowing high-precision computation only where necessary and lower-precision computation elsewhere, thereby reducing overall power consumption while maintaining model capability
Solution Approach 2:
The system dynamically adjusts computational resources and precision levels based on the current training phase and model size, allocating more resources during critical training stages and reducing resources during stable phases, enabling scalable support for models from billions to trillions of parameters
3Quantity of substance
If memory capacity is increased to support larger models, then model size and parameter count can be expanded, but memory access time and computational efficiency deteriorate
Solution Approach 1:
The system implements a nested memory hierarchy with multiple levels of caching and memory pools organized by data type and access frequency, allowing frequently accessed model parameters to be stored in faster memory layers while less frequently accessed data resides in larger capacity storage, thereby maintaining both capacity and speed
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This solution enhances computational performance, reduces power consumption, and enables efficient processing of transformer workloads by integrating computational functions and memory fabric, allowing for quick and effective implementation of AI applications across various transformer models.
Implementation Method 1
these server systems generate heat that is absorbed by the heat transfer fluid
Implementation Method 2
which will evaporate into heat transfer fluid vapor at its boiling point
Implementation Method 3
the condenser device condenses the vapor back to liquid form
Data Source
AI summary
An immersion cooling server system with AI accelerator apparatuses using in-memory compute chiplet devices. This system includes one or more immersion tanks with heat transfer fluid and configured with at least a condenser device. A plurality of AI accelerator servers is immersed in the heat transfer fluid in a bottom portion of the tanks and is configured to process transformer workloads while cooled by the immersion cooling configuration. Each of the servers includes a plurality of multiprocessors each having at least a first server central processing unit (CPU) and a second server CPU, both of which are coupled to a plurality of switch devices. Each switch device is coupled to a plurality of AI accelerator apparatuses. The apparatus includes one or more chiplets, each of which includes a plurality of digital in-memory compute (DIMC) devices configured to perform high throughput matrix computations for transformer based models.