CXL Memory Expander Access for AI Accelerator Capacity Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Graphics Processing Units (GPUs), Neural Processing Units (NPUs), and Tensor Processing Units (TPUs) face issues with insufficient memory capacity, leading to failures in running large-scale AI applications and reduced performance.
Innovation Solution
A computing device with a processor, memory, and a root complex connected via a Compute Express Link (CXL) protocol-based interconnect allows direct access to a memory expander, enabling the processor to recognize and update the physical address space to integrate memory expander space into its own, facilitating data access without a host CPU intervention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a GPU or accelerator uses its own local memory, then processing speed is improved, but memory capacity is limited
Solution Approach 1:
The patent implements a nested memory architecture where memory expanders are connected through a hierarchical structure. The GPU's local memory is nested within a larger memory system that includes memory expanders accessible via root ports and CXL interconnects. This allows the GPU to access both its fast local memory and the expanded capacity of external memory devices, resolving the contradiction between speed and capacity.
Solution Approach 2:
The patent introduces a new dimension of memory access by implementing a multi-level memory hierarchy with different access paths. The GPU can access memory through multiple dimensions: direct local memory access for high-speed operations, and expanded memory access through root ports for additional capacity. This dimensional approach allows simultaneous optimization of both speed and capacity.
2Quantity of substance
If memory expanders are added to increase capacity, then memory capacity is improved, but system complexity increases
Solution Approach 1:
The patent introduces root ports as intermediary components that manage the connection between the GPU and memory expanders. These root ports act as mediators that handle the complexity of memory management, address mapping, and data transfer protocols. By placing this intermediary layer, the system achieves expanded memory capacity while isolating the GPU from the complexity of direct memory expander management.
Solution Approach 2:
The root complex and root ports are designed with multi-functional capabilities that handle various memory management tasks including address translation, data transfer, and protocol conversion. This universal design allows a single component to manage multiple memory expanders and different types of memory devices, reducing overall system complexity while maintaining expanded capacity.
3Speed
If the processor directly accesses memory expander, then access speed is improved, but address space management becomes complex
Solution Approach 1:
The patent implements preliminary address space setup where the root complex pre-configures address mappings between the GPU's address space and the memory expander's physical addresses. This preliminary action includes creating address translation tables and configuring root ports before the GPU needs to access the expanded memory. By performing this address space management in advance, the GPU can access memory expanders directly at high speed without real-time address translation complexity.
Data Source
AI summary
Disclosed are a computing device for accessing a memory expander using a compute express link (CXL) interconnect and an operating method thereof. The computing device may include a processor, a memory, and a root complex configured to be connected with the processor, the memory, and a memory expander. The processor may recognize whether the memory expander is connected to the root complex. The processor may update a physical address space of the computing device based on a recognition that the memory expander is connected to the root complex.