CXL Memory Expander Accelerating Tensor Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cache coherent interconnect technologies, such as CXL, face performance degradation in applications like approximate nearest neighbor searches due to increased memory request and response times, which are exacerbated by the need to convert memory requests into CXL flits and transmit them through a host-to-device interconnect.
Innovation Solution
A memory expander device that acts as a type-3 device in the CXL protocol, equipped with a CXL engine, a domain-specific accelerator, and an interface register, which processes CXL flits to perform calculations on tensors stored in its memory, reducing the need to transmit entire tensors back to the host device.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If memory requests are converted into CXL flits and transmitted through host-to-device interconnect, then memory expansion capability is improved, but memory request and response times increase
Solution Approach 1:
The patent segments the memory system into host memory and device memory (via memory expander), allowing data to be distributed across multiple memory spaces. The CXL protocol divides memory operations into flits (flow control units) that can be processed independently, enabling parallel handling of multiple memory requests and reducing overall latency.
Solution Approach 2:
The patent introduces a memory expander as an intermediary device between the host and device memory. This expander includes a response queue that buffers and manages memory responses, decoupling the host memory operations from device memory operations. The response queue acts as a mediator that absorbs timing variations and prevents direct propagation of latency from device memory to host operations.
2Measurement precision
If entire tensors are transmitted back to the host device, then calculation accuracy is maintained, but data movement increases and performance degrades
Solution Approach 1:
The patent extracts only the essential calculation results from device memory and transmits them back to the host, rather than transmitting entire tensors. The memory expander identifies and extracts specific data elements (such as aggregation results or selected tensor values) that are necessary for host processing, leaving the bulk of tensor data resident in device memory. This extraction approach maintains calculation accuracy while dramatically reducing data movement volume.
Solution Approach 2:
The patent implements partial action by performing calculations entirely within the device memory space using the domain-specific accelerator, and only partially transmitting results (not full tensors) to the host. The response queue transmits a subset of data (excessive action in terms of selectivity) that is sufficient for host processing needs, avoiding the excessive data movement that would occur with full tensor transmission.
3Quantity of substance
If CXL protocol conversion is performed, then memory expansion is enabled, but device complexity increases
Solution Approach 1:
The patent implements a memory expander that serves multiple functions: it manages CXL protocol conversion, handles response queue operations, performs domain-specific calculations via accelerator, and manages both host and device memory spaces. By consolidating these diverse functions into a single multi-functional device, the system reduces overall complexity compared to having separate components for each function.
Solution Approach 2:
The patent merges the protocol conversion functionality, memory management capabilities, and domain-specific computation resources into a unified memory expander device. The CXL engine, response queue, and domain-specific accelerator are combined in a single device that interfaces with both host and device memory systems, reducing the number of separate components and simplifying the overall system architecture.
Data Source
AI summary
A memory expander is disclosed. The memory expander includes a memory, a memory controller configured to control the memory, a compute express link (CXL) engine configured to acquire a CXL flit from a host device connected to the memory expander and configured to acquire a calculation request for pieces of data stored in the memory by performing conversion on the CXL flit, and a domain-specific accelerator configured to perform a calculation in response to the calculation request.