Memory Fabric Protocol for Cache-Coherent AI Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional interconnect technologies struggle to provide fabric-wide cache coherence, in-network computation, and memory-semantic packet routing in large-scale, disaggregated AI systems, leading to architectural bottlenecks for AI and data-centric workloads due to limited scalability, inefficient memory access, and excessive message traffic.
Innovation Solution
A Memory-Fabric Transaction Layer Protocol (MF-TLP) and Memory-Centric Network Interface Controllers (MC-NICs) implement a distributed, cache-coherent memory system with predictive-prefetch and vectorized transactions, enabling efficient operations across heterogeneous nodes and federated data-center domains.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional interconnect technologies (PCIe, CXL, InfiniBand) are used for large-scale disaggregated AI systems, then basic point-to-point connectivity and limited memory sharing are achieved, but fabric-wide cache coherence, in-network computation, and memory-semantic packet routing cannot be provided
Solution Approach 1:
The patent introduces a memory fabric intermediary layer that sits between conventional interconnects and compute devices. This fabric provides memory-semantic packet routing and fabric-wide cache coherence without requiring changes to existing PCIe, CXL, or InfiniBand endpoints. The intermediary translates conventional point-to-point traffic into fabric-wide coherent operations, enabling in-network computation and collective operations while maintaining compatibility with existing devices.
Solution Approach 2:
The memory fabric is designed to provide multiple functions through a single unified architecture: it delivers fabric-wide cache coherence, enables in-network computation, supports memory-semantic packet routing, and provides collective operations (All-Reduce, All-Gather, etc.). This multi-functional approach eliminates the need for separate specialized interconnects for each function, reducing overall system complexity despite the advanced capabilities provided.
2Productivity
If operations such as gradient reductions, parameter synchronization, and large-context attention fetches traverse the network multiple times with software orchestration, then basic functionality is maintained, but bandwidth is consumed excessively and latency is increased
Solution Approach 1:
The memory fabric enables preliminary actions by providing predictive prefetch mechanisms that anticipate future memory access patterns. For large-context attention fetches and gradient reductions, the fabric proactively pre-fetches required data before it is actually needed by compute devices, reducing wait times and improving overall operation efficiency without requiring additional communication rounds.
Solution Approach 2:
The fabric provides self-service capabilities through automated collective operations. Instead of requiring software orchestration to coordinate gradient reductions or parameter synchronization across multiple devices, the memory fabric autonomously performs these operations using hardware-accelerated collective primitives. This eliminates software overhead and reduces the number of communication rounds required, directly improving productivity and reducing latency.
3Device complexity
If vectorized or scatter/gather memory operations are fragmented into discrete packets with full transport framing, then basic packet switching is maintained, but header amplification increases by 2-5× relative to payload
Solution Approach 1:
The memory fabric merges multiple discrete packet operations into single consolidated transactions. Vectorized operations and scatter/gather memory accesses that would traditionally require multiple separately-framed packets are combined into unified fabric transactions with reduced per-operation overhead. This merging approach maintains the simplicity of packet switching at the fabric level while dramatically improving bandwidth efficiency by eliminating redundant headers.
Solution Approach 2:
The fabric implements partial framing by applying transport-level framing only to the aggregate transaction rather than to each individual memory operation within the transaction. This partial action approach provides sufficient error handling and flow control at the fabric level while avoiding the excessive header amplification that would result from full framing of every discrete packet, thereby improving payload-to-header ratio without sacrificing reliability.
4Adaptability or versatility
If CXL and PCIe rely on processor-managed snoop hierarchies, then host-anchored coherence domains are achieved, but scalability is limited beyond single-node domains
Solution Approach 1:
The memory fabric introduces a new dimension to coherence management by transitioning from processor-managed snoop hierarchies to fabric-wide directory-based coherence. Instead of coherence being managed at the processor or node level (2D hierarchy), the fabric provides a 3D coherence domain that spans multiple nodes and racks. This dimensional change enables scalability beyond single-node domains while distributing coherence management across the fabric infrastructure rather than concentrating it in processors.
Solution Approach 2:
The fabric introduces an intermediary coherence management layer between processors and memory resources. This fabric-resident coherence infrastructure mediates coherence operations across multiple nodes, replacing the need for processor-managed snoop hierarchies. The intermediary fabric directories track memory state and coordinate coherence operations autonomously, enabling scalable multi-node coherence without requiring complex processor-side snoop logic, thus reducing coherence management complexity at the processor level.
Data Source
AI summary
A coherent, intelligent, packet-switched memory fabric enables predictive, cache-coherent access across distributed compute, accelerator, and memory resources using a Memory-Fabric Transaction Layer Protocol (MF-TLP). MF-TLP defines routable packet formats for read, write, vectorized, atomic, reduction, collective, and predictive-prefetch transactions executed by memory-centric network interface controllers (MC-NICs). Each MC-NIC performs packet parsing, address translation, coherence management, and near-memory arithmetic or tensor operations while coordinating with MF-TLP-aware switches providing hierarchical directory control, multi-path routing, and in-network aggregation. Vectorized and multimodal packets encode multiple addresses or tensor offsets to reduce scatter/gather overhead, and programmable caching and quality-of-service modules manage tiered memory and tenant fairness. MF-TLP supports extension headers for predictive prefetch, collective coordination, and tenant governance, operating across hierarchical leaf-spine topologies using Ultra-Ethernet Transport, InfiniBand, or CXL fabrics. The system delivers scalable, low-latency, memory-centric orchestration for large-language-model training, multimodal AI, and data-intensive analytics.