Prefetching and caching between high bandwidth memory (HBM) and non-volatile (NVM) cache on processing die

The integration of NVM and VM dies as an on-chip cache within a hybrid compute device addresses memory capacity and access speed challenges in AI/ML frameworks by optimizing data storage and access patterns, enhancing performance and reducing latency.

WO2026148074A1PCT designated stage Publication Date: 2026-07-09MICRON TECHNOLOGY INC
View PDF 0 Cites 0 Cited by

Patent Information

Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
MICRON TECHNOLOGY INC
Filing Date
2025-12-30
Publication Date
2026-07-09

AI Technical Summary

Technical Problem

Existing memory systems face challenges in meeting growing memory capacity requirements while maintaining requisite memory access bandwidth and latency, particularly in AI/ML frameworks, due to insufficient integration of volatile and non-volatile memory dies.

Method used

Integrating non-volatile memory (NVM) dies with volatile memory (VM) dies as an on-chip cache within a single hybrid compute device, where VM dies operate as a first-level cache and NVM dies as a second-level cache, with a local memory controller managing address translation and prefetching operations to predict data access patterns.

Benefits of technology

This approach enhances memory performance by reducing off-package data movement, improving data access bandwidth and latency, and optimizing data storage for predictable AI/ML operations, thus addressing memory capacity and access speed issues.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure US2025061755_09072026_PF_FP_ABST
    Figure US2025061755_09072026_PF_FP_ABST
Patent Text Reader

Abstract

A compute device includes a compute die, a on-chip cache with one or more volatile memory dies and one or more non-volatile memory dies, and a memory controller all disposed on the package substrate. The controller separates read commands, received from the compute die, according to processing threads executed by the compute die and includes prediction engines and coupled performance monitors. A first prediction engine predicts, for a first processing thread, a first data access pattern associated with a first read command received from the first processing thread. A first performance monitor determines a first statistical value corresponding to a likelihood that the first prediction engine successfully predicted the first data access pattern. The first performance monitor, responsive to the first statistical value satisfying a threshold value, forwards a prefetch command consistent with the first data access pattern to a prefetch command manager for handling.
Need to check novelty before this filing date? Find Prior Art