Streaming based generative artificial intelligence (AI) workload execution

By employing an accelerator circuit with a streaming API to directly access NVMe storage and preloading compressed weights just-in-time, the data storage and transfer challenges for machine learning models are addressed, resulting in reduced latency and improved performance.

WO2026136011A1 Publication Date: 2026-06-25ADVANCED MICRO DEVICES INC
3 Cites 0 Cited by

Patent Information

Application Number
PCT/US2025/058192
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2024-12-19
Filing Date
2025-12-04
Publication Date
2026-06-25

AI Technical Summary

Technical Problem

Computing systems face challenges in supporting the data storage and data transfer requirements of machine learning models, particularly due to the high number of parameters and data storage needs, which leads to performance issues and inability to execute applications relying on these models efficiently.

Method used

The use of an accelerator circuit with a streaming application programming interface (API) to directly access non-volatile memory express (NVMe) storage devices, preloading compressed weights, and decompressing them just-in-time for execution, bypassing the host processing circuit and system memory, thereby reducing latency and increasing performance.

Benefits of technology

This approach reduces latency and enhances performance by enabling direct and efficient data storage and transfer of machine learning data, ensuring timely availability of weights for model execution without relying on system memory, thus improving the execution efficiency of machine learning models.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

An apparatus and method for efficiently performing efficient data storage and data transfer of machine learning data. In various implementations, a host processing circuit of a computing system executes a machine learning (ML) application. The application includes a computational graph that indicates the computational order of the ML nodes, layers, and stages of the ML model. The host processing circuit translates function calls in the application to commands particular to an accelerator circuit. The accelerator circuit preloads weights to be used by ML nodes of the ML model by retrieving compressed weights from a storage device different from system memory. The accelerator circuit uses a streaming application programming interface (API) and bypasses the host processing circuit to retrieve the compressed weights from the storage device. The accelerator circuit decompresses the retrieved weights and executes the ML node using the decompressed weights.
Need to check novelty before this filing date? Find Prior Art