Virtualized Accelerator Memory Tiers for Large ML Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data processing systems face challenges in efficiently handling large models due to memory limitations, leading to increased data transfers and latency in accelerator devices, particularly in machine learning workloads.
Innovation Solution
Implementing accelerator devices with multi-tier memory systems and virtualization techniques, where models are partitioned across multiple accelerators, utilizing high-bandwidth working memory and high-capacity storage, along with virtual accelerators to manage resource allocation and data movement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If models are stored in accelerator device memory, then processing speed is improved, but memory capacity is insufficient for large models
Solution Approach 1:
The patent partitions the model into multiple segments distributed across different memory tiers (fast memory and storage memory). Different portions of the model are stored in different memory locations based on access frequency and size requirements, allowing the system to handle models larger than single-memory-capacity while maintaining efficient access patterns.
Solution Approach 2:
The patent introduces a hierarchical memory architecture that adds a vertical dimension to memory organization, with multiple tiers of memory (fast memory, storage memory) stacked in layers. This multi-tiered structure allows the system to access different portions of the model at different speeds, effectively expanding the usable memory capacity while preserving fast access paths for critical model segments.
2Quantity of substance
If data is transferred between host and accelerator device, then memory capacity is effectively increased, but latency increases
Solution Approach 1:
The patent pre-loads frequently accessed model portions into the accelerator device's fast memory before they are needed for processing. The system anticipates access patterns and prepares data in advance, reducing the need for latency-inducing data transfers during actual model execution. This preliminary positioning of data in fast memory minimizes transfer delays during inference or training operations.
Solution Approach 2:
The patent introduces a memory management unit and caching mechanisms as intermediaries between the host and accelerator device memory. These intermediary components buffer and manage data transfers, allowing the system to decouple host memory operations from accelerator processing. The intermediary layer absorbs transfer latency by maintaining local caches and managing data staging, effectively hiding the time penalty of cross-device transfers.
3Quantity of substance
If multiple accelerator devices are used to store model portions, then model size capacity is improved, but system complexity increases
Solution Approach 1:
The patent implements a unified memory management architecture that presents a single virtual address space to applications, even when the model is physically distributed across multiple accelerator devices. The memory management unit handles address translation and data routing transparently, allowing the system to scale to multiple devices without increasing application-level complexity. This universal interface abstracts the underlying multi-device complexity.
Solution Approach 2:
The patent incorporates memory access pattern analysis and dynamic data placement mechanisms that use feedback from actual model execution to optimize data distribution across multiple accelerator devices. The system monitors access patterns and automatically adjusts which model portions are stored in which devices, creating a self-optimizing system that reduces the operational complexity of managing distributed model storage.
Data Source
AI summary
A device may include an interconnect interface, a memory system including one or more first type memory devices to receive first data, one or more second type memory devices to receive second data, and an accelerator configured to perform an operation using the first data and the second data. The memory system may further include a cache configured to cache the second data for the one or more second type memory devices. A device may include an interconnect interface, a memory system coupled to the interconnect interface to receive data, an accelerator coupled to the memory system, and virtualization logic configured to partition one or more resources of the accelerator into one or more virtual accelerators, wherein a first one of the one or more virtual accelerators may be configured to perform a first operation on a first portion of the data.