Two-Tiered Accelerator Architecture for Concurrent Model Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Accelerators face limitations in processing large machine learning models concurrently due to limited storage capacity and bandwidth bottlenecks, leading to sequential execution of models despite having sufficient processor cores.
Innovation Solution
A two-tiered accelerator architecture with a high-bandwidth bus connecting accelerator memory and storage, allowing for rapid data transfer and concurrent execution of multiple models by processing data in small batches and using a prefetcher to anticipate data needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If accelerators use limited storage capacity to store models, then storage requirements are met, but concurrent model execution is prevented leading to unused processing capability
Solution Approach 1:
The storage system is segmented into two distinct tiers: accelerator memory for frequently accessed model data and storage devices for archival model storage. This segmentation allows the system to maintain multiple models across different storage tiers, enabling concurrent execution of multiple models while efficiently utilizing total storage capacity.
Solution Approach 2:
The system transitions from a single-dimensional storage hierarchy to a two-dimensional architecture by introducing a high-bandwidth bus as a new dimension for data transfer. This additional dimension enables rapid data movement between storage tiers, supporting concurrent model execution by providing sufficient bandwidth for multiple models to be loaded and processed simultaneously.
2Productivity
If accelerators transfer data between storage and memory, then data access is enabled, but bandwidth bottlenecks reduce processing efficiency
Solution Approach 1:
A high-bandwidth bus is introduced as a new dimension for data transfer between accelerator memory and storage devices. This dedicated high-bandwidth pathway eliminates the bottleneck of traditional shared buses, enabling rapid simultaneous data transfers for multiple concurrent model executions without interfering with other system operations.
3Productivity
If accelerators execute models sequentially due to storage limitations, then storage capacity constraints are satisfied, but processor core utilization is reduced
Solution Approach 1:
The system segments the model execution workflow by maintaining multiple models across different storage tiers and loading them into accelerator memory concurrently. This segmentation enables multiple processor cores to execute different models simultaneously, maximizing processor core utilization while satisfying storage capacity constraints through the two-tiered architecture.
Solution Approach 2:
The system performs preliminary actions by pre-loading model data from storage devices into accelerator memory before execution is needed. The high-bandwidth bus enables this pre-loading to occur rapidly, so that multiple models are ready in memory for immediate concurrent execution, eliminating wait times and maximizing processor core utilization.
Data Source
AI summary
An accelerator is disclosed. A circuit may process a data to produce a processed data. A first tier storage may include a first capacity and a first latency. A second tier storage may include a second capacity and a second latency. The second capacity may be larger than the first capacity, and the second latency may be slower than the first latency. A bus may be used to transfer at least one of the data or the processed data between the first tier storage and the second tier storage.