Deep Learning Inference Memory Scheduling for Lower Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The latency and throughput issues in autonomous driving systems due to large machine learning models requiring data transfer between internal and external storage units are not adequately addressed, leading to delays in real-time inference processes.
Innovation Solution
Implementing a system-on-a-chip (SoC) architecture with a shared memory (SHMEM) and an on-chip memory (OCM) to buffer intermediate data and preload model parameters, utilizing a compiler, scheduler, and data-loading firmware to optimize data transfer and reduce latency by spilling data to OCM and skipping reusable weights.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If large machine learning models are used to improve perception and decision-making capabilities, then the model accuracy and functionality are improved, but the data transfer time and inference latency increase due to limited memory capacity
Solution Approach 1:
The memory system is segmented into multiple levels: on-chip SRAM for frequently accessed data, off-chip DDR SDRAM for model parameters, and external SSD for large datasets. This hierarchical segmentation allows the system to manage large models while minimizing data transfer latency by placing different types of data in appropriate memory levels based on access patterns.
Solution Approach 2:
The system performs preliminary actions by preloading model parameters and frequently accessed data from off-chip memory into on-chip SRAM before inference begins. This preparation reduces the need for data transfer during critical inference operations, thereby lowering inference latency while supporting large models.
2Adaptability or versatility
If more data is stored in external memory to support large models, then the model functionality is improved, but the data transfer bandwidth requirements increase
Solution Approach 1:
Different memory locations are assigned different qualities based on data access patterns. Frequently accessed model parameters and intermediate results are stored in on-chip SRAM with high bandwidth, while less frequently accessed data remains in off-chip memory. This local quality differentiation optimizes the data transfer volume by ensuring high-speed access only where needed.
Solution Approach 2:
The on-chip SRAM acts as an intermediary buffer between the AI accelerator and external DDR SDRAM/SSD. It caches frequently accessed data and intermediate results, reducing the direct data transfer volume between external storage and the accelerator while supporting large model functionality.
3Quantity of substance
If data is transferred from external storage to internal memory during inference, then the model can process more data, but the transfer time introduces delays
Solution Approach 1:
Data transfer is performed as a preliminary action before inference begins. Model parameters are loaded from off-chip memory into on-chip SRAM in advance, and intermediate results are cached in SRAM during computation. This eliminates transfer delays during critical inference operations while maintaining the ability to process large datasets.
Solution Approach 2:
The system maintains continuous useful action by keeping frequently accessed data and intermediate results in on-chip SRAM throughout the inference process. This eliminates interruptions for data transfer during computation, ensuring continuous high-speed processing while supporting large data volumes through the hierarchical memory structure.
Data Source
AI summary
The disclosure provides a system and method for reducing inference latency of an artificial intelligence (AI) system. During operation, the system can obtain an AI model and compile the AI model to generate at least one Directed Acyclic Graph (DAG), which comprises determining an offset address associated with a piece of intermediate data to be transferred from a primary memory shared among multiple AI accelerators to a secondary memory. The AI accelerators, the primary memory, and the secondary memory are located on the same system on a chip (SoC). The system can then schedule computing tasks, which comprises determining a base address associated with the DAG in the secondary memory, and perform inference based on the DAG, which comprises transferring the piece of intermediate data from the primary memory to the secondary memory based on the offset address and the base address.