Deep Learning Accelerator Runtime Throughput Adaptation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning accelerators face inefficiencies when switching between artificial neural networks (ANNs) at runtime, due to overheads such as recompiling models and creating runtime schedules, which can impact robustness and real-time performance.
Innovation Solution
The solution involves dynamically changing the throughput of each ANN deployed on a deep learning accelerator (DLA) at runtime, without full recompilation and redeployment, by co-compiling multiple ANNs as a single workload and generating execution schedules that can be stored and accessed at runtime, employing techniques like weight sharing and layer fusion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple ANNs are switched at runtime on existing deep learning accelerators, then workload adaptability is improved, but system overhead increases due to recompilation and runtime schedule creation
Solution Approach 1:
The patent applies preliminary action by co-compiling multiple ANN workloads in advance into a single executable image that contains all necessary computation schedules. This pre-compilation approach eliminates the need for runtime recompilation and schedule creation, allowing the system to switch between different ANN workloads dynamically without incurring compilation overhead. The computation schedules for multiple ANNs are generated beforehand and stored together, enabling fast runtime switching while maintaining workload adaptability.
2Adaptability or versatility
If runtime switching between ANNs is implemented, then workload flexibility is improved, but real-time performance deteriorates due to compilation overhead
Solution Approach 1:
The invention resolves this contradiction by performing the time-consuming compilation and schedule generation activities beforehand, during system initialization or deployment time. The co-compiled executable image contains pre-generated computation schedules for multiple ANN workloads, enabling the system to switch between them instantly at runtime without any compilation delay. This preliminary action ensures that workload flexibility is maintained while real-time performance is preserved.
3Productivity
If compute units are allocated to multiple ANNs, then resource utilization is improved, but scheduling complexity increases
Solution Approach 1:
The patent applies merging by combining multiple ANN computation schedules into a single unified executable image that is co-compiled together. This unified schedule contains the computation instructions for multiple ANNs in an integrated format, allowing the deep learning accelerator to manage compute unit allocation across multiple workloads through a single scheduling mechanism. This approach improves resource utilization by enabling multiple ANNs to share compute units efficiently while reducing scheduling complexity compared to managing separate schedules for each ANN.
Data Source
AI summary
Multiple artificial neural networks can be compiled as a single workload. A respective throughput for each of the artificial neural networks can be changed at runtime. The multiple artificial neural networks can be partially compiled individually and then later compiled just-in-time according to changing throughput demands for the artificial neural networks. The multiple artificial neural networks can be deployed on a deep learning accelerator hardware device.