Multi-Precision NPU Planning for Mixed-Precision Model Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently executing multiple mixed-precision deep learning models on multi-precision Neural Processing Units (NPUs) without requiring additional hardware or programs, while optimizing resource utilization and minimizing execution time.
Innovation Solution
An apparatus and method that generate execution plans for multiple mixed-precision deep learning models using a multi-precision NPU, incorporating a processor with Micro-PEs, which dynamically allocates resources and adjusts precision operations through Hardware-Aware Mixed-precision Quantization (HAWQ) and dynamic programming to optimize execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple mixed-precision deep learning models are executed on a multi-precision NPU, then productivity and resource utilization are improved, but device complexity and scheduling difficulty increase
Solution Approach 1:
The NPU is segmented into multiple Micro-PEs (Processing Elements), each capable of independent precision operations. This segmentation allows multiple mixed-precision models to be executed in parallel with dedicated computational units, improving productivity while managing complexity through modular architecture.
Solution Approach 2:
The system employs dynamic programming to generate execution plans that adaptively allocate Micro-PEs to different models based on their precision requirements and execution priorities. This dynamic resource allocation optimizes productivity while the automated planning algorithm manages the complexity of coordinating multiple models.
2Loss of time
If execution plans are optimized using dynamic programming, then execution time is minimized, but computational overhead for plan generation increases
Solution Approach 1:
Execution plans are generated in advance using dynamic programming before model execution begins. The system pre-calculates optimal Micro-PE allocation and execution sequences, measuring execution times through pre-simulation or actual execution methods. This preliminary optimization minimizes runtime execution time while the computational overhead is incurred during the planning phase.
3Productivity
If hardware-aware mixed-precision quantization is applied, then model execution efficiency is improved, but model development complexity increases
Solution Approach 1:
The system implements self-service through automated execution plan generation that introspects model precision requirements and automatically allocates appropriate Micro-PEs. The dynamic programming algorithm autonomously optimizes execution strategies without requiring manual intervention, improving execution efficiency while masking the underlying complexity from model developers.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed herein are an apparatus and method for providing execution plans of multiple mixed-precision deep learning models based on a multi-precision NPU. The apparatus includes a memory, and a processor electrically connected to the memory. The processor is configured to form a multi-precision Neural Processing Unit (NPU) including a processing element (PE) composed of multiple Micro-PEs, to generate multiple mixed-precision deep learning models that perform multiple precision operations requiring different degrees of precision in a model execution process, and to generate the execution plans for executing the multiple mixed-precision deep learning models on the multi-precision NPU.