Multi-Exit ML Scheduling for Dynamic Batching on Edge NPUs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for processing multi-exit machine learning models on edge neural processing units (NPUs) face inefficiencies due to dynamic batch sizes caused by early exits, leading to hardware underutilization and increased latency, which compromises the quality of experience in latency-critical applications.
Innovation Solution
A scheduling algorithm and hardware mechanism that allows for pre-emption and dynamic batching of data items at early exits, forming larger batches to optimize NPU usage by pausing processing at pre-emption points and merging batches, while considering latency constraints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If batch processing is used to increase throughput, then processing rate is improved, but latency increases
Solution Approach 1:
The patent implements dynamic batching where the batch size and composition are adjusted in real-time based on data arrival patterns and latency requirements. The system dynamically forms batches at different stages of model processing (early exits) rather than waiting for complete batches, allowing adaptive optimization between throughput and latency.
Solution Approach 2:
The patent segments the model processing into multiple early exit points, allowing batches to be formed and processed at different depths of the model. This segmentation enables partial batch processing where some data items can exit early while others continue deeper, optimizing the trade-off between throughput and latency for different data items.
2Loss of time
If multi-exit model with early exits is used to reduce computation time, then latency is improved, but hardware utilization decreases
Solution Approach 1:
The patent merges multiple partial batches formed at different early exit points into consolidated batches for processing at deeper model layers. This merging mechanism ensures that the NPU processes data in larger, more efficient batches at each stage, maximizing hardware utilization while still benefiting from early exits for simple cases.
Solution Approach 2:
The patent maintains continuous NPU operation by ensuring that processing capacity is always utilized - either through early exit processing or by merging partial batches for continued processing. The system avoids idle NPU cycles by continuously forming and processing batches at appropriate model depths.
3Loss of time
If dynamic batch size adjustment is implemented to meet latency constraints, then latency control is improved, but system complexity increases
Solution Approach 1:
The patent performs preliminary batch formation and processing decisions at early exit points before full model processing. By making batching decisions earlier in the model pipeline, the system can prepare batches in advance and adjust sizes based on preliminary assessments, simplifying later processing stages while maintaining latency control.
Data Source
AI summary
An embodiment of the disclosure provides a method for optimising usage of a processing unit that is used to execute machine learning, ML, models. In particular, the present disclosure provides an apparatus and method for processing data using a multi-exit ML model in a way that optimises the usage of a processing unit used to execute the model.