Heterogeneous inference acceleration

Abstract

The embodiments herein describe techniques for performing ML compilation using a unified interface that combines different processors in a heterogeneous processing system which allows for intelligent partitioning of a ML model. Unlike prior solutions which rely on user preferences to assign the ML model, the unified interface can violate or break the user preferences when partitioning the ML model. The unified interface can receive information from the processors (e.g., a NPU, CPU, GPU, etc.) and determine the capabilities, current workload, power metrics, subgraphs of the ML model they can execute, and the like. With this information, the unified interface can intelligently choose when to violate or break the user-entered priority based instructions.

Images