AI Model Partitioning Across Heterogeneous Edge Accelerators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional AI model partitioning techniques do not optimize the distribution of AI models across heterogeneous accelerators on edge devices, leading to increased model load, latency, and reduced execution performance due to insufficient consideration of device-specific compute and sparsity factors.
Innovation Solution
The proposed method analyzes AI models using intermediate representation (IR) to characterize compute and sparsity, normalizing models based on device-specific metrics like precision and compute efficiency, and partitions subgraphs across heterogeneous accelerator units for optimized execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If AI models are deployed on heterogeneous accelerators without optimized partitioning, then device utilization is simplified, but model execution performance decreases and latency increases
Solution Approach 1:
The patent segments the AI model into multiple subgraphs that can be independently partitioned and executed on different heterogeneous accelerator units. This segmentation enables optimized distribution of computational tasks across devices with varying capabilities, thereby improving execution performance and reducing latency through parallel processing.
Solution Approach 2:
The patent applies local quality by analyzing device-specific metrics such as precision, sparsity, and compute efficiency for each heterogeneous accelerator, then assigning model subgraphs to devices based on their local characteristics. This ensures that each device executes tasks for which it is optimally suited, maximizing overall system performance.
2Adaptability or versatility
If conventional partitioning methods are used, then implementation is simpler, but device resource utilization is suboptimal due to ignoring device-specific compute and sparsity factors
Solution Approach 1:
The patent changes the parameters used for partitioning from generic computational metrics to device-specific parameters including precision, sparsity, and compute efficiency. This enables the system to adaptively utilize heterogeneous device resources by matching model subgraphs to devices based on their specific capabilities rather than using a one-size-fits-all approach.
Solution Approach 2:
The patent introduces an intermediate representation (IR) of the AI model as a mediator between the model and heterogeneous devices. This IR enables systematic analysis and partitioning of the model while accounting for device-specific metrics, bridging the gap between model requirements and device capabilities without requiring complex direct optimization.
3Speed
If models are executed on edge devices with limited resources, then latency is reduced, but model load increases due to insufficient optimization
Solution Approach 1:
The patent segments the AI model into subgraphs that can be selectively executed on edge devices based on their resource capabilities. This segmentation allows the system to reduce model load on edge devices by offloading computationally intensive subgraphs to cloud resources while retaining fast execution of critical subgraphs locally, thereby achieving low latency without overwhelming device resources.
Solution Approach 2:
The patent applies partial action by executing only the necessary subgraphs on edge devices rather than loading the complete model. This approach reduces the quantity of data and computational requirements on edge devices while maintaining execution speed for time-critical operations, avoiding the need to load unnecessary model components.
Data Source
AI summary
Systems and techniques for heterogeneous compute-based artificial intelligence model partitioning are described herein. An intermediate representation of an input machine learning model may be generated. The intermediate representation may be analyzed to determine compute metrics for execution of the input machine learning model. An input processing device may be analyzed to determine normalization metrics for execution of the input machine learning model on the input processing device. A partition of the intermediate representation may be generated for the input processing device based on the compute metrics and the normalization metrics. The partition may be transmitted to the input processing device for execution.