SoC ML Resource Partitioning by Inference Time
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems-on-a-chip (SOC) do not efficiently utilize multi-core processors or multiple computation engines for machine learning (ML) tasks due to the lack of effective partitioning and distribution of large ML applications into smaller subtasks that can be executed in parallel, leading to suboptimal performance and resource utilization.
Innovation Solution
A system partitions processing resources into groups with equal numbers of cores for parallel execution of ML models, determining optimal groupings based on performance profiles and minimizing communication latency and cache memory overhead, allowing for efficient execution of ML tasks by dividing them into subtasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If ML applications are executed on SOC without partitioning, then the system is simple to operate, but resource utilization is suboptimal and performance is poor
Solution Approach 1:
The patent divides large ML applications into smaller subtasks that can be distributed across multiple processing cores. The system automatically partitions the computation graph of ML models into independent subtasks, assigns them to different cores, and manages parallel execution. This segmentation enables efficient utilization of multi-core SOC resources while maintaining ease of operation through automated partitioning algorithms.
2Speed
If processing resources are partitioned into multiple groups for parallel execution, then execution speed improves, but communication latency and cache memory overhead increase
Solution Approach 1:
The patent merges adjacent computing cores into processing groups to improve cache utilization and reduce communication overhead. By combining cores that frequently interact into the same processing group, the system minimizes inter-group communication latency while maintaining parallel execution benefits. This merging strategy optimizes the trade-off between parallelism and communication efficiency.
Solution Approach 2:
The patent introduces a task scheduling system that acts as an intermediary between ML subtasks and processing groups. This scheduler optimally assigns subtasks to processing groups, considering communication patterns and data dependencies, thereby minimizing communication latency while maximizing parallel execution speed.
3Productivity
If more processing resources are allocated to ML inference, then inference performance improves, but power consumption and heat generation increase
Solution Approach 1:
The patent implements dynamic resource allocation that adjusts the number of active processing cores based on the computational requirements of different ML tasks. The system analyzes task characteristics and allocates processing resources dynamically, activating only the necessary number of cores needed to achieve target performance levels. This dynamic approach optimizes the balance between inference performance and power consumption.
Data Source
AI summary
A system obtains a performance profile corresponding to times taken to perform an inferencing by a machine learning (ML) model using a different number of processing resources from a plurality of processing resources. The system determines one or more groupings of processing resources from the plurality of processing resources, each grouping includes one or more partitions. The system calculates performance speeds corresponding to each grouping based on the performance profile. The system determines a grouping having a best performance speed from the calculated performance speeds. The system partitions the processing resources based on the determined grouping to perform the inferencing.